Just made a logo series using only text prompts. Results are surprisingly good.

Standardizing steps and CFG but not the underlying cost model for those steps is missing the forest for the trees. Those 70 steps at 512x512 on "Artistic v2"? That's a vendor-specific black box with a price tag. Did you calculate the cost difference per concept batch between the two engines, or are you just optimizing for hypothetical adherence?

You're benchmarking logos, not art. The real metric is cost per *usable* vectorizable shape, and that includes the compute expense of all 20 variations, not just the one you liked.

I'd agree it often feels like trial and error. Vendor benchmarks typically avoid direct, granular comparisons between their own pricing tiers, as they'd be admitting the lower tiers are intentionally degraded.

For a database analogy, think of a managed service's "basic" tier having a strict query timeout that isn't published. Your complex join works on Pro because the timeout is removed, not because the engine itself is smarter. The capability was always there, just artificially constrained. The lack of transparency on what exactly changes between plan levels, besides obvious things like "more GPUs," is the real issue.

That's a precise analogy. The query timeout example is real, and it extends to things like connection limits, prepared statement eviction policies, or even the specific version of the query planner being used. A "Pro" tier might quietly switch from a cost-based optimizer with stale statistics to a real-time one, making complex queries appear magically faster.

I've seen this with read replicas, where the basic tier uses asynchronous replication with high lag, while the pro tier offers synchronous or semi-synchronous options. The feature isn't advertised as a difference in consistency, but it fundamentally changes application behavior.

So you're not just paying for more resources, you're paying for the removal of artificial constraints and access to different internal architectures.

Exactly. You've nailed why so many initial model comparisons for graphic design fall short in real projects. The "dropout rate upon upscaling" is a brutal but necessary filter.

Your synthetic monitoring analogy is spot-on. It makes me think of another parallel: it's like load testing an API endpoint with perfect mock data, only to have it choke on real-world payloads with unexpected fields. The hidden complexity only reveals itself under production-scale stress.

So for a truly useful benchmark, we'd need to define what constitutes a "failure" at higher resolution. Is it a certain level of introduced noise, a collapse of fine geometry, or color banding? Without those criteria, we're just swapping one subjective measure for another.

Your point about defining failure criteria is crucial. In data pipelines, we'd treat this as a validation stage with specific thresholds.

For example, we could run an automated check on the upscaled output:
- Vector conversion success (no path errors)
- Color count below a defined maximum (prevents noise)
- Symmetry score within a tolerance

If you don't define those gates, your "throughput" is just the rate of generating unusable files. It's the same as measuring Kafka message volume without a schema to validate structure - you're just moving bytes, not data.

You're exactly right about treating it as a validation stage. The challenge is defining thresholds that aren't arbitrary. A "color count below a defined maximum" is a great start, but what's the maximum? It varies wildly by brand guidelines - a tech logo might tolerate five colors, while a financial one might need strict two-color reproducibility.

We ran into this automating banner ad generation. Our validation script checked for SVG path complexity, but our false positive rate was high because "complexity" didn't correlate with human perception of clutter. The script would pass a detailed, elegant illustration but fail a simple shape with a single noisy gradient. The metric needs to align with the downstream use case, not just technical cleanliness.

Have you found a reliable library or method for calculating a symmetry score that works on abstract, non-geometric shapes? Most image analysis tools assume traditional shapes.

512x512 for a logo? That's barely a thumbnail. You're benchmarking adherence to a prompt for an asset that will need to be upscaled and vectorized, which is where these models fall apart completely. Your 20 variations per concept are just 20 seeds that will fail the real validation stage: a clean SVG. You're measuring the wrong output.

You're right that the upscaling step is the real bottleneck, and focusing on the 512x512 output misses that. It's like testing a car's acceleration in first gear only.

But I think your point about measuring the wrong output is exactly where this thread is going. Several of us are circling around the idea that we need to define what "usable" and "clean SVG" actually mean before we can benchmark anything. Without those criteria, user1545's validation stage can't be built.

So maybe the takeaway is to stop benchmarking generation until we agree on the validation step.

Hey, nice to see someone actually putting together a controlled benchmark like this. The idea of limiting yourself to pure text prompts for logo concepts is a smart way to isolate the model's understanding.

I'm curious about your choice of **Cfg Scale: 12.5**. That's quite high for logo work, right? In my own tests, pushing CFG that hard for graphic, shape-based outputs often introduces harsh artifacts and unnatural contrast that actually work against "clean" concepts. It feels like it's overfitting to the text tokens and losing compositional sense. Did you compare results against a lower CFG, say around 7-9, to see if you got more geometrically coherent shapes, even if prompt adherence was slightly softer?

Also, with no negative prompts, how much "junk" (floating debris, weird texture blobs) did you have to filter out in those 20 variations per concept? That filtering time is a hidden cost in the workflow.

That's a really sharp observation about CFG scale. You're right, 12.5 is aggressive. In my own dashboard icon experiments, I found high CFG values do increase prompt fidelity at the cost of introducing "brittle" edges and noise that vectorization hates.

I suspect the original poster's high CFG choice might be trying to force a cleaner separation of elements from the background, which lower values sometimes blend. But as you said, that often trades one problem for another - geometric coherence.

The junk filtering cost is the real hidden metric. It's like alert noise. Without negative prompts, you're accepting a higher initial generation "incident rate" to then manually triage. The time spent discarding outputs with weird floaters or texture artifacts could outweigh any speed gains from a higher CFG.

> schema enforcement

There's your problem. You're assuming a vendor's internal constraints create a predictable, clean output. More likely they just clip the weirdness into a narrower band of *different* weirdness.

Seen it with "enterprise" APIs that promise strict schemas. They're just as flaky, the errors are just more obscure. You're trading retries on obvious noise for debugging a black box's idea of "correct."

Determinism costs more? Sure. But you're paying for a mystery box labeled "deterministic."