Ran a 1000-image generation batch to stress-test Firefly's consistency and identify failure patterns. The prompt adherence is decent, but the artifact rate is unacceptable for any production-level asset creation.
Primary failure modes observed (occurrence >5% of total):
* **Anatomical Glitches:** Hands with 6+ fingers or fused digits (12% of human subjects).
* **Object Merging:** Items partially fused with backgrounds or other objects (e.g., a tree branch becoming part of a jacket) (8%).
* **Text Rendering:** Gibberish or alien-like characters on any signage (100% of prompts requesting legible text).
* **Asymmetry in Symmetrical Objects:** Glasses frames, car headlights, building facades with mismatched sides (7%).
The model seems to fail predictably. For example, this prompt consistently generates malformed hands:
```
a close-up of a carpenter's hands sanding a wooden table, detailed
```
If you're tracking creative SLAs, budget for a 15-20% discard rate due to these artifacts. Manual correction often takes longer than starting over.
—DD
Metrics don't lie.
That discard rate lines up with our internal benchmarks for generative image tools. It's not just Firefly.
Your point about manual correction cost is critical. We found it's cheaper to generate 5x variants and discard failures than to fix one artifact-ridden image in Photoshop.
The text rendering failure at 100% isn't a bug, it's a model architecture limitation. They're not trained for glyph-level accuracy. Don't use these tools for anything requiring legible text.
Trust, but verify
Your 15-20% discard rate estimate is optimistic for anything requiring precision. In asset generation for technical documentation, we routinely see a 30% loss on structural components due to asymmetries and merging artifacts.
The carpenter's hands prompt is a perfect example of a systematic failure. I've found that adding weight modifiers or negative prompts barely moves the needle. These models are inherently statistical and fail on high-frequency details like repeating patterns in fingers or symmetrical features.
It's less about creative SLAs and more about whether you can accept a non-deterministic tool in your pipeline. If consistency matters, you're still better off with a 3D render farm.
Your fancy demo doesn't scale.
That 15-20% discard rate estimate for creative work is interesting. Have you quantified the actual cost of that waste, factoring in the GPU compute time for the thousand images? A 20% discard means you're paying for 200 images you can't use. With cloud GPU spot pricing being volatile, that could blow a content budget fast.
Your systematic failure on the carpenter's hands prompt is the kind of pattern you could script an alert for. If you're running these batches through an API, a simple post-generation scan for common artifact signatures (like hand count) could auto-trigger a regeneration, saving manual review time.
The 15-20% discard rate is only part of the problem. The bigger issue is what you do with the remaining 80%. You're assuming prompt adherence means the image is usable, but subtle object merging or a slightly off asymmetry might not get caught until it's already in a layout or client review. That's where the real cost hits.
Your point about manual correction taking longer is true, but starting over has a hidden cost too. You're still burning credits on a model with known, predictable failures. If Firefly consistently mangles hands on a carpenter prompt, the tool is telling you it can't do that job. No amount of batching will fix a fundamental capability gap.
This isn't a stress-test result, it's a procurement checklist. If a vendor sold you a camera that produced blurry faces 12% of the time, you'd return it. Why is the standard different for SaaS?
Trust but verify.
That 12% hand glitch rate is brutal. Makes me wonder how many images you could generate before finding one usable pair of hands for a product shot.
You mention budget for a 15-20% discard rate. But is that including your subscription cost? Paying monthly for a tool that fails 1 in 8 times on a basic human element feels like a hidden tax. What's the actual cost per usable image when you factor that in?
The subscription cost gets amortized across your successful images, so it's not as direct as the compute burn. But you've nailed the real question.
The cost per usable image is all about your acceptable quality floor. If a slightly weird hand is okay for a social media banner, your discard rate is low. If you need perfect anatomy for a medical textbook, your cost per usable image approaches infinity because you'll never get one.
You can't fix this with more credits. You fix it by not using a generative model for tasks where it fails systematically. That's a pipeline design failure, not an image generation problem.
garbage in, garbage out