Totally agree about that bias risk. We almost fell into the same trap where our filter started favoring the "safe", slightly blurry outputs the model was overproducing, just because they were less likely to have obvious artifacts. The style drifted way off-brief.
That human review panel you mention is crucial. We do something similar: we have our panel score on two axes - one for "correctness" (no glitches) and one for "vibe" (hits the creative brief). The filter is only trained on images that pass both. It adds a step, but it keeps the quality definition from getting watered down.
Webhooks or bust.
Glad you caught the drift away from the "vibe". Everyone obsesses over the artifact filter, but letting it kill the style is a silent fail.
That two-axis scoring is the right call, but I bet your "vibe" panel has its own drift over time. Seen a team's "good" rating slowly shift towards whatever the CEO liked in last week's demo. The ground truth isn't so ground.
Trust but verify.
Your data on predictable failure modes is exactly what we needed to formalize our vendor risk assessment for generative AI services. We've been mapping artifact rates like yours to specific contractual clauses.
For instance, that 100% text rendering failure isn't just a technical glitch, it's a functional non-compliance if the vendor's documentation implies the capability. We now require explicit carve-outs in the SLA stating text generation is unsupported, which shifts liability and forces a conversation about their roadmap.
Similarly, your 12% hand failure on a specific prompt provides a concrete test case for performance benchmarking during the evaluation period. If a vendor can't demonstrate improvement or at least consistency on that exact prompt over a new 1000-run batch, it triggers a review of their stability commitments.
Have you considered whether these failures constitute a material breach under the "fitness for purpose" doctrine in your master service agreement? It depends on your use case, but for production asset creation, a consistent 15-20% discard rate might cross that threshold.
RTFM — then ask for the audit
That's a really sharp application of the data. Formalizing those failure modes into contract terms moves the conversation from technical debt to vendor accountability.
One caveat: the "material breach" argument hinges entirely on how the purpose is defined in the agreement. A vendor could argue that a 15% discard rate is an acceptable industry baseline for creative generation, unless your use case and required throughput are explicitly specified as a schedule. Vague language like "high-quality image generation" won't hold up.
Have you found vendors pushing back on including specific, measurable test prompts in the SLA? I'd expect some resistance to that level of granularity.
—daniel