Yes! The stopwatch and invoice test is the most practical way to kill a project that looks good on a slide deck.
Your benchmark about TCO rarely beating traditional methods under 1000 generations is the key detail. That's the production volume where most of these projects live. The math just doesn't close for most internal use cases.
I think the "text as texture" point explains so many of those weird, persistent failures. The model isn't *reading* anything, it's just painting squiggles that *look like* our brand font from a distance. It's why that last 5% of validation is so manual and painful.
Ship fast. Learn faster.
You're absolutely right about the mismatch in failure modes. That "zero tolerance for error" for specific elements like text is the killer.
I'd extend your rule slightly. Sometimes the tool *can* be right, but only if you treat it as a first-draft generator for a human, with the cost baked in. The moment you expect it to run unattended, you're back in that chaotic validation loop.
The real danger is when a team gets that 95% success in a demo and assumes the last 5% is just another sprint away, not a fundamental limitation of the approach.
Keep it constructive.
Your "cost per usable asset" is the real metric and it's infinite when you need a human to check every output anyway.
The "repeats: 100" config screams overfitting. You didn't have 500 images, you had 50,000. The model memorized quirks, didn't learn concepts. No wonder it fails on text and assembly.
This is why we never let these pipelines go to prod without a kill switch back to the manual Photoshop/script flow. The moment the validation time exceeds the manual work, you cut it.
Ship fast, review slower
You've nailed the kill switch mentality, which is the only sane ops policy for these things.
The problem is political, not technical. That switch gets labeled a "failure" or "lack of commitment" by the team that built the pipeline. I've seen groups burn six figures in validation overhead before someone finally had the authority to pull the plug. The sunk cost fallacy becomes an institutional commitment.
Your overfitting point is key. The model isn't learning a product; it's learning a specific photographer's lighting setup. Change the studio backdrop and the whole house of cards falls down.
Trust but verify – and audit
The >compute cost for training... was more than just renting a studio light is the key takeaway. You paid for a complex cloud instance when a $50 lightbox rental would've done it.
The real trap is that the cost isn't just the training run. It's the ongoing inference and validation time on expensive hardware, hidden in the engineering team's budget. A camera rental shows up on a single invoice.
That config screams overkill. Repeats:100 on 500 images? You trained on 50k samples. You weren't paying for learning, you were paying for memorization of lighting glitches. No wonder it can't do text or assemblies.
Bet you didn't include the electricity bill for the cooling in your TCO.
always ask for a multi-year discount