Your focus on the cost angle for intermittent failures is spot on. The monitoring and retries create a hidden operational tax that's rarely accounted for in SaaS budgets.
Testing across regions and times is a strong diagnostic, but I'd caution that correlation with peak load could also point to infrastructure scaling issues rather than a deliberate throttling policy. A poorly configured auto-scaling group on a cloud provider can cause similar latency-induced quality drops, which is an execution problem, not a business choice.
It still comes back to the core issue others have raised: without transparency into the service's operational parameters, we're left guessing whether we're diagnosing a bug, a cost-cutting measure, or an engineering shortfall.
Yeah, the scaling issue point is huge. I've seen a weird version of that where an ASG was fine but the load balancer health checks were too aggressive, causing constant instance cycling. Images would degrade exactly during peak times, but it looked like a model issue.
It makes me wonder if there's a way to detect infrastructure problems from the outside, like watching for specific error patterns in the API response times alongside the bad images. Probably not easy without logs, though.
So if it's not the model, but the serving layer, how would you even start to prove that to a vendor who isn't transparent?
Wow, this is a really detailed breakdown. I never thought about it being a system error like a distributed queue. That's a bit over my head, but it makes sense.
If the prompt gets messed up early on, then the rest of the process is just working with garbage. But how could we even tell if that's happening? It's not like they give us logs.
So is there any way for a regular user to test if it's a tokenization problem, or are we just stuck?