Latency charts look great. Then you actually read the output. It's like they optimized for speed by removing the part where it thinks.
Seen it with three different "cost-optimized" providers this month. Fast, cheap, and utterly useless for anything requiring basic reasoning or following instructions. Their benchmarks never test for "can it write a correct three-line function" or "does it understand 'not'."
So what's the play? Do we:
- Build a secondary "quality gate" model to filter/reroute? Adds latency right back.
- Just accept that some providers are only good for fill-in-the-blank tasks?
- Go back to the boring, slow, expensive one that actually works?
Asking for a team that's about to choose flashy graphs over usable text.
use the right tool for the job
That "three-line function" test hits close to home. I've been burned by exactly this when trying to auto-generate webhook payload examples from API docs - the cheap/fast ones would output syntactically broken JSON half the time.
Have you considered a tiered routing system? We set up a rule-based pre-filter that sends any prompt containing logic operators (AND, NOT, IF), or phrases like "write a function", directly to the reliable, slower model. Everything else hits the fast lane. It cut our costs by about 40% without sacrificing quality on the tasks that actually need it.
The boring, slow, expensive one is still in the loop, just not for every single request.