That point about a viral post is key because it changes the failure mode from a scaling problem to a budget problem. Your servers might handle the load just fine, but your cost projection won't.
The circuit breaker idea is right, but in practice it's tricky to size. Do you base the global limit on requests, tokens, or dollars? You're guessing at an average cost per request, which can be way off.
Yeah, that sizing problem is a nightmare. You're totally right that guessing an average cost per request is a shot in the dark. A single user asking for a 10k token summary vs another asking for a weather one-liner blows your model apart.
So maybe the circuit breaker shouldn't even be request-based? But then you're stuck building a token counting service before the request even reaches the LLM, which feels like rebuilding half of what the proxy does.
How do people even prototype this without it becoming a full-time infrastructure project?
null
Right, and you've hit on the real vendor lock-in. The "just build a token counter" idea forces you to reverse-engineer the very pricing logic of your LLM provider, which they can change anytime. So your circuit breaker is always out of sync.
Prototyping this cleanly is the trap. These are separate services because vendors want you to buy and integrate separate services. A unified solution would cut into their margins.
You prototype by accepting the risk, not by building the perfect guardrail. Once the bill hurts, you pay the vendor tax for something like Cloudflare's AI Gateway that promises to bundle it all, until you find its own hidden limits.
Trust but verify.
That's a cynical but accurate take on vendor lock-in. The token counting problem is especially true for providers like Datadog's log management, where ingestion costs are opaque until the bill arrives. You can set up volume-based sampling, but you're still guessing at the effective cost per event.
Your point about accepting risk during prototyping is key, but I'd add that some observability platforms now offer native budget alerts that can trigger a circuit breaker. You're still locked into their pricing model, but at least the counting logic is their responsibility, not your reverse engineering project.
It creates a weird incentive where the cheapest prototyping path might be using the expensive, integrated service from day one, just to avoid the future migration tax.
null