Hey everyone, been lurking here for a bit but finally have something to share that I think this crowd will appreciate. 😅
So, like many of you, we're juggling multiple model providers (Anthropic, OpenAI, Google Vertex AI, and a couple of open-source endpoints) for different workloads. The bill was getting... interesting, but more critically, we kept hitting weird latency spikes or occasional outages that would tank our user experience. Our initial "solution" was a manual config swap, which is basically no solution at all.
I've spent the last few weeks building an automated failover system that doesn't just ping for "up/down," but actively routes requests based on *real-time, percentile-based latency* and cost. The core idea is simple: always send the request to the "best" provider for the specific task at that exact moment. The implementation, however, was a fun CI/CD-style pipeline problem.
Here's the high-level flow:
* **Health & Latency Probe:** A lightweight service runs every 30 seconds, sending a standardized prompt (a simple summarization task) to each provider's endpoint we use. It doesn't just check average latency; it tracks the p95 and p99 over a rolling 5-minute window. A 5xx error or a p99 beyond our SLO (we set it at 10s for this probe) marks a provider as "degraded."
* **Routing Decision Engine:** This is a simple Go service that ingests the probe data. Its decision logic weighs:
* Current health status (obviously).
* The p95 latency for the specific *model family* (e.g., `claude-3-5-sonnet` vs `gpt-4o`).
* The cost-per-token for the input/output of our planned request.
* A configurable "preference" score for tasks where output quality varies (e.g., we strongly prefer Claude for long-form structured JSON generation).
* **Integration:** Our apps don't call the provider APIs directly. They call our internal gateway with a task type (e.g., `"creative_writing"`, `"code_generation"`). The gateway consults the decision engine and routes the request accordingly, adding a header to trace which provider was actually used.
The config for the decision logic lives in a YAML file we can update via Git. Changing the cost factors or the model mappings triggers a pipeline that validates the config, deploys it, and runs a canary test against a shadow endpoint. Feels just like rolling out a new k8s manifest.
```yaml
# Snippet from our routing_rules.yaml
task_types:
code_generation:
primary_candidates:
- provider: "anthropic"
model: "claude-3-5-sonnet"
max_accepted_p95_ms: 4500
cost_weight: 0.7
quality_bias: 1.2
- provider: "openai"
model: "gpt-4o"
max_accepted_p95_ms: 3500
cost_weight: 1.0
quality_bias: 1.0
fallback_provider: "openai" # If primary candidates are degraded
```
The result? We've automatically cut over during two minor OpenAI slowdowns and one Anthropic regional blip without a single user complaint. The cost savings are a nice bonus, as it now naturally shifts load to the cheaper option when latencies are comparable. It's saved our dashboard daily from going red.
I'm now curious about how others are tackling this. Are you using a service mesh approach? Commercial API gateways like Zuplo? Or is everyone just building their own little orchestrator like I did? Also, how are you defining "output quality" for programmatic routing? We're still using manual task-type mappings, but I'd love to automate that based on historical performance metrics.
pipeline all the things
Love the approach of tracking p95/p99 latency instead of just averages. That's where the real pain lives.
One thing you'll want to keep an eye on: that standardized summarization probe. If your prod traffic is mostly different types of requests (e.g., long code generation vs. short classification), the probe's latency might not reflect your actual workload. I've seen systems route to the "fastest" provider based on a trivial health check, only to get hammered on real tasks because of cold starts or different model load.
Consider adding a weighting factor based on the incoming request's estimated complexity. Just a thought from the night shift 😅
NightOps
That's a solid observation, and you're absolutely right about the probe mismatch. I've audited a system that failed exactly like that - their 'latency health' was based on a 10-token classification, but the main app was generating entire documents. The routing table became a beautifully misleading artifact.
But I'd push back on the complexity weighting. Estimating that on the fly adds its own latency and complexity. And what metric do you use? Token count is a start, but different providers have vastly different scaling for, say, a 1000-token JSON structured output vs. a 1000-token creative story.
Simpler fix: probe with a few different synthetic request profiles that mirror your actual high-percentile workloads. Route based on a blended score. It's noisier, but at least you're stress-testing the same paths your users will take.
- Nina
Good start, but a 30-second probe interval is too coarse for catching real latency spikes. A provider can go from fine to drowning in 15 seconds. You need to incorporate passive latency tracking from your actual production traffic.
Also, watch for cascading failure. If Provider A gets marked slow and everyone fails over to B at once, you'll melt B. Implement some jitter and a circuit breaker pattern in your routing logic.
shift left or go home
Great approach on moving beyond simple uptime checks. Your mention of a CI/CD-style pipeline for this makes me think you're likely logging all these routing decisions and probe results.
You absolutely need to capture the decision context for each routed request. Not just which provider was chosen, but the latency scores and cost factors at that moment, plus the timestamp. When this system inevitably does something puzzling at 3 a.m., that audit trail is the only thing that'll let you reconstruct why it thought sending all traffic to a single expensive provider was a good idea.
How are you handling the logging for the probe service itself? I've seen similar setups fail because the probe's own errors or latency weren't being captured, so the routing table was operating on stale or incomplete data.
Logs don't lie.
Spot on about the audit trail. It's the difference between a "black box" and a system you can actually tune and trust. When logging, I'd add that you also need to capture the *runner-up* providers and their scores. If your top choice fails and the request goes to #2, the log should show the original decision matrix, not just the eventual destination. Otherwise, you lose the context for why #2 was suboptimal at that moment.
And yes, the probe service itself has to be treated as a first-class, monitored application. Its health metrics and execution times need to be part of the same dashboard. If your probe starts lagging or failing silently, your routing table isn't just stale - it's actively dangerous, as you said.
Have you found a logging level that's detailed enough for forensic analysis but doesn't drown you in data? That's always the tricky balance.
Keep it real.