Just built a cost calculator that factors in retries and fallbacks. You'd be surprised.

Everyone talks about cost-per-million-tokens like it's the only number that matters. They'll pick the cheapest provider, wire it into their stack, and call it a day. Then the pager goes off at 3 AM because the "cheap" endpoint is returning 429s, and your fallback chain is burning through your GPT-4 budget. The real cost isn't on the pricing page; it's in the retry logic and the cascading failure scenarios you didn't model.

I built a tool to analyze this after a particularly expensive month where our "cost-optimized" routing logic failed spectacularly. We were using a mix of providers (Claude, GPT-4o, Command R+) with automatic failover. The problem? A surge in traffic caused latency spikes on our primary, which triggered retries, which then also failed over, creating a thundering herd against our fallback providers. We paid for every single attempted token, successful or not, and the latency went through the roof.

The calculator simulates request flows under different conditions. You feed it your provider list, their per-token costs, their expected latency distributions (P50, P90, P99), and your retry/fallback policy. It then runs Monte Carlo simulations to show you the *effective* cost-per-successful-request and the realistic latency profile.

Here's a simplified version of the configuration it uses:

```yaml
providers:
- name: "openai:gpt-4o"
cost_input: 5.00 # per 1M tokens
cost_output: 15.00 # per 1M tokens
latency_p50_ms: 250
latency_p90_ms: 800
error_rate: 0.005 # 0.5%

- name: "anthropic:claude-3-5-sonnet"
cost_input: 3.00
cost_output: 15.00
latency_p50_ms: 300
latency_p90_ms: 1200
error_rate: 0.003

routing_policy:
primary: "openai:gpt-4o"
fallbacks: ["anthropic:claude-3-5-sonnet"]
retries: 2
retry_timeout_ms: 5000
fallback_on: ["5xx", "429", "timeout"]
```

The key outputs are:
* **Effective Cost per Successful 1K Tokens:** Incorporates the cost of all retry and fallback attempts. A provider with a low base cost but high error rate can become more expensive than a stable, pricier one.
* **Latency at Percentile (Actual):** Your P99 latency isn't the provider's P99. It's your provider's P99 *plus* the time spent failing over after N retries. This is what your users actually feel.
* **Budget Risk Profile:** Shows which provider volatility (error rates, latency spikes) most directly impacts your monthly spend.

The biggest surprises from my own analysis:
* Using a very fast but slightly less reliable provider as a primary can be cheaper *if* your retry logic is aggressive and your fallback is significantly slower. The retries succeed quickly most of the time, avoiding the slower fallback cost.
* Adding more fallbacks doesn't linearly increase resilience; it creates a complex cost surface. A second fallback is often only worth it if the first is fundamentally unstable, not just slow.
* For batch processing jobs, tuning for absolute lowest cost-per-token makes sense. For user-facing requests, you must optimize for the *product* of cost and high-percentile latency. Spending 20% more on tokens to cut your P99 latency in half is almost always the right trade-off.

Stop looking at static price sheets. Model the actual request flow, with all its retries and errors. The numbers you get will change your architecture.