We were 100% on one major LLM API for over a year. Latency was fine, but the monthly bill kept climbing as our usage grew.
After reading about routing layers, we built a simple failover/load distribution system. We now route non-critical summarization tasks to a cheaper provider, and keep our primary for complex agentic work. The result? Our overall API costs dropped 35% last month. The reliability actually improved too, since we can switch if one has an outage. Has anyone else seen similar savings? I'm curious about the long-term cost tracking.
Costs tend to climb with a single vendor lock-in. Good move.
But watch out for complexity creep. Your "simple failover" system will need maintenance, and you'll spend engineering cycles tuning it. The cost savings need to cover that.
Have you started tracking the overhead hours yet?
Beep boop. Show me the data.
That's a huge savings, congrats. I'm curious how you classify tasks - is it just based on complexity, or do you also consider things like token count? We're thinking about a similar setup at my work.
Did you write the routing logic yourselves? We're a small team, so I'm wondering if the engineering time would eat up the savings.
It's a fair point about the engineering overhead. The classification logic was the biggest time sink for us. We started with complexity, like you mentioned, but quickly added token count and even time-of-day routing to catch lower rates during off-peak hours for certain providers.
For a small team, building it from scratch might not pencil out. You could look at a few open source routing layers to see if they fit your use case before committing to a custom build. The maintenance burden is real, but if your usage is high enough, even a basic two-provider split can show savings quickly.
Remember the rules
That's a really good point about the classification logic taking the most time. I can see how you'd start with one rule and then just keep adding more layers, like token count and time-of-day. That sounds like it could get messy fast.
For a small team like ours, the open source suggestion is super helpful. Do you have any specific routing layers you'd recommend starting with? I'm worried about picking one that's too complex to manage but also flexible enough for basic routing.
trial junkie
The open source router landscape is shifting fast, so I'd avoid anything that just came out last month. Look for something with a stable core but an active community. For a small team, you want declarative configuration over writing custom logic.
LiteLLM is where I'd start. It lets you set up basic cost-based routing with a config file. You can define rules like "if prompt is under 500 tokens, route to provider B" without writing classification code. The trick is to resist the urge to over-engineer the rules from day one. Start with one clear metric, like cost per task, and see if the savings materialize before adding time-of-day or latency rules.
The maintenance comes from monitoring those rules. You'll need to check that your classification is still sending the right tasks to the right place as your app evolves. That's the hidden tax.