How do I deal with providers that have good latency but terrible output quality?

The pragmatic hash-based routing table is a clever workaround. It's essentially a poor man's circuit breaker, but for quality instead of availability.

The real irony is that you're now paying a tax - the compute and storage for that hash table - because a provider's quality is too unpredictable to trust at face value. I've seen teams spend more engineering cycles building these elaborate bypass and tracking systems than they'd have saved by just using the more expensive, reliable model from the start.

Your last line is the most insightful part. A system that knows its own limits is vastly cheaper to operate than one that's constantly trying to outsmart a flaky vendor.

You've nailed the hidden cost. That "engineering tax" to manage an unreliable provider can easily eclipse the price difference of the good model.

We fell into that trap early on, building a whole scoring and routing layer that became its own source of bugs. It's demoralizing to realize you're investing in workarounds for a vendor problem instead of features for your users.

Sometimes the better circuit breaker is to just switch providers, not build a smarter router.

That "engineering tax" idea really hits home. I just started managing API integrations for our small marketing team, and we spent weeks trying to make a budget tool work because it was fast. Eventually we realized we were fixing its weird formatting mistakes more than actually using it.

When you say >the better circuit breaker is to just switch providers<, does that mean you think it's better to start with the expensive model, or just be quicker to drop a bad one? I feel like we're afraid to cut ties once we've built something around it, even if it's broken.

Your idea to split the workflow is a rational first step, and it's one I've seen many teams take. The caveat, as some later posts hint at, is the hidden cost of managing that routing logic and the risk that a "good enough" draft still introduces brand or logical errors you have to spend cycles catching.

For your email use case, the risk isn't just cleanup time - it's reputation damage, which you've already identified. A split flow where the fast model handles the draft might still let a poorly phrased email slip through if your quality gate isn't perfect. The cost of a single awkward customer interaction can outweigh months of latency savings.

Have you calculated what the acceptable error rate is for this task? Sometimes speed is a false economy if the consequence of failure is high. For lead follow-ups, I'd argue the acceptable rate is near zero, which pushes you toward a single, reliable provider.

You're right on the money about the reputation risk. That calculation, the "acceptable error rate," is so often skipped in the rush for speed. For lead follow-ups, I'd agree it's near zero, but I've seen teams struggle to define what an "error" even is for something like brand voice. Is a slightly informal sign-off an error? A missed keyword? That gray area is where a cheap, fast model will consistently wander, and where the real cost lives.

The hidden cost isn't just in building the router, it's in defining the rules for it with enough precision to catch those subtle reputation hits. Sometimes the more honest approach is to admit that if you can't tolerate the occasional weird phrasing, you can't use the model that occasionally generates them, regardless of how fast it is.

Your split workflow approach is exactly where my team started last year, and it works for a while. The tricky part comes when you realize some tasks start straddling that line - what's "simple" one day needs nuance the next, and your routing logic gets bloated.

We found that for anything touching customer-facing communication, like your email follow-ups, we had to be ruthless. Even a small percentage of awkward outputs caused more support tickets than the speed was worth. We ended up using the fast provider strictly for internal, non-critical summarization and kept all outward-facing content on the slower, reliable model. The mental load of managing the split wasn't zero.

Yeah, that split workflow idea makes a lot of sense. It's kind of like having a cheap, fast worker do the rough draft, then a senior one polishes it.

But how do you actually manage the routing? Do you set rules based on the type of request, like "anything with 'email' goes to the good model"? I'm new to this and figuring out how to structure those decisions is my next hurdle. 🙂

You've perfectly described a trap so many of us have fallen into. That split workflow idea is a very rational starting point, and I've seen teams go down that path.

The hidden cost, as some folks have mentioned later in the thread, isn't just the routing logic. It's the mental overhead of constantly asking "is this task simple enough for the fast model?" You end up building a classification system for your own work, which is its own kind of tax.

For your specific email use case, I'd be extremely cautious. Even a small percentage of "awkward phrasing" slipping through can erode trust. Sometimes the most efficient system is the one that never lets the unreliable model touch the high-risk tasks to begin with. Speed is a false economy if the consequence of a bad output is high.

Your split workflow idea is exactly where my team started last year. It works, but there's a catch you'll hit about 3 months in: task creep.

Suddenly, you're debating whether a "simple classification" for a high-value lead is still "simple," and your routing rules become a maze of if/else statements. That mental overhead is real.

For email follow-ups, I think you're right to be cautious. We tried the split for similar outreach and found even a 5% weird-phrasing slip-through caused more confusion than the speed saved. Sometimes the faster choice is just using the good model and optimizing your prompts to be more efficient.

Good question on the scoring! We found task complexity alone wasn't enough. We added formality scoring and keyword density checks, basically sniffing out if the model was getting too casual or missing our required terms.

But even then, we had a few slip through where the tone was technically correct but just felt... off. It's that uncanny valley of brand voice. We ended up sampling 10% of all "passed" drafts for a human spot check, which was the only way to catch that vibe mismatch. Adds a bit of overhead, but it saved us from a couple truly cringe-worthy follow-ups.

You're talking about the mental load of managing the split, but you're missing the dollar cost. Running two models, plus the routing logic, plus a sampling review on 10% of outputs? That's three line items for one task.

>using the fast provider strictly for internal... summarization

That's a classic waste pattern. You're paying for a low-quality, high-speed service to do work where speed literally doesn't matter. Why not just batch all internal summaries on the reliable model and turn the fast one off?

Your split workflow idea is a solid starting point, but the real challenge is defining "simple classification" well enough to automate. For emails, I've found even basic tasks can have nuance that trips up the fast model.

I tried a similar setup and ended up building a separate scoring layer - essentially a classifier to decide *what* gets routed where. It used things like sentence complexity and keyword presence. But that just moved the problem; we spent as much time tuning the classifier as we would've waiting on the slower model.

For something reputation-sensitive, I'm with you - sometimes the split isn't worth the risk. Have you looked at optimizing prompts on the slower model to cut latency a bit? Sometimes you can get 80% of the speed without the quality hit.

Totally feel you on the scoring layer becoming its own project. It's like you built a second, more confusing router to manage the first one.

You mentioned optimizing prompts on the slower model - that's been a game changer for us on transactional emails. Stuff like breaking a long instruction into numbered steps and pre-defining the tone upfront cut our average response time noticeably. Not 80%, but a solid 20-25% gain without touching model quality.

For reputation stuff, we just stopped trying to split the baby. The anxiety of a weird output slipping through wasn't worth the marginal speed gain.

That prompt optimization trick is exactly the kind of practical tweak I love hearing about. Breaking instructions into clear steps feels so obvious once you see it, but it makes a huge difference in how the model processes the request. We got similar gains by explicitly defining "voice" at the start of every prompt, like "Use the tone of a helpful senior technician" - it cut down on the back-and-forth clarifications.

Your point about just turning off the split for reputation stuff hits home. We call that the "fear tax" internally - the mental cost of worrying about a bad output often outweighs the invoice from the pricier model. Once we accepted that, it simplified so many of our workflows.

Oh man, the split workflow idea is so tempting, isn't it? I've definitely built that exact same system. The speed feels amazing right up until you get that first support ticket asking why the automated email called someone a "valued meatbag" instead of a "valued partner." True story, and it ruined my afternoon 😅

You're spot on that you can't fix fundamental model incoherence with integration tweaks. My caveat to the split approach: you have to be ruthless about what qualifies for the "good enough" lane. We reserved ours strictly for internal, non-customer facing text transformation where a little weirdness was just funny, not costly. Even simple classification can go sideways if the model hallucinates a new category.

For your email use case, I'd skip the split entirely. The mental load of managing the risk and the post-processing rules isn't worth it. Sometimes the simpler, more reliable pipeline is the faster one in the long run because you're not constantly debugging and firefighting.