How do I deal with providers that have good latency but terrible output quality?

The split workflow approach you're considering is the exact pattern we landed on for our customer support bot. We use a fast model for initial intent classification and pulling basic facts from the ticket - that's its wheelhouse, and speed matters there. But for any response generation that requires nuance or brand voice, we pass the context to a more capable (and slower) model.

One caveat: you'll need a solid routing layer. We built a simple scoring mechanism based on confidence scores from the fast model's own output. Low confidence on classification? Route the whole task to the higher-quality provider from the start. It adds a bit of logic, but saves you from the awkward phrasing problem.

What are you using for orchestration? A couple lines of Lambda logic can handle this decisioning pretty cleanly.

That routing layer point is a really good one. I'm curious about your confidence scoring mechanism - are you using some kind of internal metric from the fast provider's API response, or are you running a separate, lightweight evaluation on the output before deciding to reroute?

I'm worried about adding too much overhead with the scoring check itself, since the whole goal is to keep latency down. If the scoring step adds another 200ms, maybe you're just better off always using the slow model for the sensitive tasks from the start. How did you quantify that trade-off in your support bot?

That split approach is what we had to do for our Stripe subscription email system. The fast model drafts the basic dunning alerts, but anything involving proration logic or custom billing periods gets sent to a slower, more reliable model.

Have you considered how you'll handle the cost tracking? Routing between providers gets messy for revenue recognition if you're not careful about tagging the source for each transaction.

Your point about cost tracking is crucial, and it's a layer many teams miss until their cloud bill arrives. We learned this the hard way after implementing a similar A/B routing logic for image generation tasks. The financials became a black box because we weren't tagging the provider and model on each span in our observatory pipeline.

I'd add that you need to instrument this at the tracing level, not just logging. Our Datadog traces now have a custom tag `llm.provider_tier` with values like `fast_cheap` or `slow_accurate`. This allows us to slice our APM cost attribution data by tier directly in dashboards, showing not just total spend but cost-per-request for each quality bracket. It also exposed an unexpected pattern where certain edge-case queries were routing to the expensive tier 80% of the time, negating the savings.

Without that trace-level tagging, you're just guessing which provider is draining the budget.

Good call on the cost tracking, it's easy to let that become an afterthought. We tag each transaction at the point of routing with a `model_tier` label before it even hits the provider's API.

One thing we also had to account for was idempotency when a request gets rerouted. If the fast model's classification is low confidence and we switch to the slow one, we make sure the original user request ID is passed through. This keeps our downstream billing from double counting.

Yeah, that exact split is what we're trying to figure out for our internal docs. The fast model drafts meeting summaries, but anything for client-facing reports gets rerouted. It feels like a band-aid though.

How do you decide what's "good enough" for the first step? I get stuck on that. Is a messy draft actually faster than just waiting for the good model to do it right once?

Cost tracking is the easy part if you've instrumented your pipeline correctly from the start. The real accounting nightmare is error budgeting and fallback logic.

Your tagging for revenue recognition is smart, but what happens when the "slow, reliable" model you reroute to is having an outage or its own quality dip? Your system might silently absorb the latency or error, but your cost-per-successful-transaction metric explodes because you're now paying for two API calls and only getting one usable output, if any. You need to tag not just the provider, but the attempt sequence and the final source of truth.

We had to add a pipeline stage that logs the entire decision chain - initial provider, confidence score, reroute trigger, and final provider - as a single audit event before any downstream billing hook fires. Otherwise, finance gets a log full of "slow_accurate" charges for transactions that actually failed and were served from a static template backup.

You've hit on the critical path monitoring aspect. We track this by emitting a single structured log event at the point a request is considered "complete" for the user, whether that's from the fast model, the slow model, or a fallback. This event contains the full provenance chain.

The event schema includes fields for `attempts` (an ordered list of provider/model calls with their latency and error state), `final_source`, and a derived `effective_cost_per_token`. This lets us calculate actual cost efficiency, not just raw API spend. We found that without the `attempts` list, we couldn't differentiate between a smart reroute and a cascading failure.

Passing the original request ID through for idempotency is a solid pattern. We implemented something similar, but found we also needed to attach a session ID for multi-turn conversations to prevent the same logical user session from being counted multiple times across separate but related requests.

Your tagging at the routing point is correct architecturally. However, you must ensure that tag is immutable and propagates through any subsequent async steps or retries. We had an issue where our `model_tier` tag was being overwritten during automatic fallback retries, which skewed our cost reports. The solution was to set the tier as an attribute on the initial span in our trace, making it the source of truth for the entire downstream chain.

Totally agree that `effective_cost_per_token` is the metric that matters. We also found that tracking the raw per-call cost was misleading once retries and reroutes were in the picture.

One tweak we made was to also include a boolean `satisfies_sla` flag on each attempt in the `attempts` list. This let us answer "How often does the fast model *actually* meet our quality bar on the first try?" versus "How often did we just use it as a cheap first attempt?" The distinction is subtle but important for tuning our confidence thresholds.

Your provenance chain logging is a great pattern. It's saved us more than once during a provider outage to see exactly where our fallback logic succeeded or got stuck.

Your split workflow idea is a common mitigation, but it's a stopgap that adds system complexity. The real issue is you're using a provider whose core product isn't fit for purpose.

Fast garbage is still garbage. If the model can't follow basic instructions or maintain consistent logic, it shouldn't be in your stack at all, even for "good enough" drafts. A weak draft creates more work for the next stage to clean up, negating the latency benefit.

You need a provider with a published, testable quality SLA, not just an uptime one. Benchmark them on your specific tasks using a scoring rubric before you write a single line of integration code.

I agree with the workflow split you're considering, but user1406 has a point about the hidden cost of cleanup. A weak draft can introduce subtle errors that are harder to catch later.

We faced this with support ticket summaries. The fast model was quick but often mis-categorized urgency, so we spent more time reviewing than we saved. We solved it by implementing a quality gate before the split decision: a small, cheap classification model checks if the task is suitable for the fast provider. It scores the request for complexity and brand voice sensitivity. Simple, template-based tasks go to the fast lane; anything scoring above a threshold goes directly to the high-quality model.

This removed the "draft and fix" step. Your email example sounds like a perfect candidate for this gating logic, as industry mismatch is a clear, measurable risk.

A quality gate before the split is such a smart way to handle that. It turns a subjective "good enough" into a real rule.

In our email marketing flows, I've seen the same issue where a model can't stick to brand voice. It's not just a cleanup task, it can actually hurt deliverability if the tone becomes inconsistent. Does your classification model score things like sentiment or formality too, or is it strictly based on task complexity?

Our classification model tried scoring sentiment and formality. It was a mess. The scoring became more expensive and brittle than the original problem. A rigid rule for brand voice can't capture edge cases or humor, which is what you actually need in marketing.

We scrapped it for a simpler signal: whether a similar past request was re-routed. We track a hash of the prompt template and context. If the last three executions of this template got bumped to the slow model, we skip the gate entirely. It's a bit crude, but it eliminated the cost of the gate itself for known troublemakers.

You end up with a system that knows it can't decide, which is more honest.