Troubleshooting: API responses are inconsistent between 'sonar' and 'sonar-small'.

That's not a bug, it's a feature. They're different models. Small is literally built to be a smaller, cheaper, faster inference. You'll never get identical output structures.

You can't fix this with parameters. Your parsing logic needs to handle model variance, or you need to pick one model and stick with it. For time-sensitive summaries where format matters, standardize on one.

You're right about different models, but calling it a feature is generous. It's a known pain point when switching tiers in any API. The real issue is documentation not setting clear expectations on output variance.

If format consistency is critical, you're correct they need to pick one. But they could also implement a post-processing normalization layer. That's what we did when our Slack bot switched models.

Welcome to the world of model drift. You're experiencing the core problem of switching model tiers in any service.

It's not just depth. Smaller models often strip out connective tissue and rephrase core facts, which breaks any parser expecting a consistent skeleton. Your expectation of format parity is the issue.

For time-sensitive summaries, you either eat the cost for sonar or build a much more flexible, and likely more fragile, parsing layer for small. There's no parameter fix.

Yeah, that's exactly the kind of headache I was running into last week. I tried swapping between models for a similar cost-saving idea and my whole post-processing script broke.

> not in the fundamental format of the answer

This is the tricky part. It's not just shorter sentences, the small model sometimes gives a list when the big one writes a paragraph, or swaps the order of key points entirely. Makes any kind of structured data extraction a nightmare.

Have you looked at using a system prompt to try and force a specific output format? I got *slightly* more consistent results by adding "Always structure your response with a summary sentence first, followed by bullet points." But it's still hit or miss with sonar-small.

Exactly. The system prompt trick is a band-aid, and a weak one. It adds token overhead that eats into the cost savings you wanted from using the small model in the first place.

So now you're paying more for small plus extra tokens, and the output is still unreliable. Feels like a lose-lose.

Yep, it's definitely a known thing. You're hitting the classic trade-off between model size and output stability. The parameter tuning route is limited, but I've had some luck with a two-pronged approach for my own monitoring summaries.

First, I lock down the `temperature` to 0 and set a low `top_p`. That helps a bit with the randomness. But the real fix for my parsing was adding a very explicit format directive in the *user prompt itself*, not just the system prompt. Something like:

```
Summarize the following. Provide the response in this exact JSON structure:
{"summary": "one-sentence overview", "key_facts": ["fact1", "fact2"]}
```

This nudges both models towards a similar skeleton. `sonar-small` will sometimes still miss a fact or two compared to `sonar`, but at least the JSON parsing doesn't break. It adds a few tokens, but the cost is still far below using the full `sonar` model.

The inconsistency in basic facts is the harder nut to crack. If that's a deal-breaker, you might need to accept that `sonar-small` is a different beast and treat its output as a "first draft" that needs a quick human check for critical points.

You're spot on about the normalization layer. That's the pragmatic engineering path when you can't lock to a single model. The crucial detail, from our own migration, is you have to treat it as a distillation step, not just a reformatter.

We built a lightweight service that uses a very cheap, structured extraction model (like Claude Haiku) to read the *varied* API output and repackage it into our fixed schema. It adds a bit of latency, but it's cheaper and more reliable than trying to coax perfect format from the smaller model with prompts. It essentially abstracts the model variance away from your core logic.

The documentation point is key though - you only consider this extra complexity because you've been bitten by the output drift. Clear docs on expected variance between tiers would save so many people this headache.

That's a really clever practical solution, and framing it as a *distillation* step is key. It shifts the burden of consistency from the generative model, which is fundamentally variable, to an extraction model specifically built for structured output.

I'm curious about the cost/latency trade-off you mentioned, though. When you say it was "cheaper and more reliable," did you factor in the compute and complexity cost of running the second service? For some teams, that extra architectural piece might be a bigger lift than just standardizing on the more expensive model upstream.

Your last point on documentation is so true. If a provider explicitly stated, "Output structure is not guaranteed across model tiers; treat them as different endpoints," it would set the right expectation from day one.

You're right about the system prompt band-aid. The hit-or-miss nature you described comes down to the small model's limited capacity for instruction following. It can't hold both the task and a complex format instruction as reliably.

For time-sensitive summaries, the unpredictable latency from a failed format parse often outweighs the raw inference speed gain of using `sonar-small`.