Just tried to swap out our support chatbot from GPT-4 to Mistral Large. Had to roll back within 24 hours 😅
The cost per token looked amazing on paper, but we hit two major snags:
* **Latency spikes at p99:** Our load tests showed consistent sub-200ms for GPT-4. Mistral jumped to 2-3s during even moderate concurrency. The retry logic we had to add ate any potential savings.
* **Structured output failures:** We use function calling for ticket categorization. Mistral's JSON mode kept breaking on nested fields. Example failure rate:
```json
// Target schema
{"category": "billing", "urgency": "high", "user_tier": "pro"}
// Common malformed response
{"category": "billing", "urgency": "high", "user_tier": pro} // Unquoted string
```
For now, it's back to GPT-4 for reliability. Maybe good for async batch jobs, but not for live user chat.
Anyone else run into this? Did you find a workaround, or just punt on Mistral for real-time use?
#savings
Thanks for sharing the real numbers. That latency jump is a serious blocker for any live interaction - users notice that kind of delay immediately. You're right to focus on p99, not just averages.
The structured output issue is particularly frustrating because it's so fundamental. We had a similar experience testing for a moderation helper, where the model would just omit a required field from the JSON schema entirely about 15% of the time. No amount of prompt engineering fixed it reliably. You're probably spot on with the batch job assessment - it's fine for tasks where you can validate and retry without a user waiting.
Has anyone tried running Mistral's models through a different inference provider, like together.ai or perplexity, to see if it's more about their specific infrastructure than the model itself?
Keep it civil, keep it real.
Good point about trying different infrastructure. We ran some checks through Azure's Mistral endpoints and saw the same JSON inconsistency, so it seems baked into the model's output logic.
The p99 latency is the real killer for chat. We built a retry queue with exponential backoff, but that just adds complexity. The cost saving disappears when you're paying engineers to build workarounds.
Has anyone done a direct comparison on the same hardware, like both models on VLLM? I'd be curious if the gap is smaller when you strip out the provider's orchestration layer.
null
We hit the same latency wall in our tests. The kicker is that p99 spikes get even worse when you scale - they don't just add 2 seconds, they completely blow out your auto-scaling thresholds.
On the structured output, we found the JSON issues were tied to certain input lengths. Short prompts were fine, but anything over 300 tokens and the model started dropping quotes or commas. A post-processor validation layer became mandatory, which just moved the cost problem.
The workaround we settled on was using it strictly for internal summarization tasks where a human reviews the output. Never for anything customer-facing.
Build once, deploy everywhere