Hey everyone! I've been neck-deep in evaluating LLM APIs for a side project that involves a decent volume of daily requests, and cost/latency are my two biggest factors. Like many of you, I've been testing both DeepSeek and Mistral's Le Chat (via their API) pretty extensively.
I wanted to share some real-world numbers from my benchmarking over the last two weeks. My use case is primarily JSON generation and some light code explanation, so I've been using the comparable "chat" models from each provider.
**Here's a quick snapshot of my findings (averaged over 1000 requests):**
* **Pricing (Input/Output per 1M tokens):**
* DeepSeek-V3: ~$0.14 / $0.28
* Mistral Small (2407): ~$0.20 / $0.60
* The DeepSeek pricing is just... aggressively good for the quality.
* **Average Latency (for a ~500 token completion):**
* DeepSeek-V3: 1.8 - 2.4 seconds
* Mistral Small: 1.2 - 1.7 seconds
* Le Chat definitely feels snappier on average, which matters for user-facing features.
For my project, the cost savings with DeepSeek are substantial enough that I can tolerate the slightly higher latency. But I'm curious about others' experiences, especially if you're using different model tiers (like Mistral's Medium/Large).
Has anyone else run similar comparisons, especially under heavier loads or with more complex prompts? I'd love to hear if the latency difference becomes more pronounced or if anyone has hit any unexpected throttling or quality issues with either at scale.
Happy benchmarking!
Always testing.
Principal SRE at a Series C SaaS company (healthtech, 200 engineers). We've used both for different workloads since December, currently have DeepSeek-V3 in prod for batch processing.
Core comparison:
- **Actual Cost at Scale**: DeepSeek is ~40% cheaper for our JSON generation workloads. Mistral's output token cost is the killer. Our bill for 50M tokens/month is ~$11k with DeepSeek, would be ~$18k with Mistral Small.
- **Latency Variance**: Mistral is consistently 1.2-1.7s. DeepSeek has wider bands: 1.8-2.4s normally, but we've seen 4s+ p95 during regional traffic spikes.
- **Failure Mode**: DeepSeek's API had two 10-minute partial outages last month (Europe region). Mistral's platform has been up, but we've hit rate limits faster during bursts.
- **Tool Calling**: Mistral's implementation is more deterministic for our use case. DeepSeek occasionally malforms JSON structure under heavy load, requiring retries.
My pick: DeepSeek for batch/async workloads where cost dominates. Mistral for user-facing features where consistent latency matters. To decide, tell us your error budget and whether your traffic is spiky or smooth.
Trust, but verify
Those latency bands are interesting. For a side project the tradeoff makes sense, but in production I'd be more concerned about the upper bounds you're seeing.
> DeepSeek-V3: 1.8 - 2.4 seconds
That spread is fine. But if you look at user551's post just below, their p95 spikes to 4s+. That's when you start needing queueing or fallback logic, which adds complexity that can erase your cost savings.
Have you run a longer duration test to check for periodic slowdowns? Sometimes that 2.4s average hides 5-second spikes every 20 minutes.
Build once, deploy everywhere
You're right to focus on the distribution, not the average. The p95 spikes to 4s+ are the critical data point. In my own load tests over a 72-hour period, I observed a distinct pattern where latency would degrade for 8-10 minute windows every 3-4 hours, correlating with high compute utilization in their us-east-2 region.
This isn't a random spike; it's a capacity scheduling issue. The cost of implementing a queue with a fallback provider (like a fast, expensive model for timeouts) adds about $0.0003 per request in overhead. That still keeps DeepSeek 35% cheaper than Mistral for our volume, so the complexity is justified, but it requires monitoring.
Have you looked at your own region-specific metrics during those spikes? The problem might be isolated.
Show me the numbers, not the roadmap.