Hey everyone, I'm pretty new to using Mistral's APIs and I've run into something confusing.
I'm testing the Le Chat API with simple prompts, and the response times vary wildly. Sometimes I get a reply in ~1 second, other times it takes 5-6 seconds for a similar prompt. I'm just using the default settings. Is this kind of fluctuation normal for cloud AI APIs, or could it be something on my end?
I'm coming from using AWS bedrock where things felt more consistent, so I wasn't sure what to expect here. Any insights would be super helpful! 😅
The variation you're seeing is expected with shared cloud APIs, especially at lower request volumes. Mistral likely uses dynamic load balancing across GPU clusters, which creates latency differences based on real-time routing.
Benchmarking similar services, I've recorded standard deviation in latency up to 60% of the mean for identical prompts. AWS Bedrock often feels more consistent because you're typically provisioned to a dedicated endpoint once your usage reaches a certain tier.
You could test if the variance correlates with time of day. Running 100 sequential identical requests and logging timestamps would give you a clearer performance distribution to compare against your SLA needs.
Totally normal, especially for someone just starting out. I remember having the same exact reaction when I first switched from a more expensive, provisioned service. The jump from 1 second to 6 can feel jarring.
It's probably less about your setup and more about shared resource availability at that exact moment. I've found these fluctuations smooth out a bit during off-peak hours, or if you start batching a few requests together. It's a trade-off for the flexibility and cost.
Coming from Bedrock, that's a classic apples-to-oranges comparison in terms of backend resource allocation. Have you tried checking the response headers? They sometimes give hints about the load on the routed cluster.
Always testing.
Yep, that's the classic shared-tenant tax. The ~1 second response is likely a "warm" model instance you hit, and the 5-6 seconds is probably a cold start where your request triggered a fresh container spin-up on a new GPU.
While it's normal, you shouldn't just accept it blindly if it's hurting your UX. Since you're testing, try adding a `stream=true` parameter. Even if you don't need streaming, the time-to-first-token metric you get is often more consistent and revealing than the full completion time. The variance is usually in the generation phase, not the initial computation.
Also, check if you're getting the same region each time. A quick `curl -I` might show a `x-load-balancer` or `x-region` header. That 5-second spike could be you getting routed to a different data center entirely.
pipeline all the things
That point about checking headers is key. Most people overlook it. The `x-load-balancer` header can tell you if you're being shuffled between different backend pools, which explains a lot of the variance.
You're right about the streaming parameter. It isolates the network and initialization latency from the generation time. For anyone building a user-facing app, time-to-first-token is the only metric that matters for perceived speed. The total completion time can vary based on output length and content filters, which you can't control.
A caveat: calling it "shared-tenant tax" frames it negatively. It's the fundamental trade-off of a pay-per-call model versus a provisioned endpoint. You're paying for marginal cost, not reserved capacity. If consistency becomes critical, you have to move up the pricing tier where services offer dedicated instances.