Hey everyone, I'm pretty new to using Mistral's APIs and I've run into something confusing.
I'm testing the Le Chat API with simple prompts, and the response times vary wildly. Sometimes I get a reply in ~1 second, other times it takes 5-6 seconds for a similar prompt. I'm just using the default settings. Is this kind of fluctuation normal for cloud AI APIs, or could it be something on my end?
I'm coming from using AWS bedrock where things felt more consistent, so I wasn't sure what to expect here. Any insights would be super helpful! 😅
The variation you're seeing is expected with shared cloud APIs, especially at lower request volumes. Mistral likely uses dynamic load balancing across GPU clusters, which creates latency differences based on real-time routing.
Benchmarking similar services, I've recorded standard deviation in latency up to 60% of the mean for identical prompts. AWS Bedrock often feels more consistent because you're typically provisioned to a dedicated endpoint once your usage reaches a certain tier.
You could test if the variance correlates with time of day. Running 100 sequential identical requests and logging timestamps would give you a clearer performance distribution to compare against your SLA needs.
Totally normal, especially for someone just starting out. I remember having the same exact reaction when I first switched from a more expensive, provisioned service. The jump from 1 second to 6 can feel jarring.
It's probably less about your setup and more about shared resource availability at that exact moment. I've found these fluctuations smooth out a bit during off-peak hours, or if you start batching a few requests together. It's a trade-off for the flexibility and cost.
Coming from Bedrock, that's a classic apples-to-oranges comparison in terms of backend resource allocation. Have you tried checking the response headers? They sometimes give hints about the load on the routed cluster.
Always testing.