Hey everyone, I just saw the announcement about GPT-4o mini being available on Azure OpenAI. This is actually really timely for me.
I'm in the middle of planning a migration for some of our older, rule-based customer support systems. The idea is to replace a bunch of them with a simpler, cost-effective LLM for basic classification and short response generation. The pricing for the mini model looks great on paper, but I'm nervous about real-world performance.
My team is pushing for a "lift and shift" to the cloud anyway, and if we go with Azure, using their native OpenAI service would simplify things. But I need to be sure it can handle our load. We have peaks of several thousand requests per hour.
Does anyone have any early throughput numbers or benchmarks? I'm especially curious about:
- How many TPM/RPM you're reliably getting with the mini on Azure.
- If the latency stays consistent under that load.
- Any comparison to the standard GPT-4o on Azure in terms of tokens/second/dollar.
I've been burned before by theoretical specs not matching production, so I'd really appreciate any concrete data or even anecdotal experiences. A step-by-step of how you tested it would be amazing. Trying to build a realistic timeline for this migration phase.
One step at a time
Hey, it's a smart move to be cautious about theoretical specs. I've seen a few teams in our internal community channels running quick tests. The early data I've seen shows the mini can handle a few thousand requests per hour without a sweat, but the real trick is your specific token usage per request.
For your peak load, I'd strongly recommend setting up a test deployment with your exact classification and response patterns. Run a load test that mirrors your peak hour, and watch the latency graph. That'll give you a much clearer picture than any generic RPM number.
On the cost side, the tokens-per-dollar looks solid compared to standard GPT-4o for simple tasks, but keep an eye on those quota limits Azure sets per minute. They can sneak up on you during a traffic spike. Have you planned how you'll handle throttling if you hit a limit?
Keep it civil, keep it real.
Absolutely on point about the throttling. That's where the theoretical TPM/RPM numbers fall apart. Azure's quotas are per deployment, not per resource token.
You need to watch for two limits hitting you at once: the standard Azure OpenAI service quota and the specific model's capacity quota on your deployment. If you blast through the model's quota, you get throttled hard, no matter what your overall service limit is.
Plan for a queue-and-retry pattern from day one, and monitor those 429 errors. The cheaper the model, the more aggressive they seem to be with the limits.
That's correct, and the quota architecture makes it critical to differentiate between token-based and request-based limits. The standard Azure OpenAI service quota is typically measured in Tokens Per Minute (TPM), but the deployment-level capacity is often a Requests Per Minute (RPM) limit. You can have plenty of TPM headroom but still get slammed by 429s if you exceed the RPM cap on your GPT-4o mini deployment.
A practical caveat: the retry logic needs exponential backoff with jitter, but also a fallback strategy. When you hit the model deployment's RPM limit, queuing and retrying on the same deployment just compounds the delay. The simpler, cost-effective models often have lower RPM caps, so your architecture should consider failover to a secondary deployment or a degraded operational mode, not just blind retries.
Monitoring those 429s isn't enough; you need to disaggregate them by HTTP header to know which limit you're hitting - `x-ratelimit-remaining-requests` vs. `x-ratelimit-remaining-tokens`. The Azure documentation is rarely clear on the exact RPM values for these new, smaller deployments.
Spot on about the headers. We had a nasty debugging session last month because we were only logging the 429 status code, not the specific `x-ms-client-request-id` and rate limit headers. Turns out we were constantly bumping into the RPM wall on a cheaper model deployment, while our overall TPM was at 40% usage.
Your point on fallback strategy is key. We implemented a simple failover where, after two consecutive 429s with the RPM header near zero, the system diverts the next few minutes of traffic to a secondary deployment with a different capacity SKU. It's not perfect, but it's better than a cascading retry backlog.
I'd be curious if anyone's found a reliable way to predict those hidden RPM caps for new models, or if it's purely trial and error with Azure support.