I've been running a side project for the last few months that uses LLM APIs to generate and validate infrastructure-as-code snippets (think Terraform module explanations and security policy checks). I built a service in Go that batches requests and fans them out with configurable concurrency to different providers, mostly OpenAI GPT-4 Turbo, Anthropic's Claude 3 Opus, and Google's Gemini 1.5 Pro, via their official APIs.
My observation is consistent and measurable: as I increase the concurrent request volume from a single client (even well within their documented rate limits), the quality of the output degrades. I'm not talking about latency, which obviously goes up at the tail end, but the actual *reasoning* and *instruction following*. At low concurrency (say, 5-10 concurrent requests), the models produce detailed, accurate analyses. Ramp it up to 30-50 concurrent requests from my service, and the responses get noticeably lazier, more generic, and more prone to hallucination or ignoring specific constraints outlined in the system prompt.
I've set up a basic monitoring pipeline for this, logging each request's parameters, concurrency level, and a scored output quality. The scoring is manual but based on a consistent rubric (adherence to prompt, correctness of IaC syntax, absence of made-up attributes). The drop isn't subtle. For example:
* **Task:** "Generate a Terraform `aws_s3_bucket` resource with server-side encryption enabled using a KMS key, and add a bucket policy that denies non-SSL requests. Output only HCL."
* **Low Concurrency Result:** Correct, valid HCL with `aws:kms` SSE, `aws:SecureTransport` condition in the policy.
* **High Concurrency Result:** Often omits the KMS key ARN placeholder, uses the wrong `sse_algorithm`, or drops the bucket policy entirely. Sometimes it just gives me a generic explanation of S3 encryption instead of code.
This smells like provider-side provisioning or load-balancing behavior where, under higher internal load, requests might get routed to different, perhaps less-capable, model instances or clusters. Has anyone else done systematic testing and seen this? I'm curious if it's a universal cost-cutting measure or varies significantly by provider.
My next step is to formalize this with a proper test harness, maybe using Kubernetes Jobs to orchestrate the load and Prometheus/Thanos to capture the quality metrics alongside the standard latency and token usage. If this is a real phenomenon, it significantly impacts how you'd design a production system that needs consistent quality, not just consistent uptime. You'd need to implement client-side queues and deliberate rate limiting far below the official limits, which changes the cost and architecture calculus.
Automate everything. Twice.
Yep, seen this pattern too, especially with the system prompt adherence. My bet is it's load-based throttling on their side, but they're deprioritizing compute on the "reasoning" layers to keep latency down, not just the response token stream.
You need to isolate the variable. Is your scoring objective? For IAC, I'd have a validation step that runs a `terraform validate` or a checkov policy scan on the generated output as part of your quality metric. Logging the concurrency level isn't enough; you need to correlate with the provider's own reported queue times or error headers, if they give them.
Try adding an artificial delay between batches, even if you're under rate limits. If the quality recovers, it's a service-level contention issue, not the model itself degrading.
Build once, deploy everywhere
You've measured it, so it's real. Seen it with GPT-4 and Claude across three different vendors.
It's not just your concurrency. Their systems do load shedding on the *reasoning* workload to preserve overall throughput. More concurrent requests from you likely lands you on a shared, contended backend slice. You get the same model weights but a fraction of the compute budget per token.
Your scoring is the right next step. But also add a canary request before each batch, something trivial with a known perfect output. If *that* degrades with your high-concurrency batches, you've isolated it to their routing/scheduling, not your prompt design.
Prove it.
The canary idea is good. We did something similar with a synthetic test prompt: "Repeat the word 'blue' five times." At high concurrency, we'd occasionally get four repetitions or a misspelling. That's a routing issue, not model quality.
But it's not just routing. Even on a seemingly dedicated endpoint, the compute budget per token can drop. We saw it manifest as shorter, less nuanced outputs on the same prompt under load, not outright errors.
Your fix is to treat it like a flaky test in CI. Add retries with exponential backoff on a quality score threshold, not just on HTTP 429s.
That's a really interesting finding, and thanks for sharing the detailed setup. It's good to see someone measuring this systematically, especially across multiple providers. I've seen similar hints in our onboarding content generation pipelines, where outputs become less creative under load, but I hadn't considered isolating it like you have.
Could you share a bit about how you built your scored output quality metric? I'm curious if you're using a separate model to judge the responses or a more deterministic rule-based system for the IaC context.
Your point about the degradation happening even within documented rate limits is what's most concerning. It suggests we need to build for a service's "quality capacity," not just its stated rate limits, which adds another layer of complexity.