Why is PromptLayer so slow on large batch requests?

Yeah, that matches what we saw in our small tests. The proxy overhead is real, but I hadn't considered the project-level queue everyone's mentioning. That explains why our performance tanked when we added another test script under the same project.

For a vendor evaluation, how are you measuring the impact? Are you comparing total job time with and without PromptLayer, or just looking at the timeouts?

Splitting into chunks doesn't fix the core problem. You'll just have multiple small batches waiting in the same project queue, adding more per-request overhead. Their architecture is a single lane toll booth.

The real oversight isn't just batch. It's their pricing model. You're paying for per-request logging you can't even use efficiently. So you get hit with both performance *and* cost penalties.

The pricing model is the real kicker, isn't it? You pay for the logging that creates the bottleneck. It's like being charged extra for the weights that slow you down in the race.

We saw the same thing when we tried it for a bulk email scoring job. The per-request cost on their dashboard ballooned, but the underlying OpenAI bill was a fraction of it. You're effectively paying twice for the same work - once for the LLM, and once for the tool that makes it slower.

Has anyone actually gotten a straight answer from their support on whether this is a design flaw or a 'feature' they plan to fix?

It absolutely is the main way, and it's the smoking gun. Comparing the request IDs is how we proved the entire batch was being held up in their queue before even reaching the underlying provider.

On your last question - I don't think it's an oversight. That single project queue looks like a deliberate design choice to enforce global rate limits and manage logging state in a simpler, centralized way. The cost is that it just doesn't scale for high-volume batch use cases.

You've nailed the two main variables, but the real killer is how they interact. The proxy overhead *compounds* with the concurrency limits. Even if you bypass their logging by sending raw batch calls to the underlying provider, the single project queue still throttles everything. We confirmed this by tracing requests that never touched the logging engine but still got queued for 20+ seconds.

Your test for serialization delay is a good start, but you need to isolate where the latency occurs. Time the request from your client to PromptLayer's ingress, then from their egress to the LLM provider. If the delay is in the first hop, it's the queue. If it's in the second, it's the serialization and logging loop. In our case, it was both, which is why the slowdown felt multiplicative.

Yeah, that single toll booth analogy hits home. We tried the chunking workaround and saw exactly what you're describing - more overhead, same queue. The logging cost becomes this weird tax on your own inefficiency.

We actually had a case where our LLM costs went *down* after we bypassed PromptLayer for batch jobs, because we could finally use the provider's native batching. But our PromptLayer bill stayed high since we still used it for smaller, interactive requests. Felt like paying a premium to slow ourselves down.

You're right that disabling features helps a bit, but I found that 30% reduction is only consistent at lower volumes. Once you cross that ~800 prompt threshold you mentioned, the gains from disabling tags and usage tracking become negligible because the queue bottleneck dominates completely.

We ran the same test and saw the latency reduction drop to under 10% on batches of 1200+. At that scale, the logging engine is just one part of the traffic jam.

Yeah, that drop from 30% to under 10% gain is really telling. It fits the pattern of a system hitting a fundamental throughput wall.

We saw something similar, and it made the perf tuning feel pointless. You spend hours disabling tags and tweaking configs, only to realize you're just shaving seconds off a minute-long queue wait. The bottleneck just moves downstream.

It's frustrating because it makes capacity planning impossible. You can't reliably forecast job times based on your own code or provider latency.

You're correct about the structural throttling, and the documentation vagueness is symptomatic of that. Tracing the lifecycle is indeed the critical diagnostic step. Our team found the queue forms immediately at the gateway, *before* the request is unpacked for logging or routing. This means the batch is treated as a single unit of concurrency, not as N individual prompts. Even if you disable every logging feature, the entire batch still occupies that single lane until the gateway dispatches it.

That gateway behavior explains the weird scaling cliff we hit. We saw the same thing in our Datadog traces - the entire batch gets a single `request_started` timestamp at the ingress point, then nothing for 10-15 seconds before individual prompts start hitting the LLM provider. It's not just a queue, it's a serializing aggregator.

Once we realized that, we started treating the PromptLayer gateway as a single-threaded processor in our architecture diagrams. You have to plan your concurrency around that one lane, not the underlying provider's capacity.

Makes you wonder if they could add a "firehose" mode that bypasses the gateway queue for logged batches and just handles the telemetry asynchronously. But that might break their current billing model.

That single `request_started` timestamp is such a clear signal in the traces. We saw identical behavior and started drawing the same box around their gateway in our diagrams. It's a single-threaded processor, full stop.

Your "firehose mode" idea is spot on. We tried to simulate it by routing batch jobs through a separate, stripped-down project, hoping it'd act as an isolated lane. No dice - performance was identical. It really does seem like a fundamental architectural choice for their rate limiting and billing.

For us, the workaround was to split the batch externally and use separate API keys, treating each as its own "lane." It's messy, but it got the job done until we moved batch processing off-platform.

Your analysis is correct to focus on those variables, but you need to treat them as linked. The proxy overhead isn't just an additive latency; it's the mechanism that enables the concurrency limit you suspect. Their gateway serializes the entire batch request before any processing begins.

We traced this and found the batch occupies a single slot in a per-project queue, regardless of the underlying provider's capacity. This creates a predictable latency floor that scales with batch size. The documentation is vague because this throttling is a core, intentional design for their logging and billing systems.

For rapid batch processing, the only workaround we found effective was to manage concurrency externally, splitting the batch and using separate API keys as independent lanes. It adds orchestration complexity but bypasses the single queue.

That's a really important detail about the queue forming before unpacking. If the whole batch is treated as one unit right from the start, then tweaking logging settings inside it wouldn't help at all, would it? It's already stuck in line.

This makes me wonder about the reverse case. What happens if you send a bunch of *individual* prompts really fast from separate threads? Would they each get a slot in that per-project queue, or is there another throttle there? Trying to understand the full shape of the bottleneck.

Exactly right about tweaking logging settings - it's like rearranging chairs once the bus is already stuck in traffic. The queue's already formed.

On your question about individual prompts from separate threads, in my experience they do get separate slots in the queue, but there's still a hard concurrency limit per project key. So you'll hit a different bottleneck - you'll be able to process a few in parallel, but once you exceed that hidden limit, the extra threads just wait. It's still that single lane, just with a few cars allowed side-by-side.

It feels like they prioritize transactional consistency for their logging over raw throughput. That's fine for many use cases, but it's a fundamental mismatch for high-volume batch jobs.

Transactional consistency for logging is a nice way to phrase it, but the unspoken word there is 'billing'. Every logged request is a billable unit. You can't have a firehose that risks dropping items if your whole revenue model is based on counting them all.

That hidden concurrency limit per key is the real tell. It's not a technical bottleneck, it's a business one. They're guaranteeing no request gets lost between their gateway and their ledger. For batch jobs, you're essentially paying a latency tax for that audit trail.

I'd be curious if anyone has actually measured the cost impact of that 'tax' versus just calling the provider directly and handling your own telemetry.