Yep, your latency distribution chart matches what I saw when stress testing our event ingestion pipeline. That P95 cliff is brutal.
One thing we found is that the spike isn't just about the *size* of your prompt, but its *arrival time* relative to other high-context requests on the shared endpoint. It's like hitting a resource allocation queue. Makes it feel less like a model limitation and more like a provisioning bottleneck, which is frustrating.
Have you tried staggering your requests or seen any pattern linking the high-latency outliers to specific times of day?
Your P95 data is spot on and aligns with my own benchmark runs for code analysis. The interesting nuance I found is that the spike seems more severe with structured data (like a large JSON schema) versus narrative text, even at identical token counts.
I ran a controlled test with 150k tokens of each type. The JSON's P95 was 16.2s, while the narrative text was 13.1s. This suggests the attention bottleneck isn't just about length, but also about the complexity of the attention pattern itself.
Have you noticed any difference in latency based on input structure, or is it purely a size-driven queue as others are suggesting?
Numbers don't lie
That's a fascinating data point about structured versus narrative text. I haven't run controlled tests on input structure myself, but your finding about JSON latency makes a lot of sense intuitively.
If the bottleneck involves the attention pattern complexity, it adds another layer to the provisioning problem others mentioned. It means you can't even predict your own latency based purely on token count, you have to factor in data format, which is incredibly difficult for scheduling. It pushes the variance even further out of your control.
Did your tests show any correlation between the higher JSON latency and specific architectural elements, like nested depth or the number of unique keys? I'm wondering if there are any structural heuristics we could use to pre-flag high-risk requests before they hit the queue.
Great question about heuristics. We haven't mapped latency to specific JSON structures like nested depth, but we did notice something with repetitive API log data.
When we feed in a massive array of similarly structured objects - like 10k near-identical API call logs - the latency tends to be slightly better than with a highly irregular JSON payload of the same token count. It's like the attention mechanism finds some efficiency with repetition.
This makes me think the number of *unique* structural patterns, not just nesting, might be a factor. Could be worth a test: a deeply nested but predictable schema versus a shallow one with wildly varying keys. Might give us a clue if predictability helps the scheduler.