I’m currently evaluating a potential migration to an Enterprise-tier contract with a major AI/LLM platform vendor, primarily for the advertised "unlimited context" feature. Our data science team regularly processes large documents and cross-dataset queries that would benefit from extended context windows far beyond the standard 128k tokens offered in lower tiers.
While the sales documentation and contract outline is clear on the removal of *per-request* token limits, my experience with other "unlimited" enterprise offerings (in data warehousing, API calls, etc.) suggests there are almost always *implicit* operational limits governed by compute quotas, rate-limiting algorithms, or "fair use" policies that only become apparent under sustained load.
Before proceeding with a costly negotiation, I’m seeking concrete, empirical data from teams already on such a plan. Specifically:
* **Is the "unlimited" context truly a function of available system memory for your instance, or is there a soft ceiling (e.g., 1M, 2M tokens) after which performance degrades non-linearly or requests are queued/throttled?**
* **Are there hidden compute-minute or GPU-hour quotas per user/month that, when exceeded, trigger renegotiation or automatic downgrades in model priority?** The sales rep mentioned "sustained high usage may require architectural review," which sounds like a potential limit.
* **What is the actual observed latency and throughput when consistently utilizing, say, 500k+ token contexts?** We need to model expected analyst productivity versus current chunking/embedding pipelines.
Our preliminary cost-benefit analysis hinges on whether this feature can replace our current RAG-based preprocessing for at least 70% of our analytical queries. If the effective, usable context is bottlenecked by other system parameters, the ROI calculation falls apart.
I would be particularly interested in any contractual language or technical documentation you’ve received that clarifies the following operational details:
* Definitions of "concurrent sessions" and how context is managed across them.
* Service Level Objectives (SLOs) for throughput (tokens/second) at defined context lengths.
* Any internal monitoring dashboards that show usage against allocated compute resources.
If you have performed benchmark tests, sharing your methodology or results would be invaluable. For example, a simple script to measure latency scaling:
```python
# Pseudocode for a benchmark we might run
import time
def benchmark_context_length(token_length):
prompt = generate_test_prompt(token_length) # Function to create dummy text
start = time.perf_counter()
response = model.generate(prompt, max_tokens=100)
end = time.perf_counter()
latency = end - start
return latency, response.usage.total_tokens
# Iterate over increasing context lengths: 128k, 256k, 512k, 1M tokens
# Plot latency vs. context length and note any discontinuities.
```
Any data points on these hidden dimensions would significantly improve our negotiation position and technical planning.
p-value < 0.05 or bust
Spot on about the implicit limits, that's been my experience too. In one of our contracts, "unlimited" really meant "no hard per-request cap," but they absolutely had a system-wide compute budget for our account that kicked in with sustained high throughput. We saw latency spike and got vague "resource contention" alerts when we pushed it for a few hours.
You'll want to get them to specify the *throughput* limits in writing - tokens per minute or per hour, and what the actual service level agreement (SLA) is under that sustained load. Sometimes the unlimited context is real, but the request queue just gets longer and longer. Good luck with the negotiation!
cost first, then scale
Exactly. The distinction between a per-request limit and a system-wide compute budget is critical, and it's often buried in the annexes of an SLA document, not the marketing materials.
You need to push for the specific metrics governing that budget. Ask for the sustained throughput limit (TPM/TPH) and, more importantly, the *replenishment rate*. If your budget is 10 million tokens per hour, does it refill instantly at the top of the hour, or is it a continuous trickle? The latter creates a queuing effect that destroys predictability for batch workloads.
Also, scrutinize the SLA's definition of "availability" or "performance." It likely excludes periods of "resource contention," which is the exact state you'll be in when hitting those implicit limits. You're technically "served," but with 10-second latencies, which is useless.
Good catch on the replenishment rate. That's the key detail most people miss and vendors love to keep vague. The instant vs. continuous refill determines if you can run a batch job or if you're throttled to a trickle.
Push them to define the "resource contention" clause explicitly. In one of our agreements, we got them to specify that any latency exceeding the 95th percentile baseline by a factor of 10 constituted a breach, regardless of cause. It forced them to either guarantee performance or admit the limit was a hard throttle.
Also, ask how they measure the budget. Is it based on input tokens, output tokens, or total tokens? That can double your effective consumption without warning.
Your cloud bill is 30% too high
That's a brilliant point about the *definition of the budget unit*. It bit us on a different service where "tokens processed" quietly included the system prompt tokens for every single request, which was huge for our chat-based app. Our cost projections were off by 40%.
> specify that any latency exceeding the 95th percentile baseline by a factor of 10 constituted a breach
Love this. We tried a similar tactic and framed it as "operational predictability" for our SLAs downstream. It shifted the conversation from "unlimited" to "defined performance," which is what you actually need to build on top of it. Did they push back hard on that 10x multiplier? I've found they'll sometimes counter with "degraded performance" tiers that are still basically unusable.
Infrastructure as code is the only way
Great point about how they measure the budget. That's exactly the kind of thing we'd miss during onboarding.
In my old role, we got burned because 'tokens processed' included the entire retrieved context for RAG, even the chunks we didn't use. It made forecasting impossible.
What would you recommend asking for instead? Should we push for the budget to be defined by 'input tokens submitted' or something more specific?
Absolutely push for "input tokens submitted" as the defined unit. That's the only thing you can directly control and audit from your client logs. Anything else, like "tokens processed" on their side, is a black box and gives them room to pad the numbers.
When we negotiated ours, we got it defined as "prompt tokens as counted by the client library's tokenizer, billable upon successful API response." This locks down both the source of the count and the condition for charging. You need to bake the specific tokenizer name and version into the SLA appendix.
And yes, exclude any system prompt or retrieved context that isn't part of your submitted payload. If they resist, ask them to provide a real-time metering endpoint so you can see their token count per request and reconcile it. They usually back down when faced with actual accountability.
Automate everything. Twice.
Great questions. The short answer is yes, there are almost always ceilings. In our case, the "unlimited" context is tied to a dedicated node's memory, but that node has a finite, undocumented GPU VRAM allocation. We hit a soft wall around 1.2M tokens where latency became unpredictable, not from queuing but from the model struggling with the attention mechanism across that many tokens.
For your second point, our contract has an aggregate monthly compute-hour cap, but it's expressed as a "service credit" pool. Once we use that up, we don't get throttled, but we're billed at a steep overage rate. You need to ask for the "included monthly compute units" document, not just the SLA.
Push them to define the performance envelope, not just the absence of a limit. Something like: "For contexts up to X tokens, p95 latency will not exceed Y seconds." That forces them to commit to the hardware.
The service credit pool model is a classic. It's not a technical limit, it's a financial one that kicks in at the worst time - when you're actually using the service heavily. You're paying for "unlimited" up front, then paying again on the back end.
If they won't commit to a performance envelope up to a certain token count, as you suggest, then the "unlimited" is meaningless. You're just renting access to a system that becomes unusable before you hit its theoretical limits. That's worse than a hard cap because it's unpredictable.
Show me the data