So, "128K context." Impressive number. But we've all seen the "unlimited" plans that throttle you at 10GB. Is this just marketing math, or does it actually hold up when you shove an entire AWS service specification PDF into it?
I ran a test with the **AWS S3 API reference (the full spec, ~850 pages pre-rendered)**. The goal: see if it could accurately answer a question buried deep in the middle, referencing specific error codes and request parameters from sections it would never have seen in its pre-training.
**Methodology (because reproducibility matters):**
1. Concatenated the S3 Developer Guide sections into a plaintext file. Tokenized locally to estimate ~110K tokens.
2. Fed it into the API with a system prompt: "You are a technical auditor. Answer only from the provided context."
3. Asked: "According to the provided documentation, what are the *exact* conditions under which a `POST` object upload will return a `400` error with the code `AuthorizationQueryParametersError`? List the validation rules for the `x-amz-credential` query parameter."
**The Results:**
* **First attempt:** It gave a plausible but *generic* answer about malformed credentials. Not good enough.
* **Second attempt (with more explicit positioning):** I prefaced the question with "In the section titled 'Authenticating Requests: Using Query Parameters', it states that...". This time, it nailed it. Pulled the exact validation sequence (date, region, service, terminal) and the specific formatting error that triggers that code.
**My takeaway:**
* The capacity is *there*, but retrieval quality is highly sensitive to how you frame the query. It's not a magic "find the needle in the haystack" tool yet.
* The real test isn't recall—it's **reasoning across the entire context**. Can it compare a cost-optimization suggestion from page 5 with a latency warning from page 95? My initial tests on cross-reference questions were... mixed.
* Anyone else done a *stress test* on this? I'm particularly interested in:
* Performance degradation (latency, cost) as you approach the 128K limit.
* Use with highly structured but tedious inputs (like Terraform module trees or security group JSON exports).
* Whether it consistently hallucinates less than 32K/4K models when dealing with the middle parts of massive documents.
Show me your prompts and your failure modes. The incident postmortem for this feature will be more interesting than the marketing copy.
- Nina
- Nina