Everyone's obsessed with hitting that 200k token count like it's a high score. They're shoving entire codebases and novel-length docs into a single prompt and expecting coherent answers.
Reality check: quality nosedives long before you hit the limit. The model starts:
* Hallucinating function names that don't exist
* "Forgetting" instructions from the beginning of the context
* Producing generic, waffling summaries instead of precise answers
* Mixing up details from different parts of your doc
Tried it last week with a 150k Terraform module dump. By the end, it was recommending AWS resources we'd explicitly deprecated in the first 10k tokens. Useless.
You're better off with smart chunking and a RAG pattern. Fight me.
You're spot on with the quality nosedive. I've noticed it's not just about forgetting the earliest instructions, there's also a massive drop in the model's *confidence* as the context bloats. The answers become so vague and qualified they're practically useless.
Your Terraform example is perfect. I see the same thing in design system documentation - ask about a component rule defined at the start of a massive spec, and by the end it'll start inventing its own prop names. It feels less like a recall problem and more like the model gets overwhelmed and starts averaging everything out into generic mush.
RAG isn't a silver bullet either, but you're right that it forces a better discipline. You have to think about structure and relevance first. Chasing the max token count feels like a brute force solution to a problem that needs a bit more finesse.
good UX is non-negotiable