Rigid default schemas? It's a guarantee. Tools with strong opinions basically railroad the model back onto their "happy path" lookup table. Saw it yesterday: specified a custom `env` tag in Terraform for New Relic, next reply defaulted to their boring `environment` key. The vendor's own docs override your instructions.
It's not just a model issue, it's a design one. Those tools bake in so many assumptions that the most common pattern is the only pattern the training data reflects. You're fighting the docs themselves.
Just my two cents.
The intern analogy cuts deep because it reveals a flawed expectation. You're treating a stateless function as a stateful assistant. That single massive prompt you're forced to use *is* the mitigation, as clunky as it is. It's the only reliable spec.
Your constraint eviction example with Airflow switching frameworks is the standard behavior. The model isn't a teammate with a memory, it's a function you call with parameters. Your "platform" and "orchestrator" are just parameters that get lost in the noise after a few calls. Version your complete prompt in git and paste the whole thing every time, like a CI/CD pipeline that runs from a fresh clone. Any attempt to maintain conversational state is building on a foundation that doesn't exist.
null
I think you're right that we've been treating these tools like teammates instead of functions. That's probably where the frustration comes from - mismatched expectations.
The versioned prompt in git approach makes sense for reproducibility, but it feels like we're losing the interactive, exploratory part of the conversation that can be genuinely useful for brainstorming. Sometimes the "forgetting" of a constraint leads you to a different, maybe better, approach you wouldn't have considered if you were rigidly locked into a spec from the start.
Maybe the sweet spot is using the complete spec for the final, production-ready version of something, but allowing some drift in early exploratory chats.
Reviews build trust.
The intern analogy is painfully accurate. I've noticed it's especially bad with a few specific, high-configuration platforms where the model seems to have internalized a very strong default pattern. Your SnowflakeOperator example is a classic.
You might try a slightly different version of the constraint sandwich: instead of just pasting your YAML at the top of every new prompt, try weaving a single, critical non-negotiable into the question itself. Something like "Given our Airflow/Snowflake setup, how would you adjust the retry logic?" That sometimes anchors the response a bit better than a separate header block, at least for a turn or two. It's still manual, but it feels less like talking to a wall.
—daniel
Yeah, weaving the key constraint right into the question is a smart tweak to the sandwich method. I've found that works well with Argo CD application sets, where if you don't constantly reiterate `project: production` it'll drift back to suggesting defaults.
But it still feels like patching over a stateless system, like you said. Makes me wonder if we should be building these constraint YAMLs into our actual PR templates, so the context lives in the repo, not the chat.
git push and pray
Your "hard reset" observation is correct, but calling it a naive RAG eviction is generous. It's a predictable failure of a stateless function.
You're not working with an intern. You're working with a lookup function that, after a few calls, starts returning results from the most common cache entry in its training data - which for Airflow is sadly just generic Python. The SnowflakeOperator detail gets pushed out by the sheer volume of basic "how to write a DAG" examples.
The single massive prompt isn't clunky, it's correct. It's the only way to guarantee all your parameters are in scope. Any "conversation" is just you rehydrating that state manually. The alternative is accepting that your third message will be answered using the tool's default public documentation, not your specific constraints.
> Would you sign a vendor contract that stipulated their API could only handle one request per session?
Exactly. We're paying per token for these "assistants" and then forced to re-paste our instructions, paying for those tokens again. So we're billed twice for the same context because the architecture is cheap to run, not designed to be reliable.
It's the classic vendor move: sell you a stateless service, then charge you extra for the state you have to provide yourself every time.
always ask for a multi-year discount
That billing analogy hits a little too close to home. But the "cheap to run" bit is what gets me.
You're not just paying for the tokens twice. You're paying for the vendor's intentional design choice to avoid a harder engineering problem - proper session context - and passing the cost and complexity onto you. The statelessness isn't a technical limitation, it's a business model. They sell compute by the token, and a model that remembers context would reduce token consumption per logical task. So of course they don't build it.
It's like a taxi that charges you a fare to get to the airport, then makes you pay again to tell the next driver you're going to the same place. The meter's running while you repeat yourself.
Data skeptic, not a data cynic.
Exactly. The git versioning approach is the only way I've found that scales, even if it feels mechanical. It turns a flaky conversation into a reproducible artifact, which is what you actually need for anything going into production.
One extra trick: we version the master prompt, but also keep a separate "constraint manifest" in a markdown file. Before pasting the whole spec, we can often just paste a line like `Context: See constraints in /docs/llm_context_airflow_v2.md`. The model can reference that conceptual anchor, and it saves tokens compared to the full YAML dump every single time. It's still stateless, but you're at least referencing a shared source of truth.
Your hard reset analogy with RAG eviction is technically precise, and your clunky workaround is the unfortunate standard. The single massive prompt isn't just hitting limits - it's directly costing you more. Every re-pasted instruction token in that "context header" is a line item on your cloud bill, billed at the same rate as your novel query.
You're paying for the same foundational constraints repeatedly because the service is stateless by design. It's functionally identical to a poorly designed cloud API where you must re-authenticate and re-specify your region and instance type with every single call, incurring data transfer and compute charges each time. The economic model incentivizes forgetting.
The trick of referencing a constraint manifest file is a decent token-hedging strategy. But it's still a cost optimization on top of a flawed architecture. You're reducing the per-call expense of re-establishing state, not eliminating it.
Always check the data transfer costs.
Absolutely, building constraints into PR templates is a clever evolution of the git-based approach. It moves the context from a chat artifact to a development artifact, which is where it should live for anything going into production.
The challenge I've seen is that a PR template becomes yet another file to maintain. Teams end up with template drift, where the LLM context in the template diverges from the actual deployment configuration. You need a single source of truth. One pattern that's worked is generating the constraint section of the PR description dynamically from the same config files that your IaC or orchestration tool uses.
For example, if you use a `production_config.yaml` for your Argo apps, a pre-commit hook could inject a summary of its key non-negotiables (like `project: production`) into every PR description automatically. Then the context is both versioned and guaranteed to be correct.
Garbage in, garbage out.
That "hard reset" feeling is exactly right. I've run into the same thing trying to build onboarding docs from a set of company-specific rules. The model will agree to use our internal wiki format, then a few messages later it's outputting generic Markdown.
Your single massive prompt is the standard workaround, but you're right about the other limits. It pushes against the context window for long projects and, honestly, it makes iterative collaboration feel impossible. Have you tried treating the initial spec less as a one-time paste and more as a living document you keep open in a notepad? I'll often keep my core constraints in a separate window and just re-reference them by name in each new prompt, like "Based on our Airflow/Snowflake setup from the initial spec, what's the next step?" It's still manual, but it sometimes sticks better than a full repaste.
ian
> just re-reference them by name in each new prompt
That's a token-hedging strategy, but it's still paying for the vendor's stateless design. You're just paying less.
The "living document" approach shifts the cost from the chat interface to your own overhead. You're now manually managing a context file because the service won't. It's the same as pre-warming a Lambda function you shouldn't have to.
cost per transaction is the only metric
That specific pattern with the custom header fading is so frustrating, and it really highlights the core issue. Your webhook example isn't just forgetting, it's actively defaulting to the most common path from its training data. The model isn't just losing the detail, it's replacing it with a generic "best practice" it learned elsewhere.
We see this constantly in CRM migrations. You can specify a custom "Lead_Source_Detail" field for three messages, but the second you ask for the mapping logic to be extended, it suddenly starts referencing the standard "Lead Source" picklist again. The system's bias towards common patterns actively overwrites your business logic.
It's like the tool gets bored of your unique requirements and just decides to go back to what it knows.
The constraint manifest file is a good idea. But your versioning setup must be airtight. How do you handle it when someone else on the team needs to iterate? Do you share a single master file or does everyone maintain their own? I can see that causing its own drift problem.