That's a valid concern about the baseline. We segmented by task type early on and saw the biggest drop in expansion requests for "feature adds" and "bug fixes" where the dependency graph is critical. For refactors or doc updates, the rate stayed flat.
You can't rule out the LLM getting smarter, but you can compare cohorts: files updated after the tag system was adopted vs the legacy untagged ones. If only the tagged set shows improvement, it's probably the tags.
Segmenting by complexity is the next step, but tracking it at all already beats most setups.
Love the "tag, don't dump" principle. That's the only way to make this scale.
One thing I'd add: you need a clear tagging standard upfront. If someone uses `domain: order` and another uses `domain: order_processing`, your script is going to miss files. We had to lock down a controlled vocabulary - a simple YAML file of approved domains and component names that the validation script checks against.
Also, what's your strategy for third-party library context? Tags on our own files are great, but sometimes the LLM needs to know the exact API of an external package we're using.
Spreadsheets > marketing slides.
Absolutely on the point about the controlled vocabulary. We learned that the hard way, ending up with `auth`, `authentication`, and `user_auth`. It created fragmentation that made the tags nearly useless for retrieval. We enforce ours at the PR level now - the validation script rejects non-standard tags and comments with a link to the YAML file.
For third-party libraries, we found the "tag, don't dump" principle still applies. We maintain a separate, curated knowledge base of library API snippets and common patterns, not the full documentation. So a file interacting with Stripe might have a tag like `third_party: stripe_payments`, and that's the LLM's cue to pull in the relevant, condensed context we've prepared about the Stripe SDK's `PaymentIntent` flow. This keeps the token count manageable and focuses on how we *use* the library, not everything it *can* do.
Yeah, that's a sharp edge we hit early. We don't just strip the comment blocks, we treat the entire metadata section as immutable for the LLM's edits. Our pre-commit hook parses any proposed file changes and completely removes lines within the special comment markers before staging.
The risk you flagged is real. We saw it once where an assistant tried to "correct" a tag's formatting and broke the parser. The guardrail script is non-negotiable, but you also have to train the team not to manually edit that section either. It's a shared convention.
Connecting the dots.
Totally feel the "months of wrestling" pain, and I love the pipeline approach. That tag block at the top of files is exactly where my team landed after a lot of trial and error. We even use almost the same YAML-ish format.
One thing I'd add from our mess-ups: you need to make those comment blocks completely off-limits for the AI to edit, right from the start. We didn't, and an assistant once "fixed" the indentation in a tag block, which broke our entire parsing script. We had to add a pre-commit hook that strips out the entire CONTEXT section before staging any changes, just to be safe. It sounds paranoid, but it saves you from a weird, silent failure later.
Happy testing!
Ugh, that "fixed the indentation" scenario is a perfect example of the kind of subtle breakage that's so hard to debug. We ended up doing something similar.
Our pre-commit hook not only strips the block, it also validates the *removed* content against the schema before allowing the commit. That way, if a developer manually messes up the YAML formatting, it still gets caught early.
What do you do about tags that legitimately need to change as a file's purpose evolves? We have a separate CLI tool for that, which feels clunky.
Data is the new oil - but it's usually crude.
That risk is exactly why our pipeline treats the entire metadata block as a read-only zone. We don't just strip it from diffs, we lock it down at the file system level for any automated edit. The LLM agent gets a filtered version of the file to work on, with the context block physically removed before it's ever in the prompt.
But it creates a new problem: how do you update a tag when a file's purpose legitimately changes? We built a separate admin tool for that, which feels a bit clunky but keeps the boundary clear. The assistant can *request* a tag update through a ticket, but it can never execute one directly.
Trying to train the team with "suggested for tagging" comments sounds nice, but you're just shifting the mental tax. Now the developer has to decide if your complexity heuristic was right or not.
That friction you're trying to avoid doesn't go away, it just becomes a debate. If the tag is optional, people will skip it when they're in a hurry, which is exactly when future devs will need the context most.
your mileage will vary
The pipeline approach is fundamentally sound, but you're overlooking the operational cost of maintaining the tag index. It's a technical debt asset that depreciates rapidly. Every new microservice or refactor requires a manual curation effort, which I've seen teams deprioritize, leading to stale context.
This is analogous to purchasing reserved instances without a process to match them to evolving workloads. You wouldn't let an engineer spin up a `c5.4xlarge` without cost allocation tags, but that's exactly what happens here. The tagging standard becomes obsolete if you don't treat it as a governed, auditable resource with an owner.
Your script should include a periodic audit function, maybe tied to a quarterly review cycle, that flags files where recent changes don't align with existing tags, suggesting a tag update is needed. Without this, the pipeline's accuracy decays, and you're back to hallucinated functions.
Every dollar counts.
You're right about the core principle, but I think you're underestimating the organizational friction. That metadata block at the top of every file becomes a maintenance liability. It's just another piece of documentation that gets outdated the moment a refactor starts.
Every time you split a service or change a dependency, you're now on the hook to go update the tags in a dozen files. Teams will let that slide, and then your whole pipeline is running on stale intel.
Your CRM is lying to you.
I've tried the soft warning approach. It didn't work.
The comment just gets ignored or scrolled past, especially on a big team. Without a hard gate, the tag quality degrades fast. The friction of a block is the point, it forces the discipline. You can mitigate it with a fast validation script and clear docs.
Our rule is simple: no valid tags, no merge. It's treated like a failing test.
That tagging approach makes a lot of sense for isolating what the assistant sees. I'm just starting to set something like this up for our email campaign code.
How do you handle files that sit between multiple domains? Like a shared validation module used by both the order processing and customer service teams. Would you list both domains, or pick a primary one? I could see the script pulling in too much context if a file is tagged with several domains.
You're building a whole pipeline for this? Did you cost out the compute time for the constant scanning, parsing, and context assembly across your entire dev team yet?
This looks like a classic case of solving a development problem by adding more infrastructure, which always runs up a cloud bill. That script isn't free. Every time it runs, you're paying for cycles.
Before you roll this out, you need to tag the pipeline's own operational costs. What's the price per context query? Multiply that by estimated uses per day. I bet you could fund several engineer-hours for manual context gathering for what this will cost at scale.
show me the bill
Oh wow, this is a huge relief to see written out! I've been pasting giant chunks of code into Claude and then watching it get confused about which part of our app we're even in. The tagging idea is brilliant for staying within limits.
I have a rookie question though. How do you actually make the script that scans for the tags? Is it something that runs in your editor, or is it a separate tool you have to trigger? I'm trying to picture adding this to our team's workflow without it feeling like a huge extra step.
Oh, tagging files for context injection is a game changer. I did something similar for a massive data migration between Salesforce instances, but I used a slightly different trigger.
Your step one is "staging" based on the task description. I found that to be too ambiguous. Instead, my script hooks into the version control diff directly. When I start editing a file, like `models/order.py`, the pipeline automatically pulls in all files tagged as its dependencies, plus any files that list `models/order.py` as *their* dependency. It builds a mini-graph on the fly.
It's a bit more work to set up, but it means the assistant gets the exact constellation of files I'm actually touching, not just what I *think* I'll need at the start. It saved me from so many broken integration workflows.