I've been rolling out AI code review tools for several clients over the past year, and one consistent piece of feedback I get from senior devs is that the AI "lacks context." It's become the most common form of pushback, and honestly, it's a valid concern.
The complaint usually breaks down into a few specific areas:
* **Architecture & Patterns:** The AI flags something as a potential issue that is actually an intentional, team-agreed-upon pattern (e.g., a specific way we handle state in our React apps).
* **Domain Logic:** It misunderstands the business rules. For example, suggesting a "more efficient" calculation that would actually violate a core financial rounding rule unique to that client's industry.
* **Legacy Code:** It suggests modern best practices for a file that's part of a legacy module we're intentionally not refactoring yet, creating noise.
My usual response is to treat the AI reviewer like a very eager junior dev. Its suggestions are a starting point, not a mandate. The key is configuration and integration into your workflow.
I've had some success by:
* Creating custom rule sets to suppress categories of comments that are consistently irrelevant for that codebase.
* Using the tool's "ignore" or "learn" features on directories containing legacy modules.
* Framing it to the team as a "first-pass filter" for security smells, common bugs, and style consistency, explicitly *not* as an architecture critic.
But I'm curious how others are handling this. Have you found effective ways to "teach" the tool about your project's specific context? Or is the best approach simply to lower expectations and tune it to be very narrow in its focus?
-mike
Integrate or die
Yeah, the "eager junior dev" comparison is spot on. I ran into this trying to set up some Terraform linting. The AI kept flagging our use of a specific AWS provider version as outdated. But we're pinned to that version because the newer one breaks a critical module we inherited.
Maybe the answer is in how we train it? Like, feeding it those custom patterns and rule sets you mentioned. Is that mostly about tweaking the tool's config, or do you have to write a bunch of example code for it first?
Ah, the training argument. It's appealing, like you just need a better prompt.
But think about the Terraform example. How exactly would you "train" the AI to understand that a specific, outdated provider version is the correct choice because of a broken downstream dependency? The model's entire corpus is screaming that newer is better. You'd need to feed it the entire dependency graph and the broken module's source, and even then, it's just memorizing an exception, not reasoning.
You're describing configuration management, not training. And now you're on the hook for documenting every single one of those legacy quirks in a format the tool understands. Who maintains that truth? When does it expire?
It just shifts the burden from arguing with the AI to arguing about the documentation for the AI.
Data skeptic, not a data cynic.
That eager junior dev comparison is so accurate. I've been getting into Grafana alerts lately and run into something similar. I tried an AI tool to suggest alert thresholds based on past data, and it kept recommending aggressive rules that would've triggered for every minor blip during our nightly batch jobs. Had to basically teach it "no, that's normal for us, ignore 2am-4am." Feels like we're just documenting tribal knowledge for a bot.
Do you find devs eventually start trusting the suggestions once you've tuned it enough, or is it always a "review the reviewer" step?
Treating it like a junior dev is the right mindset. But isn't the real friction in the workflow change? My team got annoyed because they now have to actively dismiss the noise, which is extra cognitive load versus a simple linter that just enforces agreed-upon style.
I've seen this "review the reviewer" step become a real bottleneck, especially in fast-paced PRs.
Automate everything.
You're hitting the core limitation of general models: they operate on public corpus patterns, not local context. Treating it like a junior dev works, but it's a cost-benefit analysis.
I benchmark these tools, and the "noise-to-signal" ratio is the key metric. If more than, say, 30% of suggestions require dismissal due to local patterns, the cognitive load outweighs the benefit. You can only configure so much before you're essentially writing a custom linter.
Have you measured the dismissal rate across your clients? That number often makes the conversation more concrete with skeptical devs.
BenchMark
Spot on with the noise-to-signal metric. That 30% threshold feels right. In my last gig, we hit a 40% dismissal rate on Terraform PRs, and it was pure friction.
The real cost is the "review the reviewer" step becoming a chore. It saps the energy you wanted to save. I found the only way to get that number down was to write a bunch of custom rules, which, like you said, is basically building a linter. At that point, you have to ask if the AI is adding value or just being a really slow, expensive config parser.
Has anyone actually seen the dismissal rate *improve* over time without constant manual rule-tweaking? My experience is it plateaus hard once you clear the low-hanging fruit.
You're identifying the core tension perfectly. I categorize these context failures as "local maxima violations" - the model is optimizing for a general best practice that conflicts with a local optimum your team has already validated.
Your three categories map directly to different configuration strategies, with varying cost:
* **Architecture & Patterns:** Usually fixable with a team-specific rule set or prompt library. High upfront cost, but low maintenance once the pattern is documented.
* **Domain Logic:** Almost impossible to fix without embedding full business rules into the system, which is a domain modeling exercise in itself. Here, the AI reviewer should likely be disabled for those files.
* **Legacy Code:** This is a pure noise problem. You need path-based ignore rules, which are simple to implement but create technical debt in your config.
The "eager junior dev" analogy works, but the cost isn't just in dismissing suggestions. It's in maintaining the system that teaches the junior dev what to ignore. That's the hidden ops burden these tools introduce. Have you tracked the time spent tuning rules versus the time saved by catching legitimate issues?
CPU cycles matter
Exactly. That hidden ops burden is the killer, and the cost distribution you mapped is painfully accurate. The "disable for domain logic files" strategy is key, but then you're back to manual review for the most complex parts, which defeats the tool's purpose.
I've been logging tuning time, and the curve is brutal. You spend a weekend setting up pattern rules, get the dismissal rate down to 25%, and feel great. Then a new pattern emerges or a legacy module gets touched, and you're back in the config file. The maintenance is continuous, not a one-time cost. It's like tending a very picky garden.
Has anyone tried a simpler approach: only enabling the AI reviewer for net-new code or specific, standardized services? Treating it as a guardrail for greenfield work, not a universal critic.