Our team recently completed a six-month analysis of bot-generated comments across three major automated code review tools (SonarQube, GitHub Advanced Security, and a popular commercial SAST tool). The initial promise was a significant reduction in vulnerability introduction and code smell propagation. However, the observed outcome was a phenomenon we're calling "bot blindness"—after approximately 8-12 weeks, our senior developers began exhibiting a statistically significant drop in engagement with *any* automated comment, regardless of severity or tool.
The core issue appears to be signal-to-noise ratio degradation. Even with meticulous tuning, false positives and pedantic style nitpicks (line length, trivial variable naming in non-public APIs) create a background hum. When a high-severity, legitimate security finding appears, it's buried in the same visual pattern and impersonal phrasing as a hundred previous low-value comments. The human reviewer's pattern-matching brain learns to filter the entire pattern.
Our data from a sample of 427 merged pull requests shows:
* **Week 1-4:** ~92% of bot-generated comments received a human response (acknowledgement, fix, or dispute).
* **Week 9-12:** This dropped to ~31%.
* **Week 17-24:** Plateaued at ~15%, mostly only on critical security alerts, and even those were sometimes missed.
We attempted to mitigate this by implementing a priority tagging system in our pipeline, attempting to gate comments to only those above a certain severity. The configuration looked something like this for our SonarQube quality gate:
```yaml
sonar.qualitygate:
ignore-patterns:
- "common-*"
severity-gate:
- blocker: true
- critical: true
- major: false
- minor: false
- info: false
scope:
- new-code-only: true
```
This helped reduce volume by ~60%, but the residual noise still led to the same filtering behavior, just on a longer timeline (~16 weeks). The fundamental conflict is that for a bot to be comprehensive, it must be verbose; for a human to remain effective, input must be curated.
I'm interested in whether other teams have hit this wall and what, if any, process or tooling strategies have shown measurable success in maintaining engagement. Specifically:
* Have you moved to a "bot-first, human-second" review where bots create a summary report *outside* the PR line comments?
* Is there a tool that effectively learns from dismissals to silence repeat false positives across the team?
* Does aggregating findings into a single, formatted PR comment from a bot (instead of 50 inline ones) change the engagement dynamic?
The goal is to preserve the benefit of automated scanning without training our engineers to ignore a continuous stream of automated feedback.
-ck
This is fascinating, and that drop-off statistic is pretty stark. It reminds me of something similar we saw with our initial ETL job alerts. When everything from a critical pipeline failure to a minor schema drift triggered the same Slack channel ping, the team just started muting the channel entirely.
So is the solution maybe to have the bots "speak" less, or to change how they signal severity? Like, could a genuine security finding trigger a different, rarer kind of notification entirely, outside the standard comment stream? That way it breaks the visual pattern you mentioned.
Curious, did your analysis track whether comments from *other human reviewers* also saw a drop in engagement, or was it isolated to the bot-generated ones?
rookie
Your point about tracking engagement with *human* comments is crucial, and we did measure that as a control. The drop-off was isolated to bot-generated content. Engagement with comments from other team members remained consistently high across the same time period, which strongly suggests the issue is specific to the automated source and its presentation.
The visual pattern you identified is exactly right. We found the impersonal, templated phrasing from bots creates a uniform "wall of text" in the review interface. The brain starts skimming past anything that matches that template. Simply changing the notification channel, as you suggested, might help for critical items, but it doesn't address the core review hygiene problem where important but non-critical findings are still missed.
A technical nuance from our data: the drop wasn't linear. There was a sharp cliff around week 10, correlating with a period where three separate tools flagged the same PR with over 50 combined comments, most about formatting. After that event, engagement never recovered, even on clean PRs.