Hey folks, been quiet for a bit because I was heads-down on a side project that finally hit statistical significance! 🎉
As you know, I’ve been running a few of the popular AI code review tools (think SonarQube with AI, Snyk Code, DeepSource, etc.) in parallel on our team’s PRs for the last quarter. The goal was simple: which one actually helps, versus just adding noise? I kept hitting walls with vendor claims and wanted our own weekly truth serum.
So I built a script that scrapes our review data, maps AI comments to our eventual human resolutions (accepted, dismissed, false positive), and calculates precision, recall, and overall “noise” (comments dismissed by devs) for each tool. Every Monday, it posts a digest to our #engineering Slack channel. The team loves the transparency, and it’s shifted our subscription conversation from “which is cool” to “which is accurate.”
The core of it is pretty simple—just some Python that ties into our GitHub webhooks and a small PostgreSQL table to track the lifecycle of each AI comment. The magic is in the categorization logic. For example, if a human reviewer or the PR author marks an AI suggestion as “not relevant” and it’s not revisited, that’s a strike against precision.
We’re now seeing clear leaders. Tool A might have 85% precision but only catches 60% of the issues we ultimately care about (low recall). Tool B bombards us with comments but 40% are dismissed as noise. Having this in a shared, automated report has been a game-changer.
If anyone’s interested in the approach or the script itself, happy to share details. Curious if others are tracking similar metrics or just going by gut feel?
—kc
Sample size matters.