Check out my benchmark: false positive rates across 4 SAST tools on our code.

Alright, I've finally had enough. After sitting through yet another vendor presentation where they touted a "99% accuracy rate" with no concrete definition of what that even means, I decided to run our own internal benchmark. We took a snapshot of our primary monorepo—about 1.2 million lines of a mix of Java, Go, and Python—and ran it through four of the usual suspects: the current market leader, the open-source darling everyone recommends but never actually runs at scale, the "AI-powered" newcomer, and the one our security team is pushing for because of some bundle deal with our container scanner.

The methodology was straightforward: we ran each tool with its default ruleset against a fixed codebase. We then manually reviewed every single finding from each tool across a statistically significant sample of the codebase to classify true positives, false positives, and the debatable ones where the tool is technically correct but the finding is irrelevant (like a hardcoded password in a unit test mock object). We counted those last ones as false positives for the purpose of this exercise, because if I have to spend time reviewing it, it's noise.

The results were, frankly, predictable but worse than I expected. The market leader had the lowest outright false positive rate at around 22%, but that's because it flagged so little overall. It missed three known vulnerabilities we have intentionally left in a test module for this purpose. The open-source tool flagged everything, including the kitchen sink, resulting in a false positive rate north of 68%. The noise was so immense the report was essentially useless without massive tuning. The AI tool was a fascinating disaster; its false positive rate was about 45%, but its explanations for the findings were often comically wrong, citing CWE patterns that didn't match the code at all. The bundle-deal tool sat in the middle at 35% false positives, but its performance on the monorepo was abysmal, taking three times longer than any other and failing to correlate findings across service boundaries.

What this tells me is that the advertised numbers are borderline meaningless. Every one of these tools requires such significant investment in tuning—creating custom rules, maintaining extensive allow-lists, and constantly adjusting for framework updates—that the total cost of ownership is being wildly underestimated. You're not just buying a license; you're buying a multi-year commitment to feed and care for a very noisy, very needy pet. And by the time you've tuned it to be useful, you're so deeply locked into that tool's specific rule syntax and workflow that migrating is a non-starter.

I'm particularly skeptical of the "out-of-the-box" experience they all sell. Out-of-the-box, you get either a deluge of nonsense or a trickle of obvious findings. The real work begins after the purchase order is signed, and I rarely see that factored into the business case. Has anyone else done similar benchmarking and found a tool that doesn't require a small army of consultants to make it operational? Or are we all just accepting that 30-40% of our AppSec team's time will be spent curating tool output rather than actually improving security?

Just my two cents