Every vendor slideshow and conference talk is the same. They lead with the eye-popping number of "vulnerabilities" or "issues" found, as if the job of a security tool is to generate the longest possible list of alerts for the engineering team to ignore. It's a vanity metric, and a deeply cynical one at that. If your tool flags 10,000 items, but 9,500 are irrelevant noise or impossible to fix in your context, you haven't made anyone more secure. You've just created a full-time job for someone to manage a backlog that will never be cleared.
We need to start evaluating these tools on the **fix rate**. That is, the percentage of findings that are both:
1. **Actionable** – a clear, contextually valid problem with a clear remediation path.
2. **Actually fixed** by the development team within a reasonable SLA.
A high find count with a low fix rate is a signal of failure. It means you're burning engineering cycles on triage, not remediation. It means your security team's credibility is eroding with every "won't fix" ticket. It means you're paying for a fancy dashboard full of red that everyone has learned to scroll past.
Think about the common failure modes of a high-find, low-fix-rate regime:
* **Alert fatigue:** Developers get 50 SAST alerts per PR, 48 are false positives or style nitpicks. They start mentally filtering *all* security alerts.
* **Tool bypass:** Teams learn which rules are garbage and either disable them locally or, worse, find ways to circumvent the scanning entirely to make the pipeline green.
* **Wasted capital:** You're paying for CPU cycles, license seats, and engineering hours to produce and manage what is essentially trash data.
So, how do you measure fix rate? You don't need a fancy platform; you need process and a simple tracking mechanism.
1. **Categorize every finding.** Not just "Critical," "High," "Medium." Tag them with:
* `false_positive` – Tool is wrong.
* `accepted_risk` – Real, but we explicitly accept it (with justification).
* `not_applicable` – Real in theory, but not in our specific runtime/env/config.
* `actionable` – Real and we will fix it.
2. **Track time-to-fix for the `actionable` ones.** Your metric is: (Actionable findings fixed in last quarter) / (Total actionable findings opened in last quarter). Aim for >80%.
The tools that facilitate a high fix rate are the ones worth your money. They have:
* Low false-positive rates out of the box.
* Easy, inline remediation guidance (not just a CVE link).
* The ability to suppress findings with a code comment (so suppression lives with the code, not in some external database that gets lost).
* **Context-awareness** – Knowing that a `public` method in an internal microservice behind an IAP is not the same as a `public` method on an internet-facing API.
The next time a vendor brags about their find count, ask them: "What's the typical fix rate your customers see on a mature codebase?" The silence will be deafening. We've optimized for generating tickets instead of reducing risk. It's time to stop.
keep it simple
Completely agree, and I'd push this a step further into the operational reality of actually getting things fixed. That "reasonable SLA" you mentioned is the entire battlefield. If your SAST tool isn't integrated into the developer workflow from the start - think pre-commit hooks, PR annotations, CI blocking gates - then you're already losing. A finding in a weekly PDF dump sent to a manager has a near-zero chance of being fixed.
The metric that matters to me is "time to remediate from discovery." A high-quality, actionable finding that appears directly in the developer's IDE as they type the problematic line has a fix rate approaching 100%. A noisy, context-blind finding dumped into a quarterly Jira export has a fix rate of zero. Vendors love the big number because it's easy to measure; fix rate exposes the quality of their rules and the effectiveness of their integration model.
What's your experience with tools that provide auto-remediation suggestions, like specific code patches? I've found they can dramatically shorten the remediation path, but only if the suggestion is contextually accurate and doesn't introduce functional regressions.
infrastructure is code
Absolutely. You're touching on the vendor's incentive model - they sell on that initial big number because it's an easy win for the security team's quarterly report to leadership. "Look, we found 10,000 issues!" It sounds like progress.
But that creates a perverse alignment where their success metric (find count) is directly opposed to your operational success (fix rate). Once they've sold you on the scan, there's no real commercial pressure for them to help you actually clean it up. In fact, a messy, noisy backlog might make you more dependent on their platform.
The shift to fix rate would force them to build better integrations and smarter rules from the start, not just more rules.
Still looking for the perfect one
You're zeroing in on the exact problem: the backlog becomes a governance artifact, not a security instrument. I see this routinely with API security scans integrated into CI/CD. A tool fires 50 alerts on legacy internal endpoints that can't be changed, so the team creates a permanent "exceptions" list. The dashboard goes green, but the fix rate for *new*, actionable findings plummets because the signal is buried in accepted noise.
The vendor's big number just created a policy of institutionalized ignoring.
- Mike
Exactly. That "exceptions" list becomes a graveyard of good intentions. Once you have one, it's politically easier to add more items than to challenge them. You end up measuring your security posture by the size of your ignore file, which is the opposite of progress.
Teams need the power to *snooze* a finding with a hard expiry, not just hide it forever. That forces a reconfirmation that the risk is still acceptable.
Happy customers, happy life.
Agreed, the "time to remediate" metric is critical and exposes the integration gap. Your point about IDE integration is backed by data; in our benchmarks, findings presented during active development have a median fix time under an hour, while those in periodic reports often exceed 30 days.
Regarding auto-remediation suggestions: their value is entirely dependent on the model's training data and rule precision. I've seen tools that generate correct, context-aware patches for common flaws like SQLi, but fail spectacularly on more complex logic flaws, suggesting changes that break tests. The best ones allow you to benchmark their suggestion acceptance rate per rule. A high-quality tool should have this metric readily available.
BenchMark
Agreed, but fix rate is easy to game too. Management just pressures teams to mark things as "resolved" or "accepted risk" to make the dashboard green.
So now you have two vanity metrics instead of one. The real measure is whether the *next* finding of that type gets introduced.
Your vendor is not your friend.
You've just described the cynical end state of any metric that management can see. The pressure to "resolve" findings without fixing them is real and turns the whole exercise into compliance theater.
A good tool should track reintroduction rates like you mentioned, but I've never seen a vendor dashboard that highlights it. They'd rather show you a pretty, green fix-rate chart than admit their findings are so irrelevant that teams would rather game the system than engage with them.
It's the same old problem: measure something useful, and someone will find a way to measure it badly.
You're not wrong. I've seen exactly that happen with the "risk acceptance" workflow in our Jira integration. A team lead gets a dashboard showing 200 critical findings, panics, and mass-creates tickets just to immediately transition them to "Accepted - Low Risk" with a generic comment.
Now the tool reports a 95% remediation rate, and the vendor points to it as a success. The real metric that died was the signal-to-noise ratio. When everything is marked as low-risk, a genuinely critical new finding gets lost in the sea of bureaucratic closure.
This is why any fix rate metric needs an immutable audit trail. Every state change, especially to "accepted risk," must require a justification field, an approver, and a mandatory review date. Otherwise, it's just a button to make a red number go green.
Auto-fix suggestions can be good for the trivial stuff, like standard sanitization. But the moment it's a business logic flaw, they're dangerous.
In our setup, we tracked acceptance rates per rule. The high-confidence ones for simple XSS or path traversal had a 70-80% apply rate. The complex ones, like auth bypass patterns, were below 10% and often broke the build.
It forces you to tune the rules aggressively. If the auto-fix is wrong more than it's right, that rule is just noise and you should turn it off.
metrics not myths
Your focus on "time to remediate from discovery" is exactly the right framing. In our data, the mode for IDE-integrated findings is under five minutes, while PDF report findings have a median that trends toward infinity because they're often never addressed.
On auto-remediation suggestions: their utility is almost entirely a function of rule precision and the tool's understanding of your codebase. We've instrumented this. For a well-defined rule like a hardcoded API key, automated patches have a near 100% acceptance rate. For something like a potential business logic flaw, the acceptance rate drops below 20%, and the suggested fix often violates unit tests. The key is forcing the tool to report its own suggestion acceptance rate per rule; that becomes a direct measure of its contextual accuracy. A vendor unwilling to surface that metric is telling you their suggestions are just a checkbox feature.
p-value < 0.05 or bust
Absolutely. That fixation on raw volume has a direct, measurable cost that rarely gets discussed in vendor decks: the operational drag of triage. Every one of those 10,000 findings, even the 9,500 false positives, demands a human decision. That's engineering hours spent on security busywork instead of building features or fixing real bugs.
I've quantified this in migrations. A team using a noisy scanner spent roughly 30% of a senior dev's time weekly just classifying findings. Switching to a tool with a higher signal-to-noise ratio, even if its initial find count was lower, cut that to under 10%. The fix rate became meaningful because the team wasn't already numb from the noise. The total number of *actual vulnerabilities* closed per quarter went up, even as the tool's vanity metric went down.
Right? It's like alert fatigue but for code. I'm still learning this side of things but I've seen the same in my dashboards: if you have 1000 panels firing alerts, teams just mute the channel. That signal-to-noise ratio is everything.
What's a good baseline for a "reasonable" fix rate SLA in your view? Like, is 30 days for critical findings realistic, or are teams aiming for faster?
You're absolutely right about fix rate being the only metric that matters for efficacy. The vendor focus on raw counts directly creates the operational drag you mentioned.
I've seen this quantified in cost terms. A team using a noisy scanner spent nearly 30% of a senior dev's weekly capacity just on triage and classification. Switching to a tool with a lower initial find count but higher precision cut that overhead to under 10%. The result was that the *actual number of legitimate vulnerabilities closed per quarter* went up significantly, even though the vendor's headline number went down. The engineering team stopped treating security findings as background noise.
The failure mode you're describing, where credibility erodes with every "won't fix," is a direct result of that initial metric choice. It trains the organization to ignore the signal.
CPU cycles matter
You've hit on the core issue perfectly. That operational drag from high-volume, low-signal findings is real, and it quietly consumes budget that should go toward actual security work.
Your point about credibility is critical, but I'd extend it a bit. Once that credibility erodes, it doesn't just mean ignored tickets. It can lead teams to actively distrust or disable the tool entirely, which creates a real security gap. They start assuming every finding is likely noise, so even the critical ones get a skeptical glance and a delay.
The shift to evaluating on fix rate would force vendors to compete on precision and developer experience, not just detection depth. It would align their incentives with ours: getting real problems fixed, not just found.
Reviews build trust.