We just wrapped up a 3-month experiment where we mandated SAST scans (using Checkmarx) for every single PR, blocking merges on any high/critical findings.
The security team's report looks great: vulnerability count dropped sharply. But the team's energy is way down.
The big issues we hit:
* **Noise overload:** So many low-severity or false-positive findings (e.g., "hardcoded passwords" that were just example configs in test files).
* **Context blindness:** The tool couldn't understand our internal libraries or safe patterns, so devs spent hours writing suppression comments.
* **PR delays:** Simple fixes waited days because a scan queue was backed up.
It felt like we traded security for velocity *and* morale. Devs started dreading PRs.
Has anyone found a balance? Maybe only scanning on schedule? Or a way to drastically cut the noise before it hits the PR? I'm worried we just trained people to click "suppress."
Let's build better workflows.
Your experience mirrors what I've measured in several teams. The critical metric you're missing is the signal-to-noise ratio of your findings. If more than, say, 30% of high-severity alerts are false positives or require contextual suppression, the tool is governing on bad data and developer resentment is a rational outcome.
You asked about cutting noise before the PR. The most effective method I've implemented is a two-tiered scanning strategy:
1. A pre-commit hook running a fast, focused scan (using a linter like Semgrep with a highly curated rule set) that catches egregious issues like hardcoped secrets in new code.
2. The full, slower SAST scan only on the main branch nightly, with findings triaged by a dedicated security engineer before being filed as tickets.
This reduced PR-blocking events by about 70% in our case, because the nightly scan could use path-based exclusions and benefit from batch analysis. It also created a separation of concerns: developers fix what's in their immediate control, and systematic issues are handled as prioritized tech debt.
Mandating the tool on every PR assumes the tool's output is production-ready. In my experience, it rarely is without significant configuration and rule tuning, which should be done *before* enforcement, not during. Have you quantified the hours spent writing suppression comments versus fixing actual vulnerabilities? That ratio is often shocking.
Data first, decisions later.