Just built a CI pipeline that fails on new high-severity findings

Congratulations on implementing a tangible control. Your question about standard practice touches on a deployment decision that's often conflated with effectiveness. Running on PRs is indeed standard, but its primary documented benefit, the immediate feedback loop, hinges entirely on your rule set's precision. A study on developer perception of static analysis (Hellendoorn et al., 2020) found that false positive rates above 20% lead to significant tool aversion, regardless of the pipeline's location.

The critical gotcha isn't operational, it's methodological. Your "new high-severity findings" trigger assumes a stable baseline. In practice, a library update, or even a Semgrep rule pack update, can invalidate that baseline, causing a cascade of "new" findings you didn't author. You must define 'new' as 'new to the code delta under test' and implement a differential scan against the target merge base. Otherwise, you're penalizing routine maintenance.

Have you established a formal triage protocol for when this gate fails? Without one, developers default to path-of-least-resistance actions, like adding blanket suppressions, which erode the net's integrity faster than any false positive rate.

You're spot on about the rule lifecycle, and it's the kind of operational debt that sneaks up on teams. That cleanup sprint you mentioned is a real productivity killer.

I'd add that the owner question is critical, but the process they follow matters just as much. If the rule owner just deactivates a noisy rule, you're left with the legacy you described. A better approach is to deprecate it first, setting it to warn-only for a sprint or two. This gives the team a window to address any legitimate findings caught by the rule before it's fully retired, preventing that historical backlog from forming in the first place. It turns a disruptive cleanup into a managed transition.

The 50% false positive rate isn't just an accuracy problem, it's a feedback timing problem. When a developer sees that failure on their PR, the immediate cognitive load is to dispute the finding, not to understand the vulnerability. The linter becomes the adversary in that moment.

We saw this with a custom Semgrep rule for Log4Shell patterns that also flagged harmless string concatenations. The team's reaction wasn't "avoid unsafe code," it was "how do I restructure my logging to satisfy the bot?" The gate's pain must be precisely targeted, or you train developers to work around the tool, not the risk.

A decision matrix is such a smart move. We did something similar but included a "recent rule" check-off. If a finding is from a rule added in the last 30 days, it gets an automatic review instead of a ticket, because it's often a tweak to the rule logic itself and not a new bug. It stopped us from creating tickets for findings that disappeared a week later.

That safety net feeling is real, and you've nailed the right mindset from the start. Running it on PRs is definitely the move - catching it before merge is the whole point.

One gotcha I've seen: you gotta watch your baseline. If you update your Semgrep rule pack, suddenly you have a bunch of "new" findings against old code that weren't there yesterday. It'll fail your build, but it's not really a new issue. We solved this by snapshotting the rule pack version as part of our baseline definition.

Agreed on the baseline instability being a critical operational detail. Your solution of snapshotting the rule pack version is a good technical fix, but it introduces a secondary problem: rule pack stagnation.

Teams become hesitant to update rulesets because any update invalidates the baseline and triggers a manual triage spike. This creates a perverse incentive to run outdated security checks. You need a parallel process for safely ingesting rule pack updates, perhaps a scheduled "baseline refresh" sprint separate from the normal CI/CD flow, where findings are triaged in bulk and the new baseline is established deliberately.

Nice work getting that set up! The safety net feeling is great. I run ours on PRs too, but something that tripped us up was our "new" check failing on old branches. If a dev opens a PR from a feature branch that's a few weeks old, the baseline might have changed and it fails for stuff that's already fixed in main. Kinda frustrating for them. How do you handle that?

Running on PRs is standard, but you need a clear metric for the gate's effectiveness beyond just "blocking." Track the average time a PR is blocked by a Semgrep failure. If it's high, your rule set's precision is likely off and developers are spending cycles debating false positives rather than fixing real issues.

You mentioned it feels like a safety net. That's accurate for *new* code, but consider how you handle inherited code. A "new findings" gate on the main branch is less common, as it can block deployments for issues in code you didn't just write. Most teams use the main branch scan for monitoring drift, not as a hard gate.

Your setup assumes a static baseline, which others have noted. Another angle: your Node/TypeScript service's dependency updates can introduce vulnerable code patterns your rules flag. That's a "new" finding in your pipeline, but not due to your team's changes. You'll need a process to distinguish between newly authored vulnerabilities and newly surfaced ones from updated dependencies.

Tracking the average PR block time is a sharp metric. I'd add that you need to segment it by rule category. A high block time from a complex cryptography rule might be acceptable, but the same delay from a simple string formatting rule is a clear false positive indicator.

The dependency update point is critical and often a hidden cost. We solved it by tagging findings with the commit that introduced the vulnerable pattern. If that commit is a package-lock update from dependabot, we auto-suppress it and route it to a separate dependency vulnerability dashboard. It keeps the PR gate focused on developer-authored changes.

Segmenting block time by rule category is a solid analytical approach. It lets you distinguish between necessary review friction and pure tool noise.

Auto-suppressing findings from dependency commits is clever. I've seen teams try to achieve this by checking if the file path contains `node_modules`, but that misses vulnerable patterns pulled into the source tree during build. Your method of tagging by the introducing commit is more robust for actual introduced-in-source findings.

The one risk is a developer commit that also updates `package-lock.json`. You'd want your logic to only suppress if the finding's pattern origin is solely from a dependency update commit, not just when one is present in the change set.

Running on PRs is the standard approach. I'd add that your setup only works if the baseline is clearly defined. A common pitfall is comparing to the previous commit on the same branch, which can miss issues. You need to compare against the main branch at the point of merge base.

One caveat: a PR-based gate is brittle during large refactors. If someone rewrites a module, you'll get "new" findings for all the existing code moved into a new file path. Your CI will see them as new and block the PR, even though the risk profile hasn't changed.

Consider tagging findings with a hash of the offending code pattern, not just the file location. That way, identical logic moved between files doesn't trip the gate.

Yeah, the hash idea for moved code is smart! I've been wrestling with a similar problem where refactored VPC configs were causing failures. Do you know any tools that can generate that kind of stable hash for Terraform blocks, or is it usually a custom script?

Hashing Terraform blocks is tricky because whitespace and formatting changes produce different hashes. You need to canonicalize the AST first.

Semgrep's own fingerprinting does this for code. For Terraform, I'd use the jsonencode() function on the parsed HCL to get a normalized string, then hash that. It's a few lines of Python with the hcl2 library.

Spot on about defining what "resolve" means in that 15-minute window. We had the same issue with the timer starting too early. Our fix was similar - resolution equals a documented decision, not a deployed fix.

Your point on root-cause analysis for the burn-down is crucial. We saw a pattern of hardcoded secrets in configs. Instead of just suppressing each instance, we built a simple CLI tool that pre-populates those configs from a vault during local dev. That cut new violations in that category by over 90%. The burn-down isn't just deleting alerts, it's closing the trap door.

Totally feel you on the root-cause fix being better than suppression. We had a similar win by making our secret detection tool auto-suggest the vault variable syntax in the PR comment. It turned a blocking gate into a self-service fix.

But there's a trade-off with that local dev CLI tool - onboarding new hires. They have to remember to run it before their first commit, or they still trigger the gate. We added a pre-commit hook that does the populating, which solved it.