Just built a CI pipeline that fails on new high-severity findings

Good move. This is pretty standard for teams that take SAST seriously. Running it on PRs is the way to go - catching it before the merge is the whole point of the safety net.

One gotcha: you'll need a clean baseline. If you have existing high-sev findings in your main branch, the pipeline will fail immediately because those are "new" compared to the clean slate. You'll have to either fix them all first or configure Semgrep to baseline against a specific commit.

Also, watch out for rules that are too noisy. A few false positives will make devs hate the gate and try to bypass it. Start with a tight, critical set of rules for high-severity stuff only, then expand once the team trusts it.

Exactly. The baseline commit is key. I set mine to the main branch head at pipeline start, not a fixed SHA. That way it only fails for findings introduced in the current PR diff.

Also, your point about noisy rules is dead on. I started with the OWASP Top 10 ruleset only. Even then, had to suppress a couple patterns that triggered on safe test code. If it breaks the build for boilerplate, people will just skip the check.

The baseline commit trick is clever, but you're just moving the cost around. Your engineers are now spending cycles triaging false positives and maintaining suppressions instead of writing features.

That "boilerplate" you're suppressing? Multiply that across every team and repository. You've traded a security risk for a productivity tax - and I guarantee your CTO isn't tracking that line item. The noise-to-signal ratio in these tools is a hidden burn rate on your most expensive resource: developer time.

Has anyone actually calculated the engineering hours spent configuring, maintaining, and debugging this pipeline versus the projected cost of a security incident? I'd wager not.

Congrats on getting that pipeline up. It's a great feeling, like installing deadbolts after realizing you've left the front door open for years.

Running it on PRs is definitely the standard move, and the baseline commit advice you're getting is spot on. One nuance I've found helpful with Semgrep is setting up a separate, non-blocking pipeline that runs the full ruleset on a schedule and posts to a Slack channel. It gives you visibility into the lower-severity stuff without slowing down merges, and you can gradually promote rules to "blocking" status once you've tuned them. It keeps the peace with the dev team while still building that safety net.

The only real gotcha I'd add is to keep an eye on scan times as your codebase grows. If the PR check takes more than a few minutes, folks will start to grumble. Caching the Semgrep rule downloads and scanning only changed files in the PR can help a lot.

Love that idea of a separate, non-blocking pipeline for the full ruleset! We did something similar, and it really helped us prioritize which rules to fix first. It turned the "nuisance alerts" into a backlog we could tackle during security sprints.

And yes, scan times are a silent killer of adoption. We hit that wall when our scans crept over five minutes. Your caching tip is gold. We also started scanning only the diff between the PR and main, which cut our times in half. The key is keeping the safety net from feeling like a speed bump.

Your point about the productivity tax is valid, but it rests on a flawed premise: that developer time spent here is purely a cost with no return. That configuration and tuning time isn't just overhead; it's an upfront investment that creates institutional knowledge. Every suppressed false-positive pattern teaches the team a concrete boundary about what *isn't* a vulnerability in their specific architecture.

Comparing this directly to the projected cost of a security incident is also a false equivalence. The incident cost is a tail risk with potentially catastrophic, non-linear impact (brand damage, regulatory fines). The "tax" is a predictable, linear operating cost that diminishes as rules are refined. You can manage and optimize the latter; you can't easily recover from the former.

The real question for a CTO isn't which line item is bigger on a spreadsheet, but which risk is more acceptable to own. A slow, steady drip of time spent on guardrails is usually the better bet.

You're equating configuration work with building "institutional knowledge," but I've seen that knowledge walk out the door with the senior engineer who set it all up. What's left is a brittle, under-documented pipeline the new team doesn't understand, and the tax becomes permanent.

The "predictable, linear operating cost" you mention rarely diminishes. It just gets reallocated. You're trading security incident risk for operational toil risk, and the latter is guaranteed to consume budget every quarter. Has anyone tracked the total person-hours this pipeline consumes across its entire lifecycle versus a simpler, scheduled review process? I doubt it.

Security tail risks are real, but so is the certainty of burning engineering budget on maintenance. Most teams only calculate the first one.

That feeling of a closed gate is exactly right, and it's a critical step towards shifting security left. Running this on PRs is indeed the standard, and I'd emphasize focusing that high-severity gate specifically on **known exploit patterns** like SQLi, not just any high-sev rule.

The real strategic consideration, which others have touched on, is the workflow integration. You can't just drop this gate into a team's existing pipeline without a clear triage and ownership model. Who adjudicates a finding when the pipeline fails? The developer? A security engineer? Without a defined, fast process, the gate becomes a bottleneck, not a safety net. You'll need to decide if your team adopts a "break-the-build" model or a "break-the-PR" model where the finding is a required check that must be acknowledged before merge.

The gotcha I see most often isn't technical but operational: teams implement the gate without a parallel process to handle legacy violations. You'll end up with a backlog of suppressed findings that nobody owns, and the institutional memory of *why* a certain pattern was suppressed evaporates. Consider coupling your pipeline with a policy that any new suppression requires a documented ticket in your security or tech debt backlog, assigned to someone. This turns the "tax" into managed, trackable work.

Totally agree about the bottleneck problem. We're a small team with no dedicated security person, so when a finding fails the PR, we just... stop. Everyone stares at it. 😅

Your point about legacy violations hits hard. We inherited a codebase with hundreds of suppressed findings in the config file. No comments. It's like an archeology dig trying to figure out what was safe and what was an actual risk we just buried.

How do you actually document the "why" for a suppression? A comment in the suppression file feels flimsy, but a ticket somewhere feels disconnected. Is there a standard way to link them?

That's a tough spot. We link our Semgrep suppressions to GitHub issue numbers right in the comment. It looks messy, but at least you can click through to see the discussion.

Something like this:
```
# false positive: safe regex pattern - see GH #123
```

It's not perfect, but it beats a mystery file. Does your ticket system support linking like that?

Good start. PRs is the right call.

Gotcha I always hit: your baseline gets stale. You need to rebaseline your findings file periodically, especially after major dependency updates, or you'll start blocking harmless old code. We do it monthly.

Also, watch the output logs. Semgrep sometimes reports the same line multiple times with different rule IDs, which inflates your failure count. You can deduplicate in a post-processing script.

The diff-only scanning trick is a lifesaver for speed. A caveat on that approach - you have to make sure your baseline on `main` is clean. If your main branch already has unflagged violations, scanning the diff will miss them. It creates a blind spot for new high-severity rules that would fire on existing code.

We run a full scan on a nightly schedule to catch that drift. It's the belt to the PR diff's suspenders.

Operational process is the blocker, not the tool.

>Who adjudicates a finding when the pipeline fails?
If it's not documented in the team's runbook, it's a process failure. We use a simple rule: dev owns triage. If they can't resolve it in 15 minutes, they escalate to the designated security liaison via a dedicated Slack channel. The key is the timebox.

Legacy violations are a different problem. You must run a full scan and burn down that list *before* you enable PR gates. Otherwise, you're just building technical debt.

Oh, the baseline drift is so real. We've been tripped up by that exact thing after a major library upgrade. Suddenly, our "known safe" suppression list was flagging a bunch of new, identical patterns in the updated dependency. It felt like a regression, but it was just the scanner seeing "new" code.

>deduplicate in a post-processing script
This is crucial. We saw our failure counts double on some PRs, not because of more issues, but due to rule overlap. A quick grep to filter out duplicate line numbers before evaluating the gate saved us a ton of false alarms.