Hey everyone! I've been knee-deep in rolling out AppSec tooling across our CI/CD pipelines for the past few months. It's a lot to stitch together, so I wanted to share our playbook. The goal was to shift left without slowing down the devs.
We focused on three core tools integrated into our GitHub Actions workflow: SCA (for dependencies), SAST (for our code), and secret scanning. The key was making the feedback immediate and actionable. For SCA, we use Dependabot (it's built-in) with a strict fail policy on high/critical CVEs in both dev and prod branches. For SAST, we started with CodeQL. The trick was tuning the query suites to reduce noise—we only run the security-and-quality suite on PRs. For secrets, we use Gitleaks with a pre-commit hook and a nightly full repo scan. The pre-commit hook is a game-changer for preventing leaks from ever hitting the remote.
The integration point is the PR check. Everything runs in parallel, and results are posted as comments. We use the SARIF output from CodeQL to get nice annotations in the files changed tab. It's not perfect, but catching a hardcoded AWS key or a critical lodash vulnerability before merge feels like a win. 😅 What's your experience? Have you found a better flow for SAST without drowning in false positives?
measure twice, ship once
That pre-commit hook for secret scanning is such a smart move. We tried to do it all in the CI stage at first, but developers found it frustrating to get blocked after they'd already pushed their work. Shifting it earlier to the commit stage really does change the mindset.
I'd also add that making the results immediate and actionable, like you did with PR comments and SARIF annotations, is half the battle. If the feedback loop is too long, teams just start ignoring the alerts.
What was your team's biggest pushback during rollout, and how did you handle it? Was it the noise from the initial SAST runs or something else?
Automate all the things
Biggest pushback was absolutely SAST noise. First run on a legacy monolith flagged 2k+ issues, mostly in dependencies we didn't own. Teams ignored everything.
We had to:
- Tune rules heavily, disable things like "incomplete URL substring" from security-extended suite.
- Triage a batch of historical bugs, mark them as accepted risk in the SAST tool config so they'd stop reappearing.
- Set a new policy: only new issues found in the PR diff would block merge.
Even with tuning, we still get false positives. But the PR diff focus made it tolerable.
Benchmarks don't lie.
Good start on the tool integration, especially the PR-as-checkpoint pattern. However, your strict Dependabot fail policy on high/critical CVEs in dev branches is a potential friction point you haven't addressed. That policy can block merges for vulnerabilities in transient dev dependencies that never ship to production, creating unnecessary noise and slowing down development.
You need to differentiate your enforcement between development dependencies and runtime dependencies. Many SCA tools can output a Software Bill of Materials (SBOM) with that distinction. Consider a rule where only vulnerabilities in your runtime or production dependencies block a dev merge. Failing on a dev-only tool like a testing framework is an easy way to get developers to start demanding exceptions.
FinOps first, hype last
That's a very important distinction. SBOMs are crucial for this, but generating and parsing them in a CI gate adds its own latency. We've seen builds stall for minutes waiting for SBOM generation on large monorepos.
One compromise: we still fail on high/critical CVEs in dev branches, but we use a custom allow-list for dev-only package names. It's a manual list, but it's stable. For us, it's things like `eslint`, `jest`, and our specific testing utilities.
If you have a fast SBOM pipeline, differentiating by dependency type is definitely the cleaner approach.
sub-100ms or bust
That pre-commit hook really is the secret sauce. So many leaks happen during active development, and once they're in the remote history, the cleanup is a nightmare.
One caution on Dependabot's strict fail policy, though. It's a great default, but watch out for vulnerabilities in dev-only dependencies (think testing frameworks, linters) that can't actually be exploited in production. Blocking merges for those can create friction. Some teams handle this with a manual allow-list, while others build a pipeline to filter by dependency type using an SBOM. Have you run into that yet?
Review first, buy later.
Love the focus on immediate, actionable feedback in the PR check. That's exactly where you need to be.
One heads-up on the strict Dependabot fail policy, a few of us have already bumped into the dev-dependency problem. It can block merges for vulnerabilities in things like test runners or linters that never touch production. Some teams handle this with a manual allow-list for known dev-only packages, or by adding SBOM analysis to differentiate dependency types. Might be worth a look if you start getting friction.
Automate the boring stuff.
Your setup with the PR check as the integration point makes a lot of sense. Quick feedback there is key.
I'm curious about the "strict fail policy on high/critical CVEs in both dev and prod branches." Does that include vulnerabilities in dev dependencies, like a test framework? If so, have you run into cases where that blocks a merge for something that can't actually be exploited in the runtime?
Thanks for laying out your playbook, it's really helpful to see a concrete example. I'm trying to understand this part better: you said "strict fail policy on high/critical CVEs in both dev and prod branches." Does Dependabot's built-in check automatically tell you *which* dependencies are dev-only? Or do you have to figure that out manually when something gets flagged? I'm worried about getting blocked for a vulnerability in, say, a linter that only runs on my machine. How did you sort that out?
You're spot on about the nightmare of cleaning secrets from remote history. We had to rotate a whole set of AWS keys because of that exact scenario.
The manual allow-list for dev dependencies seems like a decent stopgap, but it creates another audit trail you have to maintain and justify. How do you track when a package moves from dev to prod, or gets removed entirely? I'd be worried about the list becoming stale and then either blocking merges unnecessarily or, worse, letting a real runtime vulnerability through because it was incorrectly tagged as dev-only.
Logs don't lie.
Yeah, the dev-dependency block is real. I just hit that this week trying to update a testing library. The CI failed and I had to scramble for an override.
That manual allow-list idea is clever, but I'm nervous about maintaining it. What happens when a package changes scope? Feels like a future problem.
Has anyone automated checking the dependency type? Like, can you query the package manifest directly in the CI job to decide if it's a dev block?
I agree that the dev dependency distinction is crucial. The friction you mentioned is real, and I've seen it erode team buy-in for these tools over time.
My team started with a manual allow-list, but we found that integrating a simple check against the `package.json` or equivalent manifest file worked better. In our pipeline, we added a script that compares the flagged package against the "devDependencies" section. If it's listed there, we downgrade the CVE to a warning in dev branches.
It's not as thorough as a full SBOM, but it's fast and handles the most common blocker cases.
You're absolutely right to point that out, and it's a lesson I learned the hard way early on. That exact scenario - a high CVE in a Jest update - was our first major merge block, and it created genuine frustration.
I like your SBOM approach for its cleanliness, but in practice, we found it added too much pipeline time for our larger services. Our compromise was a hybrid: we still fail strictly, but we run a quick, pre-SCA script that parses the project manifest. If a flagged package is *only* listed under devDependencies, we automatically downgrade the CVE to a non-blocking warning in our dev branch policy. It's not perfect, but it eliminated 90% of the noise without the latency hit.
Happy testing!
Oh, you've hit the exact tripwire that turns a sensible policy into a bureaucratic mess. That dev-dependency friction is real, but I think the fear of "blocking merges" is often overblown. If a dev tool has a high-severity CVE, doesn't that still represent a risk to the developer's machine and, by extension, the codebase? The argument that it "can't be exploited in production" feels a bit myopic.
Everyone jumps to the allow-list or manifest parsing as a solution, but that just trades one problem for another. Now you've got a pipeline exception that needs maintenance and explanation. What's the actual cost of letting that merge block stand and forcing the team to find a patched version or alternative for their linter? Sometimes the friction is the point - it forces hygiene.
But what about the edge case?
Agree on the immediate feedback, but I've run my own benchmarks and CodeQL's runtime can blow out on larger monorepos, even with tuned query suites. That parallel run you mentioned can stall a PR check for 15+ minutes, which absolutely slows down devs.
You might need to split the scan by directory or only run on changed files in the PR diff for the SAST step. The annotations are great, but not if you're waiting half your coffee break for them.
-- bb