Real experience with GitHub secret scanning: false positive rates

Your initial experience matches what I've observed across several teams. The first week can generate an alarming volume of alerts, primarily because the default patterns are quite broad to catch everything.

The false positive rate does improve significantly with tuning, but it's not automatic. You need to actively manage it. Start by creating and pushing a `.github/secret-scanning.yml` file to your repositories. This lets you define custom patterns for your internal test tokens and exclude specific paths, like your documentation directories. Also, don't forget to mark the initial batch of example placeholder alerts as false positives in the UI, as this feedback helps train the system.

A common oversight is neglecting to adjust the scan for dependency files. If you're using package managers, secrets can be flagged in `package-lock.json` or `yarn.lock`. You'll likely want to add a pattern exclusion for those. The noise will drop from a flood to a trickle, but you must commit to that initial configuration effort.

The initial noise you're experiencing is entirely normal, and I can quantify that statement. In my team's deployment across roughly 300 repositories, our initial false positive rate was approximately 85%. The key is to treat the first wave of alerts not as incidents, but as a data collection phase for your tuning process.

Focus your first tuning efforts on path-based exclusions. Create a `.github/secret-scanning.yml` file and start by excluding entire directories known for noise, such as `docs/`, `test-fixtures/`, and `vendor/`. This single action can reduce alert volume by 40-60% immediately. Next, systematically review the flagged patterns; you'll likely find a handful of high-frequency offenders, like internal UUID formats or example `CONFIG=xxx` strings in documentation, that you can add as custom patterns with a lower confidence level.

The system doesn't automatically "learn" from dismissals in a machine learning sense, but your manual categorization builds the historical data you need to refine rules. After two weeks of diligent categorization and rule adjustment, we brought our false positive rate down to under 15%. The ongoing maintenance is minimal, mostly reviewing net-new pattern alerts.

Totally agree with treating the first wave as a data collection phase. That's a crucial mindset shift. The 85% initial false positive rate you mentioned lines up with what I've seen, especially in larger, older repos with legacy test data.

One caveat on path exclusions: be a bit careful with blanket `vendor/` or `node_modules/` exclusions if you're scanning dependencies for secrets (which you can opt into). We found a few real, sneaky secrets buried in a vendored library's test file once, so we ended up using more granular exclusions for our own code only.

Your point about the system not "learning" from dismissals is key - it's a manual feedback loop. I'd add that committing time to build those custom patterns for internal token formats pays off exponentially down the line. It turns the scanner from a noisy alarm into a useful guardrail.

Your experience is common. We saw the same flood of alerts for test fixtures and example strings. It does get better, but only if you commit to tuning.

Our team decided to audit every alert from the first week, even though it took time. We found three recurring patterns in our false positives and built custom patterns for them. That cut our weekly review time down by about 70% after a month.

What's your process for reviewing the alerts? Are you doing it solo, or can you spread the initial audit across the team?

Yeah, the initial flood is real. My tip for handling the noise is to sort the alerts by pattern first, not by repo. You'll quickly see which specific detector is causing most of the doc/placeholder hits. Then you can push a custom pattern to block that one noisy rule across all repos at once, which is faster than path exclusions piece by piece.

Did you know you can also pause alerts for a specific pattern for 24 hours while you write the fix? Gives you a bit of breathing room.

Good call on flagging dependency files - that's a nuance teams often miss right away. I'd say excluding lockfiles is a solid first step, but if you're using the dependency scanning feature, you might want a more targeted approach later.

> the system not "learning" from dismissals
Exactly this. I've seen teams assume marking alerts as false positives creates some ML model adjustment, but it really just suppresses that specific instance. You have to build those custom patterns yourself, which is both a limitation and a feature, since you maintain full control over what your org's "normal" looks like.

Calling it a feature to manually build patterns is a generous spin. It's more like a tax on your engineering hours. GitHub sells this as a security automation, then bills you again in labor to make it stop yelling at your own docs.

And that targeted approach for dependencies? Good luck. You'll spend more time writing exclusion regex than you would just grepping the code yourself quarterly. The whole promise falls apart when you realize the tuning work scales linearly with your repo count.

"Tax on your engineering hours" is a bit dramatic, but the core frustration is real. The real catch-22 is that the teams who most need this automation - the ones with sprawling, legacy repos - are the ones who get hit hardest by the tuning tax.

You're right about the scaling problem. I've seen it: the promise of a centralized security tool breaks down when each team's "internal test token" format is slightly different, forcing you into regex-whack-a-mole or blanket exclusions that defeat the purpose.

So is the answer to just grep quarterly? Maybe for a small, tidy codebase. But for most, that's a compliance nightmare waiting to happen. You're stuck with the lesser of two evils.

The "audit every alert from the first week" approach is the only way to do it, though I question if a week is enough for a truly messy repo. We gave it a full sprint. The real trick is turning that audit into something executable for the team.

You asked about process. We made it a rotating duty, a "secret scanning sheriff" for a week. They'd triage the queue, build the custom patterns, and update the central config. It spreads the pain and builds institutional knowledge. Without that rotation, the tuning tax user536 mentioned becomes a permanent burden on one poor soul.

The 70% reduction tracks, but it only holds if you treat those custom patterns as living docs. New services with new dummy credential formats will sneak in and you'll be back to square one if you don't have a process to catch them during code review.

You're in the classic onboarding phase everyone hits. That initial 85%+ false positive rate others mentioned? That's been my experience too, it's just the cost of entry.

The tuning does work, but think of it as an ongoing FinOps-style cost allocation exercise. You're investing engineering time now to reduce future alert noise, which has a tangible operational cost. The key is tracking that time and the reduction in alerts to prove the ROI to your team, otherwise it feels like wasted effort.

One tip I didn't see mentioned: treat those internal test tokens and example placeholders as a policy failure. Why are those in the repo at all? The best custom pattern is one that prevents the dummy data from being committed in the first place. Can you push for using environment variables or verified patterns in your docs? That's a longer-term fix, but it reduces the tuning surface area.

Absolutely love the "policy failure" framing. It reframes the whole problem from cleanup to prevention, which is so much more sustainable.

The ROI tracking you mentioned is a great discipline. We logged our triage hours for the first quarter, then showed leadership the trend line dropping month over month. That hard data turned it from a "noisy security tool" complaint into a funded, ongoing initiative.

One pushback, though. For a fast-moving startup, "never commit dummy data" can be a heavy lift. We got more traction with a lightweight git hook that just warns you at commit time if a new internal token pattern appears. It's not perfect, but it nudges behavior without grinding dev speed to a halt.

Great point about the git hook as a lightweight nudge. We tried that, but hit friction when developers would just skip the hook locally. The real win was adding it as a required status check in CI, so it couldn't be bypassed. It sounds heavy, but it's the only way we got the "policy failure" idea to stick.

> ROI tracking...turned it from a "noisy security tool" complaint into a funded, ongoing initiative.

This was the only way we got budget for a proper secret manager. Once leadership saw the triage hours as a recurring cost, funding a real fix became an easy decision.

Required CI checks are the hammer, but you have to be careful not to hit your own thumb. The big trade-off is dev velocity. If that check flags a legit pattern in a hotfix branch, you've just added a frustrating blocker.

That's why we treat the required check as a final gate, but lean on the local hook for 95% of the nudges. The CI one just catches the folks who --no-verify out of habit. The real trick is making the local hook fast enough that it's not a nuisance itself.