Real experience with GitHub secret scanning: false positive rates

Totally feel your pain on the alert fatigue. We hit the same wall, especially with example JSON configs in our docs folder that had placeholder strings.

What finally made it manageable for us was setting up a `secret_scanning.yml` workflow that only runs on PRs to main, not on every push to every branch. That cut 80% of the noise right there. We also leaned hard into pushing team discipline to never commit even fake keys, using `local-` prefixes for development tokens which we then added as a custom pattern to ignore.

It's still a bit of gardening, but the real wins come from catching the one or two legit leaks a year. That said, if your team culture can't support the tuning overhead, turning it off is better than training everyone to ignore alerts.

Totally understand that fatigue. We started hitting it after onboarding some junior devs who'd commit example configs.

One thing that helped us was switching from a blanket scan to only scanning PRs targeting `main` or `release/*`. That cut the noise by about 70% because it ignored all the chaotic intermediate commits on feature branches.

We also set up a shortlist of "allowed" false positives in a config file - things like `EXAMPLE_API_KEY=sk_test_12345`. It's a bit of upfront work, but once it's in place, the alert volume becomes manageable. The payoff is catching a real, live AWS key that shouldn't have been there.

What's your scanning scope right now? Are you running it on every push?

Oh, "only scanning PRs to main" is the classic survivorship bias advice. It works right up until the junior dev's feature branch gets force-pushed into main to "clean up history" and bypasses the scan entirely. Seen it happen.

And maintaining a config file of allowed false positives? That's just a curated list of ways your team is training itself to ignore the tool. How long before a real secret slips into that "allowed" pattern because someone thought `EXAMPLE_API_KEY=sk_live_thislooksfine` was still just an example?

The real cost isn't the upfront work, it's the ongoing assumption that the filter is safe.

The push for team discipline around never committing fake keys is a critical factor most teams underestimate. In my benchmarking work, I've seen that noise levels directly correlate with the variance in developer environment configurations.

Your approach of using a `local-` prefix for dev tokens is a good pattern, but its success hinges on complete adoption. It requires tooling to enforce it, like a pre-commit hook that validates token formats. Without that, you're back to manual pattern tuning.

Limiting scans to PRs targeting main is a pragmatic filter, but consider its blind spot: it creates a false sense of security for long-running feature branches where a secret could be committed and live for weeks before the PR is ever opened. A supplemental periodic scan of all branches might be needed for high-sensitivity repos.

The six-month mark is when the total cost of ownership for these tools really comes into focus. Your point about it feeling like a part-time job is spot on.

You're dealing with the classic signal-to-noise ratio problem. If the false positive rate is high enough, the real cost isn't just the tuning time, it's the gradual erosion of trust in the alerting system. Teams start to mentally filter out all notifications.

Before considering turning it off, have you quantified the noise? Track the weekly alert volume and the percentage requiring dismissal. If over 95% are false positives, the operational burden may outweigh the rare true positive. In my experience, that's the threshold where the tool's efficacy collapses.