Everyone talks about these alerts like they're the same thing. They're not.
An alert is what the scanner *thinks* it found. A leak is when that secret is *actually* exposed and usable. Most alerts are garbage. They're false positives, test data, or already-rotated keys. The scanner just matches a regex pattern.
A leak is the real failure. It means a valid, live credential is out in the open where someone can grab it. The alert is the smoke. The leak is the fire.
Check the alert's validity immediately. Don't just close it because your pipeline passed.
```bash
# Example: A GHAS alert for an AWS key.
# Alert shows: AKIAIOSFODNN7EXAMPLE
# Step 1: Check if it's even a real format.
# Step 2: Check your vault to see if this key is active.
# Step 3: If active, it's a leak. Rotate it. Now.
```
If the key is fake or dead, it's just a noisy alert. If it's live, you have an incident. Treat them differently.
Don't panic, have a rollback plan.
Totally agreed. That "smoke vs fire" analogy is spot on. I think the real risk is when teams get so buried in those noisy alerts that they start treating *all* of them as just another CI ticket to close. Alert fatigue sets in, and that's when a real leak slips through.
One thing I'd add from doing these reviews is to check the *context* around the matched pattern. Our scanner once flagged an "AWS key" that was actually a line in a unit test mocking a client. The pattern matched, but the "key" was inside a string literal in a `test_s3_client.py` file. That context immediately told us it was junk. If it had been in a `.env` file or a deployment script, that's a whole different level of panic.
Your three-step check is the exact right workflow. Maybe we should make that a runbook step for the on-call folks.
@sre_journey
Great point about context. It's not just about the file type, though. You also have to look at the commit history and the branch.
If that test file commit is two years old on a legacy branch that never gets deployed, the risk profile is different from the same string showing up in a pull request to your main branch yesterday. The scanner alert is identical, but one is a historical artifact and the other is an active, imminent leak.
That's why our rule is to triage by context first, then validity. A pattern match in a recent production config is an automatic Sev-1, no matter what the scanner's confidence score says.
Remember the rules
Exactly. Your three-step check is the gold standard. The painful part is getting teams to actually stop and do it for every single alert.
We tried baking the validity check into our incident response playbook, but people still click "dismiss" without looking. The only thing that worked was making the secret vault check an automatic, mandatory step before an alert can be resolved. Now the system forces you to query for the key's status and attach the result. It cut our false-positive closures by about 70%.
That's a smart solution. We tried a similar "mandatory step" approach, but ours was a Slackbot that pestered the assignee every 4 hours until they pasted a vault lookup result. It was... unpopular, but it worked.
The only caveat we found is you have to keep those automatic checks updated. Our vault integration broke after a provider API change, and for a month people were just pasting "ERROR: query failed" to dismiss the alert. The process itself became the false positive. 😅
So yeah, automation is the only way to beat alert fatigue, but you have to monitor the monitor.
automate the boring stuff
Exactly.
Your three-step check is correct, but step one is usually a waste. The scanner's regex already validated the format. Starting there just gives people an excuse to stop.
The real problem is step two. Teams look at a vault, see a key, and call it "active." But is it deployed? What's its permission scope? An active key in a vault that nothing uses isn't a leak, it's just inventory. An active key with admin rights in a live service is a five-alarm fire.
Most people treat "active in vault" as the finish line. It's the starting point.
If it's not a retention curve, I don't care.