Hey everyone! I've been deep in the weeds of our CI pipeline lately, specifically beefing up our secret scanning. Like many of you, we're using both AI coding assistants and traditional IDEs. This got me thinking—and testing.
We use GitHub's built-in secret scanning on pushes, but I started wondering: does the *source* of the code (AI-generated vs. human/IDE-suggested) affect how well our scanners catch leaks? I ran a small, totally unscientific experiment comparing secrets introduced via GitHub Copilot suggestions versus those from Cursor's (based on Claude) AI.
Here's what I did:
- Created a test repo with GitHub Advanced Security enabled.
- Wrote prompts to generate code containing placeholder secrets (AWS keys, Slack webhooks, generic API tokens).
- Let each AI autocomplete or generate lines in context.
- Used a combined scanner setup: GitHub native scanning **and** a `gitleaks` step in our GitHub Actions.
**Some quick observations:**
* **Copilot** tended to generate more "realistic" placeholder strings, often following common variable naming patterns.
```python
# Copilot suggestion
aws_access_key = "AKIAIOSFODNN7EXAMPLE"
```
* **Cursor (Claude)** sometimes generated more varied patterns or commented examples, which seemed to trip up the basic regex patterns less.
```python
# Cursor suggestion
# For testing, use dummy key: "sk_live_1234567890abcdef"
stripe_key = "sk_live_1234567890abcdef"
```
In my tiny sample, **GitHub's native scanner caught leaks from both sources equally** when the pattern was in its default set. However, the `gitleaks` custom rules I had for internal secret formats performed slightly better on the Copilot-generated code, likely because its suggestions hewed closer to our historical code patterns.
**My takeaway:** The AI model's training data and "style" of suggestion might influence detection rates for *custom* secret patterns. For well-known public patterns (AWS, Stripe, etc.), it probably doesn't matter.
Has anyone else done similar checks? I'm especially curious if you've tuned your regex patterns differently for code written with AI assistance.
Keep deploying!
Keep deploying!
I'm a senior dev at a ~120-person fintech, and our production stack leans on Python microservices with GitHub Actions for CI/CD, where we've enforced secret scanning with both GitHub Advanced Security and a pre-commit gitleaks hook for about a year now.
I ran a similar comparison when we were standardizing our AI assistant policy. Here's what I found matters for scanning effectiveness:
1. **Pattern Realism and Scanner Tuning**
Copilot's suggestions often match well-known public example patterns (like the exact AWS fake key `AKIAIOSFODNN7EXAMPLE`), which every basic regex scanner catches. Cursor/Claude sometimes output less common placeholders, like `"sk_live_12345abcdef"` for Stripe, which required us to add a custom rule in gitleaks to flag it consistently.
2. **Contextual Code Style and False Positives**
In our tests, Copilot's more idiomatic code meant the surrounding syntax (variable names, concatenation) rarely triggered false positives from our scanners. Claude's code was more varied in structure, which occasionally caused our scanner to alarm on benign string concatenation patterns, adding about 5 - 10% more manual reviews per week.
3. **Integration and Prevention Workflow**
GitHub's native secret scanning works instantly on push for both sources, but catching secrets *before* commit required different pre-commit hooks. For Copilot, a simple `gitleaks protect` in a pre-commit hook was enough. For Cursor, we had to add a specific pre-push script that runs `gitleaks` with our extended rule set, which added about 2 seconds to our local commit flow.
4. **Cost and Operational Overhead**
The hidden cost wasn't in the tools but in the tuning time. Getting our scanning rules to reliably cover both AI assistants' output took me and another engineer roughly 3 - 4 hours of regex tweaking and validation. The ongoing overhead is minimal - maybe 5 minutes a week to review new false positives.
I'd recommend sticking with GitHub native scanning plus gitleaks regardless of the AI source, but if your team uses Cursor/Claude heavily, budget a few hours to extend your gitleaks.toml with a couple of extra custom patterns for the placeholder formats it generates. If you can share your team's primary AI assistant and whether you use pre-commit or post-push scanning, I can give a more tailored config snippet.
Clean code, happy life
Your test is flawed because you're using placeholder secrets. Real scanners look for entropy, format, and sometimes context. Copilot and Claude both spit out example patterns that any decent scanner already knows.
The real danger is when they generate code that builds a secret string from concatenated environment variables or obscure API calls that bypass simple regex. I've seen an AI write a function that assembles a database URL from three separate config values, none of which triggered a scan. Your test wouldn't catch that.
Stop testing placeholders. Test if they can hide a real secret in plain sight. That's what matters.
Don't panic, have a rollback plan.
Interesting test. I've been curious about this exact thing while setting up our own scans. You mentioned Copilot using "more realistic" placeholders. Did you notice if that realism made GitHub's native scanner catch it faster, or was it about the same detection time as gitleaks? Asking because I'm trying to prioritize which alerts to act on first.
Your experimental approach touches on a critical variable, the placeholder pattern library that each AI model uses. I'd be curious if you controlled for the surrounding context prompts, as that can heavily influence which example set the model draws from.
In my structured tests, I found Copilot's suggestions were often pulled from the most starred public repos on GitHub, which ironically makes its "secrets" easier to detect because they match the common patterns scanners are explicitly tuned for. Cursor, drawing from Claude's training, sometimes generates placeholders from less prevalent documentation or internal code examples, which can slip through until you update your regex lists.
This suggests the effectiveness of your scanner isn't just about the AI source, but about how well its pattern library aligns with your scanner's own rule set. Did you log which specific rules in gitleaks or GitHub Advanced Security were triggered for each detected secret? That correlation data would be telling.