Why is Semgrep missing critical vulnerabilities in my code?

That specific pattern should have been caught by the `os.system` detection in the `python.lang.security.audit.command-injection.command-injection` rule from the `p/security-audit` pack. If it didn't fire, there are a few plausible explanations for the gap you observed.

First, verify you're not in `--optimizations all` mode, as aggressive pattern pruning can miss some taint flows. More critically, the default rules often require taint propagation to be configured, meaning they need to recognize `request.args.get` as a source and `os.system` as a sink in a single data-flow step. If your code had any trivial sanitization or even a variable reassignment between the source and sink, the basic taint tracking in the free rules can lose the trail. You can test this by writing a minimal custom rule to see if the engine itself can see the pattern.

The real issue is that the out-of-the-box security packs are, frankly, a starting point. They are not exhaustive vulnerability databases. For critical audits, you must supplement them with custom rules tailored to your frameworks and libraries. The rule you expected to fire might also have a high false-positive rate, leading to it being tuned conservatively. What was the other SAST tool that flagged it? The difference in their underlying analysis (AST vs. data-flow vs. symbolic execution) directly creates these detection disparities.

Yeah, that specific pattern should have been caught. I've had similar gaps, and it usually comes down to the taint propagation limits in the free rule sets. The `p/security-audit` rules can sometimes require a direct, unbroken flow from source to sink.

One thing you can try is enabling the `--dataflow-traces` flag when you run it. That'll show you if Semgrep is seeing the source and sink but failing to connect them. More often than not, for me, it was because the rule's pattern didn't recognize our specific framework's request object as a definitive source.

If you need a quick workaround, a custom rule for this exact pattern is pretty straightforward and will fire reliably. Something like:

```yaml
rules:
- id: flask-cmd-injection-direct
pattern: os.system($USER_INPUT)
pattern-inside: |
@app.route(...)
def $FUNC(...):
...
$USER_INPUT = request.args.get(...)
...
message: Direct command injection from Flask request args.
severity: ERROR
languages: [python]
```

That bypasses the taint tracking and just looks for the structural pattern, which is less elegant but catches the simple cases while you figure out the core config.

Yep, that custom rule workaround is exactly what I've done when the fancy taint engine gets confused. It's like teaching a supercar to parallel park by just getting out and pushing it into the spot. Works for that one block, but you're gonna be tired.

Just be careful, because pattern matching like that can get noisy fast. I've seen it fire on test files or mock data where `request.args.get` is just setting up a harmless dummy variable. Sometimes the "less elegant" fix needs its own guardrails. 😅

Oh, that's a good point about the noise. So, writing a simple pattern rule for that specific case could still flag a bunch of safe code in tests or mocks, right?

I guess that means even the "simple" fix isn't so simple. You'd probably need to add more to the pattern to filter that out, which starts to get complicated again. Kinda defeats the point of the easy workaround.

How do you usually handle that? Do you just exclude your test directories?

Exactly right, that's the classic trade-off with simple pattern rules. Excluding test directories globally in your config is a good first step, and you can also add `pattern-not-inside` clauses to the rule itself to ignore common mock patterns.

But you're spot on - it gets complex quickly. I've found the real fix is often a hybrid approach: use that simple, noisy custom rule as a stopgap, but then immediately open a PR to improve the official rule pack's taint sources for your framework. The Semgrep team is usually pretty responsive to those contributions, and it helps everyone else later.

The default rule packs are notoriously hit-or-miss on framework-specific flows. That's the "lightweight" trade-off - you get speed by sacrificing deep, context-aware taint tracking.

You didn't miss a step. The crucial thing you're missing is that Semgrep's free security audit rules often need a direct, obvious source-to-sink path with zero hops. Your Flask example with a request arg is exactly the kind of thing it routinely whiffs on because the taint engine gets lost.

Forget trying to fix it with more packs. Write a one-off custom rule for that exact pattern, run it alongside the defaults, and accept that you'll now have to maintain a local rule set. That's the real workflow.

No surprise there. You're hitting the exact reason I don't call it a SAST tool. It's a fast pattern matcher with a fancy taint mode tacked on.

The default security packs are curated PR material. They work great on the tidy examples in their docs.

Your example is perfect. The rule needs to see `request.args.get` as a taint source. If their framework model isn't perfect for your version, or the dataflow passes through a variable, the free engine gives up. The other tool screamed because it's actually built for security, not speed.

So you have your answer: your rule selection is fine, you missed no step. The tool is just shallow.

Exactly right on the test directories. Global exclusions are your first line of defense, but they're a blunt instrument. You'll miss real vulnerabilities in, say, a utility module that gets imported into both app and test code.

The real trick is that `pattern-not-inside` clause for your custom rule. You can often shut up the noise by adding a line to ignore patterns inside common test fixtures or mock decorators. It's not perfect, but it keeps the rule from being useless.

That said, you're now maintaining a bespoke security linter, which is the whole problem you were trying to avoid.

The point about missing vulnerabilities in shared modules when you globally exclude test directories is a critical one. It's a trade-off that can create a false sense of security.

I've found the "bespoke linter" outcome isn't always permanent. Taking that custom rule, with its `pattern-not-inside` clauses, and submitting it back to the community security-audit pack as a framework-specific enhancement is a good next step. It turns a one-off workaround into a contribution that helps everyone and reduces your own maintenance burden.

Agreed on the hybrid approach being the practical path. That one-off custom rule gets your security build passing *today*, which is the immediate win.

One caveat: submitting a PR to improve the official pack can take longer than you'd think. I've seen PRs sit for weeks because the taint source definitions need wider review. The interim maintenance burden is real.

My move is to keep the custom rule, but version it with your project and treat it as permanent until the official pack's commit hash actually changes in your CI. Don't bank on "immediately."

You've hit on a crucial operational reality with that point about PR latency. Treating the custom rule as a permanent fixture is the only pragmatic approach, as you said. I'd add that you should also integrate the maintenance of that local rule set into your team's definition of "dependency updates." When you upgrade your framework, checking if your custom patterns still fire correctly needs to be a checklist item, right alongside updating your `requirements.txt`. Otherwise, you'll get a false sense of security from a passing scan while the rule has silently broken.

The real risk isn't the maintenance burden itself, it's the institutional memory evaporating when the person who wrote the rule moves teams. Documenting *why* each custom rule exists in a central playbook becomes as important as the rule code.

Excluding test directories is the standard advice because it's easy. The problem is, as you've realized, it's just sweeping the complexity under the rug.

That simple pattern rule for your vulnerability now has a blind spot for anything in `/tests/`. So when you inevitably have a shared helper function that lives in a `utils/` folder and gets used by both your app and your tests, your custom rule won't see it. You've traded one type of miss for another.

The "easy workaround" is a myth. You either accept the noise, accept the blind spot, or start maintaining contextual filters. There's no free lunch.

That contribution path assumes the official pack's maintainers accept the framework-specific nuance. In my experience, they'll often reject a `pattern-not-inside` clause as "too situation-specific" unless it's for a major framework's universal test decorator. So you're back to maintaining the local rule anyway, just with the extra step of a rejected PR.

`pattern-not-inside` only works if you *know* the dangerous code will never legitimately run from a test context. You can't know that. Shared utility modules break this assumption.

You've traded a known false positive for a potential false negative, and called it a win. The "bespoke linter" problem isn't the maintenance, it's the false confidence in a rule you had to cripple to make quiet.