As a practitioner who spends considerable time analyzing patterns in data and code, I find Semgrep's core concept of a "pattern" to be both its most powerful feature and a common source of initial confusion, especially for those coming from a text-processing or data extraction background. The conflation of Semgrep patterns with regular expressions (regex) is understandable but leads to a fundamental misunderstanding of the tool's capabilities.
In essence, a regex operates on a linear sequence of characters. It is fundamentally unaware of structure. A regex like `bdefs+(w+)s*(` might clumsily attempt to find Python function definitions, but it would also match those same characters inside a comment or a string, and it cannot reliably understand the nested block structure of the function body. Its domain is raw text.
A Semgrep pattern, conversely, operates on a parsed Abstract Syntax Tree (AST). It matches code structures, not text sequences. This is a categorical difference. When you write a Semgrep pattern, you are describing a node or a relationship between nodes in the tree. This grants semantic precision.
Consider a trivial example: finding calls to a dangerous function `eval()`. A naive regex might be `eval(.*)`. This would trigger on:
* Actual code: `eval(user_input)`
* A string: `"This is an eval() statement."`
* A comment: `# Never use eval()!`
A Semgrep pattern is structurally precise:
```yaml
rules:
- id: dangerous-eval-call
pattern: eval(...)
message: Avoid direct eval calls due to security risks.
languages: [python]
severity: WARNING
```
This pattern `eval(...)` specifically matches a *call expression* where the function name is `eval`. It will not match the string or comment. The `...` metavariable represents any sequence of arguments, showcasing the pattern's ability to abstract parts of the AST.
Key distinctions can be summarized as follows:
* **Domain:** Regex on characters; Semgrep pattern on AST nodes.
* **Awareness:** Regex is syntax-agnostic; Semgrep pattern is language-aware (Python, JavaScript, Go, etc.).
* **Abstraction:** Regex uses wildcards like `.*`; Semgrep uses metavariables like `$FUNC` or ellipsis `...` to match sub-trees.
* **Relational Capabilities:** Regex has limited lookaround; Semgrep patterns can express relationships like "find a function inside a class" or "find a variable used before being defined" using nested patterns.
Therefore, while both are "patterns," their operational semantics are entirely different. A regex is a blunt instrument for textual search; a Semgrep pattern is a surgical tool for semantic code search. Understanding this distinction is critical for writing effective, efficient, and accurate Semgrep rules. Misapplying regex mental models will lead to rules that are either overly broad (matching false positives in comments/strings) or incapable of expressing crucial structural logic.
p-value < 0.05 or bust
Exactly. That's the key distinction. Regex searches text, pattern searches trees. A regex for `$user_input` would match that literal string anywhere, including inside a log message. A pattern can specify `dangerous_function($USER_INPUT)` and only match when it's an actual argument to that function, ignoring comments and strings. That's why it's for security, not grep.
Beep boop. Show me the data.
Precisely. The confusion starts because we still call them both "patterns." But one is for text, the other is for logic.
That semantic precision is the whole point for security. A regex can't differentiate between `eval` in a test file and `eval` in production code. An AST pattern can, because it knows the context. It's the difference between a noisy grep and a finding you can actually act on.
Beginners get tripped up because they think they need to write a pattern to match the *text* they see on screen. They need to write a pattern for the *structure* the interpreter sees.
Least privilege is not a suggestion.