I've been integrating Semgrep into our CI/CD pipeline for a few months now, mostly using the default output. It's great for human review, but I'm hitting a wall with automation. I need to parse the results programmatically to fail builds on specific high-severity findings, log them to our internal dashboard, and maybe even auto-create tickets.
The `--json` flag seems like the obvious answer, but the output structure is... substantial. I ran a test scan and got a massive JSON blob. I'm trying to write a lightweight Python script to extract just the critical bits—rule ID, severity, file path, and line number—but navigating the nested structure feels a bit cumbersome.
Has anyone built a robust parser for this? I'm debating between:
* Writing a custom Python module that digs into `results` > `extra` > `metadata`.
* Using `jq` in a bash script before passing data to my service.
* Maybe there's a cleaner way using `--output` with a custom format I haven't discovered?
What's your preferred approach for consuming Semgrep data in an automated workflow? Here's a snippet of what I'm starting with:
```python
import json
import subprocess
# Run semgrep
cmd = ["semgrep", "scan", "--config", "auto", "--json"]
result = subprocess.run(cmd, capture_output=True, text=True)
data = json.loads(result.stdout)
for finding in data.get("results", []):
rule_id = finding.get("check_id")
severity = finding.get("extra", {}).get("metadata", {}).get("severity")
# ... more parsing
```
It works, but I'm wondering if I'm missing a more elegant solution, or if there are common pitfalls (like handling the different `severity` string formats) I should watch for.
--builder
Latency is the enemy, but consistency is the goal.
Yeah, the JSON output is dense. I use jq for initial parsing in the pipeline because it's quick to prototype.
Something like:
`semgrep scan --json --config auto . | jq -c '.results[] | {check_id, severity, path, start_line}' > filtered.json`
That gets you a clean stream of objects. Then your Python script can just read that filtered file. It keeps the heavy lifting out of your main code.
I've also seen people use the `--output` flag with a custom format string, but I find the JSON+jq combo more reliable for branching logic, like failing builds only on "ERROR" severity.
Still looking for the perfect one
Yeah, parsing that raw JSON directly in Python can be a slog. I went the custom module route initially, but honestly, `jq` as a pre-filter is way more efficient for pipelines. Lets you keep the Python script simple.
One caveat: if you need data from deep in `extra.metadata`, like CVE IDs, the jq query gets messy fast. For that, I have a small helper function that loads the full JSON and extracts just those nested fields. You could combine both approaches - use jq for the 90% of common fields, and Python for the weird metadata.
Here's a snippet of my fallback parser for the deep stuff:
```python
def get_metadata_findings(raw_json_path):
with open(raw_json_path) as f:
data = json.load(f)
for result in data.get('results', []):
meta = result.get('extra', {}).get('metadata', {})
cve = meta.get('cve')
# ... do something with it
```
Keeps the main flow clean.
Data is the new oil - but it's usually crude.
Yeah, `jq` for the win. I pipe it to `jq -C` in my local sandbox to get colored output for quick reviews before I commit to the pipeline filter.
Your two-stage approach is smart. I hit the same snag with metadata. My jq queries for anything past `extra.metadata` looked like a cat walked on the keyboard. The performance hit of loading the full JSON in a helper is negligible for CI, and it keeps the logic readable.
One thing to watch: the schema can shift between major Semgrep versions. I had a script break because `severity` moved from a string to an enum in the JSON. Now I run a schema check in a dev container first. 😅
My sandbox is bigger than yours.