I've been trying to roll out semgrep-agent across our projects, and it's been a solid tool for most of our services. However, it consistently fails in our GitLab CI pipelines for two specific monolithic repositories. These are legacy beasts, each over 5GB with hundreds of thousands of files. The job just times out or gets killed.
The error isn't consistent. Sometimes it's a `SCAN_TIMEOUT`, other times the runner runs out of memory, and occasionally it just hangs until the job is canceled. I've tried throwing bigger runners at the problem (16GB RAM, 8 vCPUs), but it only delays the inevitable. The smaller repos scan in under 5 minutes.
Here's the core of my `.gitlab-ci.yml` configuration:
```yaml
semgrep:
image: returntocorp/semgrep-agent
variables:
SEMGREP_APP_TOKEN: $SEMGREP_APP_TOKEN
script:
- /home/semgrep/agent/semgrep-agent
```
I've tried adding some obvious flags with `SEMGREP_AGENT_EXTRA_ARGS`, like `--no-git-ignore` and `--timeout 0`, but no luck.
Has anyone successfully got this running against a truly massive codebase? My specific questions are:
* Is there a way to properly exclude huge directories (like `vendor/`, `node_modules/`, built binaries) that the agent seems to be trying to scan? The `.semgrepignore` doesn't seem to be respected early enough in the process.
* Are there any hidden tuning knobs for memory or file-descriptor limits inside the container?
* Should I be pre-filtering the diff or scanning only changed files in a different way for these giants?
At this point, I'm considering writing a wrapper script that first does a `find` to prune the tree before the agent runs, but that feels like a hack. Looking for proven strategies.
Build once, deploy everywhere
Oh wow, 5GB repos - I feel your pain. Been there with some ancient codebases.
You're on the right track with excluding directories, but the semgrep-agent itself can be a blunt instrument on monoliths. The trick isn't just the excludes, it's preventing the agent from even trying to index those paths in the first place. Have you tried setting up a `.semgrepignore` file at the root of your repo? It works like a `.gitignore` and the agent should respect it. You can toss in patterns for `vendor/**`, `node_modules/**`, `*.min.js`, `*.jar` - anything you know is pure noise.
Also, could you split the scan? Maybe run it only on diffed files (`SEMGREP_AGENT_EXTRA_ARGS="--gitlab-sast"` might help, though I think that's for MRs) or target a specific language per job? Scanning everything at once is what's murdering your memory.
One last hail mary - did you check the runner's disk space? Sometimes it's not just RAM, but the disk IO and temp files that cause the hang.