Skip to content
Notifications
Clear all

Hot take: Mend's agent is bloated for our lean CI.

3 Posts
3 Users
0 Reactions
1 Views
(@alexm)
Reputable Member
Joined: 2 weeks ago
Posts: 160
Topic starter   [#21838]

After six months of integrating Mend (formerly WhiteSource) into our CI pipeline for a suite of microservices, I've reached a conclusion that contradicts much of the marketing: the Mend Unified Agent is a resource hog that fundamentally conflicts with a lean, container-native CI/CD philosophy. While the vulnerability database is comprehensive, the execution overhead is untenable for high-velocity development.

Our primary issue is with the agent's startup and scan duration. In a typical ephemeral CI runner environment (GitLab CI, Kubernetes pods), the time and resource tax per pipeline job is significant. Consider this representative sample from our `.gitlab-ci.yml`:

```yaml
sbom_scan:
stage: test
image: docker.io/mend/ua:latest
script:
- /usr/local/bin/ws_scanner.sh
variables:
WS_APIKEY: $WS_APIKEY
WS_WSS_URL: $WS_WSS_URL
WS_PROJECTNAME: $CI_PROJECT_NAME
artifacts:
reports:
cyclonedx: gl-sbom-report.json
```

The `docker.io/mend/ua:latest` image is over 1.2GB. Pulling this on a fresh runner adds 60-90 seconds before any code even executes. The scan itself then consumes substantial CPU and memory:

* Peak memory usage: 1.8-2.4 GB per scan (for a ~500 file Java/Node.js project)
* Average scan time: 4-7 minutes
* Agent process spawns 20+ threads

This directly impacts our developer experience and infrastructure costs:
* **Pipeline Bloat:** A merge request that previously ran unit tests in 3 minutes now takes 10+ minutes, creating a feedback loop bottleneck.
* **Runner Resource Contention:** We must configure larger, more expensive CI runner instances to accommodate the Mend agent's memory footprint, starving other concurrent jobs.
* **Cost Amplification:** In cloud-based CI systems (GitHub Actions, CircleCI), where compute time is monetized, these extra minutes compound across hundreds of daily builds.

We attempted mitigations:
* **Caching the Agent Image:** This saved pull time but not the in-scan resource consumption.
* **Using the "Quick Scan" Option:** This reduced time marginally but at the cost of scan depth, which defeats the purpose.
* **Scheduled vs. PR Scans:** This decouples security from the commit, increasing risk exposure.

The architectural critique is that the agent is monolithic. It bundles its own JVM, dependency resolvers for 20+ languages, and a full local database copy. In an era where SCA tools like Trivy or Grype leverage lightweight, single-binary designs and can run as a simple step without a heavyweight container, Mend's approach feels anachronistic.

For organizations operating at scale with a focus on CI efficiency, the Mend agent's bloat is a critical flaw. The value of the findings is undermined by the friction of obtaining them. I'm interested to hear if other teams have quantified this overhead or found successful patterns to tame it without resorting to a separate, asynchronous scanning workflow that breaks the shift-left paradigm.



   
Quote
(@devops_dad)
Estimable Member
Joined: 5 months ago
Posts: 144
 

Yeah, that image size is a real killer for ephemeral runners. I hit the same wall a while back. Pulling a gig-plus image just to start a security scan feels like it defeats the purpose of a fast CI loop.

We ended up scripting around it by using a much smaller, custom image with just the Mend CLI jar and its dependencies. It's a bit more maintenance, but it cut our pull times down to seconds. The memory footprint is still rough, though - sounds like you're seeing the same 2GB spikes we did. Makes you miss the days when a scanner was a simple binary.

Have you looked at running it as a sidecar or daemonset in your k8s cluster instead of per-pipeline? It doesn't solve the bloat, but at least you're not paying the pull tax on every single job.


it worked on my machine


   
ReplyQuote
(@garethp)
Trusted Member
Joined: 2 weeks ago
Posts: 47
 

The sidecar or daemonset approach is a valid workaround for the image pull tax, but it introduces a different set of architectural trade-offs. You're now managing a persistent service with its own lifecycle, which can complicate scaling and version synchronization with your CI runners. The memory overhead becomes a constant baseline cost on your nodes, rather than a spike contained to the pipeline.

I've seen teams try the custom smaller image route as well. The maintenance burden you mention is non-trivial, especially when Mend updates their agent or its dependency tree. You often end up replicating a significant portion of the bloat anyway, because the Java runtime and the comprehensive vulnerability database are the core resource drivers. You might shave off 700MB from the image, but you're still left with the 2GB heap during scan execution.

A more effective pattern, in my experience, is to shift the scanning to a later stage in the pipeline - treating it as a gating mechanism for merge requests rather than a per-commit job. This doesn't reduce the agent's footprint, but it drastically cuts the number of times you incur the cost, preserving the lean CI loop for the majority of development commits.


Plan the exit before entry.


   
ReplyQuote