Skip to content
Notifications
Clear all

Why is Mend so slow in CI for large monorepos? Any fixes?

2 Posts
2 Users
0 Reactions
2 Views
(@alexg)
Reputable Member
Joined: 1 week ago
Posts: 154
Topic starter   [#10168]

I've been conducting a performance analysis of SCA tooling in CI/CD pipelines for the last quarter, specifically focusing on polyglot monorepos with 1000+ dependencies. Mend (formerly WhiteSource) consistently emerges as a significant bottleneck. A scan that takes `trivy` or `grype` 90 seconds can stretch to 25+ minutes with Mend's `ws` scanner in a similarly configured job. This isn't just anecdotal; the data from our distributed tracing is unequivocal.

The primary culprits for this latency in a large monorepo context appear to be architectural:

* **Synchronous, Sequential Analysis:** Mend's scanner often processes languages and manifests sequentially rather than leveraging parallel execution pipelines. In a monorepo with 15 microservices across Go, Java, Python, and JS, this linear approach compounds wait time.
* **Agent-Controller Overhead:** The `ws` CLI agent performs local analysis, but there is a non-trivial synchronous handshake with the Mend backend for policy checks, vulnerability correlation, and license compliance. Network latency and backend queueing during peak times directly inflate CI job duration.
* **Inefficient Delta Scanning:** While Mend supports incremental scans, its mechanism for determining the delta in a monorepo—especially when a shared library or root `package.json` is updated—can be inefficient. It frequently falls back to re-evaluating large portions of the dependency tree.
* **Persistent Cache Invalidation:** The local cache seems to be invalidated more aggressively than necessary. A change in one service module can trigger a re-download and re-scan of cached dependency data for unrelated modules, suggesting poor granularity in its cache tagging.

We've implemented several mitigations with mixed results:

1. **Aggressive Caching of the Mend Directory:** We now persist the `.mend` and `~/.ws` directories using CI cache mechanisms (e.g., GitHub Actions cache, GitLab CI cache). This is crucial.
```yaml
# Example GitHub Actions step
- name: Cache Mend artifacts
uses: actions/cache@v3
with:
path: |
~/.ws
.mend
**/node_modules/.cache/ws
key: ${{ runner.os }}-mend-${{ hashFiles('**/package-lock.json', '**/pom.xml', '**/go.mod') }}
```
2. **Splitting the Scan Job:** We abandoned a single monolithic scan job. Instead, we trigger language or service-specific scans in parallel jobs based on changed paths. This requires a custom orchestration layer but reduced our worst-case scan time from ~28 minutes to ~8 minutes.
3. **Tuning Scan Parameters:** Using `-c . -r .` to force a recursive scan from the monorepo root proved faster than letting the agent auto-detect, which seemed to add filesystem traversal overhead. We also explicitly disable agents for languages not present (`--no-golang`, `--no-pip`).

My question to the community is whether you've dissected similar performance issues and what your solutions were. Specifically:

* Have you found a way to genuinely enable a parallel, distributed scan across a monorepo's subprojects?
* Are there configuration flags or environment variables (e.g., `WS_PARALLEL_SCANS`) that are not well-documented but improve throughput?
* Has anyone moved to a model where Mend scans are performed asynchronously outside the critical CI path, and if so, how do you enforce policy gates (e.g., block merges on critical vulnerabilities)?

The vendor's standard response is to "increase resources" or "use the SaaS accelerator," but neither addresses the fundamental sequential analysis pattern. I'm looking for engineering deep-dives, not sales pitches.

-- alex



   
Quote
(@infra_architect_rebel)
Estimable Member
Joined: 3 months ago
Posts: 122
 

You nailed the core issue. The agent-backend chatter kills it.

I've seen teams hack around it by running scans async in a nightly job, then using the API to check PRs against those cached results. Cuts CI time to under a minute.

But that's a workaround for a tool not built for monorepo scale. You're better off with a scanner designed for local speed first.


Simplicity is the ultimate sophistication


   
ReplyQuote