What's the best way to handle mono-repos without creating 100 separate projects?

backend_perf_guru · 2026-08-21T21:21:16Z

Having recently migrated our primary codebase to a monorepo structure (approximately 1.2M lines across 45 distinct services and libraries), I've been conducting a deep-dive evaluation of Black Duck's scanning efficiency and project management overhead in such an environment. The canonical, but naive, approach of creating a 1:1 mapping between a Black Duck "project" and every discrete component (e.g., each `package.json`, `Cargo.toml`, `go.mod`) quickly becomes untenable. The administrative burden and UI noise are secondary concerns; the primary performance hit comes from the duplicated scanning of shared dependencies and the orchestration overhead of 100+ concurrent scans. My hypothesis, which I've been testing across several iterations, is that the optimal configuration balances scanning granularity with logical grouping to minimize total scan time and maximize actionable result grouping. Here are the strategies I've A/B tested: * **Single Monolithic Project Scan:** Point Black Duck at the repository root. This performs a single, deep scan. * **Result:** Unacceptably long scan duration (~2.1 hours for our codebase). The resulting "Bill of Materials" is a colossal, undifferentiated list, making it impossible to triage vulnerabilities per deployable unit. Policy violations become meaningless as they cannot be mapped to specific service owners. **Latency and actionability are poor.** * **Per-Service/Per-Library Project (The Naive Approach):** Automated project creation via API based on build manifests. * **Result:** High initialization overhead. While subsequent scans can be parallelized, we observed significant resource contention on the scan engine (we're self-hosted). More critically, common dependencies (e.g., `lodash`, `react`, `aws-sdk`) are scanned *repeatedly* for each project, a pure waste of CPU cycles. Total aggregate scan time was ~45 minutes, but with high system load. * **Hybrid, Logic-Driven Grouping:** This is where I've found the most promising latency/profile results. We group components based on: 1. **Shared Dependency Profile:** Microservices with identical or near-identical dependency lists (e.g., a suite of Node.js services all using the same framework version) are scanned as a single project. 2. **Deployment Pipeline:** All artifacts built by a single CI/CD pipeline are grouped. 3. **Criticality Tier:** Isolate high-criticality services (public-facing APIs) into their own projects for finer-grained policy control, while grouping lower-risk internal tools. Implementation requires a pre-scan parsing step. We use a simple script to analyze manifests and generate a scan plan: ```bash # Pseudo-logic for grouping (simplified) # 1. Hash the dependency list for each component # 2. Group components by hash # 3. For each unique hash, create one Black Duck project for group in $(find ./services -name "package.json" -exec jq -c '.dependencies' {} ; | sort | uniq -d); do # Create Black Duck project via API # Scan all service directories containing this exact dependencies hash done ``` **Result:** Reduced our total scan footprint from 45 to 12 logical projects. Aggregate scan time dropped to ~18 minutes due to eliminated redundancy. Triage is simplified as each project maps to a logical dependency set and team. The remaining challenge is managing the lifecycle of these grouped projects as dependencies drift apart over time. We've implemented a monthly reconciliation job that re-computes groupings and merges or splits Black Duck projects via their API. I'm keen to hear from others who have tackled this at scale. What grouping heuristics have you found effective? Have you measured the overhead of project creation/deletion via API versus the scanning savings? Is there a point where the complexity of a custom pre-scan orchestrator outweighs the Black Duck performance gains? --perf

Page 2 / 2 Prev

Black Duck Reviews

Last Post by emmaw 1 month ago

16 Posts

16 Users

0 Reactions

24 Views

RSS

emmaw

(@emmaw)

Estimable Member

Joined: 3 months ago

Posts: 139

22/08/2026 1:23 pm

That's a clever use of policy rules to keep alerts clean. But doesn't that create a sort of "single point of failure" for security updates? If the library team is slow to patch, every downstream service is just carrying that vulnerability silently in their SBOM, right?

ReplyQuote

Page 2 / 2 Prev

Forum Icons: Forum contains no unread posts Forum contains unread posts

Topic Icons: Not Replied Replied Active Hot Sticky Unapproved Solved Private Closed