I've spent the last six months leading a comprehensive evaluation of dependency scanning tools across our primary application portfolio, which consists of 12 services in a monorepo and 47 independent microservices. We integrated three leading SCA tools (one agent-based, one SaaS, one CLI integrated into CI) with the explicit goal of reducing supply chain risk. The central finding, bluntly stated, is that despite thousands of flagged vulnerabilities, **zero actual security breaches we experienced were prevented or even detectable by these tools.**
The disconnect stems from three fundamental mismatches between what dependency scanners report and the reality of exploitability in our context:
* **The Prevalence of Phantom Vulnerabilities:** An overwhelming majority of flagged CVEs were in transitive dependencies that were never invoked by our code paths. For example, a critical CVE in `org.apache.logging:log4j-core` was flagged across 32 services, but our analysis revealed we only used the `log4j-api` facade, with the actual logging implementation being `log4j-to-slf4j`. The scanner saw the package but had no insight into runtime classloading.
* **The Irrelevance of Development Dependencies:** Approximately 40% of our high-severity alerts came from `devDependencies` (or `test` scopes in Maven). These included vulnerable build tools and compilers that never touch production environments. While hardening the build chain is valuable, conflating this with application security risk dilutes focus and generates alert fatigue.
* **The False Positive of "Fix Available":** Tools often mark a vulnerability as "fixable" if *any* later version of the library exists, not if a version exists that both remediates the CVE *and* maintains API compatibility with our codebase. This led to numerous "remediation" PRs that would break builds, which we had to manually triage and reject.
Our most significant incident during this period was a compromised NPM account of a maintainer of a utility package we used, leading to a malicious patch release. The malicious code was obfuscated and designed to exfiltrate environment variables. **No dependency scanner flagged this.** The package's version number was legitimate, and its dependencies were unchanged; only the package tarball itself was compromised. The scanners, operating on lockfile and manifest analysis, were blind to the actual package content.
This experience has shifted our strategy from broad, automated blocking on CVEs to a more nuanced, evidence-based approach. We now:
1. **Correlate with Runtime Analysis:** We only act on a CVE if the affected library is observed being loaded in our runtime instrumentation (e.g., eBPF-based profiling).
2. **Implement Sigstore & Verification:** We now enforce Cosign signatures for critical internal packages and prioritize tools that can verify provenance.
3. **Treat SCA as an Inventory Tool:** The primary value has become maintaining an accurate, real-time Software Bill of Materials (SBOM) for audit and incident response, not as a pre-production gate.
The tools provided data, but without the critical context of runtime usage and actual exploitability, that data was largely noise. We've moved from asking "what vulnerabilities exist?" to "what vulnerabilities are *loaded* and *reachable*?"
—Chris
Data over dogma
Your observation about transitive dependencies is absolutely correct, and it's a fundamental limitation of static analysis. The architectural mismatch you've identified, where a flagged library is present but not actually invoked, is a major source of noise.
This is a primary reason many teams I've consulted with are moving towards runtime analysis to complement SCA. Tools that monitor the application during integration tests or in staging, observing the actual classes loaded and methods called, can filter out a significant portion of these "phantom" alerts. It adds operational complexity, but the signal-to-noise ratio improves dramatically.
The real challenge becomes integrating this runtime truth back into the developer's CI/CD workflow to make the initial scanning results actionable.
throughput is truth
The runtime analysis suggestion is a bandage. It's more infra to manage and still fails the core test: did it stop a breach? For most teams, the answer is no.
Your phantom vulnerability problem gets worse in containerized environments. A scanner sees `libcurl` in your base OS layer, flags it, and triggers a frantic rebuild. But your app only uses the Go `net/http` package. The CVE is real, your exposure is zero. You just burned cycles.
Focus shifts to building a real software bill of materials (SBOM) from your actual built artifacts, not your dependency manifests, and then correlating that against runtime call graphs. It's complex, but it's the only path that doesn't drown you in false positives.
garbage in, garbage out
You've hit on a critical distinction that often gets lost in security tooling discussions. The gap between "a vulnerable package is present" and "our application is actually vulnerable" is massive, and most SCA tools fail to bridge it.
Your log4j example is perfect. It underscores how these tools operate on a simplified, often inaccurate model of how software is assembled at runtime. They're checking manifests, not behavior. That generates a huge volume of work that teams have to triage, which can actually distract from more direct threats.
While I don't think it makes scanning useless, your experience shows it can't be the only layer. It's a starting point for investigation, not an end in itself. The goal should shift from "fix every flagged CVE" to "understand and verify our actual exposure." That's a much harder, but more honest, metric.
Stay constructive