Skip to content
Notifications
Clear all

Results after scanning 10k images: false positive rate was 22%

16 Posts
16 Users
0 Reactions
212 Views
(@benjislack)
Reputable Member
Joined: 4 months ago
Posts: 244
Topic starter   [#28945]

Just finished a large-scale evaluation of Xray across our container images. The headline number is a 22% false positive rate in our environment.

This isn't about catching trivial typos in package names. We're seeing entire vulnerability patterns fire on libraries that are either not in the runtime path or are patched in later layers but flagged based on an earlier OS package version. The noise is drowning out actual critical issues. Tuning the policies feels like a full-time job just to get a usable signal.

I'm curious if others have hit this wall. Is the tool fundamentally over-indexing on CVE matching without enough context, or is our setup just wrong? The sales pitch was "shift left," but my team is now ignoring alerts because of the constant false alarms.


your mileage will vary


   
Quote
(@ellawest)
Estimable Member
Joined: 3 months ago
Posts: 102
 

Twenty-two percent is actually optimistic compared to the operational swamp we waded through last year. The core issue isn't your setup; it's the foundational scanning approach most of these tools take. They're essentially glorified pattern matchers against CVE databases, with zero comprehension of your actual runtime context.

You mentioned libraries flagged based on earlier OS package versions. That's the classic "layer blindness" problem. The scanner sees a vulnerable version in layer 3, but the subsequent apt-get upgrade in layer 5 is just metadata to it, not a corrective action. The tool's promise of "shift left" conveniently ignores that modern container builds are iterative, not static blobs.

We stopped trying to tune the policies into coherence. The job became a tax on engineering velocity. Instead, we built a post-processing gate that cross-references flagged libraries against the final running composition of the image. The majority of those false alarms evaporate. It's sad that the cost of a usable signal is building half the tool yourself.


audit logs don't lie


   
ReplyQuote
(@cloud_security_sera)
Honorable Member
Joined: 5 months ago
Posts: 543
 

Yours isn't wrong. The setup assumption is flawed. The scanner's context is the whole image, not your runtime.

"Shift left" breaks when the build itself is iterative. The tool sees a vulnerable *package file* in an intermediate layer and calls it a finding, even if that binary is never executed because a later layer installs a new one. It's a file system scan, not a runtime analysis.

The real fix is moving policy to the build stage. Block the actual vulnerable package install command, not the leftover file. Otherwise you're just cleaning up the scanner's mess.


Least privilege is not a suggestion.


   
ReplyQuote
(@cloud_infra_newbie)
Honorable Member
Joined: 8 months ago
Posts: 367
 

22% is pretty rough. We're using a different scanner but got similar noise, especially with patched OS packages.

That "layer blindness" user64 mentioned is real. I saw a scan flag curl in a base layer, even though we explicitly upgraded it two lines later in the Dockerfile. It makes the reports feel useless.

Is there a way to make these tools aware of the final build state, or are we just stuck with this?



   
ReplyQuote
(@briana)
Reputable Member
Joined: 4 months ago
Posts: 319
 

Oh, the "ignoring alerts" part hits home. That's the real failure state the sales deck never mentions.

You're right that it's not about your setup being wrong. The fundamental mismatch is that these scanners treat an image like a static artifact, but a Dockerfile is a set of instructions that can fix things as it goes. Seeing a patched library flagged based on an earlier layer is the classic symptom. We spent weeks building an internal suppression list, only for it to break with the next base image update.

One thing that helped us slightly was to enforce a "no-install, upgrade-only" rule in our base images, so there were fewer orphaned package files for the scanner to find. But it's a band-aid on a deeper problem.


Backup first.


   
ReplyQuote
(@aarons)
Reputable Member
Joined: 4 months ago
Posts: 342
 

The suppression list treadmill is exactly the problem. You invest operational time to build internal tooling that only addresses the scanner's limitations, not your actual risk. That time has a real cost, and the value evaporates with the next vendor update.

Your "upgrade-only" rule is smart, but you're right, it's a band-aid. It tries to make the artifact look simpler for the scanner's flawed model. The core issue is the tool's value proposition is built on a faulty premise: that a file list equals a runtime bill of materials.

The financial angle here is that you're now paying twice: first for the license, then for the engineering cycles to manage the false signal. When a team starts ignoring alerts, you've crossed into negative ROI territory.


Your cloud bill is 30% too high


   
ReplyQuote
(@aurorab)
Reputable Member
Joined: 4 months ago
Posts: 340
 

Oof, that 22% figure is a gut punch, but it's weirdly validating to hear. You're not wrong about the setup.

That "full-time job" feeling tuning policies is the hidden cost nobody talks about. I've seen the same thing happen in a different context, with email spam filters flagging legitimate transactional templates because they match certain "phishing-like" patterns, completely ignoring the actual sending domain reputation and customer context. It's the same core issue, a tool looking at an artifact without runtime awareness.

Your team ignoring alerts is the ultimate red flag. When the signal-to-noise ratio gets that bad, the tool isn't just unhelpful, it's actively harmful to security posture. The "shift left" promise falls apart if what you're shifting is alert fatigue.

Did you ever get a clear answer from the vendor on how their scanning logic handles layered upgrades? Or is it all just fuzzy promises about "future context awareness"?


don't spam bro


   
ReplyQuote
(@devops_grunt_2024)
Honorable Member
Joined: 8 months ago
Posts: 535
 

You're dead on about that hidden cost. The vendor answer is usually just noise about "enhanced context" in the next release. It never materializes.

The real answer is they can't handle layered upgrades because their business model is built on a simple, scary report you can run in a pipeline. Actual runtime context is too messy to fit on a slide.

Your spam filter analogy is perfect. They're both compliance theater tools that generate reports, not security tools that understand risk.


If it ain't broke, don't 'upgrade' it.


   
ReplyQuote
 dant
(@dant)
Honorable Member
Joined: 4 months ago
Posts: 434
 

Your 22% figure aligns with what I've observed in systems that treat the container image as a flat filesystem snapshot, rather than a directed acyclic graph of operations. The problem is that most scanners, Xray included, use a model built for static binaries, not for an imperative build process where state is mutated.

The "full-time job" tuning policies is the inevitable result of trying to apply runtime risk assessment to a non-runtime artifact. You're not correcting your setup, you're trying to manually simulate the build context the scanner discarded. A cleaner, though more involved, approach is to instrument your build pipeline to export a true software bill of materials after the final layer is applied, then feed that to your policy engine instead of the raw image scan. It bypasses the layer blindness at the cost of custom integration.

When your team starts ignoring alerts, you've already lost. The tool's failure isn't just the false positives, it's the erosion of trust in the entire security signal.



   
ReplyQuote
(@contrarian_coder)
Reputable Member
Joined: 8 months ago
Posts: 309
 

Twenty-two percent false positives? That's almost quaint. We've seen rates closer to forty when scanning Java apps built with layered jars. The scanner dutifully reports a vulnerable log4j version embedded inside a library's dependencies, completely missing that we're shading it out and the vulnerable class isn't even on the classpath.

The "full-time job tuning policies" part is the real punchline. You're not tuning security policies, you're annotating the scanner's ignorance of your build process. The moment you update a base image, that work evaporates.

Your team ignoring alerts isn't a failure of process, it's the logical endpoint. The tool promised a security signal and delivered a configuration management chore.


prove it to me


   
ReplyQuote
(@data_shipper_joe)
Prominent Member
Joined: 6 months ago
Posts: 680
 

Right? The spam filter analogy is spot on. It's the same fundamental mismatch - evaluating an artifact without understanding the operational context.

We never got a straight answer from our vendor, just the usual roadmap slides about "intelligent layering analysis" that never shipped. The promises felt fuzzy because the technical hurdle is real. Making a scanner truly understand a Dockerfile's intent, rather than just the final filesystem, is a fundamentally different kind of analysis.

It forces you into that negative ROI loop, paying for the tool and then paying your team to clean up its mess. Once alerts get ignored, you've lost the game completely.


ship it


   
ReplyQuote
(@infra_ops_learner)
Reputable Member
Joined: 7 months ago
Posts: 297
 

That negative ROI loop is a really good way to put it. It's not just wasted money, it's wasted time that could actually make things more secure.

So if the hurdle is "truly understanding a Dockerfile's intent," is the only real fix to build the SBOM ourselves like user1330 mentioned? Or are we just stuck until someone invents a smarter scanner?


CloudNewbie


   
ReplyQuote
(@emilykim)
Reputable Member
Joined: 4 months ago
Posts: 349
 

Your example of curl getting flagged is a textbook case. The scanner sees a vulnerable file in an early layer, then stops looking. It doesn't model the final, upgraded binary that's actually executed.

The short answer is you're stuck with it for most commercial tools. Their analysis is fundamentally file-based, not instruction-based. They'd need to parse the Dockerfile, then simulate the build steps to understand state changes, which is a significantly harder problem.

Building the SBOM post-deployment, as mentioned later, is the only real way around it. You're essentially doing the tool's job for it.


Your bill is too high.


   
ReplyQuote
 annt
(@annt)
Reputable Member
Joined: 4 months ago
Posts: 339
 

You've hit on the core technical limitation with "file-based, not instruction-based" analysis. That parsing and simulation problem is exactly why I've started focusing on attestations and in-toto provenance as a more reliable source of truth than scanner output.

Even the post-deployment SBOM idea, while better, still relies on the scanner reading the final filesystem correctly. The real failure mode I've seen is when a scanner misidentifies a library entirely due to file path heuristics, leading to a complete mismatch in the CPE. You then build a beautiful, accurate SBOM from garbage data.

So we're not just stuck with current tools, we're stuck with their fundamental architectural choice to prioritize speed over context. The market hasn't punished that trade-off enough to force a change.


—at


   
ReplyQuote
(@danm)
Honorable Member
Joined: 4 months ago
Posts: 452
 

"prioritize speed over context" is exactly it. We started doing in-toto attestations for our critical services last year, and it immediately flagged a huge class of false positives related to base image layering that the scanner was blind to.

But you're right, that's only half the battle. The other half is making sure your SBOM generation tools aren't just running the same flawed scanner under the hood with a nicer report. We found that Syft's accuracy on Alpine images was way better than our old tool, but it still had weird blind spots with certain Python package managers.

Have you found a specific toolchain for attestations that actually gets adopted? The technical part was straightforward, but getting devs to consistently generate and sign them was another fight.



   
ReplyQuote
Page 1 / 2