I've been looking into JFrog Xray for scanning our Artifactory repositories. We're a small team, and the promise of automated CVE detection sounds great, but I'm a bit skeptical.
For those using it in production, does it actually catch real vulnerabilities effectively? I'm especially curious about:
* False positives – do you get a lot of noise?
* How well does it handle transitive dependencies in Java or npm packages?
* Is the default configuration good enough, or does it need heavy tuning?
I'd love to hear about your actual experience with its findings versus just trusting the marketing.
We've been running Xray on our Artifactory for about eight months, and it definitely catches the big, scary vulnerabilities. I was skeptical too, but it flagged a critical log4j variant we'd missed in a nested dependency, which was a real "oh wow" moment.
On your specific points:
* False positives: There's some noise, especially with lower-severity CVEs. We had to tweak the severity filters to reduce alert fatigue. The default policy flagged a lot of "medium" issues we considered acceptable risk for internal tools.
* Transitive dependencies: It's solid for Java (Maven/Gradle) and decent for npm. The deep recursive scan works, but you need to make sure your build info is published to Artifactory for the full graph. For npm, it sometimes struggles with lockfile vs package.json mismatches.
* Configuration: The default config is a good starting point, but you'll need to adjust the watch policies and severity thresholds to match your team's risk appetite. It's not "heavy" tuning, more like an afternoon of setup.
That's a really helpful question to ask. We've been using Xray for about six months on our team, and I agree with being skeptical of the marketing claims.
From our experience, it does catch real vulnerabilities effectively, especially the high severity ones. For the default configuration, I'd say it's a good starting point but definitely needs tuning. We created separate policies for our production repos versus development to cut down on the noise for lower-severity issues in internal tools, like user1384 mentioned. It made the alerts much more actionable.
One thing I'd add about transitive dependencies is to double-check your CI/CD integration. We found its accuracy improved a lot once we configured our builds to publish detailed module information consistently.
I appreciate the detail about CI/CD integration. The point about consistency in publishing module information resonates with a hurdle we encountered. In our setup, we found that slight variations in how teams configured their Gradle or Maven publishing tasks led to gaps in the dependency graph Xray could analyze, particularly for multi-module projects.
This makes me wonder, in your tuning process for separate policies, did you find any specific criteria beyond severity that helped prioritize which findings to address first? For instance, did you incorporate factors like exploit maturity or the presence of a known fix directly into your policy logic, or was severity filtering sufficient for making alerts actionable?
It absolutely catches real stuff. That skepticism is healthy, but it paid off for us when it found a nasty spring-shell vulnerability buried three layers deep in a docker image.
The noise from false positives is real, though. The defaults are pretty aggressive. We had to create a custom policy that ignored certain low/critical combos for old, internal apps that never face the internet. That cut the noise by maybe 70%.
For npm, just make sure you're scanning the actual lockfile, not just the package.json. That was the key for us to get accurate transitive dependency mapping.
That's a great point about scanning the actual lockfile. We found the same, but I'd add that even with the lockfile, you need to watch out for cases where teams use `npm ci` versus `npm install` inconsistently across environments. If the lockfile isn't regenerated properly in some pipeline stages, you can get a mismatch between what was scanned and what actually runs, leading to missed vulnerabilities. It's a process issue Xray can't solve for you.
Extract, transform, trust
Your skepticism is spot on. It works, but you'll fight the defaults.
>False positives - do you get a lot of noise?
Yes. The default policies are junk for a small team. You'll get spammed on every medium CVE in old dev dependencies. Create a stripped-down policy for day one. Ignore severities below "High" for non-production repos.
>How well does it handle transitive dependencies?
It's fine if your build tool publishes the full graph. For npm, lockfile scanning is critical. But if your CI doesn't generate a fresh lockfile, the scan is useless. That's on your process, not Xray.
The tuning is mandatory, not optional. Set aside a day to configure policies and watch permissions.
metrics not myths
Your skepticism is really the right approach here. It does catch the high-severity, critical issues, which is the main value for a small team - you can't manually track all those nested dependencies.
The tuning point is crucial though. The default setup is a one-size-fits-all that doesn't fit anyone perfectly. I'd suggest creating your first policy by working backwards: start by manually reviewing a week's worth of its "critical" alerts to see what's genuinely relevant to your stack, then build your rules from that real data. It saves a lot of initial frustration with noise.
And on transitive dependencies, echoing what others said about process - its accuracy is only as good as the dependency data you feed it. If your builds aren't consistent, you'll have blind spots.
Keep it civil, keep it real.
That backwards approach to policy building is smart. It mirrors how we treat alert tuning in our monitoring systems - start with raw data, then filter.
One nuance on dependency data quality: even with consistent builds, if you're pulling from a mix of public registries and internal mirrors, you need to verify Xray's watch coverage includes them all. Missed that once and had a false sense of security for a whole repo group.