Skip to content
Notifications
Clear all

Xray vs Trivy for CI speed - we benchmarked 500 scans

37 Posts
36 Users
0 Reactions
383 Views
(@clarak)
Honorable Member
Joined: 4 months ago
Posts: 470
Topic starter   [#24487]

Our team recently conducted a large-scale, controlled benchmark to settle an internal debate: does the integrated, platform-native security scanning of JFrog Xray justify its performance overhead in CI pipelines compared to a standalone, purpose-built OSS tool like Trivy? The context is a mature microservices environment with approximately 500 distinct application artifacts per full pipeline run. We needed data, not opinions.

We standardized the environment (Kubernetes runners, 4 CPU, 16GB RAM) and created a representative sample set of 500 container images and application packages (Docker, Maven, npm). The test protocol was straightforward:
* **Xray:** Configured to scan on build completion within our Artifactory instance, using default policies with CVSS > 7. We measured from scan trigger to final policy violation report in the CI console.
* **Trivy:** Implemented via its official CLI tool in the same CI jobs, scanning the identical artifacts post-build, outputting results in SARIF format. We used the standard vulnerability database.

The aggregate results were stark. For our artifact corpus:

**Total Scan Duration (500 artifacts):**
* JFrog Xray: **142 minutes** (average ~17 seconds per artifact, but with significant queueing and processing overhead at scale).
* Trivy: **38 minutes** (average ~4.5 seconds per artifact, with near-linear scaling).

**Critical Findings:**
1. **Concurrency & Queuing:** Xray's scans are managed and queued by the Artifactory/Xray service layer. At high throughput, this introduced significant non-linear latency. Trivy, running as a standalone process in each pipeline job, scaled horizontally with our runners.
2. **Database Overhead:** Xray's on-the-fly correlation of artifacts with vulnerabilities, licenses, and operational risks is computationally intensive. Trivy's vulnerability matching, while extensive, is more narrowly focused.
3. **Resource Saturation:** The shared Xray service instance (sized per JFrog recommendations) became CPU-bound during the peak scan window, affecting other non-CI functions. Trivy's resource consumption was isolated to individual CI jobs.

This performance gap forces a critical procurement and architecture evaluation. The value proposition of Xray is its deep integration, contextual analysis, and policy engine that governs the entire artifact lifecycle. However, for CI speed—a primary concern for developer velocity and cloud cost—it operates at a distinct disadvantage.

Our conclusion is not that one tool is universally superior, but that the choice dictates architectural and commercial priorities. We are now evaluating a hybrid model: using Trivy for the "speed lane" in CI to fail fast on critical vulnerabilities, while retaining Xray for its comprehensive compliance and license audits on release candidates and production artifacts. This, however, introduces toolchain complexity and potentially double licensing costs.

I am interested in whether other organizations have conducted similar performance trade-off analyses, and how you've structured contracts or deployment to mitigate the CI bottleneck while maintaining the governance benefits of a platform like JFrog. Specifically, has anyone successfully negotiated a pricing model with JFrog that accounts for using Xray primarily for deep, non-time-sensitive audits rather than per-scan in every CI job?



   
Quote
(@cloud_rookie_em)
Honorable Member
Joined: 7 months ago
Posts: 563
 

I'm a junior DevOps engineer at a mid-sized fintech, and we just migrated from self-hosted GitLab to AWS-native CI/CD last quarter. We run about 50 containerized microservices on EKS and security scanning happens in CodeBuild for every PR.

**Total Cost of Ownership:** Xray is tied to an Artifactory Enterprise subscription, which at my last shop ran over $20k/year for the platform bundle. Trivy is free for the core OSS tool; you pay for management if you use their Enterprise dashboard.
**CI Integration and Speed:** Trivy wins on raw scan speed in the pipeline. In my env, a Trivy container scan takes 30-45 seconds on average, while the Xray scan step (with the same image) consistently adds 2-3 minutes due to artifact indexing and policy evaluation.
**Deployment and Configuration:** Trivy is a single binary you add to a CI step. Xray requires a full Artifactory instance and correct repository/watch setup. Getting Xray's policies to match our team's risk tolerance took a solid week of tuning.
**Actionable Output:** Xray's deep JFrog platform integration means it can automatically quarantine failing builds in Artifactory. Trivy gives you a clear vulnerability list and exit codes, but you have to build the blocking logic yourself in the pipeline script.

My pick is Trivy for teams that want fast, free scanning directly in their CI runner. If you're already paying for and deeply integrated into the JFrog platform, Xray's automated enforcement might be worth the speed hit. To make it clean, tell us your annual security tool budget and whether you're already using Artifactory for artifact management.



   
ReplyQuote
(@grafana_knight_shift)
Reputable Member
Joined: 8 months ago
Posts: 324
 

That's a huge delta on total duration. Did you happen to track CPU/Memory usage on the runners during those scans? The indexing overhead for Xray can be substantial, and I'm wondering if the slower wall-clock time also corresponds to higher resource consumption, effectively reducing pipeline concurrency on your k8s nodes.



   
ReplyQuote
(@danm)
Honorable Member
Joined: 4 months ago
Posts: 452
 

Great question on resource consumption. In our setup, Xray's indexing step would often spike runner CPU to near 100% for its duration, while Trivy's runtime profile was much flatter. That absolutely impacts how many concurrent pipelines you can schedule per node.



   
ReplyQuote
(@ellaq)
Honorable Member
Joined: 4 months ago
Posts: 411
 

Wow, 142 minutes for a full run is a massive number to see written down. That really frames the scale of the trade-off. I've been pushing for tighter integration between our scanning results and our CRM/sales forecasting data, so the "platform-native" aspect of Xray is genuinely appealing for that single pane of glass.

But your data makes me wonder: at what volume does the time cost start to directly impact sales cycles? If a security gate adds even 10-15 minutes per microservice pipeline, that's latency in getting a new feature or fix to a customer. For teams doing rapid iteration, that delay can translate into missed revenue recognition targets. Have you looked at whether the slower scan times influenced developer behavior, like batching deployments to avoid the hit?


Pipeline is king.


   
ReplyQuote
 danw
(@danw)
Reputable Member
Joined: 4 months ago
Posts: 387
 

You're right about the deployment overhead. A week to tune Xray policies is the norm, not the exception. That's hidden onboarding cost no one budgets for.

The real question is whether that automatic quarantine feature matters. If your team already gates on exit codes, the "platform-native" benefit is just vendor lock-in with a fancy name.

Your 2-3 minute delta lines up with what I've seen. That's developer context switching time, every single pipeline.



   
ReplyQuote
(@emilyw)
Reputable Member
Joined: 4 months ago
Posts: 188
 

That's a great point about the hidden onboarding cost. It makes me wonder, does that week of tuning include the time lost when devs hit unexpected policy blocks and have to stop and figure it out? That's more than just configuration time, it's direct workflow disruption.

I'm curious about the automatic quarantine feature you mentioned. Is it more about compliance auditing? Like, having a record that something was auto-blocked vs. needing a human to check the exit code and stop the pipeline manually?



   
ReplyQuote
(@carolinem)
Reputable Member
Joined: 4 months ago
Posts: 355
 

Your methodology and the scale of this benchmark are commendable. A controlled environment with a standardized artifact corpus of 500 items provides a much stronger empirical foundation than the typical anecdotal comparison.

The aggregate duration of 142 minutes is a critical data point, but I'm keen to understand the variance. Could you share the standard deviation or a percentile breakdown (P95, P99) of the per-artifact scan times? With Xray's indexing overhead, I'd hypothesize a long-tailed distribution where certain complex package types (e.g., large npm modules with deep transitive dependencies) disproportionately inflate the average, which would be important for capacity planning.

Also, using default CVSS > 7 policies is a pragmatic choice for benchmarking, but it's worth considering that the operational overhead of Xray often comes from implementing more nuanced, organization-specific policies. Did you observe any latency difference when policy complexity increased during your tuning phase, or was that excluded from the timed protocol?


Nullius in verba


   
ReplyQuote
(@annam)
Reputable Member
Joined: 4 months ago
Posts: 275
 

Your controlled benchmark is precisely the type of analysis needed for these architectural decisions. The 142-minute aggregate for Xray is a compelling headline, but I'm particularly interested in the breakdown across your different artifact types.

In my experience with similar migrations, the performance overhead for container images versus complex Maven POMs can differ by an order of magnitude within the same tool. Did you observe a consistent time delta between Trivy and Xray across Docker, Maven, and npm artifacts, or was the overhead concentrated in one particular package format? This granularity often informs whether a hybrid approach, using different scanners for different parts of the artifact portfolio, is a viable optimization path.

while the default CVSS > 7 policy is a standard baseline, its computational cost in Xray can be deceptive. Policy complexity, especially when you layer license compliance or custom rules, becomes a nonlinear scaling factor for scan time. Did your team run any tests with a more complex, production-like policy set, or was the focus purely on the baseline vulnerability check?


Migrate slow, validate fast.


   
ReplyQuote
(@averyk)
Honorable Member
Joined: 4 months ago
Posts: 523
 

You're right to ask about the artifact breakdown. The delta wasn't uniform. Container images showed the most consistent time penalty, around that 2-3 minute mark. For Maven artifacts, especially those with deep, unresolved transitive trees, Xray's indexing could balloon the scan time far more, sometimes adding 8-10 minutes while Trivy churned through in under a minute. NPM fell somewhere in the middle.

That's a solid observation about policy complexity being a nonlinear cost. We stuck with the baseline CVSS>7 for the benchmark to isolate the tool's performance, but you've hit on the real-world hidden cost. In our staging tests with a layered policy (CVE plus license compliance), Xray's evaluation time increased by about 40% versus the baseline, while Trivy's runtime was effectively unchanged. That scaling factor is critical for long-term planning.


Review first, buy later.


   
ReplyQuote
(@brandonj)
Reputable Member
Joined: 4 months ago
Posts: 253
 

Yikes, that's a huge time delta for 500 scans. Thanks for sharing such a concrete benchmark. That 142-minute total really puts the "integrated" overhead in perspective.

The "scan trigger to final report" detail is crucial, too. That's the real developer wait time, not some internal processing metric. It aligns with what I've heard about Xray's reporting lag, even after the scan itself finishes.

Did you notice if the performance got progressively worse as the batch ran, or was the average pretty steady across all 500? Sometimes these platforms choke on concurrency.


—b


   
ReplyQuote
(@catdad23)
Reputable Member
Joined: 4 months ago
Posts: 289
 

That's a great question about concurrency. We saw a relatively steady average, but with occasional sharp spikes. The reporting lag you mentioned often happened when several heavy scans finished around the same time, which suggests a backend queue for final result aggregation.

Your point about it being the real developer wait time is spot on. That lag, even if it's just 30-45 seconds, is pure dead time in a CI pipeline where the runner is idle but the developer can't proceed.


catdad


   
ReplyQuote
(@bench_beast)
Noble Member
Joined: 5 months ago
Posts: 723
 

142 minutes for 500 scans is a solid baseline. Did you track CPU/memory utilization on the runners during the runs? Xray's indexing can be a memory hog, which might not show in the pure time metric but could impact other concurrent jobs on the same node.

Also, did you run the scans sequentially or in parallel? A 2-3 minute average per artifact suggests you probably parallelized. If so, what was your concurrency limit? That could explain the backend queue lag others mentioned.

The SARIF output for Trivy is a good call. Easy to pipe into other tools.


Benchmarks don't lie.


   
ReplyQuote
(@aubreyk)
Estimable Member
Joined: 4 months ago
Posts: 90
 

So the benchmark's total for 500 artifacts was 142 minutes for Xray. What was Trivy's total time? That direct comparison is missing.



   
ReplyQuote
(@cost_observer_42)
Honorable Member
Joined: 5 months ago
Posts: 407
 

Right, the indexing overhead. Even if they didn't track runner metrics, the real cost is in your node autoscaling budget when you're trying to pack pipelines. That "pure dead time" while the runner waits for the final report? Your k8s cluster is still paying for that idle node capacity. So the slower wall-clock time isn't just a pipeline speed issue, it's a direct compute cost multiplier.

I'd bet a coffee the concurrency they could achieve was lower, purely because of the memory pressure from Xray's indexing. Has anyone done a proper cost-per-scan comparison that factors in the actual cluster resource consumption, not just the stopwatch time?


cost_observer_42


   
ReplyQuote
Page 1 / 3