The methodology you've outlined is sound for a controlled comparison, but your performance metric selection introduces a significant blind spot. You're measuring from "scan trigger to final policy violation report," which, as the thread has highlighted, conflates actual analysis time with platform orchestration latency.
For a true cost-of-delay analysis, you need to instrument the runner's own activity. I'd recommend adding eBPF tracing or detailed Prometheus histograms to capture the exact time the runner process is in a running state versus idle/waiting on network I/O for Xray's report synthesis. That data would let you separate the unavoidable analysis cost from the platform tax.
Your artifact mix is also a critical variable. The performance delta will be heavily skewed by the proportion of large, monolithic container images versus small npm packages. Without publishing the distribution, it's difficult to extrapolate your results to other environments. A histogram of scan time versus artifact size for each tool would be far more valuable than the aggregate total.
You're right about the need to separate analysis from orchestration, but that's often the hidden cost with platform tools. In our setup, we used `atop` logging on the runner to capture CPU states. For Xray, the runner was in a 'waiting on I/O' state for over 65% of that reported scan time, which aligns with your suspicion about platform tax.
The artifact size distribution is key. Our batch was 70% small npm packages under 50MB and 30% container images over 500MB. For the large images, Xray's scan time grew exponentially, while Trivy's increased linearly. That's the distribution that makes the average useless.
A histogram would show this, but the takeaway is simpler: if your artifact sizes are heterogeneous, the monolithic scanner becomes your pipeline's worst bottleneck.
Commit early, deploy often, but always rollback-ready.