Hey folks, ran into a real headache this week and wanted to see if anyone else has battled this. We’ve been rolling out Xray across our services, and it’s been smooth for most discrete microservices. But our legacy monorepo—a beast with hundreds of Python and Go modules, a mix of Dockerfiles, and tons of lock files—causes Xray scans to consistently time out after about 30 minutes.
Our setup is pretty standard: Xray v3.x on-prem, integrated with Artifactory, scanning on Docker push and during CI builds. The scan starts fine, chugs through dependencies, then just hangs until the timeout hits. We’re not seeing obvious errors in the logs, just that the scan task expires. I suspect it’s either hitting a resource limit or some path depth/complexity issue.
Here’s what we’ve tried so far:
* Increased the `scan.timeout` and `scan.maxTime` values in the `system.yaml` configuration.
* Boosted the heap for the Xray service (gave it 8GB).
* Broke the scan into smaller chunks by targeting specific subdirectories in the CI script, which works but feels like a workaround.
Has anyone successfully tuned Xray for a massive, nested codebase? I’m especially curious about:
* Whether adjusting the Garbage Collection parameters for the JVM made a difference.
* If there’s a known bottleneck with certain file types (like scanning `go.mod` vs `requirements.txt` at scale).
* Any configuration tweaks to the `indexer` or `persist` services that improved throughput.
Our current CI snippet for the partial scan workaround looks like this:
```bash
# We loop through top-level service dirs to scan separately
for dir in services/*/; do
jf xray scan --repo my-repo-local --path "$dir" --format json_v2
done
```
It gets the job done but defeats the purpose of a unified report. Would love to hear your experiences or if we’re missing a key config knob.
--builder
Latency is the enemy, but consistency is the goal.
We hit the same wall with a monorepo that had a massive node_modules footprint. Tuning the JVM heap helped a little, but the real fix for us was adjusting the scan's internal thread pool and connection timeouts, which aren't in the main system.yaml. Check your router-service.log for deadlock patterns. We added these to the xray.yaml under the service config:
```
xray:
...
scan:
maxParallelScanners: 2
dbUpdatePoolSize: 10
```
Dropping maxParallelScanners actually increased throughput because it reduced thrashing. Also, verify your storage isn't getting hammered - if it's NFS, those latency spikes will silently murder scan tasks.
Automate everything. Twice.
That thread pool adjustment is a crucial insight, especially the part about reducing parallelism to combat thrashing. It mirrors behavior I've seen in other scanners when dealing with deeply nested dependency trees, where context switching overhead becomes a silent killer.
Your mention of storage latency is spot on. Even with local SSDs, I've observed that the sheer volume of small I/O operations during a scan of a massive monorepo can saturate IOPS, causing the scan to appear hung. Monitoring `iowait` during a scan attempt became a required diagnostic step for us.
One caveat I'd add: lowering `maxParallelScanners` can backfire if your bottleneck is actually network latency to external registries or databases, as it leaves scanner threads idle waiting for responses. You really need to correlate those router-service deadlock patterns with external service response times.
Great tips already on the thread pool settings, that's something I wouldn't have thought to check. I'm new to this whole Xray setup, so apologies for the basic question, but when you mention targeting subdirectories as a workaround, does that mean you're manually triggering separate scan jobs for different parts of the monorepo? I'm wondering if there's a way to configure that partitioning more permanently, or if you just have to script it. Also, have you looked at whether the timeout happens at a predictable point, like after processing a certain number of files? Could help pinpoint if it's a specific problematic module.
Yeah, splitting scans by subdirectory was our workaround too. It does feel clunky, but scripting it in CI was the only thing that got us past the timeouts initially.
For your question on predictable failure points - ours always choked on a few specific deeply nested Docker builds with long chain dependencies. Might be worth letting one scan run to the limit and checking the router-service.log right before timeout; it sometimes shows the last module it was trying to index.
Beyond thread pools, have you checked the Xray database performance? At that scale, we found indexing bottlenecks that didn't show up until the scan was mostly done.
Automate everything.
The database angle is critical and often overlooked. In our deployment, the timeout wasn't in the scan phase itself, but during the final analysis and persistence step where Xray correlates all the indexed components. A scan can appear to hang because it's silently blocked on a database transaction.
You can verify this by checking for long-running queries on your PostgreSQL instance during an active scan. We had to increase `work_mem` and tweak autovacuum settings specifically for the Xray database to prevent table bloat from causing lock contention during these bulk inserts.
That > 30-minute hang is exactly what we saw with our monorepo before pinpointing the database as the culprit. Since you've already bumped timeouts and heap, and splitting the scan works, I'd bet your issue is also in that final correlation/persistence phase.
Did you check if the timeout coincides with a specific log pattern in router-service.log, like a transition from "indexing" to "persisting" or "correlating"? That's usually the giveaway. If it's hanging there, tuning the JVM won't help, you need to look at the Postgres side like user777 mentioned.
I'm curious if you've monitored the Xray database connections during a scan. When ours timed out, we'd see a bunch of idle-in-transaction connections that hinted at the lock contention.