I'm currently conducting a detailed performance and cost analysis of our container security scanning pipeline, with Aqua Security's scanner as a central component. A significant operational bottleneck has emerged during scans of repositories containing large, complex multi-stage Dockerfiles, and I am seeking community input on potential root causes and mitigation strategies.
The observed behavior is that the Aqua scanner (we are running the `scanner` CLI version 2022.04) will consistently "hang" or become unresponsive for extended periods (exceeding 45 minutes) when processing specific Dockerfiles. This is not a universal failure but is correlated with Dockerfiles that exhibit the following characteristics:
* Multi-stage builds with more than 8 discrete `FROM` statements.
* A high number of intermediate image layers, often due to extensive `COPY --from=` operations between stages.
* Cumulative image size (sum of all base images and artifacts) exceeding 5 GB.
* The use of `RUN` instructions that generate substantial artifact output within the build context.
Our standard scan command, executed within our CI runner (a c5.2xlarge EC2 instance with 16GB RAM), is:
```bash
./scannercli scan --htmlfile report.html
--dockerfile-path ./Dockerfile
--registry-auth
--check-for-update=false
--host
--user
--password
```
The process typically stalls after printing the "Fetching image layers..." or "Analyzing image layers..." log line. CPU utilization on the runner drops to near-zero, while memory usage remains constant at approximately 2-3GB, suggesting a possible I/O wait state or a deadlock in layer analysis.
**Troubleshooting steps we have already undertaken:**
* Increased CI runner instance size to c5.4xlarge (16 vCPU, 32GB RAM) with no improvement, ruling out pure resource saturation.
* Isolated the Dockerfile and built the image locally, then scanned from a local registry. The hang persists, eliminating network latency to our upstream registry as the cause.
* Enabled `--debug` logging, which provides no additional output after the aforementioned log line.
* Tested with a simplified version of the same Dockerfile by sequentially removing stages. The scan succeeds with 6 stages but consistently hangs with 7 or more.
The operational cost impact is non-trivial. These extended scan durations block CI/CD pipeline stages, leading to:
1. Increased compute costs for the stalled CI runners.
2. Development team productivity loss due to delayed feedback.
3. Potential for resource exhaustion in our CI queue during concurrent scans.
Has anyone performed a similar deep-dive on Aqua scanner performance with complex images? I am particularly interested in:
* Known internal thresholds within the scanner (e.g., maximum layers, maximum total file count) that could trigger this behavior.
* Optimal scanner configuration flags for large, multi-stage images (e.g., `--memory`, `--threads`, though documentation is sparse).
* Whether the issue is more likely tied to the Dockerfile parsing phase versus the subsequent image manifest analysis.
* Any workarounds, such as splitting the Dockerfile or employing a different scanning pattern for such "monolithic" build definitions.
Spreadsheets or it didn't happen.
Your correlation between the hang condition and multi-stage complexity with many `COPY --from` operations is a strong data point. I've observed similar resource exhaustion patterns in other dependency resolution tools when they reconstruct a directed acyclic graph from a Dockerfile. The scanner may be attempting to map every artifact flow between stages to build a complete software bill of materials, and a high number of cross-stage references can cause combinatorial explosion.
You might check if the hang correlates with a specific memory threshold on your runner. I'd be curious if you see the scanner process gradually increasing its RSS memory usage before becoming unresponsive, or if it's stuck in an apparent CPU-bound loop. This could help differentiate between a memory leak and a logic flaw in parsing the stage dependencies.
A pragmatic workaround, albeit not a fix, could be to split the monolithic Dockerfile for scanning purposes only. Generate a separate file for each final stage image and scan those individually, though this loses the ability to track inherited vulnerabilities through the build stages.
prove it with data