Alright, who's been running the new container scanning engine in production? I've had it enabled on a couple of our lower environments for the past three weeks, and I have to say, the performance hit is significant. We're not talking about a minor delay; we're talking about pipeline stages that used to take 90 seconds now blowing out to 8-9 minutes. This is on a standard CI/CD runner with decent resources.
The old engine would chew through a moderately sized application image (say, a Node app with ~300 packages) in a timeframe that kept developers from complaining. The new one seems to be doing a much deeper layer analysis, which is great for completeness, but it's grinding our automated deployment processes to a halt. My team is now facing the classic trade-off: more thorough security data versus the velocity of actually shipping code.
I've dug into the logs and the process. A few observations:
* The scanner now appears to be pulling and analyzing **every layer independently**, even for images that are already in the registry and haven't changed. There's no apparent caching of base layer results.
* The memory footprint on the runner has nearly doubled. We're seeing OOM kills on some of our smaller, memory-constrained build pods.
* The API response for scan results is more detailed, which is good, but the polling interval to get a 'COMPLETE' status has increased exponentially.
Here's a snippet from a pipeline log that illustrates the wait. This is just for the scanning phase to finish, not the actual build:
```bash
INFO: Starting container vulnerability scan for image: my-registry/app:feature-branch-123
... [90 seconds of build output] ...
STATUS: SCANNING - Layer 12/45
... [4 minutes later] ...
STATUS: SCANNING - Layer 32/45
... [another 3 minutes] ...
INFO: Scan completed. Vulnerabilities found: 17
Total scan time: ~8m 30s
```
Has anyone else seen this? More importantly, has anyone found a workable configuration or a flag to revert to the previous engine's performance profile? I'm all for better security, but if this becomes a blocker, teams will just start bypassing the scan or demanding exceptions, which defeats the whole purpose. I'm curious if this is a universal experience or if our setup is somehow suboptimal. What are your production numbers looking like?
Yeah, the layer analysis without caching is a killer. I've seen this pattern before with other tools that prioritize "freshness" over performance. It's like they forget we're running this in CI, where the same base layers are used across dozens of builds a day.
You might want to check if your setup allows for a dependency caching proxy. For some scanners, routing those layer pulls through a local cache can shave off a huge chunk of time, even if the engine still analyzes them. Not a perfect fix, but it got us from 7 minutes back down to about 3.
That memory spike you mentioned is concerning though. OOM kills in CI are the worst. Makes me wonder if they're holding the entire dependency tree in memory now instead of streaming results.
one stack at a time