Hey everyone, I've been digging into our team's CI/CD costs over the last quarter and noticed something pretty wild. Our monthly bill from our cloud-based CI provider spiked by almost 40% last month. After pulling the detailed usage data, I traced it back to the security scanning steps we added to our pipelines. We're now running SAST, container scanning, and secret detection on every single merge request build.
I set up a simple query to see where the compute minutes were going:
```sql
SELECT
pipeline_name,
SUM(duration_seconds)/60 as total_minutes,
AVG(duration_seconds)/60 as avg_minutes_per_run,
COUNT(*) as run_count
FROM ci_jobs
WHERE created_at >= DATE_TRUNC('month', CURRENT_DATE - INTERVAL '3 months')
AND job_name LIKE '%scan%'
GROUP BY pipeline_name
ORDER BY total_minutes DESC;
```
The results were eye-opening. The scanning jobs alone consumed more minutes than the actual application build and test stages combined for our medium-sized microservice. The container scan, in particular, is a resource hog, sometimes taking 8-9 minutes by itself.
I'm trying to build a cost-benefit analysis for my lead. Has anyone else done a deep dive on this? I'm wondering about a few things:
1. Is it more cost-effective to run these scans on a schedule (e.g., nightly) rather than on every push?
2. Are there ways to optimize the scanner configurations to run faster without sacrificing too much coverage? I'm worried about tuning things and missing a critical vulnerability.
3. For those self-hosting runners, do you find the fixed cost outweighs the variable, usage-based cost of managed CI for these intensive scanning tasks?
I love the security posture improvement, but the cost implications feel significant, especially as we scale up the number of projects and developers. Any data or experiences you can share would be super helpful.
That SQL query is telling you exactly what you need for your analysis. You've identified the container scan as the primary bottleneck; that's a classic pattern. The layer-by-layer analysis, especially if you're pulling base images from remote registries each time, burns through compute minutes.
For your cost-benefit case, you need to separate pipeline efficiency from security necessity. Running a full container scan on every MR for a service that hasn't changed its `Dockerfile` or dependencies is pure waste. A more granular approach we implemented was to trigger the heavy container scan only when the `Dockerfile`, `package.json`, or `requirements.txt` changed, using pipeline rules. The SAST and secret detection can stay on every run, as they're usually faster.
The other dimension is cache utilization. If your CI provider charges for compute time, see if you can mount a persistent volume for the scanner's vulnerability database. Pulling that multi-gigabyte DB on every job run adds significant overhead. We saw a 60% reduction in container scan duration just by implementing that.
You're right to focus on the data from that query, that's where the real story is. The jump from a few seconds for SAST to several minutes for a container scan can completely change the economics.
We've seen teams get caught in this exact trap. The initial security improvement feels like a win, but the long-term operational cost gets forgotten until the bill arrives. You might find it useful to compare those compute minutes to the actual number of new vulnerabilities caught per pipeline run over the same period. It often reveals a law of diminishing returns, where most of the cost is spent re-scanning unchanged layers for no new findings.
Your cost-benefit analysis will be stronger if you frame it around optimization rather than removal. No lead will argue against necessary security, but they will listen to a proposal that achieves the same coverage for half the cost.
—HR