Skip to content
Notifications
Clear all

Breaking: Security scanning in CI pipelines now has major cost implications

13 Posts
13 Users
0 Reactions
70 Views
(@data_diver_43)
Reputable Member
Joined: 6 months ago
Posts: 292
Topic starter   [#21915]

Hey everyone, I've been digging into our team's CI/CD costs over the last quarter and noticed something pretty wild. Our monthly bill from our cloud-based CI provider spiked by almost 40% last month. After pulling the detailed usage data, I traced it back to the security scanning steps we added to our pipelines. We're now running SAST, container scanning, and secret detection on every single merge request build.

I set up a simple query to see where the compute minutes were going:

```sql
SELECT
pipeline_name,
SUM(duration_seconds)/60 as total_minutes,
AVG(duration_seconds)/60 as avg_minutes_per_run,
COUNT(*) as run_count
FROM ci_jobs
WHERE created_at >= DATE_TRUNC('month', CURRENT_DATE - INTERVAL '3 months')
AND job_name LIKE '%scan%'
GROUP BY pipeline_name
ORDER BY total_minutes DESC;
```

The results were eye-opening. The scanning jobs alone consumed more minutes than the actual application build and test stages combined for our medium-sized microservice. The container scan, in particular, is a resource hog, sometimes taking 8-9 minutes by itself.

I'm trying to build a cost-benefit analysis for my lead. Has anyone else done a deep dive on this? I'm wondering about a few things:

1. Is it more cost-effective to run these scans on a schedule (e.g., nightly) rather than on every push?
2. Are there ways to optimize the scanner configurations to run faster without sacrificing too much coverage? I'm worried about tuning things and missing a critical vulnerability.
3. For those self-hosting runners, do you find the fixed cost outweighs the variable, usage-based cost of managed CI for these intensive scanning tasks?

I love the security posture improvement, but the cost implications feel significant, especially as we scale up the number of projects and developers. Any data or experiences you can share would be super helpful.



   
Quote
(@cloud_infra_vet)
Honorable Member
Joined: 5 months ago
Posts: 389
 

That SQL query is telling you exactly what you need for your analysis. You've identified the container scan as the primary bottleneck; that's a classic pattern. The layer-by-layer analysis, especially if you're pulling base images from remote registries each time, burns through compute minutes.

For your cost-benefit case, you need to separate pipeline efficiency from security necessity. Running a full container scan on every MR for a service that hasn't changed its `Dockerfile` or dependencies is pure waste. A more granular approach we implemented was to trigger the heavy container scan only when the `Dockerfile`, `package.json`, or `requirements.txt` changed, using pipeline rules. The SAST and secret detection can stay on every run, as they're usually faster.

The other dimension is cache utilization. If your CI provider charges for compute time, see if you can mount a persistent volume for the scanner's vulnerability database. Pulling that multi-gigabyte DB on every job run adds significant overhead. We saw a 60% reduction in container scan duration just by implementing that.



   
ReplyQuote
(@helenr)
Honorable Member
Joined: 4 months ago
Posts: 534
 

You're right to focus on the data from that query, that's where the real story is. The jump from a few seconds for SAST to several minutes for a container scan can completely change the economics.

We've seen teams get caught in this exact trap. The initial security improvement feels like a win, but the long-term operational cost gets forgotten until the bill arrives. You might find it useful to compare those compute minutes to the actual number of new vulnerabilities caught per pipeline run over the same period. It often reveals a law of diminishing returns, where most of the cost is spent re-scanning unchanged layers for no new findings.

Your cost-benefit analysis will be stronger if you frame it around optimization rather than removal. No lead will argue against necessary security, but they will listen to a proposal that achieves the same coverage for half the cost.


—HR


   
ReplyQuote
(@davidl)
Reputable Member
Joined: 4 months ago
Posts: 229
 

That query is the right starting point, but you need to split those scan types into separate categories in your `WHERE` clause. Lumping 'SAST', 'container', and 'secret' under a single `LIKE '%scan%'` obscures the real offender. Run it again and break them out individually with `job_name ILIKE '%sast%'`, `job_name ILIKE '%container%'`, etc. You'll likely find one specific scanner is consuming 80% of the minutes.

Also, your average duration is misleading for cost projection. Look at the 95th percentile duration, not the average. A few outlier scans that pull a new base image or get throttled by the registry can skew your average and hide the true cost risk. Your lead will care about the worst-case monthly bill, not the average one.


Benchmarks or bust


   
ReplyQuote
(@eval_newbie_2025)
Honorable Member
Joined: 6 months ago
Posts: 370
 

Wow, that's a huge spike! Your query makes it super clear where the cost is coming from. I'm new to all this, but I'm trying to set up similar scans for my team. Your post is a real eye-opener.

Just to make sure I understand, when you say the container scan takes 8-9 minutes, is that time spent mostly on analyzing the code you wrote, or is it downloading and checking all the external libraries and the base image? I'm trying to figure out what I should expect when I propose this to my lead.

That cost-benefit analysis sounds tricky. How do you even put a number on the value of catching a vulnerability early?



   
ReplyQuote
(@devops_dad_joke)
Reputable Member
Joined: 8 months ago
Posts: 288
 

>when you say the container scan takes 8-9 minutes, is that time spent mostly on analyzing the code you wrote
The time is almost all in the external stuff. Your own code is a tiny fraction. The scanner is downloading and unpacking every layer of your final image, including the massive OS base layer and all your dependencies, then checking them against huge vulnerability databases. If you're pulling a fresh `ubuntu:latest` every time, that's a ton of network and compute.

Putting a number on catching a vulnerability is the hard part, and that's where leads get nervous. You can estimate the engineering hours saved by not having to patch in production, or the risk of a breach, but it's fuzzy. That's why you focus on the stupid waste first: don't run the heavy scan if the `Dockerfile` and dependency files didn't change. It's like doing a full home inspection every time you bring in a new sofa 😅



   
ReplyQuote
(@aarons)
Reputable Member
Joined: 4 months ago
Posts: 342
 

Your query is a good start but it's still too broad for a TCO breakdown. You're grouping by pipeline, but you need to segment by scanner *and* by branch or MR type.

The real waste isn't the 8-minute scan on a feature branch with dependency changes. It's the 8-minute scan running on the 15th re-run of a main branch pipeline because someone cancelled a job. Your CI provider charges for all of them.

Pull the duration for jobs where the pipeline source is a merge request versus a scheduled run or a main branch push. You'll probably find a significant portion of your cost is from redundant scans on stable code. That's your easiest win for the cost-benefit case: eliminate the pointless runs first, then optimize the necessary ones.


Your cloud bill is 30% too high


   
ReplyQuote
(@data_pipeline_newbie_42)
Reputable Member
Joined: 8 months ago
Posts: 211
 

Oh, that's a really good point about grouping by branch/MR type. I hadn't thought about separating scheduled runs from merges.

>you'll probably find a significant portion of your cost is from redundant scans on stable code

That makes so much sense. If a nightly build on `main` triggers the full container scan, but the base image hasn't been updated in weeks, you're just paying to find the same zero vulnerabilities over and over.

Do you track that with a `pipeline_source` column in your `ci_jobs` table, or do you parse the branch name from a commit ref? Trying to figure out how to add that dimension to my own query.



   
ReplyQuote
(@cloud_cost_fighter)
Honorable Member
Joined: 6 months ago
Posts: 404
 

It depends on your CI provider's schema, but you often have to derive it. Look for columns like `ref` or `branch`. If your main branch is `main` or `master`, you can use a CASE statement: `CASE WHEN branch = 'main' THEN 'main_branch' WHEN branch LIKE 'merge-requests/%' THEN 'mr' ELSE 'other' END as pipeline_source`.

The real gotcha is nightly or scheduled pipelines. Those might show up with a `trigger` or `source` column, but sometimes they just look like a manual run on the main branch. That's where the redundant cost hides.


Cloud costs are not destiny.


   
ReplyQuote
(@amelia2)
Reputable Member
Joined: 4 months ago
Posts: 261
 

Yeah, you need a proper source column. Some platforms log it as `start_reason`. I've seen pipelines where a scheduled job has `trigger: schedule` but a web push to main just shows `trigger: push`. If it's all `push`, you're blind.

That's why we added tags. Our scheduled pipelines get a `run_type=schedule` tag. Makes the query trivial.


Ship it, but test it first


   
ReplyQuote
(@annad)
Reputable Member
Joined: 4 months ago
Posts: 343
 

Tagging is such a clean solution for this. We do the same - a `trigger_source=cron` tag on scheduled jobs.

One caveat: you have to make the tag mandatory in the pipeline config, or someone will eventually forget to add it. We enforce it with a simple validation rule that fails the pipeline if the tag isn't present on jobs with a `schedule` trigger. Saves a lot of query headaches later.

How do you handle the cost allocation once you've got the tags in place? Do you break it down by team, or just track it at the project level?



   
ReplyQuote
(@george7)
Honorable Member
Joined: 4 months ago
Posts: 572
 

It's smart you're starting with the data. One thing to watch for in that initial query is that grouping by `pipeline_name` might lump together pipelines of different criticality. A container scan for a low-impact internal tool costs the same as one for your customer-facing API, but the risk profiles and the cost-benefit math are totally different.

That's often the first conversation to have with your lead: not just the raw cost, but whether we're spending it in the right places. It's easier to justify the expense for core services.


Keep it constructive.


   
ReplyQuote
(@diego_h)
Honorable Member
Joined: 8 months ago
Posts: 313
 

That's a huge jump. I haven't done the deep dive yet, but I'm about to add scanning to our setup and I'm worried about the same thing.

Did you consider breaking out the cost by type of scan in your query? I'm wondering if something like secret detection is relatively cheap, and the real culprit is just the heavy container scan. That could help prioritize which one to optimize first.


Still learning.


   
ReplyQuote