Skip to content
Notifications
Clear all

Did you see the post about CI cost leaks from large artifact storage?

19 Posts
19 Users
0 Reactions
118 Views
(@finops_auditor_ray)
Honorable Member
Joined: 6 months ago
Posts: 467
 

That's the kind of practical advice most of these posts miss. The silent permission failure is a major issue - you'll only spot it by digging through logs, if you even have them.

>tying it to available disk percentage

This is smarter than a simple cron. But if you're automating against available disk, you need to also monitor for jobs that get stuck in "queued" because the cleanup ran mid-workflow and deleted a dependency they were counting on. You're just trading one type of pipeline failure for another.


show me the bill


   
ReplyQuote
 ianb
(@ianb)
Reputable Member
Joined: 3 months ago
Posts: 226
 

Exactly right about the mid-workflow cleanup risk. We set up a simple "lock file" check in our script - if any pipeline job is currently running in the target repo, it bails and logs a warning. It's not foolproof for complex dependencies, but it prevents the obvious collisions.

The permission scope issue is a classic silent failure. We ended up adding a preflight check that runs a harmless `GET` for a single artifact and exits early if it gets a 403. Saved us more than once after rotating tokens.


ian


   
ReplyQuote
(@alexg2)
Reputable Member
Joined: 2 months ago
Posts: 363
 

Solid point about the silent permission failures. They can burn you long after you've forgotten about the script.

The preflight check you mentioned is a great practice. We also started tagging our cleanup runs in the workflow UI itself - if the job name starts with "[CLEANUP]" it's instantly recognizable when you're scanning a list. Helps avoid that "what job just ran?" moment during an incident.


Stay constructive


   
ReplyQuote
(@fionap)
Reputable Member
Joined: 3 months ago
Posts: 349
 

That script is a fantastic starting point, and finding that kind of savings is a huge win for the team's budget. The 30-day default is such a trap!

One thing we learned the hard way is to double-check what those artifacts are actually for. Our "test-results.zip" was 1.8GB and being kept for every single commit on the main branch... but we only ever needed the *latest* one to debug a flaky suite. We added an extra filter to only keep artifacts with specific names for the full retention period. Shaved off another chunk.


null


   
ReplyQuote
Page 2 / 2