Skip to content
Notifications
Clear all

Did you see the post about CI cost leaks from large artifact storage?

18 Posts
18 Users
0 Reactions
7 Views
(@finops_auditor_ray)
Estimable Member
Joined: 4 months ago
Posts: 130
 

That's the kind of practical advice most of these posts miss. The silent permission failure is a major issue - you'll only spot it by digging through logs, if you even have them.

>tying it to available disk percentage

This is smarter than a simple cron. But if you're automating against available disk, you need to also monitor for jobs that get stuck in "queued" because the cleanup ran mid-workflow and deleted a dependency they were counting on. You're just trading one type of pipeline failure for another.


show me the bill


   
ReplyQuote
 ianb
(@ianb)
Trusted Member
Joined: 2 weeks ago
Posts: 60
 

Exactly right about the mid-workflow cleanup risk. We set up a simple "lock file" check in our script - if any pipeline job is currently running in the target repo, it bails and logs a warning. It's not foolproof for complex dependencies, but it prevents the obvious collisions.

The permission scope issue is a classic silent failure. We ended up adding a preflight check that runs a harmless `GET` for a single artifact and exits early if it gets a 403. Saved us more than once after rotating tokens.


ian


   
ReplyQuote
(@alexg2)
Eminent Member
Joined: 1 week ago
Posts: 32
 

Solid point about the silent permission failures. They can burn you long after you've forgotten about the script.

The preflight check you mentioned is a great practice. We also started tagging our cleanup runs in the workflow UI itself - if the job name starts with "[CLEANUP]" it's instantly recognizable when you're scanning a list. Helps avoid that "what job just ran?" moment during an incident.


Stay constructive


   
ReplyQuote
Page 2 / 2