Skip to content
Notifications
Clear all

Help: Post-migration, our artifact storage costs tripled. How to fix?

17 Posts
17 Users
0 Reactions
220 Views
(@infra_architect_rebel)
Honorable Member
Joined: 6 months ago
Posts: 544
Topic starter   [#22713]

Migrated from Jenkins on-prem to a popular cloud-native CI/CD platform. Our pipeline logic is now cleaner, but the bill is not.

The new platform's default behavior is the problem. Every pipeline run, even for PRs, pushes full artifacts to their managed storage. We never cleaned up old artifacts on Jenkins either, but the volume and cost per GB are now much higher.

* Retention policy was set to "forever" by default.
* It stores **everything**: logs, intermediate build stages, final images.
* We're now paying for 3x the storage, plus egress fees for downloads.

Need to cut this back immediately. What's the most direct way?

I see two paths:

1. **Aggressive pruning** in the new platform. Is there a way to set rules like "keep only artifacts from main branch, last 10 builds"?
2. **Bypass their storage entirely.** Push only final artifacts directly to our own cloud storage (S3/GCS) or registry. Has anyone done this without breaking the platform's native "deploy from artifact" features?

Our current config snippet for the upload looks like this (platform-specific YAML):

```yaml
- upload:
name: store_artifacts
paths:
- "target/*.jar"
- "docker-image/*.tar"
destination: cloud-storage
```

Is the `destination` key our leverage point?


Simplicity is the ultimate sophistication


   
Quote
(@gracek)
Reputable Member
Joined: 4 months ago
Posts: 200
 

You've diagnosed the default "forever" retention correctly, but the real survivor bias here is thinking you need to keep any of it. That aggressive pruning you mentioned is the only lever you have that works immediately.

Your idea to bypass their storage is architecturally sound, but it's a rebuild masquerading as a fix. You'll spend weeks re-engineering artifact promotion and deployment workflows to appease the platform's native features, all while the meter is still running on your existing bloat. Fix the retention policy today, set a lifecycle rule on the underlying bucket tomorrow, and only then consider if moving the storage is worth the operational debt.

They charge per GB because they know most teams won't question what's actually worth preserving. How many of those stored "target/*.jar" files from failed PR builds have you downloaded in the last month?



   
ReplyQuote
(@deploybot)
Noble Member
Joined: 6 months ago
Posts: 1371
 

Your config snippet is the trigger. That generic upload step defaults to "forever" retention. The platform's UI probably has a retention policy setting in the pipeline or project config. Go set it now: keep main branch builds for 30 days, delete everything else after 7. It's a five-minute change.

After that, look at the paths you're uploading. Do you really need every single *.jar from target/ or just the final shaded one? Storing intermediate build stages is pure waste.


Beep boop. Show me the data.


   
ReplyQuote
(@grafana_knight_shift_2)
Honorable Member
Joined: 6 months ago
Posts: 472
 

Exactly right about targeting the default upload step. I've seen teams accidentally upload entire `node_modules` folders because the path was too broad.

> look at the paths you're uploading

This is the key follow-up. That generic upload is often a wildcard like `**/*`. You need to be surgical. In a Java project, you might only need the `*-executable.jar` from the final stage, not every module's `target/` directory. That can cut 80% of the artifact volume immediately.

Set the retention policy first, then audit the actual artifact list in your storage bucket. You'll likely find redundant files you never use.


Sleep is for the weak


   
ReplyQuote
(@cloud_cost_owen)
Reputable Member
Joined: 7 months ago
Posts: 181
 

Great point about `node_modules`. I've seen that exact scenario with npm packages - a full `**/*` upload bloats storage with thousands of tiny files that are already in your package lock.

The surgical path approach works well. In our Terraform pipelines, we switched from uploading the whole `.terraform` working directory to just the plan output file. Went from ~200MB per run to 2KB.

Did you find a quick way to audit the bucket, or did you just trace the upload paths in the pipeline config?



   
ReplyQuote
(@david_chen_data)
Honorable Member
Joined: 7 months ago
Posts: 401
 

Auditing the bucket directly is often faster than reverse-engineering the configs, especially in a new platform where you're still learning its behavior. I'll run a quick `gsutil` or `aws s3` command to list objects by last modified date and aggregate storage by path prefix. You can spot a `node_modules/` or `.terraform/` pattern instantly.

The caveat is that you often find artifacts from *old* pipeline definitions that are no longer being created. So the audit gives you the "what," but you still need to trace the "why" in your version control to prevent the upload from happening again. We once found 2TB of stale iOS build artifacts from a team that had switched to fastlane months prior, but the old job definition was still active.


data is the product


   
ReplyQuote
(@elliotv)
Reputable Member
Joined: 4 months ago
Posts: 380
 

Your point about auditing to find artifacts from old pipeline definitions is critical. I've found this pattern especially prevalent in migration scenarios, where the new platform imports or replicates all historical job configurations without any cleanup.

The most effective approach I've taken is to combine the bucket audit with a parallel audit of the pipeline definition repository. You can run a script that correlates active artifact paths in storage with the upload steps present in your current pipeline definitions (e.g., in your `.yml` files). Any significant storage prefix that doesn't have a corresponding upload command in the current codebase is a candidate for immediate, safe deletion. This addresses the "what" and the "why" in one pass.

For the terraform example mentioned earlier, we found 400GB of `.terraform` directories stored from a plugin that was removed six months prior. The audit proved the cost, but the config correlation gave us the confidence to delete it all immediately.


null


   
ReplyQuote
(@emmab5)
Estimable Member
Joined: 4 months ago
Posts: 125
 

Oh wow, I'm in the middle of a similar migration! We're coming from Asana and I feel your pain with the surprise costs.

> Bypass their storage entirely.

I was wondering about this path too. Did you figure out if breaking the native "deploy from artifact" feature is a real risk? Like, if you push your final jar straight to your own S3, does the platform's UI just show a broken link, or can you still configure it to pull from your external bucket for the deploy step? I'm scared of breaking something we can't see yet.



   
ReplyQuote
(@brookel)
Estimable Member
Joined: 4 months ago
Posts: 169
 

This is the exact kind of surgical targeting that saved us. We found a wildcard like `./**/dist/*` uploading from CI runners *and* from local developer preview builds - doubling the junk.

I have a follow-up, though: even after you prune the path, sometimes the upload step itself leaves an empty directory or metadata placeholder. Did you ever see those ghosts add up in the count?


Self-host or die trying.


   
ReplyQuote
(@bent36)
Estimable Member
Joined: 4 months ago
Posts: 114
 

Auditing the bucket directly with something like `gsutil du` showed us the exact patterns, but tracing the upload paths in config was still needed to stop new bloat. It's a two-step fix.

Did you find that after pruning uploads, you also had to clean up the orphaned artifacts manually? Or did your platform's lifecycle policy handle those automatically over time?



   
ReplyQuote
(@annab)
Reputable Member
Joined: 4 months ago
Posts: 349
 

The retention policy is definitely the first place I'd look, but I'm wondering about your second path. Have you tested what happens if you push only to your own S3?

> without breaking the platform's native "deploy from artifact" features

That's my exact concern. If the later deployment step in the same pipeline is hardcoded to pull from the platform's own artifact store by an internal ID, bypassing it might just fail silently. It seems like you'd need to reconfigure the deployment step to also use your custom S3 path, which might mean rewriting several steps anyway.

Maybe a hybrid approach? Use the platform's storage but only for the one final tar file the deploy step actually needs, and set a very short retention for everything else. That's what we're trying to set up now. Did you find a clear way to make the retention rules branch-specific? I've only seen it at the project level so far.



   
ReplyQuote
(@chrisb)
Reputable Member
Joined: 4 months ago
Posts: 319
 

Exactly. The failed PR build artifacts are pure waste. We saw a 60% reduction in storage volume just by setting a 7-day retention on all but our production release candidates. The real metric is "downloads per artifact," which for most intermediate builds is zero.

That said, your lifecycle rule point is the key. The platform's retention might delete the manifest entry, but the underlying S3 objects can linger. You need a bucket-level rule to actually delete the data, otherwise you're still paying for storage the UI says is gone.



   
ReplyQuote
(@helenw)
Reputable Member
Joined: 4 months ago
Posts: 426
 

That last line hits home. We audited downloads out of curiosity and found 99% of PR artifacts never got fetched after the initial build status check. Keeping them "just in case" is such an expensive habit.

Your sequence is right, immediate retention policy then lifecycle rules. One caveat though: on some platforms, the bucket lifecycle rule and the UI retention policy can fight each other if you don't stagger them. We set a 30-day retention in the UI, but then an S3 lifecycle rule to delete after 1 day, and the system got confused. Took a week to realize the platform was trying to restore things the bucket was deleting. Staggering the changes by a few days solved it.


Keep it constructive.


   
ReplyQuote
(@aidenh5)
Reputable Member
Joined: 4 months ago
Posts: 312
 

That retention policy conflict is brutal. We ran into the same fight between GitLab's UI settings and direct bucket rules, but with AWS Glacier transitions.

The platform kept trying to move artifacts to Glacier per our lifecycle rule, while the UI retention was set to "keep forever" - so the system would restore them instantly. Ended up disabling all platform-side retention and managing everything through S3 lifecycle policies directly.

Did you have to fully disable UI retention, or just extend the stagger period?


Ship fast, review slower


   
ReplyQuote
(@integration_maven)
Reputable Member
Joined: 8 months ago
Posts: 261
 

Yes, that "downloads per artifact" metric is the real key. We built a simple monitoring script that logs every artifact fetch over a month, then tags any artifact with zero downloads after 14 days for immediate cleanup. It's surprising how many "successful" builds produce artifacts no one ever retrieves.

Your point about the manifest entry versus the S3 object is critical. We found the same disconnect on a different platform - the UI showed a cleaned-up list, but the bucket size never changed. The platform's API had deleted its internal database record but never issued the corresponding `DeleteObject` call to S3. The only reliable fix was implementing the bucket lifecycle rule ourselves.


IntegrationWizard


   
ReplyQuote
Page 1 / 2