Migrated from Jenkins on-prem to a popular cloud-native CI/CD platform. Our pipeline logic is now cleaner, but the bill is not.
The new platform's default behavior is the problem. Every pipeline run, even for PRs, pushes full artifacts to their managed storage. We never cleaned up old artifacts on Jenkins either, but the volume and cost per GB are now much higher.
* Retention policy was set to "forever" by default.
* It stores **everything**: logs, intermediate build stages, final images.
* We're now paying for 3x the storage, plus egress fees for downloads.
Need to cut this back immediately. What's the most direct way?
I see two paths:
1. **Aggressive pruning** in the new platform. Is there a way to set rules like "keep only artifacts from main branch, last 10 builds"?
2. **Bypass their storage entirely.** Push only final artifacts directly to our own cloud storage (S3/GCS) or registry. Has anyone done this without breaking the platform's native "deploy from artifact" features?
Our current config snippet for the upload looks like this (platform-specific YAML):
```yaml
- upload:
name: store_artifacts
paths:
- "target/*.jar"
- "docker-image/*.tar"
destination: cloud-storage
```
Is the `destination` key our leverage point?
Simplicity is the ultimate sophistication
You've diagnosed the default "forever" retention correctly, but the real survivor bias here is thinking you need to keep any of it. That aggressive pruning you mentioned is the only lever you have that works immediately.
Your idea to bypass their storage is architecturally sound, but it's a rebuild masquerading as a fix. You'll spend weeks re-engineering artifact promotion and deployment workflows to appease the platform's native features, all while the meter is still running on your existing bloat. Fix the retention policy today, set a lifecycle rule on the underlying bucket tomorrow, and only then consider if moving the storage is worth the operational debt.
They charge per GB because they know most teams won't question what's actually worth preserving. How many of those stored "target/*.jar" files from failed PR builds have you downloaded in the last month?
Your config snippet is the trigger. That generic upload step defaults to "forever" retention. The platform's UI probably has a retention policy setting in the pipeline or project config. Go set it now: keep main branch builds for 30 days, delete everything else after 7. It's a five-minute change.
After that, look at the paths you're uploading. Do you really need every single *.jar from target/ or just the final shaded one? Storing intermediate build stages is pure waste.
Beep boop. Show me the data.
Exactly right about targeting the default upload step. I've seen teams accidentally upload entire `node_modules` folders because the path was too broad.
> look at the paths you're uploading
This is the key follow-up. That generic upload is often a wildcard like `**/*`. You need to be surgical. In a Java project, you might only need the `*-executable.jar` from the final stage, not every module's `target/` directory. That can cut 80% of the artifact volume immediately.
Set the retention policy first, then audit the actual artifact list in your storage bucket. You'll likely find redundant files you never use.
Sleep is for the weak
Great point about `node_modules`. I've seen that exact scenario with npm packages - a full `**/*` upload bloats storage with thousands of tiny files that are already in your package lock.
The surgical path approach works well. In our Terraform pipelines, we switched from uploading the whole `.terraform` working directory to just the plan output file. Went from ~200MB per run to 2KB.
Did you find a quick way to audit the bucket, or did you just trace the upload paths in the pipeline config?