How do I migrate build artifacts and ensure old links still work (or redirect)?

Your initial inventory script is a good start for identifying the obvious candidates, but it misses the artifacts that don't have standard extensions. We had a significant number of raw build logs and custom report files with no extension at all in our Jenkins setup.

Beyond that, the `stat` and `ls` approach for file size and timestamps can be unreliable if you're crawling a live system during active builds. We found it necessary to run the inventory against a replicated, read-only snapshot of the artifact storage to avoid corrupting data or getting inconsistent results. The cost estimation from this inventory is also only half the picture; you need to factor in the ongoing storage pricing differences between your old and new platforms, not just the one-time transfer cost.

That inventory script is a good starting point, but it can break on spaces or unusual characters in filenames, which are more common than you'd think. I've had to use `find -print0 | xargs -0` to handle that safely.

Also, for estimating transfer costs, don't forget to add a buffer for retries and failed transfers. Our actual egress was about 15% higher than the raw sum of file sizes due to that overhead.

The dual-write phase you mentioned is key. We used an S3 event notification on the old bucket to copy new artifacts to the new system as they were created during the transition, which kept the delta manageable at cutover time.

Oh, the `find -print0` trick is a lifesaver. I hit that exact problem with our old CI server's logs - filenames with brackets and spaces just disappeared from the inventory.

That 15% buffer is smart. I only budgeted for raw size and now I'm worried about overspending. Do you think that overhead percentage holds for large, individual files too, or is it more about the count of small transfers?

Your script's first line is already a problem. Don't pipe `find` into `xargs` like that for a file inventory, it'll choke and give you incomplete data at your scale.

You're also missing the most important column: the full source URI path that downstream systems are actually calling. The basename is useless for mapping. You need the full path relative to your artifact server's root. For 120k artifacts, write this directly to a database or at least a structured file, not stdout. A CSV won't cut it when you need to start doing lookups for the redirect map.

The real bottleneck won't be the transfer, it'll be validating that map. We had to write a suite of integration tests that impersonated every known consumer, from the deployment hooks to the internal dashboard widgets, and hit the old URLs against our redirect service before we flipped the switch. Found dozens of path mismatches the inventory missed.

Wow, that's a lot of artifacts to track! That inventory script looks like a great start.

But like others said, the file extensions might not cover everything. What about build logs or custom outputs? Should the `find` command maybe just list all files and then filter later? Also, I'm worried about spaces in filenames breaking it.

One question: how did you handle the actual download links? Is mapping the filename enough, or do you need the full path where the old CI server hosted it?

They absolutely tried to set dates years out. We countered by adding an automated review for any `valid_until` date exceeding 180 days, which triggered a ticket requiring a business justification from the owning team. The key was linking it to our quarterly access audit - any map entry with a future expiry beyond the next audit cycle got flagged as a risk.

You also need to consider the cleanup mechanism itself. A timestamp is just metadata. We paired it with a lightweight service that ran daily, querying for expired entries and moving the corresponding artifacts to a quarantine bucket with a 30-day grace period before deletion. This created a tangible consequence for missing the deadline, beyond just a broken link.

Linking future expiry dates to the quarterly audit cycle is clever. It turns a technical configuration into a governance issue.

But what happens when a team submits a justification that's just "we need it for a legacy client"? Your ticket system now has a permanent, recurring exception that everyone will cite. We found that requiring a concrete, incremental sunset plan with verifiable checkpoints was the only way to stop that loophole.

Also, a 30-day grace period after quarantine is generous. We dropped ours to 7 days and made the notification email go to the entire team alias, not just the ticket owner. The social pressure from that alone fixed most of the lazy expiries.

Great point about starting with inventory, it's the foundation for everything after. That find command is a good start, but I'd skip trying to list extensions upfront - just grab all files and parse them later in Python or something. At that scale, you'll need to handle edge cases like symlinks or broken permissions anyway.

Also, don't sleep on capturing the actual HTTP access patterns. Your artifact server logs will show you which old URLs are still being hit, which is way more important than just listing files. That'll tell you what really needs a redirect and what you can archive cold.

Absolutely agree that artifact inventory is the critical first step, and your estimate of 40% timeline allocation is spot on. We had a similar scale of artifacts and the URI mapping phase took far longer than the physical data transfer.

Your script's focus on common archive extensions is a great starting point, but I'd add that you also need to account for ephemeral or "latest" symlinks that Jenkins (and other systems) often create. These symbolic links pointing to the most recent successful build artifact are heavily referenced by dashboards, and their paths need to be captured and replicated in the new system's structure, not just the actual file they point to.

Also, while you're building that initial inventory, it's the perfect time to start tagging artifacts with metadata like pipeline name, build number, and creation date. That extra context becomes invaluable later when you're trying to prune or validate the redirect map - you can make decisions based on actual artifact age and usage, not just filename patterns.

Your 40% timeline estimate for artifact management matches our experience, though ours was closer to 50% once we factored in the AWS egress costs for the transfer. That inventory script is a good starting point, but you need to run a cost simulation on the `stat -c %s` output before you begin. For 120k artifacts, even at an average of 100MB, you're looking at 12TB. At AWS's standard egress rate, that's a significant budget line just for the data movement, not including any compute for the migration logic.

Have you considered using S3 Batch Operations with inventory manifests for the physical copy? It can handle retries and cost reporting natively, which might save you from building that 15% buffer into your estimates. Also, for the URI mapping, storing the results in DynamoDB with the full source path as the key made our redirect service lookups trivial and cheap.

Spot on about the timeline split - we saw something similar moving to HubSpot's ecosystem. Your phased approach is solid, but that first step is where everyone gets bogged down.

The URI mapping is the killer. We found dashboards and old email campaign reports had hardcoded links to artifact paths that nobody had documented. Capturing the actual access patterns from server logs, like user1371 mentioned, became our single most useful dataset. It showed us what was truly 'live' versus just archived bulk.

One thing that helped us: we started the redirect layer (a simple nginx map file) *before* the physical transfer was done. That way we could test the mapping logic with a subset of artifacts and fix the path matching rules early. Did you run into any weird rules for how Jenkins structured those artifact directory paths? We had to account for some legacy job naming that used characters the new system wouldn't accept.

Totally agree that relying on logs alone gives you a skewed sample. It's not just monthly dashboards either - think about compliance artifacts pulled once a year for an audit.

But I'm less ruthless on the "delete and let 404s tell you" approach. In regulated environments, you can't always let broken links be the discovery mechanism. Sometimes you need to know *before* you cut off access. We paired our proxy logs with a manual attestation process from team leads for their critical legacy paths. It was a pain, but it caught things logs never would.

That script will miss symlinks, which are critical. Jenkins `lastSuccessfulBuild` artifact dir is a symlink. If you don't preserve the link target path in your mapping, your redirects will fail.

Also, for cost, run that `stat -c %s` output through `awk` to sum total size. Multiply by your cloud provider's egress rate. You'll find your 15% buffer evaporates. Consider a phased transfer based on access logs to prioritize hot data.

The symlink point is absolutely correct, but you have to be careful how you preserve them. Simply replicating the link target path in a mapping table won't work if your new storage system uses a different scheme for immutable references. You need to decide if you're mapping the *logical* path that users hit, or the *physical* path to the final artifact. We mapped both, which doubled the size of our manifest but prevented the "link chain" problem.

On the phased transfer, prioritizing by logs is logical, but it assumes your logs are complete. In our case, we had third-party systems pulling artifacts via service accounts that didn't log in the standard proxy. We still got burned by surprise egress charges from a batch job we'd forgotten about, so your buffer evaporated either way.

> We used a similar pattern, but our alert triggered a workflow that created a ticket in the repo of the service that owned the broken link.

That's a solid escalation path. We tried tagging CloudFormation stacks with a `LinkOwner` tag, so our decaying proxy could send the alert directly to the team's Slack channel based on the ARN. It cut down the ticket noise a lot.

I agree six months can be aggressive. We landed on eight months for a similar reason - it gave us a full two-quarter buffer for teams that only update their integrations during planning cycles. The dashboard visibility was key; we just made sure the countdown clock was on the team's main operational dashboard, not buried in a separate tool.