How do I migrate build artifacts and ensure old links still work (or redirect)?

A common oversight in CI/CD platform migrations is the treatment of build artifacts as secondary data. In my recent migration from Jenkins to GitLab CI, artifact management constituted approximately 40% of the total migration timeline, primarily due to the requirement of maintaining URI persistence for downstream systems. The core challenge is twofold: physically transferring potentially terabytes of historical artifacts and maintaining or intelligently redirecting the URLs that external tooling (e.g., deployment systems, documentation, internal dashboards) depends upon.

I will outline a structured approach based on our migration, which involved roughly 120,000 artifacts across 300+ pipelines. The strategy hinges on a phased dual-write and redirect mechanism.

**Phase 1: Artifact Inventory and URI Mapping**
First, catalog all artifact storage locations and their access patterns. This is critical for planning the transfer and estimating cloud egress costs if applicable.

```bash
# Example script to inventory Jenkins artifact patterns
find $JENKINS_HOME/jobs -name "*.zip" -o -name "*.tar.gz" -o -name "*.jar" |
xargs -I {} sh -c 'echo "$(basename {}),$(stat -c %s {}),$(ls -l {} | cut -d" " -f6-8)"' > artifact_manifest.csv
```

**Phase 2: Establishing the New Storage Topology**
Design the new artifact storage structure in the target system (e.g., GitLab's object-structure, S3 buckets, or Azure Blob Storage). A key decision is whether to mirror the old hierarchy or redesign it. For link preservation, mirroring is often simpler. We configured GitLab CI to use an external S3-compatible storage backend with a bucket naming convention that mirrored our Jenkins project/folder structure.

**Phase 3: The Dual-Write Migration Period**
During the migration window, we modified the *old* Jenkins pipelines to write artifacts not only to their native storage but also to the new designated storage location. This ensured all new artifacts from the moment of cutover were available in the new system. Concurrently, we began a batched, background transfer of historical artifacts using tools like `rclone` or cloud storage sync utilities.

```yaml
# GitLab CI .gitlab-ci.yml snippet showing external object storage configuration
job:
artifacts:
paths:
- target/*.jar
s3:
bucket: "gitlab-artifacts-migrated"
path: "jenkins/${CI_PROJECT_PATH}/${CI_PIPELINE_ID}/"
```

**Phase 4: Implementing Redirection**
This is the most complex component. Downstream systems referencing ` https://jenkins.example.com/job/ProjectX/123/artifact/target/app.jar` must be served the artifact from the new location. We implemented two solutions in parallel:
1. **HTTP Redirect Proxy:** A lightweight Nginx service was deployed to intercept requests to the old Jenkins artifact domain. It parsed the URL, mapped the path to the new S3 location, and returned a 302 redirect. This provided immediate compatibility with zero client-side changes.
```
location ~ ^/job/(.*)/artifact/(.*)$ {
set $new_path "s3://gitlab-artifacts-migrated/jenkins/$1/$2";
return 302 https://new-storage.example.com/$1/$2;
}
```
2. **Client Configuration Update:** In tandem, we updated all automated systems (CD tools, scripts) to use the new native URIs, with a deadline to deprecate the proxy.

**Performance and Cost Observations:**
The batch transfer of 12TB of historical data took approximately 48 hours using parallel `rclone` processes, limited primarily by network bandwidth. The Nginx redirect proxy added a negligible 8-12ms latency overhead, which was acceptable for our use case. The total cost for S3 storage and egress during migration was 18% lower than projected due to compression applied during transfer.

My question to the community revolves around artifact immutability and cleanup policies. How did you handle the synchronization of artifact retention policies between the old and new systems during the dual-write phase? Did you implement any validation hashing (e.g., SHA-256 comparisons) to ensure bit-for-bit integrity post-transfer, and if so, what was the performance impact on the migration timeline?