Just wrapped up a six-month slog moving 200+ pipelines from Jenkins to GitLab CI. Everyone obsesses over YAML translation, but that's the easy part. The real time-sink was proving the new system actually worked *exactly* like the old one.
We spent most of our time building a parallel run framework. Every pipeline change had to execute in both Jenkins and GitLab CI, comparing artifacts, exit codes, and even side-effects. Found so many "hidden" dependencies on Jenkins plugins and workspace paths that weren't in any docs. The vendor's "migration guide" was useless here—pure marketing fluff. If you're not testing like a maniac, you're just hoping it works. And hope isn't a strategy. 😒
Just my two cents.
Just my two cents.
> "Found so many 'hidden' dependencies on Jenkins plugins and workspace paths"
This hits home. I'm a few months into my first CI gig and we're still finding random things like a plugin that injected environment variables into the build shell that nobody remembered to document. Then GitLab just... didn't have them. Took us a week to track down a failing test because of a missing `BUILD_TAG` variable.
One thing I'm curious about - how did you handle the parallel run framework for things like secrets or network calls that might have side effects? Did you have to stub out external services in both systems, or just accept that some tests would be slightly different? I'm trying to build a similar comparison for a smaller migration and I'm worried about false positives from timing differences.
Learning by breaking
Your question about handling secrets and side-effects in parallel runs is the critical operational detail most migration plans gloss over. We approached it by implementing a strict environment isolation policy for the comparison phase.
We created a mirrored staging environment for all external dependencies (databases, APIs, object storage) and used short-lived, scoped credentials that were identical between Jenkins and GitLab runners. For network calls, we recorded a week's worth of traffic from the Jenkins production runs, then replayed it against our staging services during the parallel GitLab execution. This eliminated timing and data-state variances as a source of false positives. The key was treating the migration not as an infrastructure change, but as a full-blown data replication problem.
If you're building a smaller comparison, I'd recommend starting with traffic capture/replay for any outbound calls. For secrets, the only safe path is to generate new, migration-specific credentials for both systems to use against your isolated staging endpoints. Accepting any differences in side-effects means your validation isn't actually validation.
show me the SLA
Totally agree on the mirrored staging and traffic replay. That's the gold standard. But man, creating a full mirrored staging environment can be a cost monster nobody budgets for. I've seen smaller teams get tripped up there.
One hack that helped us was to use VCR-style recording at the network layer *from the runners themselves* for our smaller services. You run the Jenkins job once, capture all the outbound calls, then let the GitLab CI job run against the recorded tapes. It's not perfect for stateful things, but it catches 90% of the "oh, we were calling *that* deprecated API" moments without standing up a whole parallel universe. It also highlights if your new runner environment has different network egress rules, which is a sneaky one.
Your point about generating new credentials is spot on. Using the old prod secrets for testing is like practicing a fire drill by setting your desk on fire.
The VCR hack is clever for catching those blind calls, but it creates a new risk if you're not careful. I used it once and the team got a false sense of security because the tapes were made from a jenkins executor running on a specific subnet. When the GitLab runners finally went live in a different VPC, the firewall rules blocked everything the tapes said should work.
You have to record the network calls from a runner environment that's network-identical to your target, or you're just moving the problem. That means spinning up a temporary runner with the new config anyway, which eats back into some of that cost saving.
Spot on about the egress rules, though. That's killed us on two separate cloud migrations.
Automate everything. Twice.
> you have to record the network calls from a runner environment that's network-identical to your target
This is the precise pivot from a functional test to a full infrastructure validation. The cost of building that network-identical environment, even temporarily, is unavoidable and often the line item that gets cut.
A strategy I've used to contain cost is to treat the network recording as a distinct, finite project. You don't need a full staging suite for this. You spin up a single, short-lived GitLab runner instance in the exact target network (subnet, security groups, route tables), run a curated subset of pipeline jobs that are known to make external calls, and capture the traffic. You then tear it down immediately. The cost is minutes of compute, not months of a full parallel environment. The resulting tapes become your source of truth for allowed egress patterns, which you can then codify into your new environment's security group rules. This validates the network path and pays for itself by preventing production outages from blocked egress.
every dollar counts
Totally makes sense to isolate the network validation as a focused, tear-down task. That "curated subset" is key though - how do you pick which jobs make the cut? I've seen teams miss a critical pipeline that only runs monthly.
One extra step we added was to run a quick analysis on the Jenkins server logs for outbound connections over, say, the last 90 days. It helped us find some obscure API calls from a cleanup script that ran quarterly. Without that, we wouldn't have added it to the recording subset, and it would've blown up later.
Your point about codifying the egress patterns is spot on. We turned the captured traffic into a simple allow-list for the security group, which the platform team loved. It shifted the conversation from "does it work?" to "here's exactly what it needs."
> treat the network recording as a distinct, finite project
See, that's the part that always gets me. A "finite project" implies you know the boundaries, but you never do. You run a subset of jobs, you capture traffic, you tear it down. Then three months later a job you *didn't* record hits an API that changed, or a new feature introduces a call to a different service domain. Your tapes are now stale specs, and the "allow-list for the security group" you built from them becomes a liability.
The cost isn't in the minutes of compute. It's in the organizational delusion that you can ever truly freeze your infrastructure's interactions in time. You just traded one hidden dependency for another.
FOSS advocate