Everyone's posting about moving from Jenkins to GitLab or CircleCI. The real story is the black box period after the switch. Your new pipeline is a mystery box until you prove it's stable.
We built a Grafana dashboard to track the migration itself. Key panels:
* Build duration comparison (old vs new platform) for the same jobs.
* Failure rate by pipeline stage, mapped to the new platform's equivalent steps.
* Secrets/credential retrieval errors, because that's always the hidden landmine.
The dashboard pulled from both systems during the parallel run. Took about a week to get the metrics right. The graphs showed us exactly where the translation was bad—turns out our "equivalent" caching strategy wasn't equivalent at all. Saved us from a full cutover with a 40% failure rate.
your mileage will vary
Your approach of instrumenting the migration itself is spot on. Too many teams treat it as a binary event and only monitor the new system after the switch. The parallel run phase is the only time you get a true A/B test.
I'd add that the "build duration comparison" panel is crucial, but can be misleading if you don't also track resource consumption metrics from the underlying nodes. We saw nearly identical build times in our migration, but the new system was consuming 2-3x the CPU seconds due to a configuration mismatch in the ephemeral runner scaling. A simple duration graph would have missed the cost impact entirely.
Your point about secrets retrieval errors is a universal truth. It's often the implicit environmental context that doesn't map. Did you also track the latency of those retrieval calls? We found our new system had added a 500ms overhead per secret due to a different library's connection pooling, which became a major bottleneck for jobs with many secrets.
You're right about duration hiding cost. We had the inverse, new platform showed slower builds but lower infra cost per build hour. That mismatch flagged a config issue where the old system's metrics were inflated by pre-emptible nodes stalling but not reporting it.
We didn't track secret latency granularly, just success/failure. The 500ms overhead per secret you found is brutal. That would have killed our data jobs. Adding that to our checklist.
Your point on CPU seconds is why we added a cost per successful build panel. Duration is just one variable.
Benchmarks don't lie.