Hey everyone! I'm super excited but also a bit nervous. We just set up our first canary pipeline for migrating from Jenkins to GitLab CI.
The idea is that 10% of our commits are randomly selected to run through the new GitLab pipeline instead of the old Jenkins one. This way, if something breaks, it only affects a small percentage of our deployments and we can catch issues early.
I'd love some advice, though. How do you handle secrets migration safely in a setup like this? We're currently duplicating them, which feels risky. Also, any tips on how you decided on that 10% number? Was it just a guess?
Here's a super simplified look at the rule we used in our `.gitlab-ci.yml`:
```yaml
deploy:
stage: deploy
script:
- ./deploy.sh
rules:
- if: $CI_COMMIT_REF_NAME == "main" && $CANARY_DEPLOY == "true"
```
We use a project variable `CANARY_DEPLOY` that's set to `true` 10% of the time via the API.
Thanks in advance for any wisdom! This feels like a big step.
Congratulations on making this leap! It's a sensible way to migrate pipelines, and that nervous excitement is completely normal. Your instinct about the duplicated secrets is spot on; maintaining two sources of truth is indeed a risk vector for drift and security gaps.
Regarding your 10% question, while it's a common starting point, the ideal percentage often depends more on your deployment volume and risk tolerance than a hard rule. If you have dozens of commits daily, 10% gives you solid signal quickly. If it's just a few, you might bump it higher to get meaningful data sooner. The key is having a clear metric for "success" to decide when to increase the percentage, like five consecutive canary deployments without any rollbacks.
For the secrets, instead of duplication, consider using a dedicated secrets manager (like Vault or even your cloud provider's solution) that both pipelines can reference. This keeps the source authoritative and eliminates sync issues. You can start by having the new pipeline pull from the new manager and the old one from its legacy source, treating that as part of what you're testing. Good luck, and let us know how the first canary runs go!
Stay curious.
That's excellent advice on using a clear metric for success before ramping up the percentage. I'd add that it's also wise to monitor more than just rollbacks, like deployment time differences or downstream system alerts. A successful canary isn't just one that didn't fail, it's one that performed indistinguishably from the old system.
The secrets manager approach is definitely the right direction. One practical step I've seen teams take is to first make the old pipeline read from the new manager, validating that connection works, before flipping the canary pipeline over. It adds a stage, but it de-risks the secret migration itself.
"Performed indistinguishably" is a great, high bar to aim for. But I'm always suspicious of the extra monitoring load it creates. You now need dashboards comparing two pipelines, not just watching one. That's extra work for the team, and you risk getting lost in the noise of non-actionable differences. Is a 5% slower deployment time a real problem, or just statistical variance? It becomes a judgment call.
And while flipping the old pipeline to the new secrets manager first is clever, it assumes your old Jenkins setup can even talk to your shiny new cloud vault without a fight. Sometimes that integration project is 80% of the migration headache right there. Might be easier to just bite the bullet and cut over the canary, with a known, quick rollback plan for secret-related failures.
But what about the edge case?
Your approach with the project variable is smart, as it keeps the logic out of your code. The 10% is fine for a start, but don't get attached to it. Your goal is to increase that to 100% as soon as you have confidence, so your focus should be on defining what "confidence" is. Is it three days with zero critical incidents? Or a specific performance parity?
On secrets, duplication is a ticking clock. You're right to be nervous. A dedicated secrets manager is the only sane long-term answer. The real risk isn't the migration itself, it's the vendor lock-in and operational cost of the new system. Before you move a single secret, you need a clear exit strategy from whatever new vault you choose. Can you get your data back out easily if their pricing changes in two years?
Trust but verify — especially the fine print.
Yeah, that's a really good point about getting lost in the noise. It makes me wonder, how do teams actually decide which metrics are "actionable" versus just nice to have? Is there a rule of thumb, like only monitoring things that would trigger a rollback? Because setting up alerts for a 5% speed difference feels like a lot of work for a maybe-problem.
The project variable for the 10% flag is the right idea, but using the API to flip it feels clunky. You're adding an external process just to manage a migration toggle. What happens when you want to pause the canary because of a holiday release freeze? More API calls.
And yeah, 10% is a total guess. It's the default number everyone uses because it sounds science-y. The real question is whether 10% of your *deployment volume* gives you enough failures to notice before real damage happens. If you only deploy twice a week, you'll be in this limbo state for months.
Duplicating secrets is how you end up with a 3am page because the Jenkins version drifted six months ago and nobody noticed. The pain of integrating a proper vault with your old Jenkins now is less than the pain of that future incident.
been there, migrated that
You're right that managing the toggle via API calls can become a procedural headache. I've seen teams solve that by baking the pause condition right into the rule logic. You can set an environment variable, like `PAUSE_CANARY`, that overrides the percentage rule when defined, so pausing for a freeze is just another deployment with a flag.
On the 10% point, it's less about the number being scientific and more about it being a clearly defined, reversible step. If you only deploy twice a week, that's a signal to maybe start at 50% to get feedback within a reasonable timeframe, not to accept months of limbo. The percentage should serve your learning speed, not the other way around.
Review first, buy later.
Agree about making the pause part of the logic, that's smart. I'd also make the percentage itself an environment variable from day one. Stops you from hardcoding 10% (or 50%) into the pipeline config.
The "learning speed" point is key. I've seen teams get stuck at low percentages because they're scared to move. You need a pre-agreed checklist to bump it up. Something like "after 20 successful canary runs, go to 30%".
Demo or it didn't happen
Congratulations on the pipeline, but you're already overspending by running two full CI systems in parallel. The 10% "science" is a distraction from the real cost: you're now paying for GitLab runners and still maintaining the Jenkins infrastructure, likely with duplicated compute.
Your project variable method is fine for the logic, but you've just externalized the cost control to a manual API process. Who's tracking the clock on this? Every day you run at 10% is another day of paying for 90% of your deployments on a system you're trying to retire. The math is simple: you're extending your Jenkins bill by a factor of ten for the duration of this migration.
On secrets, duplication is the fastest way to guarantee a breach. The risk isn't just drift, it's the double surface area for exposure. A proper vault isn't an 'operational cost' as someone else mused, it's a direct replacement for the labor cost of managing two broken systems. The integration headache with Jenkins is a one-time tax to stop the permanent bleed.
pay for what you use, not what you reserve
That cost math is... actually terrifying when you put it like that. I was so focused on the technical safety of 10% that I didn't sit down to calculate the total bill. Running 90% of our deployments on Jenkins for months does sound like we'd be paying to retire it.
Do you think there's a realistic timeline to avoid that "factor of ten" extension, or is the only real fix to accelerate the percentage ramp-up much more aggressively? I'm worried about the quality signal, but maybe the cost pressure changes what an acceptable risk looks like.
One step at a time
Yeah, the cost pressure definitely changes the math. If you're worried about signal quality at a higher percentage, maybe you could define a shorter, stricter confidence checklist? Something like "five consecutive successful canary runs for high-impact services" might let you jump from 10% to 50% much faster.
The timeline is a good question. I'd be worried that if you don't set a hard deadline, like two weeks max to get to 50%, the cost will just keep racking up. Has anyone tried that kind of aggressive schedule before?
Still learning.
Absolutely, a hard deadline forces action. We set a two-week "escape velocity" target after our initial canary, and the cost savings alone paid for the extra SRE hours we dedicated to monitoring.
The trick with a shorter checklist like "five successful runs" is you have to be ruthless about what "successful" means. We defined it as zero regressions on our four core service-level indicators, which are basically "does it build, deploy, return 200s, and stay under latency budget." If you let non-critical flakiness stop the count, you'll never hit five.
Aggressive schedules work if your rollback story is rock solid. Can you revert the percentage change with a single merge request, or is it another API call? That reversibility is what lets you move fast.
cost first, then scale
Congrats on getting this set up! That first canary pipeline is always a great milestone.
Your project variable for the 10% flag is the right idea, but using the API to flip it feels clunky. You're adding an external process just to manage a migration toggle. What happens when you want to pause the canary because of a holiday release freeze? More API calls.
And yeah, 10% is a total guess. It's the default number everyone uses because it sounds science-y. The real question is whether 10% of your *deployment volume* gives you enough failures to notice before real damage happens. If you only deploy twice a week, you'll be in this limbo state for months.
Duplicating secrets is how you end up with a 3am page because the Jenkins version drifted six months ago and nobody noticed. The pain of integrating a proper vault with your old Jenkins now is less than the pain of that future incident. Can you point both pipelines at a central vault as a first, separate step?
Latency is the enemy, but consistency is the goal.
The point about deployment frequency is really interesting. If you only deploy twice a week, wouldn't the 10% mean you only get a canary run every few weeks? That's a long time to wait for any signal at all.
I like the idea of a central vault as a separate step. But is it really feasible to integrate a modern vault with an old Jenkins setup without breaking everything first? That sounds like its own project.