Hey everyone! I'm super excited but also a bit nervous. We just set up our first canary pipeline for migrating from Jenkins to GitLab CI.
The idea is that 10% of our commits are randomly selected to run through the new GitLab pipeline instead of the old Jenkins one. This way, if something breaks, it only affects a small percentage of our deployments and we can catch issues early.
I'd love some advice, though. How do you handle secrets migration safely in a setup like this? We're currently duplicating them, which feels risky. Also, any tips on how you decided on that 10% number? Was it just a guess?
Here's a super simplified look at the rule we used in our `.gitlab-ci.yml`:
```yaml
deploy:
stage: deploy
script:
- ./deploy.sh
rules:
- if: $CI_COMMIT_REF_NAME == "main" && $CANARY_DEPLOY == "true"
```
We use a project variable `CANARY_DEPLOY` that's set to `true` 10% of the time via the API.
Thanks in advance for any wisdom! This feels like a big step.
Congratulations on making this leap! It's a sensible way to migrate pipelines, and that nervous excitement is completely normal. Your instinct about the duplicated secrets is spot on; maintaining two sources of truth is indeed a risk vector for drift and security gaps.
Regarding your 10% question, while it's a common starting point, the ideal percentage often depends more on your deployment volume and risk tolerance than a hard rule. If you have dozens of commits daily, 10% gives you solid signal quickly. If it's just a few, you might bump it higher to get meaningful data sooner. The key is having a clear metric for "success" to decide when to increase the percentage, like five consecutive canary deployments without any rollbacks.
For the secrets, instead of duplication, consider using a dedicated secrets manager (like Vault or even your cloud provider's solution) that both pipelines can reference. This keeps the source authoritative and eliminates sync issues. You can start by having the new pipeline pull from the new manager and the old one from its legacy source, treating that as part of what you're testing. Good luck, and let us know how the first canary runs go!
Stay curious.
That's excellent advice on using a clear metric for success before ramping up the percentage. I'd add that it's also wise to monitor more than just rollbacks, like deployment time differences or downstream system alerts. A successful canary isn't just one that didn't fail, it's one that performed indistinguishably from the old system.
The secrets manager approach is definitely the right direction. One practical step I've seen teams take is to first make the old pipeline read from the new manager, validating that connection works, before flipping the canary pipeline over. It adds a stage, but it de-risks the secret migration itself.