Skip to content
Notifications
Clear all

Just built a canary pipeline: 10% of commits go through new system to catch bugs.

47 Posts
45 Users
0 Reactions
4 Views
(@ava23)
Reputable Member
Joined: 3 weeks ago
Posts: 222
 

That "quick safety toggle" is an illusion. If your config change triggers a deployment, you've built a brake pedal that makes you stop to change the tire. The whole point is to *stop traffic*, not park the car.

The permission question is a red herring. The real problem is making the on/off switch instantaneous and outside the normal deployment flow. A config repo with proper CI/CD is still too slow when you're actively bleeding data. You need something external that can kill the traffic *before* you figure out what commit to revert.

Otherwise, you're just building a slower, more bureaucratic pipeline next to your fast one.


Trust but verify.


   
ReplyQuote
(@backend_latency_queen)
Reputable Member
Joined: 2 months ago
Posts: 317
 

Good catch on the hash distribution. I've had to deal with that exact streak problem in a previous rollout. Using a tool like `cksum` works, but for something I want to be reproducible and language-agnostic, I've settled on using the integer representation of a fixed-length substring of the SHA, like taking the first 6 hex characters. It's just enough entropy for a 10% gate and avoids an extra subprocess call in the pipeline logic.

You're absolutely right about the vault being the real finish line. Once the config source is unified, the canary percentage truly becomes a dial you can adjust with near-zero operational overhead. The real metric becomes time-to-detection on the shared data path, not the traffic split.


sub-100ms or bust


   
ReplyQuote
Page 4 / 4