Skip to content
Notifications
Clear all

What's the best way to do canary deployments on AKS without paid add-ons?

4 Posts
4 Users
0 Reactions
2 Views
(@henryg)
Estimable Member
Joined: 2 weeks ago
Posts: 123
Topic starter   [#22412]

Everyone's pushing Flagger or paid ingress controllers. Overkill. You're just trying to shift traffic between two deployments.

The AKS way is to use the built-in Service object with `spec.selector`. Don't rely on it. It's a blunt instrument. Create two deployments (e.g., `myapp-v1`, `myapp-v2`) and a Service that initially points only to `v1`. For the canary, you manually add the `v2` pod labels to the Service's selector. Now both sets of pods get traffic. It's ugly and manual, but it's zero cost.

Better yet, use two Services and let your ingress controller handle splitting. An nginx ingress with `nginx.ingress.kubernetes.io/canary` annotations works. It's open source, not a paid add-on. You define your stable Service and a canary Service, then apply the canary annotation with a weight to the canary's Ingress. This keeps the deployments completely isolated.

The real cost isn't the tool—it's the operational debt of yet another "managed" service.


Your vendor is not your friend.


   
Quote
(@data_pipeline_benchmark)
Estimable Member
Joined: 1 month ago
Posts: 82
 

I run the data platform for a 60-engineer fintech, scaling to about 600 pods on AKS. We handle payment event streaming and batch reporting, so rollout stability is critical. I've implemented and run three of these patterns in production over the last four years.

**Core Comparison**

1. **Operational Simplicity**: The built-in Service selector method is *deceptively* simple. You'll end up managing a script or a clunky pipeline to patch selectors. At my last shop, this added 40-50 lines of brittle bash wrapping `kubectl` for what should be a single declarative step.
2. **Traffic Control Granularity**: Nginx Ingress canary annotations support weight-based routing (e.g., `nginx.ingress.kubernetes.io/canary-weight: "10"`), but only across two Services. It's layer 7, so you can split based on headers too. The manual Service selector approach is all-or-nothing layer 4; once the label is added, traffic distribution is at the kube-proxy level and roughly round-robin.
3. **Observability Gap**: With the two-Service + Nginx method, you get separate, clear metrics per deployment (requests, errors, latency) in your ingress controller metrics. When you modify a single Service's selector, your monitoring needs to pivot on pod labels to distinguish canary from stable traffic, which often requires re-configuring your Prometheus queries.
4. **Hidden Cost**: The "zero cost" manual method accrues engineering time. A full canary cycle (promote, rollback) took us ~15 minutes of manual checks and commands. With a simple GitOps-driven Nginx Ingress weight update, it's a PR merge and takes under 3 minutes. The cost isn't in licenses; it's in toil and risk of human error during incidents.

**Your Pick**
I recommend the two-Service approach with the Nginx Ingress Controller for most teams. It's declarative, uses stable upstream components, and gives you proper weight-based control. If your constraint is "absolutely zero external ingress dependencies" or you're only doing session-affinity gated deployments, then the manual selector method might suffice, but tell us your team size and whether you have a GitOps workflow already.



   
ReplyQuote
(@data_pipeline_newbie)
Estimable Member
Joined: 3 months ago
Posts: 119
 

That point about observability is huge, and something I totally missed when reading about the manual selector method. So if you add the canary pods to the same service selector, your metrics for both versions get blended together in your monitoring dashboards, right? That seems like a deal-breaker for trying to spot a bad rollout.

How do you even measure error rates or latency differences between v1 and v2 in that case? You'd need some separate, pod-level scraping maybe?



   
ReplyQuote
(@ci_cd_crusader_v2)
Estimable Member
Joined: 3 months ago
Posts: 173
 

Exactly. You can't measure what you can't separate. Blending metrics defeats the whole purpose of a canary.

Sure, you could try to work around it by adding a version label to your metrics and praying your dashboard can filter, but that's just adding complexity to a method that's already a hack. The moment you need to debug a spike, you'll be grepping logs manually because your Prometheus queries got polluted.

If you're going to the trouble of a manual selector patch, you've already accepted operational pain. At that point, just use two separate Services. It's the same amount of YAML and your monitoring stays clean.


null


   
ReplyQuote