I've been running our customer-facing analytics workloads on AKS for about a year now. Our base load is predictable, but we get sharp, unpredictable spikes when several clients run their end-of-quarter reports simultaneously. Scaling the main node pool feels slow and expensive for these short bursts.
I've been reading the docs on AKS virtual nodes, which seem to be backed by Azure Container Instances. The promise of scaling out to handle a burst in seconds without managing additional VMs is exactly what we're looking at on paper.
However, I'm wary of the operational gotchas. I've seen a few older threads mentioning network latency between pods on virtual nodes and pods in the main pool, and some constraints on supported features.
For those using this in production:
- How have you found the reliability of the pod scheduling onto virtual nodes during a rapid scale-out event?
- Is the networking overhead actually noticeable for a stateless HTTP service, or is it overblown?
- What was the biggest hurdle in getting your workloads to run on them? I'm thinking specifically around storage or network security policies.
We're a B2B SaaS shop, so my main concern is maintaining performance isolation and reliability during these bursts. Any concrete experiences would be really helpful.
We've been using AKS virtual nodes for a similar burst scenario for about 8 months. The scheduling speed is the real deal, pods spin up on the virtual node in under 30 seconds when the cluster autoscaler is waiting for VM nodes.
On networking, for stateless HTTP services, the overhead hasn't been a problem for us. We're talking maybe 1-2ms extra latency. The bigger issue we hit was with network policies. If you're using Azure Network Policies (not Calico), they won't be enforced on pods scheduled on virtual nodes. That was a gotcha for our security review. Also, remember they can't mount Azure Disk volumes, only Azure Files.
For your use case, I'd suggest a test with a canary deployment that tolerates the virtual node. Set up a Horizontal Pod Autoscaler and watch it during a simulated spike. It's been a cost-saver for us, but you need to work within its fences.
Keep shipping.