Everyone seems to default to "they're all just Kubernetes, the underlying cloud shouldn't matter." Tell that to my pager after last Thursday's latency spike.
We were migrating a stateful workload between AKS and EKS, same node specs, same pod specs, same everything. The performance disparity in pod-to-pod TCP throughput wasn't subtle. It was the difference between a highway and a dirt road with periodic toll booths.
Let's start with the raw numbers from our `iperf3` runs across pods in different nodes within the same cluster:
**EKS (using the AWS VPC CNI):**
```
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 5.23 GBytes 4.49 Gbits/sec 143 sender
```
**AKS (using Azure CNI):**
```
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 3.41 GBytes 2.93 Gbits/sec 451 sender
```
Notice the retransmits. That's the story right there. Azure CNI's default configuration routes every pod packet through the host network stack for a source NAT check. It's an extra hop, a bottleneck, and a single point of congestion. The AWS VPC CNI assigns a real VPC IP to each pod, so it's essentially native EC2 network performance.
But wait, the Azure evangelist says, "just enable accelerated networking and use the Azure CNI with transparent mode!" Sure. Let's talk about the operational tax on that:
* It requires specific VM series (not all are supported).
* Your cluster creation time doubles.
* You're now locked into a specific subnet sizing strategy that can bite you later during scaling.
The managed control plane is fine, but the data plane is where you live. AKS feels like you're renting a car where the accelerator is connected via a long, elastic band. EKS gives you direct drive, but you better be comfortable managing your own CNI daemonsets and worrying about your IP exhaustion yourself.
So no, they're not "all just Kubernetes." The cloud provider's plumbing is the foundation. Choose based on whether you want raw speed (EKS) or the illusion of simplicity that later becomes a throughput ceiling (AKS).
I'm a platform engineer at a mid-sized e-commerce company, running a mix of event-driven microservices and batch data pipelines on Kubernetes in production, with a heavy focus on inter-service latency.
* **Pod-to-pod throughput:** Your iperf3 results mirror our stress tests. AKS with Azure CNI consistently showed 30-40% lower throughput and 3-4x higher retransmits compared to EKS for East-West traffic. The performance delta became a real constraint for our Kafka brokers and internal gRPC services.
* **Network policy enforcement cost:** Enforcing network policies on AKS (using Calico on Azure CNI) added a measurable 0.5ms-1ms of latency to policy-matched flows. On EKS, using the VPC CNI with Security Groups for pods, the policy enforcement is at the hypervisor layer and we saw no perceptible added latency.
* **IP address management overhead:** The Azure CNI model (where pods consume VNet IPs) requires significant address space planning upfront and became a scaling headache. We had to resize our VNet twice. EKS's VPC CNI has the same constraint, but AWS's larger default VPC CIDR made initial planning less painful.
* **Observability and troubleshooting:** The extra hop in the Azure CNI data path made tcpdump and flow log correlation more complex, needing host and pod-level captures. AWS VPC Flow Logs captured pod-level traffic directly, which cut our average network incident triage time in half.
For raw network performance and operational clarity, I'd recommend EKS for latency-sensitive, high-throughput workloads like your stateful service. The call gets harder if you're deeply integrated with other Azure services (like Managed Identities or ACR), so tell us about your service mesh plans and whether you use other cloud-native services beyond the cluster.
Webhooks or bust.
Your retransmit counts are damning, and you've correctly identified the root cause: Azure CNI's default mode forces all pod traffic through the host's conntrack for SNAT, which is a serialization point that doesn't exist with AWS VPC CNI's direct ENI attachment. We've seen similar numbers in our own tests, though I'd add that the gap narrows if you switch AKS to the Azure CNI Overlay mode (preview, but GA now in some regions) which uses an overlay network and avoids that host stack hop. Still, it introduces encapsulation overhead, so you're trading retransmits for MTU fragmentation.
One thing I'd be curious about: did you run these benchmarks with any monitoring agent overhead? If you're running a DaemonSet for metrics or traces, the agent's own network usage can interact with the conntrack bottleneck on Azure. We've seen customers accidentally increase retransmits further by having heavy log collection agents on the same node. What was your agent footprint during the test?
null