Skip to content
Notifications
Clear all

My results after load testing our app on EKS, GKE, and AKS.

2 Posts
2 Users
0 Reactions
1 Views
(@ci_cd_junkie)
Estimable Member
Joined: 5 months ago
Posts: 134
Topic starter   [#2919]

Alright folks, buckle up. I just spent the last three weeks in a deep, deep rabbit hole, and I have some data that I think will spark a *fantastic* discussion. We've been planning a major regional expansion for our primary customer-facing API, and the directive was to evaluate the big three managed Kubernetes services—AWS EKS, Google GKE, and Microsoft AKS—on a level playing field. Not just "can it run," but how does it hold up under sustained, ugly load?

The goal was simple: deploy an identical application stack (a Go API with a Redis cache and Postgres backend) on each, using their respective managed database/services where possible, and then hammer it with Locust to simulate a realistic traffic pattern ramping up to 10,000 concurrent users. All clusters were configured with three node pools (system, app, cache) on comparable general-purpose VMs (e.g., 4 vCPU, 16GB mem). All used the latest generally available Kubernetes version at the time (1.28). The IaC was, of course, all Terraform, because I'm not a monster.

Here's the high-level application architecture we deployed everywhere:
```yaml
# Kustomize overlay snippet for the app deployment
apiVersion: apps/v1
kind: Deployment
spec:
replicas: 10
template:
spec:
containers:
- name: api
image: our-registry/api:v1.5.0
resources:
requests:
memory: "256Mi"
cpu: "250m"
limits:
memory: "512Mi"
cpu: "500m"
livenessProbe:
httpGet:
path: /health
port: 8080
initialDelaySeconds: 30
periodSeconds: 10
```

Now, for the juicy bits. The load test revealed some fascinating operational differences:

* **Networking & Load Balancer Performance:** This was the biggest differentiator. GKE's native integration with its cloud load balancer was *blisteringly* fast for pod-to-pod traffic and external ingress. The latency under load was consistently 15-20% lower than the others for internal service calls. AKS (using Azure CNI) had the most consistent throughput but required more careful tuning of concurrency limits on its Application Gateway. EKS (with the AWS VPC CNI) was solid, but we observed more TCP connection drops during the steepest ramp-up phase, which we eventually traced to needing to adjust the `aws-node` DaemonSet's `MAX_ENI` and warm-pool settings.

* **Node Autoscaling & Pod Scheduling Speed:** When the Horizontal Pod Autoscaler kicked in and new nodes were needed, GKE's cluster autoscaler was the undisputed winner in reaction time. New nodes were ready for scheduling in ~90 seconds. AKS was close behind at ~2 minutes. EKS took noticeably longer, often 3.5-4 minutes, which forced us to over-provision node pools slightly to handle our spike profiles. The pod scheduling latency itself (kube-scheduler performance) was negligible across all three once the nodes were ready.

* **Observability & Debugging Overhead:** This is more subjective, but the operational feel differed. GKE's built-in metrics in Cloud Operations felt seamless. AKS's integration with Azure Monitor was good, but required more manual dashboard building. For EKS, we had to rely on a full DIY Prometheus/Grafana/CloudWatch agent setup, which is more flexible but adds non-trivial operational and cost overhead just to get to the same starting line.

So, what's the verdict? There's no universal "best." If your priority is raw performance and minimal tuning for rapid scaling, **GKE** felt like the smoothest experience out of the box. If you're deeply embedded in the Azure ecosystem and value consistency, **AKS** is incredibly competent. If you need fine-grained control over the networking layer and are willing to invest the time in tuning, **EKS** is a powerhouse.

I'm left with a bunch of new questions, though. Has anyone else done similar comparative load testing? Did you find specific CNI configurations (like Cilium on EKS) that completely changed the game? I'm especially curious about how these services handle node disruptions and rolling upgrades under load—that's my next test phase.


pipeline all the things


   
Quote
(@late_night_lurker)
Trusted Member
Joined: 5 months ago
Posts: 33
 

Looking forward to the data. Did you run the tests from a single region or distribute your load generators to avoid network latency skewing the results? That detail always trips me up.



   
ReplyQuote