I've been running a mid-sized Kafka deployment on GKE for a while now, and one of the trickiest parts has been balancing cost against latency for our stream processing jobs. We have some workloads that are super latency-sensitive (think fraud detection), and others that are batchy and can wait for cheaper spot nodes. The default cluster autoscaler just wasn't cutting it—it would spin up expensive on-demand nodes for the batch jobs if they asked first.
So, I dove into setting up priority-based expansions. The core idea is simple: you define node groups with different priorities, and the autoscaler will try to scale the highest-priority group that can schedule the pending pods first. This lets you have a "cheap" pool (spot/Preemptible VMs) and a "fallback" expensive pool.
Here's the mental model and a few key config snippets that made it work for us:
* **Node Pool Setup:** In GKE (or your cloud's equivalent), create your node pools. For us, it's `nodepool-spot` (priority 10, spot VMs, tainted) and `nodepool-ondemand` (priority 1, regular VMs).
* **The Crucial Annotation:** Your Pods need to express their *tolerations* for the cheaper pool's taints, and a hint for the autoscaler. This was the "aha" piece:
```yaml
spec:
tolerations:
- key: "instance-type"
operator: "Equal"
value: "spot"
effect: "NoSchedule"
priorityClassName: "batch-job" # Or "latency-sensitive"
```
* **Autoscaler Manifest:** The real magic is in the autoscaler's `expander` configuration. You must set `--expander=priority`. This tells it to respect the `cloud.google.com/instance-type-priority` annotation (or your cloud's equivalent) on the node groups.
The tricky part? Getting the pod disruption budgets and pod affinity right so your latency-sensitive pods *don't* end up on a spot node that vanishes, forcing a reschedule to the on-demand pool and causing a blip. We ended up using an `anti-affinity` against the spot taint for our critical consumers.
Has anyone else implemented this pattern, especially for stateful streaming workloads? I'm curious about how you handle the "fallback" scaling delay when the cheap pool is exhausted.
—Claire
That's a solid approach, and it's great you're sharing the config specifics. The tolerations on the Pods are indeed key to making this work - without them, everything just schedules to the on-demand pool and defeats the purpose.
One nuance we've seen: watch out for node startup time on your spot/preemptible pool. If it's significantly slower than on-demand, your latency-sensitive pods might have to wait longer than you'd like if the "cheap" pool is scaled to zero. Some teams set a minimum node count of 1 or 2 in the spot pool as a buffer for that initial scaling event.
—HR
Great, so now you're paying for spot nodes even when they're idle. That minimum node buffer defeats half the point.
Watch your cloud bill next month.
Your stack is too complicated.
That's a really helpful breakdown, thanks! The priority ordering makes sense now.
> The default cluster autoscaler just wasn't cutting it
I've seen this exact thing happen during our vendor demos. It's so frustrating to watch a cheap batch job trigger expensive nodes.
A quick question, since we're looking at similar setups: how did you handle the resource requests on your batch pods? Did you have to adjust them to fit the smaller spot machine types, or did you keep them the same?
Learning every day
Oh, the resource request dance. That's where the real cost bloodshed happens if you get it wrong.
We kept the requests identical, but it forces you into a specific node shape strategy. If your batch pods ask for 4 CPU and your spot pool uses smaller instance types, the autoscaler will just spin up multiple nodes to fit one pod, which is hilariously wasteful. You end up with a bunch of half-empty, cheap nodes, which kinda defeats the purpose.
Our move was to standardize on a single, larger spot instance type for the "cheap" pool (think `n2d-standard-8`) so the pods fit. That way you're still getting the spot discount, just on a bigger machine. The trick is making sure your latency-sensitive jobs *can't* schedule on that big, cheap shape via taints/tolerations, or your priority order is meaningless. It's a balancing act between granularity and billing complexity.