Okay, maybe I'm missing something obvious here, but I've been trying to get Cluster Autoscaler working reliably with a mix of spot and on-demand nodes for a few weeks now, and it feels way more complex than it should be.
The docs make it sound straightforward: set some priorities, define some expansion policies, and you're good. But in practice, I'm constantly battling with either over-provisioning (and blowing our budget) or under-provisioning (and having pods stuck pending forever because spot capacity isn't available). The whole "balance similar nodes" and "skip nodes with local storage" logic gets really fuzzy with spot.
For example, my main pain point is the delay. Spot termination happens, CA sees the pending pods, but then it seems to hesitate before deciding to scale up a replacement. By the time it does, the workload is already degraded. I've been tweaking `--scan-interval`, `--max-node-provision-time`, and the `priority expander` weights, but it feels like guesswork.
Is this just a rite of passage? Are there specific settings for spot-heavy clusters that I should be looking at? I'm especially curious about how folks handle different instance families – do you configure multiple node groups for different spot types, or rely on a single, flexible group? Any real-world config snippets or lessons learned would be super helpful. I'm worried I'm overcomplicating this.
Oh man, this is really timely for me! I've been trying to move some of our non-critical Asana automation workloads to spot instances on GKE to save costs, and I've run into the same hesitation problem you mentioned. The delay feels unpredictable.
I'm so new to this that I've just been using the default settings - I didn't even know about tweaking the `--max-node-provision-time`. Could that delay also be because it's waiting to see if another spot instance becomes available first, before maybe falling back to on-demand? That's what I'm guessing happens sometimes.
How do you decide when to just give up on spot for a particular pod and let it schedule on-demand instead? Is that something you configure in the autoscaler itself, or in the workload? I'm worried about blowing our budget too.
You're absolutely right about the tuning feeling like guesswork; it often is because the defaults are built for stable, on-demand environments. The delay you're describing is a fundamental trade-off in the design. The autoscaler evaluates scaling actions on its scan interval, but for spot, the more critical setting is often the `--max-node-provision-time`. For spot instance pools, you should set this significantly lower than the default (often 15 mins or less) to force a faster fallback to your on-demand expansion option. Otherwise, it will wait the full duration hoping for the spot capacity to materialize.
Regarding instance families, this is where the priority expander configuration becomes crucial. You can't just rely on a single 'spot' priority. You need to define multiple expansion options in your node groups, each with different instance types and pricing models, and assign them priority weights that reflect your cost vs. speed tolerance. For example, give your preferred spot instance type the highest priority, but include a diverse set of other spot families (with lower priority) and a final on-demand fallback. This reduces the "hesitation" by giving the autoscaler a clear, ordered list of alternatives to try when the first choice is unavailable.
What's your fallback strategy configured as in the expander? A common pattern is to use a priority list like: spot-preferred-family, spot-alternate-families, then on-demand. This structure specifically addresses the under-provisioning problem when a specific spot capacity is gone.
Wow, that's a really clear explanation, thank you. So it's not just about lowering one time setting, but giving it a whole menu of backup options to pick from quickly. That makes the "fallback" logic you mentioned click for me.
I have a follow-up, though. When you create those multiple expansion options with different priorities, how do you handle labeling or taints so the right workloads land on the right nodes? I'm worried that if I give it too many choices, pods might end up scheduling on a less-optimal instance type just because it was available first, even if a cheaper spot option becomes available a minute later. Does the autoscaler rebalance after the fact, or is that something I'd have to manage separately?