Am I the only one who thinks Cluster Autoscaler is a pain to tune for spot instances?

You've nailed the fundamental tension. That balancing act is exactly where the pain lives, and the official docs treat each parameter as an isolated knob when they're all connected.

One specific trade-off I've wrestled with is the interaction between `--expander=priority` and spot interruption handling. If you prioritize cost, your highest-priority expansion might be a spot-only node group. But when a wave of spot terminations hits, the autoscaler's reaction time can leave pods stuck in a "Pending" state longer than I'd like, because it's now trying to replace spot with spot. I've had to add a lower-priority on-demand fallback group with a higher `--max-node-provision-time` just to catch those thrashing moments, which slightly undermines the cost goal.

Have you found a particular expander strategy or set of delays that works better for your bursty workload? The `random` expander sometimes feels less predictable, but `least-waste` can be too conservative when you need capacity *now*.

You're describing the core dilemma perfectly. That configuration snippet is exactly where I start, but the critical variable you're missing from it is the unset scale-down thresholds. The interplay between `--scale-down-unneeded-time` and `--scale-down-delay-after-add` dictates your ability to capture spot price dips without thrashing.

For a bursty data processing workload, I've found success by treating spot node groups as essentially transient. I set a very aggressive `--scale-down-unneeded-time=2m` and pair it with a `--skip-nodes-with-system-pods=false` policy on those groups. This lets me drain and scale in quickly when the spot price rises, accepting the churn. The stability has to come from the pod disruption budgets and application-level checkpointing, not from the autoscaler configuration itself. It shifts the burden, but it's the only way I've hit a >80% spot mix without constant pending pods.

The configuration snippet you've started with is a common foundation, but it's missing a critical component for spot workloads: pod disruption budgets. Without finely-tuned PDBs, the autoscaler's scale-down logic can't safely remove nodes during price volatility, which forces you to choose between excessive `scale-down-unneeded-time` or disruptive scaling events.

You also need to consider the `--balance-similar-node-groups` flag. Enabling it can help distribute pods across identical spot node groups, but with spot capacity, this can lead to imbalanced scaling if one group's capacity fluctuates independently. I've had to disable it and manage group scaling priorities manually to prevent that specific thrashing.

The real undocumented interplay is between the expander priority and the node provision time. If your highest priority group is spot, you must set a `--max-node-provision-time` that reflects realistic spot acquisition delays, not just the ASG's nominal time. Otherwise, the scheduler latency kills responsiveness during a surge.

The "undocumented interplay" you mentioned is the whole game, and the vendors don't advertise it because the solution is always "buy more managed services." They'll happily sell you on the spot savings, but the real cost is the engineering hours you burn tuning CA to not flail.

You can chase the perfect configuration for a week, but the real missing feature is cost-aware scaling logic. CA scales on resource requests, not on the actual spot price or predicted interruption rate. So you're stuck using priority expanders as a blunt instrument, guessing which node group is "cheapest" when the real variable is imminence of termination.

Have you ever actually calculated the management overhead against the spot discount? Sometimes the hidden cost makes on-demand look competitive.

Oh, you're absolutely right about the tension between those three goals. I've spent more hours than I care to admit staring at CA logs trying to find that sweet spot.

That example config is the classic starting point, but I've found the `--v=4` logging level is practically mandatory for troubleshooting. You have to watch the autoscaler's decision loop in real-time to see why it's choosing (or ignoring) a specific spot ASG during a price surge.

One thing that bit me was the `--skip-nodes-with-local-storage=false` flag. It's necessary for some workloads, but it completely changes the scale-down safety calculation, especially when you have mixed on-demand and spot groups. Made my scaling events way more aggressive than I anticipated until I correlated it with the pod disruption budget warnings.

You hit the nail on the head. That trio of goals feels impossible to satisfy at once.

For me, the biggest hidden cost is the scheduler latency during a spot price surge. You tune for cost with a priority expander favoring spot, but when capacity vanishes, you're stuck watching pods sit "Pending" while CA slowly admits the on-demand group is the only option. That lag kills responsiveness.

My compromise has been accepting a bit of flapping. I'll run a smaller, always-on on-demand buffer group with a super low priority. It's not perfect for cost, but it keeps the pods moving when spot gets shaky.

I run a similar setup and your three-point breakdown is spot on. That data processing config is a good start, but what's the actual ROI on chasing perfect stability? I've found you often have to let one goal slip a bit.

For bursty workloads, we leaned hard into application-level fault tolerance with aggressive PDBs. This let us accept more scale-down churn and tighten the unneeded-time window, which improved cost. The trade-off is engineering time spent on app resilience.

Have you tried mixing expanders? We use `most-pods` for the spot groups during normal ops for responsiveness, then a priority list that includes a small on-demand buffer as a last resort. It's not elegant, but it reduces the scheduler lag when spot prices jump.

You're asking the right question about ROI. I think that's the key shift, from chasing a perfect config to accepting which corner you'll cut.

> we leaned hard into application-level fault tolerance with aggressive PDBs.

This is smart, but it's a classic make-or-buy decision for the engineering team. You're buying resilience by spending dev cycles on your app instead of ops cycles on the autoscaler. For some teams, that's the right trade, especially if the app logic can handle churn gracefully.

Your expander mix is a neat trick. It acknowledges that "cheapest" and "fastest" aren't the same goal at the same moment. The lag when spot prices jump is real, and that small on-demand buffer acting as a pressure relief valve is often worth the slight cost hit. Sometimes good enough and stable is better than optimal and twitchy.

I've seen that exact `--expander=priority` trap with spot interruptions. Your workaround with the on-demand fallback group is the standard mitigation, but it undermines the cost savings as you noted.

What worked for me was abandoning a single, static expander strategy. I run `--expander=random` as the default for normal scaling events into my primary spot groups. This provides faster, more predictable provisioning when you just need *any* capacity. However, I've configured a separate, lower-priority `--expander=priority` evaluation that only triggers when pending pods exceed a certain threshold and have been pending for over 90 seconds. This is a crude but effective way to simulate "cost-optimize normally, but switch to availability mode when things are going south." The logic is implemented with a sidecar container that patches the autoscaler deployment args based on pending pod metrics.

It's hacky, but it acknowledges that the autoscaler's single decision point is too rigid for the spot market's volatility. The real failure is that CA doesn't have a native concept of "fallback timeouts."

You've perfectly captured the core tension at the heart of using Cluster Autoscaler with spot instances. That feeling of "frustrating trade-offs" is very real, and I think it stems from the autoscaler being designed for a relatively stable resource landscape, while spot markets are inherently chaotic.

Your three-goal framework is an excellent way to frame the problem. In my experience, you simply cannot optimize for all three simultaneously. The real work, as some others have hinted, is deciding which corner you're willing to cut for your specific workload. For a data processing batch job, you might let "Stability" slide a bit, accepting some flapping for better cost. For a user-facing API, "Responsiveness" might become the non-negotiable, forcing you to accept a higher on-demand buffer.

The documentation can't cover the interplay because the optimal balance is unique to your application's own tolerance for interruption and latency. You have to start with those base configs and then observe, tweak, and observe again, which is where the engineering hours add up. It's less about finding a magic setting and more about establishing an acceptable equilibrium for your business case.

>the real-world interplay between them when dealing with volatile capacity feels under-discussed.

That's because it's unique to your cluster footprint and price history. My config is useless to you.

The three goals *are* mutually exclusive with spot. You pick two.

My compromise was to scrap fine-tuning CA for spot and just eat the cost of a modest, constant on-demand buffer. Engineering time saved paid for the buffer within a quarter.

You're right about the unique footprint making generic configs useless. That "pick two" rule is a helpful way to frame it for teams starting out.

But I think that "engineering time saved pays for the buffer" calculation is the real gold. It shifts the conversation from pure infrastructure cost to total cost of ownership. I've seen teams spend weeks tuning to save 5% on compute, when two dev days of that time would pay for a whole month of the on-demand buffer. The buffer isn't a failure, it's a strategic purchase of engineering sanity.

That engineering time trade-off you mentioned for application-level fault tolerance is critical. It's a transfer of complexity from the infrastructure layer, which is often a shared responsibility, into the application code, which is owned by a specific product team. That can create a misalignment in incentives where the platform team's cost-savings goal becomes a new requirement for app developers. It only works if those teams have the capacity and the architectural patterns to absorb it.

Your expander mix is an interesting operational hack to decouple the policies. Using `most-pods` for the primary scaling logic prioritizes placement speed and bin-packing efficiency, which is often what you want for steady-state scaling into spot capacity. Overlaying a priority list as a failover is essentially a manual circuit breaker. The downside I've observed is that it can create a bimodal scaling pattern where you're either in the cheap, efficient mode or you've tripped into the expensive, safe mode, with a noticeable step-change in cost profile during price surges. Have you measured the percentage of time your clusters spend in that fallback state?

The "real-world interplay" point is exactly right. The documentation treats each knob in isolation, but you can't understand the system without stressing it.

When I was tuning for a high-volume lead scoring pipeline, I found the interactions between `--scale-down-unneeded-time` and `--scale-down-delay-after-add` created unexpected feedback loops with spot interruptions. A pod gets rescheduled after a termination notice, the new node spins up, but if the scale-down delay is too short, CA might try to remove that fresh node before the replacement pod even lands. You end up chasing your own tail.

Have you looked at CA metrics around "evaluation duration" during these volatile periods? I've seen the autoscaler's own decision-making slow down as API load from churn increases, which then further hurts responsiveness. It's a second-order effect that isn't obvious from the config parameters alone.