We've been running the Kubernetes Cluster Autoscaler (CA) version 1.27.3 on our AWS EKS cluster for several months, and a persistent issue has begun impacting our production workloads. Despite our configuration, the autoscaler repeatedly attempts to scale down nodes that have running, non-evictable pods. This results in a continuous loop of scale-down simulations logged as blocked by pods, causing unnecessary churn and preventing legitimate scale-in operations. I've conducted a thorough review of the documented scale-down constraints and believe our pod configurations are compliant, yet the behavior persists.
Our primary goal is to achieve efficient cluster resource utilization, but we must safeguard stateful workloads and pods with local storage. The cluster uses a mix of node groups: one for general purpose applications and a separate, labeled group for stateful services. The problematic scale-down attempts are consistently targeting nodes in the general purpose group.
Here is a simplified version of our Cluster Autoscaler deployment configuration, with sensitive details redacted:
```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: cluster-autoscaler
namespace: kube-system
spec:
...
template:
spec:
containers:
- command:
- ./cluster-autoscaler
- --v=4
- --stderrthreshold=info
- --cloud-provider=aws
- --skip-nodes-with-local-storage=false
- --expander=least-waste
- --node-group-auto-discovery=asg:tag-key=k8s.io/cluster-autoscaler/enabled,tag-value=true
- --balance-similar-node-groups
- --scale-down-unneeded-time=10m
- --scale-down-delay-after-add=10m
- --scale-down-unready-time=20m
```
The most telling evidence comes from the CA logs. The following pattern repeats every 30 seconds for specific nodes (e.g., `ip-10-0-5-122.ec2.internal`):
```
I0920 14:45:32.987456 1 scale_down.go:789] Node ip-10-0-5-122.ec2.internal - utilization 0.825000
I0920 14:45:32.987512 1 scale_down.go:799] Node ip-10-0-5-122.ec2.internal is not suitable for removal - predicate checking failed: non-movable kube-system pod: aws-node-xyz12
I0920 14:45:32.987532 1 scale_down.go:799] Node ip-10-0-5-122.ec2.internal is not suitable for removal - predicate checking failed: non-movable kube-system pod: kube-proxy-abc34
```
My analysis of the pods mentioned (`aws-node`, `kube-proxy`) confirms they do not have `cluster-autoscaler.kubernetes.io/safe-to-evict: "false"` annotations, nor do they explicitly request local storage. They are, however, critical system pods. According to the CA documentation, DaemonSet pods are non-evictable by default, which is correct. The issue is that these nodes also host several other evictable, low-utilization pods. I expected the CA to cordon the node and evict those other pods first, freeing the node for removal, but it seems to evaluate the node as unsuitable immediately due to the presence of any non-evictable pod.
My questions for the community are:
* Is the observed log behavior (`non-movable kube-system pod`) the definitive block, or is the CA merely listing *one* reason among many? Could there be other hidden constraints not being logged at this verbosity level?
* Has anyone successfully configured CA to drain nodes that host system DaemonSet pods, provided all other pods are evictable? Does the `--skip-nodes-with-system-pods` flag (defaulting to true) entirely preclude scaling down such nodes, or does it only apply to pods in the `kube-system` namespace with a non-null `PriorityClassName`?
* What is the recommended pattern for achieving scale-down on general-purpose worker nodes that, by necessity, run the AWS VPC CNI and kube-proxy DaemonSets? Must we isolate these system components to a dedicated, non-scalable node group?
I am prepared to provide additional log snippets, node descriptions, and pod Disruption Budget configurations if it aids in debugging. My objective is to reconcile the theoretical model of the Cluster Autoscaler with its operational behavior in a real-world, multi-tenant EKS environment.
-ck
That's a classic pain point. The CA's logs saying a scale-down is "blocked by pods" can be a bit misleading; it's often not about eviction policy, but about whether the scheduler can find a suitable *destination* for those pods on remaining nodes.
Since you're targeting the general purpose group, double-check the pod-to-node affinities and tolerations. Even a `nodeSelector` for a generic label that's *only* on that specific node group will pin a pod there. If those pods can't tolerate the taints on your stateful group, the CA sees them as unmovable.
Also, if those pods have requested resources (CPU/memory) that simply don't fit on any other node after the hypothetical removal, the simulation will fail. It's worth checking if any of these pods have unusually large requests, even if their actual usage is low.
Keep it civil, keep it real.
The affinity point is valid, but it assumes a stable node label landscape. What happens when a node's labels get stripped during a rolling update? The CA's simulation sees a movable pod, scales the node down, and your workload ends up stranded. Seen it happen with a daemonset that managed custom labels.
Your focus on resource requests misses the more common trap: pod disruption budgets. A PDB blocking voluntary evictions will stop the scale-down cold, and the logs aren't always clear about it.
Doubt everything
You're right about the simulation needing a viable destination, but focusing solely on CPU and memory requests can overlook the actual constraint. The autoscaler also accounts for allocatable resources after subtracting node system daemons. A pod with a large, standard request might fit numerically, but if the destination node has a high allocatable overhead, the simulation still fails.
Check the allocatable capacity versus total capacity on your nodes. A disparity there, often from kube-reserved or system-reserved settings, means a node with 8GB total might only offer 6GB to pods. That's where the "doesn't fit" calculation happens, even if the raw numbers look fine.
independent eye