We're running a critical marketing automation stack on AKS. The cluster autoscaler is making demonstrably bad decisions, scaling down nodes while there are pods stuck in `Pending` state due to insufficient CPU. This is causing pipeline delays and alert storms.
Our setup:
- AKS v1.27
- Multiple node pools (system pool, `standard_d4s_v3` for apps, `standard_d8s_v3` for memory-heavy workloads).
- Cluster autoscaler enabled via `--enable-cluster-autoscaler` on the node pools.
- Pods have both resource `requests` and `limits` defined.
- No pod disruption budgets are blocking evictions (checked).
Here's a typical event sequence from `kubectl get events`:
```yaml
LAST SEEN TYPE REASON OBJECT MESSAGE
2m Warning ScaleDown node/aks-apps-1234 node removed by cluster autoscaler
5m Warning FailedScheduling pod/data-sync-xyz 0/5 nodes are available: 3 Insufficient cpu, 2 node(s) had untolerated taint...
```
The autoscaler seems to evaluate a node as "unneeded" based on the sum of requests of *scheduled* pods, ignoring these unscheduled `Pending` pods that are waiting for resources. It scales down, the pending pods then have even fewer nodes to target, and the cycle continues.
What we've tried/verified:
1. Confirmed CA logs show it sees the pending pods (`Pod doesn't fit on any node` messages).
2. No affinity/anti-affinity rules are overly restrictive.
3. `--skip-nodes-with-system-pods=false` is set (we have daemonsets).
Is the root cause the default 10-minute `scale-down-delay-after-add`? Or is there a known issue with how AKS's CA calculates "resource utilization" for scale-down decisions? We need reproducible fixes, not just "restart the autoscaler pod."
What specific metrics or log patterns should we isolate to prove the bug to Azure support? Has anyone solved this with a custom scale-down utilization threshold?
Show me the query.
You're describing a known subtlety of the cluster autoscaler's scale-down decision logic. The autoscaler calculates node utilization by summing the resource requests of pods that are *scheduled and running* on that node. It does not incorporate unscheduled pending pods into that utilization calculation because from its perspective, those pods are not yet placed anywhere. The pending pods are considered only during the scale-up trigger: if they remain unschedulable, the autoscaler will attempt to add nodes. But scale-down evaluations run on a separate timer and look at current utilization of each node. If a node's utilization dips below the scale-down threshold (default 50% for CPU) and the pods on it can be rescheduled elsewhere, it gets flagged as unneeded, even if there are pending pods that *could* have fit on that node.
The key question is why those pending pods are not being scheduled on the very node the autoscaler is about to remove. If the node has free CPU, the scheduler should have placed them there unless there is a constraint blocking it -- node selector, toleration mismatch, pod affinity/anti-affinity, or a topology spread constraint that prefers spreading across nodes. The `FailedScheduling` event you showed lists "3 Insufficient cpu" on other nodes, but it doesn't say why the node that was later scaled down wasn't a candidate. My guess is that either the pending pods have a node affinity or taint toleration that excludes that specific node pool, or the node had some other condition (e.g., a taint that the pending pods don't tolerate) that prevented scheduling even though CPU was available.
I would look at the full pod spec for one of those pending pods, specifically `nodeSelector`, `affinity`, and `tolerations`. Also check the node's taints and labels. If the node being scaled down has, say, a `CriticalAddonsOnly=true:NoSchedule` taint (common on system pools), and your data-sync pod doesn't tolerate it, that would explain the gap. The autoscaler sees the node as underutilized (only system pods running) and decides it's safe to remove, while the pending pods can't use that node anyway.
Another common cause: if your pending pods request more CPU than any single node has available after accounting for the pods already running, the autoscaler might consider the node underutilized yet still not schedule the pending pod there. But that would be caught by `Insufficient cpu` on all nodes, as you see. In that scenario the autoscaler *should* trigger a scale-up event. The fact that it's scaling down instead suggests the pending pods are not even considered as candidates for that particular node due to some scheduling constraint.
One debugging step I'd recommend: temporarily add a pending pod that has *no* constraints and requests a small amount of CPU, then watch whether the autoscaler still scales down that node. If it does, there may be a bug or a misconfiguration in the autoscaler's `--scale-down-unneeded-time` or `--scan-interval` settings. If it doesn't, the constraints on your data-sync pod are the culprit.
Could you share the output of `kubectl describe pod ` for a pod that was in the Pending state when the node was scaled down? Omitting the pod name and objectives, just the scheduling constraints part.
null