What's the best way to monitor Cluster Autoscaler decisions in production?

Ops lead at a 350-person fintech. We run 80+ node GKE clusters with Cluster Autoscaler handling spot and regular node pools. We tried dashboards, then built our own.

**Alerting vs. Dashboarding**: The official Helm dashboard is useless for alerts. It's a static snapshot. You need time-series. We built four custom Grafana panels, but the only one that matters is a stacked graph of `unschedulable_pods_count` per priority class. That shows *why* it's scaling.
**The Critical Log Most Miss**: You're filtering for scale events, but you must also grep for `"msg"="Failed to scale up"` with the sub-string `"no available machines"`. This triggers our pager when cloud quota is hit, which metrics alone won't show for minutes.
**Scale-Down Lag is a Cost Killer**: The metric `scale_down_in_cooldown` is a placeholder. Real delay is in `--scale-down-delay-after-add` (default 10m) plus `--scale-down-unneeded-time` (default 10m). That's 20 minutes minimum before an empty node is even evaluated. We reduced it to 12m total and saved ~18% on a test cluster.
**The Silent Failure Mode**: `cluster_safe_to_autoscale=0` is a catch-all. Root cause is usually a PDB being too restrictive or a pod with `topologySpreadConstraints` it can't satisfy. We alert on this, but the runbook is 20 steps long. The dashboard never helps here.

I'd recommend skipping the canned dashboard. Build one panel for unschedulable pods and alert on the two log patterns (`Failed to scale up`, `no available machines`). Tell me your node group strategy (single pool vs. multiple) and if you use PDBs heavily; my advice changes.

You're dead on about the scale-down lag being a cost killer. That 20-minute default floor caught us off guard, too. We found the bigger waste wasn't even empty nodes, but underutilized ones - we had a bunch sitting at like 15% CPU for those 10 minutes because `scale-down-unneeded-time` was only looking at the average. Setting a lower threshold for `scale-down-utilization-threshold` (we use 0.5) got us another small chunk of savings.

And yes, `cluster_safe_to_autoscale=0` is the worst kind of alert. The last time it fired for us, it wasn't a PDB but a daemonset pod with an overly greedy memory request that was blocking all scale-downs. Took a couple hours to trace through the logs. Do you have a standard drill-down checklist for when that flag flips?

Great question about the checklist. When that flag flips, we run down a quick triage list that's saved us a ton of time. It's basically a search for pods that are "anchored" to a node.

First, check for pods with `cluster-autoscaler.kubernetes.io/safe-to-evict: false`. That's the common culprit, often set on things like monitoring agents. Next, look for any PodDisruptionBudget that's blocking eviction because `maxUnavailable` is zero. After that, we verify daemonset pods, especially ones with large resource requests that can't be fit elsewhere.

But the real time-saver was setting up a log query that runs automatically when the alert fires. It surfaces any log lines from the autoscaler with "not eligible for removal" or "pod with local storage", which points you directly to the blocking pod. Saved us from manually grepping through JSON logs every time.

That automated log query is the only sane way to handle it. The checklist is good, but it's still manual labor for an alert that means your cluster is actively burning money.

We went a step further and built a small dashboard panel that runs a variant of that query continuously, just for the "safe to evict" and "local storage" cases. It lists the offending pod names and namespaces directly. It surfaces the problem before the `cluster_safe_to_autoscale` metric even flips, because the autoscaler logs those evaluation warnings on every loop.

The one thing your checklist misses, and our query initially did too, is the "kube-system pods with no PDB" scenario. Sometimes it's not an explicit annotation or a PDB, it's that the autoscaler simply won't evict certain system pods unless you explicitly tell it to. We added a third regex to catch the log line about "kube-system pod" and now the triage is almost fully automated.

You still have to go kill the pod, but at least you're not spending twenty minutes figuring out which one it is.

Good call on scraping the logs directly. I built something similar, but I found the `"event"="ScaleDownEmpty"` logs were actually hiding the real cost issue for us - they only fire when a node is completely empty.

The bigger savings came from chasing `"event"="ScaleDown"` which includes nodes that are *mostly* empty but have those last one or two system pods. We set up a separate Loki alert for those to track how long nodes are stuck in that state. It's usually a PDB or a daemonset with a big memory request, like you mentioned.

For dashboards, I scrapped the Helm one completely. My main panel is just the `unschedulable_pods_count` stacked by node group, with a log feed from the scale events right below it. Keeps it simple.

Metrics alone are fine if you stop worshiping `increase()`. That alert you wrote will fire on any tiny blip - use `rate() > 0.01` or you're paging yourself for a single transient failure that auto-heals. And you don't need Promtail pipelines to catch "Failed to scale up". The counter `cluster_autoscaler_failed_scale_ups_total` tells you that already. If you want context, correlate with `cloud_resource_quota_exceeded` metrics if your provider exports them. Parsing logs is adding a fragile middleman that breaks every other release. You can get 90% of the value from metrics alone if you build proper histograms of `unschedulable_pods_count` and rate of scale failures. The Helm dashboard is useless, sure, but so is any dashboard that just shows counters. Focus on rate of change, not static numbers. What are you actually alerting on that metrics can't give you within a minute?