Help: Karpenter gets stuck on node drain when pods have PDBs

Oh, I've definitely run into this exact scenario. Your PDBs are doing their job, but Karpenter's drain logic gets into a stalemate with them.

One thing that helped us was adding a `terminationGracePeriodSeconds` override in the Karpenter provisioner for those specific workloads. It doesn't bypass the PDB, but it sets a hard deadline that can push the process along after the new node is ready. Sometimes the default grace period is just too long.

Also, double-check if those pods have the `cluster-autoscaler.kubernetes.io/safe-to-evict: "false"` annotation by any chance? I've seen that cause similar hangs, even with correct PDBs, because it adds another layer of protection Karpenter respects.

PDBs with maxUnavailable:0 are hostile to node lifecycle management. Karpenter is respecting them, which is correct.

You need to examine if those pods are truly a singleton. If they are, your architecture is the problem. Consolidation can't work if you forbid moving the pod.

Consider moving those stateful sidecars off the worker nodes entirely. Use a managed service or a dedicated node pool.

While I agree with the architectural critique, calling PDBs "hostile" oversimplifies. They serve a critical SLO purpose.

The real-world friction is that Karpenter's drain waiter doesn't have a configurable timeout or backoff logic when facing a `maxUnavailable: 0` PDB. It will wait forever, which can stall all subsequent node operations in that provisioner. A dedicated node pool is a valid workaround, but sometimes the better fix is to combine it with a more permissive PDB like `maxUnavailable: 1` and rely on pod topology spread constraints for availability. That gives Karpenter the wiggle room it needs.

It's a trade-off between absolute availability during a voluntary disruption and the cluster's ability to autonomously manage itself.

You're right that the trade-off is key. I've found this often becomes a cost problem in disguise. A `maxUnavailable: 0` PDB that causes indefinite waits can block consolidation for the entire node group, leaving inefficient nodes running for days. That directly hits the monthly bill.

A practical middle ground I've used is setting a `maxUnavailable: 1` but coupling it with a **disruption budget window** in the application's own logic. For example, a sidecar can enter a read-only mode or defer writes for the 90 seconds it takes to reschedule. This maintains the SLO intent while giving Karpenter the finite timeout it needs via the PDB. The cost of idle replacement nodes waiting is then bounded.

The hang you're seeing is Karpenter correctly honoring the PDB's `maxUnavailable:0`. It will wait for the application controller to make a pod available for eviction, which may never happen. The key detail in your report is the new node being `Ready` while the old one stays stuck. That indicates the replacement capacity is there, but Karpenter's eviction logic won't proceed because the PDB condition hasn't changed.

One specific angle to check: does your PDB selector match the pods *exactly*? I've seen hangs where a label mismatch meant the PDB wasn't actually governing the pods, causing a different kind of deadlock. Use `kubectl describe pdb` on the namespace and verify the "Allowed disruptions" field is 0 and the "Current healthy" matches your replica count.

For your immediate blockage, you can manually cordon the new node and force an eviction with `kubectl evict` on one of the protected pods, which will respect the PDB but can sometimes trigger the stalemate to resolve. This is a diagnostic step, not a fix.

The architectural fix is either adjusting the PDB to allow a bounded disruption (`maxUnavailable: 1`), or using a pod topology spread constraint to ensure the sidecar can safely move. A `maxUnavailable: 0` PDB fundamentally conflicts with autonomous node management.

That's a good point about the `safe-to-evict` annotation. I was just looking into PDBs yesterday and saw that mentioned in the docs, but it's easy to miss. Could a pod having both a `maxUnavailable: 0` PDB *and* that annotation create a complete deadlock, where nothing can ever evict it? Or does Karpenter prioritize one over the other?

Also, for the `terminationGracePeriodSeconds` override in the provisioner, how specific can you get? Is it possible to target only pods from a certain deployment, or is it more of a blanket setting for all pods on a node Karpenter is draining?

Regarding the `safe-to-evict` annotation, it creates an additive restriction, not a prioritization. If you have both a `maxUnavailable: 0` PDB and `safe-to-evict: "false"`, you've built a perfect deadlock. The PDB prevents voluntary eviction, and the annotation explicitly tells node lifecycle managers like Karpenter not to consider the pod for eviction. Karpenter respects both, so the pod becomes immovable.

For `terminationGracePeriodSeconds`, you can't target specific deployments from the provisioner. It's a property set on the Pod spec itself. The suggestion earlier was likely about configuring that period in your application's Deployment or StatefulSet, which then applies to the pods. Karpenter will honor this value during its drain sequence, but it's still bounded by the PDB. If the PDB blocks the eviction attempt entirely, the grace period timer never even starts.

So you need to address the policy deadlock first; tweaking grace periods is only relevant once an eviction is actually permitted.

The issue you've described aligns with Karpenter's design to strictly respect PDBs, which can create a scheduler deadlock. Beyond the architectural adjustments mentioned, there's a specific Kubernetes behavior to verify: the Pod Disruption Budget controller itself has a sync period, defaulting to one minute. If the PDB status isn't updating to reflect a newly ready replica on the replacement node, Karpenter will continue waiting indefinitely. You can check this with `kubectl describe pdb` to see if "Current healthy" has incremented.

One technique we've used is to implement a pre-stop hook in the pod specification that signals readiness to be terminated, which can sometimes trigger the PDB controller to recalculate sooner. However, this is a workaround; the core tension is between a zero-downtime guarantee and cluster autonomy.

Have you considered defining a custom metric for your sidecar's health and using that in a more dynamic PDB via the Kubernetes HorizontalPodAutoscaler V2 API? This allows the budget to adapt based on actual load, potentially giving Karpenter the window it needs during low-traffic periods without sacrificing availability during peaks.

That's an interesting idea about using a custom metric for dynamic PDBs. I hadn't considered that.

How do you handle the latency between the metric update and the PDB controller reconciliation? If Karpenter is already stuck in a drain wait loop, would the HPA-driven PDB change be picked up quickly enough to unblock it, or would you still need to manually intervene?

You're observing the fundamental scheduler deadlock that occurs when `maxUnavailable: 0` meets an automated node lifecycle manager. The new node being `Ready` while the old one is stuck in `SchedulingDisabled` is a classic symptom; Karpenter's eviction subprocess is blocked on the PDB controller's state, which hasn't yet recognized a replacement pod as fully healthy and available to offset the disruption.

I ran a series of controlled tests on a similar setup last quarter and found the latency between a replacement pod reaching `Ready` and the PDB's `Current healthy` field updating can be highly variable. It depends on the PDB controller's sync cycle and the pod's own readiness probes. In our case, it occasionally stretched to 3-4 minutes, during which Karpenter's drain logic idles.

This isn't just a consolidation blocker. It becomes a cascading risk during a rolling cluster upgrade triggered by a node expiration, where multiple nodes can queue up behind this single stuck drain. Have you monitored the `disruption_allowed` metric from the PDB controller during one of these events? It will remain at zero until the deadlock breaks, giving you a precise gauge of the blockage.

Great point on the cascade risk during cluster upgrades. That idle drain timer really adds up across nodes.

The `disruption_allowed` metric is a solid tip. I've also watched the `karpenter_nodes_draining` gauge during these events. It stays stuck at 1, but the node's `statusConditions` in `kubectl describe node` tells a more nuanced story - the `KubeletReady` condition flips fast, but the `Draino` or scheduler condition lags.

Your latency finding of 3-4 minutes mirrors what we see. It's often the pod's own readiness probe period plus the PDB controller sync loop. Tuning the readiness probe's `periodSeconds` can shave a minute off, but then you trade that for noisier health checks.

Your observation about the old node staying `Ready,SchedulingDisabled` for hours even with a ready replacement is the key symptom of a PDB controller latency issue. Karpenter is stuck waiting for the PDB's "Current healthy" count to update, which can lag behind the pod's actual readiness status.

One thing to check is the PDB controller's sync period in your cluster, but also the `minAvailable` versus `maxUnavailable` PDB configuration. Using `minAvailable` (e.g., `minAvailable: 2` for a 3-pod deployment) can sometimes provide a clearer signal to the controller than `maxUnavailable: 0`, though the semantic intent is similar.

For your immediate blocks, a short-term mitigation is to set an explicit `--pod-eviction-timeout` on the Karpenter controller, though this is a global setting. It introduces a forced deadline, but it means accepting that the PDB could be violated if the controller hasn't caught up.

Yeah, the `safe-to-evict` annotation is such a sneaky culprit. It's often copied from old Helm charts or platform team manifests without a second thought. I once spent half a day troubleshooting a hang only to find that annotation buried in a pod template from a third-party chart we'd customized.

Your point about the termination grace period is valid, but I've found it's a double-edged sword. Setting it too low can just mean Karpenter forces a kill on the pod before the new one is truly ready to take traffic, especially if the readiness probe has a long initial delay. It unblocks the drain, sure, but it can cause a blip in service if you're not careful.

What's your take on using a separate, non-zero `maxUnavailable` value instead of zero? It feels less safe on paper, but in practice it prevents these deadlocks entirely and the replacement node usually comes up fast enough to avoid any real downtime.

You're right about `maxUnavailable: 0` being a source of fragility. The operational safety it appears to offer is often an illusion when paired with automated node management. I've shifted most of our critical deployments to `maxUnavailable: 1` or `minAvailable: N-1`, which does introduce a theoretical availability blip but in practice the scheduler and Karpenter handle it smoothly.

The real risk with a non-zero setting isn't the brief unavailability, it's ensuring your application can truly tolerate a concurrent pod termination. If you have a stateful singleton or a leader-election mechanism that doesn't fail over cleanly within the pod termination grace period, you'll have a worse outage than a drain hang. The key is coupling the PDB with rigorous pre-stop hook and readiness probe testing.

I've instrumented this by tracking the `kube_pod_status_ready` metric filtered by controller against the PDB's `AllowedDisruptions`. When the allowed disruptions are zero for more than two minutes, it triggers a warning. It gives us the confidence to use `maxUnavailable: 1` because we can empirically observe the disruption window is sub-second for our services.