Help: Karpenter gets stuck on node drain when pods have PDBs

The disruption budget window in app logic is a clever approach. I've tried something similar with a sidecar that flips a deployment's pod spec label when it gets a SIGTERM, which triggers an HPA scale-up to cover the window. But then you need to coordinate the scale-down after the new pod's ready.

It adds some moving parts, but it does bound the cost like you said. Have you seen any issues with the sidecar itself getting killed before it can signal the app to enter read-only mode?

We saw the same thing in our cluster. The new node being ready while the old one is stuck draining is exactly the PDB controller latency others have mentioned. For our stateful sidecars, we added a simple readiness gate that the pod only passes after it's confirmed it can serve traffic, which made the "current healthy" count update much faster for the PDB controller.

Have you looked at the events on the PDB itself with `kubectl describe pdb` during one of these hangs? It often shows the controller still waiting, even though a replacement pod looks ready from a kubelet perspective.

You've hit on a very common pain point. The hang isn't Karpenter being broken, it's waiting for a signal from the Kubernetes PDB controller that never arrives quickly enough when `maxUnavailable: 0` is set.

The new node being ready while the old one is stuck in `SchedulingDisabled` is the classic symptom. The PDB controller's "current healthy" count lags behind the pod's actual readiness status. I've seen this latency stretch to several minutes, as others noted, dictated by the PDB sync period and your pod's own readiness probe timing.

A short-term mitigation is to set a `pod-eviction-timeout` on the Karpenter controller to bound the wait. For a longer-term fix, consider if you can shift from `maxUnavailable: 0` to `minAvailable: N-1`. It provides a similar safety guarantee for your stateful sidecars but gives the system a clearer path forward during drains.

N-1 is the right pattern, but it still needs tight readiness probes. A pod marked ready by a kubelet might not be ready for the PDB controller's math. That's where the hang happens.

Setting a pod-eviction-timeout just papers over the controller lag. It might force the drain, but you risk killing a pod before its replacement is truly in-service if your probe intervals are long.

Exactly right. That readiness probe gap is where the PDB controller gets stuck. We started annotating our pods with a custom condition that the PDB controller can read - it's a bit of a hack, but it bridges that "kubelet ready" vs. "app ready" delay.

Have you seen the `pdb-controller` logs during one of these hangs? They often show the sync loop waiting on the exact pod you think is ready, but the pod's `Ready` condition in the API hasn't propagated yet.