We've been running Karpenter in production for about six months, and it's been mostly smooth. However, we've hit a recurring snag during node termination that I'm hoping someone else has solved.
Our setup: Karpenter v0.33.1 on EKS, with a fairly standard provisioner. The issue occurs when Karpenter decides to drain a node for consolidation or because it's unhealthy. If pods on that node are protected by a PodDisruptionBudget (PDB) with `maxUnavailable: 0` (or a low number), the drain process seems to hang indefinitely. Karpenter logs show it's waiting, but it never seems to re-evaluate or force the issue, even when replacement nodes are already provisioned and ready.
* The pods in question are often stateful workloads (like databases in sidecars) where we need a strict PDB.
* We've confirmed the PDBs are correctly configured and allow voluntary disruptions.
* The new replacement node is scheduled and becomes `Ready`, but the old node stays `Ready,SchedulingDisabled` for hours.
This defeats the purpose of consolidation and causes resource bloat. Has anyone else encountered this? I'm particularly curious about:
* Is there a known interaction between Karpenter's drain logic and PDBs that we've misconfigured?
* What's the expected behavior—should Karpenter eventually time out and proceed?
* Are there proven workarounds besides relaxing the PDB (which isn't an option for us)?
We're considering writing a custom controller to intervene, but that feels like we're missing a native solution.