I've seen this pattern suggested a few times in vendor blogs as some kind of "advanced hybrid strategy," usually with a heavy dose of marketing fluff about "intelligent failover" and "maximizing resilience." Let's cut through that. The core idea—using Karpenter for rapid spot node provisioning while keeping Cluster Autoscaler (CA) for reliable on-demand fallback—is sound, but the implementation details are where you'll get burned. It's not a magic bullet; it's a specific operational trade-off for cost-conscious workloads that can tolerate a complex, multi-controller setup.
The primary use case is when you have a baseline of on-demand nodes managed by CA and you want Karpenter to handle burst or batch workloads on spot instances, with the understanding that if spot capacity is revoked or unavailable, Karpenter's pods will eventually land on the on-demand pool, which CA can then scale up if needed. You're running two node group concepts: one static group for CA (e.g., an AWS Managed Node Group) and Karpenter provisioning from its own provisioner specs.
Here’s the critical configuration most get wrong. You must set disjoint scheduling constraints to prevent the systems from fighting. Karpenter must NOT be allowed to schedule pods that are intended for the CA-managed node group, and vice-versa. This is done via node selectors and taints/tolerations.
Example: Your CA-managed node group should have a dedicated label/taint.
**CA-Managed Node Group (e.g., in Terraform for an EKS Managed Node Group):**
```yaml
# This taint is key. Karpenter pods won't tolerate it by default.
taints = [
{
key = "capacity-type"
value = "on-demand"
effect = "NO_SCHEDULE"
}
]
labels = {
"node-group-type" = "ca-managed"
}
```
**Karpenter Provisioner manifest:**
```yaml
apiVersion: karpenter.sh/v1beta1
kind: Provisioner
metadata:
name: default
spec:
requirements:
- key: node.kubernetes.io/instance-type
operator: In
values: [c5.large, c5a.large, c6i.large]
- key: topology.kubernetes.io/zone
operator: In
values: [us-east-1a, us-east-1b]
- key: karpenter.sh/capacity-type
operator: In
values: ["spot"]
# This is the crucial part: Karpenter pods should NOT tolerate the CA taint
taints:
- key: "karpenter.sh/capacity-type"
value: "spot"
effect: "NO_SCHEDULE"
labels:
karpenter.sh/provisioner-name: "default"
consolidationPolicy: WhenEmpty
ttlSecondsAfterEmpty: 30
```
Your pod specs then dictate where they land. Batch jobs that can use spot get the appropriate toleration for `karpenter.sh/capacity-type: spot`. Your core, stable services tolerate the `capacity-type: on-demand` taint and are scheduled on the CA group.
The real failure scenario isn't during smooth operation—it's during a spot interruption cascade. What happens when Karpenter's spot nodes get reclaimed and it tries to rapidly re-create them, but spot capacity is exhausted in your chosen instance types? You'll see:
* A flurry of Karpenter provisioning attempts.
* Pods stuck in `Pending` because Karpenter can't satisfy their spot-specific constraints.
* CA will NOT help these pods, because they have a toleration for `spot` taint but not for the `on-demand` taint on CA nodes.
The "failover" only works if you've designed your pod specs with a fallback strategy. This usually means defining two pod topology spread constraints or writing your deployments to have a spot-tolerant spec and an on-demand spec, which is complex. Alternatively, you can let the spot pods just fail and rely on higher-level controllers (like a Job queue) to resubmit them with different tolerations after a timeout.
Running both controllers adds measurable overhead:
* You now have two scaling logic engines to monitor and tune.
* You pay for Karpenter's control plane costs (AWS charges for EC2 Fleet API calls).
* Your cluster's node composition becomes harder to reason about.
* Debugging requires checking two sets of logs and events.
Is it worth it? For a significant batch workload where spot savings pay for the operational complexity, yes. For a typical web service cluster, likely not. The benchmarks I've run show that for pure on-demand scaling, CA is slightly slower but more predictable. Karpenter excels at rapid, diverse spot provisioning. Combining them forces you to manage the seams between two different paradigms.
—emma
FinOps first, hype last
Absolutely spot on about the scheduling constraints being the critical failure point. I've implemented this twice, and both times the initial conflict came down to overlapping taints or labels that let pods schedule onto the wrong node group.
One subtlety I'd add: your CA-managed node group needs a `karpenter.sh/do-not-consolidate: "true"` taint to prevent Karpenter from trying to evict pods from those stable, on-demand nodes. Without that, you can get into a loop where Karpenter sees pods on a CA node and tries to move them to a spot node it provisions, defeating the whole fallback purpose.
- Mike
Great breakdown of the core idea. You're absolutely right that the *disjoint scheduling constraints* are the make-or-break part. I've found the taint/toleration setup is only half the battle.
The other half is making sure your Karpenter provisioner's `nodeSelector` or `nodeAffinity` explicitly *excludes* the CA-managed node group's labels. Otherwise, during a spot interruption, the Karpenter pods can get stuck in a Pending state, waiting for a node that matches the provisioner spec, even though there's capacity in the CA pool. The controllers don't fight, but the pods can't move.
Data doesn't lie, but dashboards sometimes do.
You're right about that explicit exclusion in the provisioner being a silent killer. I learned that the hard way when a spot outage led to a backlog because the pods' nodeSelector was set to a generic role label that was on both CA and Karpenter nodes.
What finally worked for me was adding a distinct label like `provisioner-group: karpenter-burst` and making sure it's in the Karpenter provisioner template, but also setting a node affinity "requiredDuringSchedulingIgnoredDuringExecution" that specifically matches that label. That way it's a positive match for Karpenter nodes, and a non-match for the CA nodes which have a different label value. Just excluding the CA label can still leave ambiguity if you scale up other node groups later.
That `karpenter.sh/do-not-consolidate: "true"` taint is critical, but I'd treat it as a safety belt rather than the primary steering mechanism. The real problem often surfaces with the other built-in controllers like the `disruption` controller, not just consolidation. Even with that taint, if you haven't explicitly disabled drift in your Karpenter provisioner, it can still decide those on-demand nodes are "drifted" and try to replace them, leading to similar eviction chaos.
The more holistic approach I've settled on is a multi-layer isolation strategy:
- The taint you mentioned.
- A provisioner config with `consolidation: { enabled: false }` and `drift: { enabled: false }`.
- A `nodeSelector` on the provisioner that only matches nodes with a `karpenter-burst` label, which my CA node groups explicitly do not have.
It adds configuration overhead, but it turns a soft, annotation-based boundary into a hard, declarative one. The controllers co-exist by never even considering each other's territory.