Karpenter is one of those tools that's easy to describe and easy to get wrong. "It adds nodes when pods are pending" is true, but it hides the parts that matter in production: what it's allowed to launch, how much it's allowed to spend, and when it takes nodes away again.
So instead of describing it, we ran it. A real Amazon EKS cluster on Kubernetes 1.36 in us-east-2, Karpenter 1.15.1 installed with the official getting-started guide, and a NodePool with a hard limit. Five pods that don't fit, one new node in about 30 seconds, and consolidation removing it once the pods were gone. All the output below is from that run.
Karpenter vs the cluster autoscaler
First, where Karpenter sits. HPA changes the number of pods. VPA changes the size of pods (their requests). The cluster autoscaler and Karpenter change the number of nodes.
The cluster autoscaler works through node groups. On AWS those are Auto Scaling groups, each with a fixed instance type. It decides which group to grow, and the group launches a node. Karpenter skips node groups. It looks at the pending pods, works out what they need, and launches a suitable instance directly from a wide range of types. That usually means faster scale-up, better-fitting machines, and no longer managing dozens of node groups.
The starting point
Two managed m5.large nodes run the system pods and Karpenter itself:
$ kubectl get nodes -L node.kubernetes.io/instance-type,karpenter.sh/nodepool NAME STATUS ROLES AGE VERSION INSTANCE-TYPE NODEPOOL ip-192-168-43-107.us-east-2.compute.internal Ready <none> 13m v1.36.5-eks-39aca56 m5.large ip-192-168-77-160.us-east-2.compute.internal Ready <none> 13m v1.36.5-eks-39aca56 m5.large
The empty NODEPOOL column matters. These nodes are not managed by Karpenter, and that's on purpose. Karpenter should never run on nodes it manages. If it decides to remove the node it's running on, you can end up with nothing left to launch capacity. A small managed node group (or Fargate) for Karpenter and critical system pods is the standard pattern.
The NodePool: what Karpenter may launch, and how much
A NodePool is the contract between you and Karpenter. Here's the spec of the one we used, a karpenter.sh/v1 NodePool named default (the EC2NodeClass it references is trimmed):
spec:
template:
spec:
requirements:
- key: karpenter.sh/capacity-type
operator: In
values: ["on-demand"]
- key: karpenter.k8s.aws/instance-category
operator: In
values: ["c", "m", "r"]
- key: karpenter.k8s.aws/instance-cpu
operator: Lt
values: ["9"]
...
expireAfter: 720h
limits:
cpu: 8
disruption:
consolidationPolicy: WhenEmptyOrUnderutilized
consolidateAfter: 1mRead it top to bottom:
- Requirements. AMD64 (trimmed above), on-demand only, C, M or R instance families, fewer than 9 vCPUs per instance.
- nodeClassRef points at an EC2NodeClass with the AWS details: the node IAM role, the AMI alias (
al2023@v20261023), and subnets and security groups discovered by thekarpenter.sh/discoverytag. - expireAfter: 720h. Nodes are replaced after 30 days, so you pick up patched AMIs and never have a two-year-old server nobody dares to touch.
- limits.cpu: 8. Karpenter can never run more than 8 vCPUs in total from this pool.
- disruption. Consolidate nodes that are empty or underutilized, after one minute.
Why NodePool limits matter
The limits block is the line we'd insist on in every review. Without it, a bug that creates ten thousand pods becomes ten thousand pods' worth of EC2 instances. The limit is your cost blast radius. When the pool reaches it, Karpenter stops launching and pods stay Pending, which is a much better failure mode than a surprise bill.
Applying it creates both objects:
$ kubectl apply -f nodepool.yaml nodepool.karpenter.sh/default created ec2nodeclass.karpenter.k8s.aws/default created
Five pods that don't fit
The test workload is inflate, a deployment of pause containers where each pod requests one full CPU. Scale it to five:
$ kubectl scale deployment inflate --replicas 5 deployment.apps/inflate scaled $ kubectl get pods -l app=inflate NAME READY STATUS RESTARTS AGE inflate-56f87d64bc-2zjjl 0/1 Pending 0 6s inflate-56f87d64bc-6sfz7 0/1 Pending 0 6s inflate-56f87d64bc-f8tmw 0/1 Pending 0 6s inflate-56f87d64bc-gpv7w 0/1 Pending 0 6s inflate-56f87d64bc-rxzls 0/1 Pending 0 6s
The scheduler explains why:
FailedScheduling 0/2 nodes are available: 2 Insufficient cpu. no new claims to deallocate, preemption: 0/2 nodes are available: 2 No preemption victims found for incoming pod.
Note that "Insufficient cpu" is about requests, not real usage. Pause containers use almost nothing, but the scheduler only adds up what pods ask for.
From pending pods to a NodeClaim
Within seconds Karpenter has created a NodeClaim, its record of a machine it's launching, and nominated the pending pods to it:
$ kubectl get nodeclaims NAME TYPE CAPACITY ZONE NODE READY AGE default-p9w68 c6a.2xlarge on-demand us-east-2c Unknown 14s
The events show the pods being matched to that claim before the node even exists:
Nominated inflate-56f87d64bc-2zjjl Pod should schedule on: nodeclaim/default-p9w68 Nominated inflate-56f87d64bc-6sfz7 Pod should schedule on: nodeclaim/default-p9w68
Reading the Karpenter controller logs
The controller logs are JSON, which makes them noisy to read raw but easy to filter. This pulls out just the lifecycle messages:
$ kubectl logs -n kube-system -l app.kubernetes.io/name=karpenter -c controller --tail=-1 | jq -rR 'fromjson? | select(.message|test("found provisionable|computed new nodeclaim|created nodeclaim|launched nodeclaim|registered nodeclaim|initialized nodeclaim")) | [.time[11:19], .message, (.NodeClaim.name // ""), (."instance-type" // ""), (."capacity-type" // ""), (.zone // "")] | @tsv' | column -t -s $'\t' | tail -8
11:07:53 found provisionable pod(s)
11:07:53 computed new nodeclaim(s) to fit pod(s)
11:07:54 created nodeclaim default-p9w68
11:07:56 launched nodeclaim default-p9w68 c6a.2xlarge on-demand us-east-2c
11:08:12 registered nodeclaim default-p9w68
11:08:23 initialized nodeclaim default-p9w68 From found provisionable pod(s) at 11:07:53 to initialized nodeclaim at 11:08:23 is 30 seconds. That's pending pods to a ready node, including launching the EC2 instance and the kubelet joining the cluster.
One machine for five pods: bin-packing
Look at what it picked. Five pods each requesting one CPU. Karpenter didn't launch five small machines. It launched one c6a.2xlarge (8 vCPUs), which fits all five plus the per-node overhead, and stays inside both the "fewer than 9 vCPUs" requirement and the 8 CPU pool limit.
$ kubectl get nodes -L node.kubernetes.io/instance-type,karpenter.sh/capacity-type,karpenter.sh/nodepool NAME STATUS ROLES AGE VERSION INSTANCE-TYPE CAPACITY-TYPE NODEPOOL ip-192-168-101-185.us-east-2.compute.internal Ready <none> 21s v1.36.5-eks-39aca56 c6a.2xlarge on-demand default ip-192-168-43-107.us-east-2.compute.internal Ready <none> 14m v1.36.5-eks-39aca56 m5.large ip-192-168-77-160.us-east-2.compute.internal Ready <none> 14m v1.36.5-eks-39aca56 m5.large
All five inflate pods were Running on ip-192-168-101-185, the new node, labelled with NODEPOOL default.
Consolidation: removing the node again
Scaling up is half the job. The other half is giving machines back. We scaled inflate down to two, then deleted the deployment:
$ kubectl logs -n kube-system -l app.kubernetes.io/name=karpenter -c controller --tail=-1 | jq -rR 'fromjson? | select(.message|test("disrupting|tainted|deleted node|deleted nodeclaim|terminat")) | [.time[11:19], .message, (.NodeClaim.name // ""), (."instance-type" // ""), (."capacity-type" // ""), (.zone // "")] | @tsv' | column -t -s $'\t' | tail -8
11:09:52 disrupting node(s)
11:09:53 tainted node default-p9w68
11:10:24 deleted node default-p9w68
11:10:24 deleted nodeclaim default-p9w68 Karpenter noticed the node was empty, tainted it so nothing new would land there, and about 30 seconds later the node and the NodeClaim were gone. The instance is terminated, so you stop paying for it. kubectl get nodeclaims now returns No resources found, and we're back to the two original m5.large nodes.
Controlling disruption in production
Consolidation saves money, but it moves pods. When a team complains their service restarts too often, there are three levers:
- PodDisruptionBudgets on every service, so Karpenter can only take down a limited number of pods at once.
- Disruption budgets on the NodePool, for example at most 10% of nodes at a time, or no disruption during business hours.
- The
karpenter.sh/do-not-disruptannotation for things that truly can't be interrupted, like a long batch job.
Don't put do-not-disrupt on everything. That turns consolidation off and you're back to paying for empty machines.
Spot instances and interruptions
Our pool was on-demand only, but spot is where much of the saving is. AWS gives a two-minute warning before reclaiming a spot instance. Karpenter listens for those warnings through an SQS interruption queue fed by EventBridge (the queue the getting-started CloudFormation stack creates). When a warning arrives, it cordons the node, launches a replacement and drains the pods inside those two minutes.
Only put workloads on spot that can handle it: stateless, multiple replicas, spread across zones, and allowed to run on many instance types. The more types you allow, the less likely they are all reclaimed at once.
When pods stay Pending and Karpenter does nothing
Start with the pod events, then the Karpenter logs, which usually say why. The common causes:
- The NodePool requirements don't match the pod (the pod wants ARM, the pool only allows AMD64).
- The NodePool limit is already reached.
- A node selector or toleration no pool satisfies.
- A hard topology spread across zones your subnets don't cover.
- On the AWS side, no capacity for the allowed instance types, or the account's vCPU quota. The lab lists an on-demand vCPU quota of at least 12 as a prerequisite for exactly this reason.
Clean up
This creates billable resources, so tear it down when you're done:
kubectl delete deployment inflate kubectl delete nodepool default helm uninstall karpenter -n kube-system eksctl delete cluster --name "$CLUSTER_NAME" --wait aws cloudformation delete-stack --stack-name "Karpenter-$CLUSTER_NAME"
Go deeper
The [Karpenter on EKS lab](/labs/karpenter-eks/) has the NodePool, the inflate deployment and every step of this run, including the full teardown. For the interview angle (spot interruptions, topology spread, expireAfter and the hard follow-up questions), watch the autoscaling round of the [Platform Engineer Mock Interview](/episodes/platform-engineer-mock-interview/).