CCloudSuccessTechworld with Randy
Labs / Karpenter on EKS: From Pending Pods to a New Node in 30 Seconds
Lab · Advanced · 45 min (+ ~20 min cluster creation)

Karpenter on EKS: From Pending Pods to a New Node in 30 Seconds

A real Amazon EKS cluster with Karpenter 1.15. Five pods that don't fit, a NodePool with a hard CPU limit, Karpenter picking one right-sized instance in about 30 seconds, and consolidation removing it again once the pods are gone. Costs a few dollars if you clean up straight after.

Real recorded runFrom: Platform Engineer Mock Interview
The output on this page is from our actual run, not edited. Your pod names, IPs and timings will differ; the behaviour won't.

What you'll learn

  • NodePool and EC2NodeClass, and why limits matter
  • How Karpenter turns pending pods into a NodeClaim
  • Reading the Karpenter controller logs
  • Consolidation and how to control disruption
  • A full, clean teardown

Before you start

  • An AWS account (this creates billable resources)
  • AWS CLI v2, eksctl, helm, kubectl
  • On-demand vCPU quota of at least 12 in your region

Setup

Follow the official getting-started guide

bash
# karpenter.sh → Getting Started (v1.15.1, Kubernetes 1.36)
# 1. CloudFormation stack for Karpenter's IAM + interruption queue
# 2. eksctl create cluster (2 x m5.large managed nodes for system pods)
# 3. helm install karpenter oci://public.ecr.aws/karpenter/karpenter --version 1.15.1

Clean up when you're done

bash
kubectl delete deployment inflate
kubectl delete nodepool default
helm uninstall karpenter -n kube-system
eksctl delete cluster --name "$CLUSTER_NAME" --wait
aws cloudformation delete-stack --stack-name "Karpenter-$CLUSTER_NAME"

Lab files

nodepool.yamlinflate.yaml

The lab

01The starting point

Two managed m5.large nodes run the system pods and Karpenter itself. Karpenter never runs on nodes it manages.

randy — zsh
randy@macbook ~ % kubectl get nodes -L node.kubernetes.io/instance-type,karpenter.sh/nodepool
NAME                                           STATUS   ROLES    AGE   VERSION               INSTANCE-TYPE   NODEPOOL
ip-192-168-43-107.us-east-2.compute.internal   Ready    <none>   13m   v1.36.5-eks-39aca56   m5.large        
ip-192-168-77-160.us-east-2.compute.internal   Ready    <none>   13m   v1.36.5-eks-39aca56   m5.large
randy — zsh
randy@macbook ~ % kubectl -n kube-system get pods -l app.kubernetes.io/name=karpenter
NAME                         READY   STATUS    RESTARTS   AGE
karpenter-6c7d478946-4j8m6   1/1     Running   0          10m
karpenter-6c7d478946-s6hcl   1/1     Running   0          10m

02The NodePool

On-demand only, C/M/R families, fewer than 9 vCPUs, and a hard limit of 8 CPUs in total, your cost blast radius. Consolidate when empty or underutilized after one minute.

randy — zsh
randy@macbook ~ % cat nodepool.yaml
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: default
spec:
  template:
    spec:
      requirements:
        - key: kubernetes.io/arch
          operator: In
          values: ["amd64"]
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["on-demand"]
        - key: karpenter.k8s.aws/instance-category
          operator: In
          values: ["c", "m", "r"]
        - key: karpenter.k8s.aws/instance-cpu
          operator: Lt
          values: ["9"]
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: default
      expireAfter: 720h
  limits:
    cpu: 8
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 1m
---
apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
  name: default
spec:
  role: "KarpenterNodeRole-twr-karpenter-demo"
  amiSelectorTerms:
    - alias: "al2023@v20261023"
  subnetSelectorTerms:
    - tags:
        karpenter.sh/discovery: "twr-karpenter-demo"
  securityGroupSelectorTerms:
    - tags:
        karpenter.sh/discovery: "twr-karpenter-demo"
randy — zsh
randy@macbook ~ % kubectl apply -f nodepool.yaml
nodepool.karpenter.sh/default created
ec2nodeclass.karpenter.k8s.aws/default created
randy — zsh
randy@macbook ~ % kubectl get nodepools,ec2nodeclasses
NAME                            NODECLASS   NODES   READY   AGE
nodepool.karpenter.sh/default   default     0       False   7s

NAME                                     READY     AGE
ec2nodeclass.karpenter.k8s.aws/default   Unknown   7s

03Five pods that don't fit

Each inflate pod asks for one CPU. The existing nodes are full, so all five go Pending.

randy — zsh
randy@macbook ~ % kubectl apply -f inflate.yaml
deployment.apps/inflate created
randy — zsh
randy@macbook ~ % kubectl scale deployment inflate --replicas 5
deployment.apps/inflate scaled
randy — zsh
randy@macbook ~ % kubectl get pods -l app=inflate
NAME                       READY   STATUS    RESTARTS   AGE
inflate-56f87d64bc-2zjjl   0/1     Pending   0          6s
inflate-56f87d64bc-6sfz7   0/1     Pending   0          6s
inflate-56f87d64bc-f8tmw   0/1     Pending   0          6s
inflate-56f87d64bc-gpv7w   0/1     Pending   0          6s
inflate-56f87d64bc-rxzls   0/1     Pending   0          6s
randy — zsh
randy@macbook ~ % kubectl get events --field-selector reason=FailedScheduling -o custom-columns=REASON:.reason,MESSAGE:.message | head -3
REASON             MESSAGE
FailedScheduling   0/2 nodes are available: 2 Insufficient cpu. no new claims to deallocate, preemption: 0/2 nodes are available: 2 No preemption victims found for incoming pod.
FailedScheduling   0/2 nodes are available: 2 Insufficient cpu. no new claims to deallocate, preemption: 0/2 nodes are available: 2 No preemption victims found for incoming pod.

04Karpenter launches one right-sized node

A NodeClaim appears within seconds. The logs show the whole lifecycle, and pending pods became a ready c6a.2xlarge in about 30 seconds. One machine for five pods: bin-packing.

randy — zsh
randy@macbook ~ % kubectl get events -o custom-columns=REASON:.reason,OBJECT:.involvedObject.name,MESSAGE:.message | grep -E 'Nominated|Launched' | head -4
Launched                    default-p9w68                                  Status condition transitioned, Type: Launched, Status: Unknown -> True, Reason: Launched
Nominated                   inflate-56f87d64bc-2zjjl                       Pod should schedule on: nodeclaim/default-p9w68
Nominated                   inflate-56f87d64bc-6sfz7                       Pod should schedule on: nodeclaim/default-p9w68
Nominated                   inflate-56f87d64bc-f8tmw                       Pod should schedule on: nodeclaim/default-p9w68
randy — zsh
randy@macbook ~ % kubectl get nodeclaims
NAME            TYPE          CAPACITY    ZONE         NODE   READY     AGE
default-p9w68   c6a.2xlarge   on-demand   us-east-2c          Unknown   14s
randy — zsh
randy@macbook ~ % kubectl logs -n kube-system -l app.kubernetes.io/name=karpenter -c controller --tail=-1 | jq -rR 'fromjson? | select(.message|test("found provisionable|computed new nodeclaim|created nodeclaim|launched nodeclaim|registered nodeclaim|initialized nodeclaim")) | [.time[11:19], .message, (.NodeClaim.name // ""), (."instance-type" // ""), (."capacity-type" // ""), (.zone // "")] | @tsv' | column -t -s $'\t' | tail -8
11:07:53  found provisionable pod(s)                                                      
11:07:53  computed new nodeclaim(s) to fit pod(s)                                         
11:07:54  created nodeclaim                        default-p9w68                          
11:07:56  launched nodeclaim                       default-p9w68  c6a.2xlarge  on-demand  us-east-2c
11:08:12  registered nodeclaim                     default-p9w68                          
11:08:23  initialized nodeclaim                    default-p9w68
randy — zsh
randy@macbook ~ % kubectl get pods -l app=inflate -o wide | awk '{print $1,$2,$3,$7}' | column -t
NAME                      READY  STATUS   NODE
inflate-56f87d64bc-2zjjl  1/1    Running  ip-192-168-101-185.us-east-2.compute.internal
inflate-56f87d64bc-6sfz7  1/1    Running  ip-192-168-101-185.us-east-2.compute.internal
inflate-56f87d64bc-f8tmw  1/1    Running  ip-192-168-101-185.us-east-2.compute.internal
inflate-56f87d64bc-gpv7w  1/1    Running  ip-192-168-101-185.us-east-2.compute.internal
inflate-56f87d64bc-rxzls  1/1    Running  ip-192-168-101-185.us-east-2.compute.internal
randy — zsh
randy@macbook ~ % kubectl get nodes -L node.kubernetes.io/instance-type,karpenter.sh/capacity-type,karpenter.sh/nodepool
NAME                                            STATUS   ROLES    AGE   VERSION               INSTANCE-TYPE   CAPACITY-TYPE   NODEPOOL
ip-192-168-101-185.us-east-2.compute.internal   Ready    <none>   21s   v1.36.5-eks-39aca56   c6a.2xlarge     on-demand       default
ip-192-168-43-107.us-east-2.compute.internal    Ready    <none>   14m   v1.36.5-eks-39aca56   m5.large                        
ip-192-168-77-160.us-east-2.compute.internal    Ready    <none>   14m   v1.36.5-eks-39aca56   m5.large
randy — zsh
randy@macbook ~ % kubectl get nodeclaims -o wide
NAME            TYPE          CAPACITY    ZONE         NODE                                            READY   AGE   IMAGEID                 ID                                      NODEPOOL   NODECLASS   DRIFTED
default-p9w68   c6a.2xlarge   on-demand   us-east-2c   ip-192-168-101-185.us-east-2.compute.internal   True    41s   ami-06e6e691d7d05d43a   aws:///us-east-2c/i-05bd4d6e6cdd323ed   default    default

05Consolidation

Scale down, delete the deployment, and Karpenter taints, drains and deletes the empty node about 30 seconds later. Back to the two original nodes.

randy — zsh
randy@macbook ~ % kubectl scale deployment inflate --replicas 2
deployment.apps/inflate scaled
randy — zsh
randy@macbook ~ % kubectl delete deployment inflate
deployment.apps "inflate" deleted from default namespace
randy — zsh
randy@macbook ~ % kubectl logs -n kube-system -l app.kubernetes.io/name=karpenter -c controller --tail=-1 | jq -rR 'fromjson? | select(.message|test("disrupting|tainted|deleted node|deleted nodeclaim|terminat")) | [.time[11:19], .message, (.NodeClaim.name // ""), (."instance-type" // ""), (."capacity-type" // ""), (.zone // "")] | @tsv' | column -t -s $'\t' | tail -8
11:09:52  disrupting node(s)                     
11:09:53  tainted node        default-p9w68      
11:10:24  deleted node        default-p9w68      
11:10:24  deleted nodeclaim   default-p9w68
randy — zsh
randy@macbook ~ % kubectl get nodes -L node.kubernetes.io/instance-type,karpenter.sh/nodepool
NAME                                           STATUS   ROLES    AGE   VERSION               INSTANCE-TYPE   NODEPOOL
ip-192-168-43-107.us-east-2.compute.internal   Ready    <none>   15m   v1.36.5-eks-39aca56   m5.large        
ip-192-168-77-160.us-east-2.compute.internal   Ready    <none>   15m   v1.36.5-eks-39aca56   m5.large
randy — zsh
randy@macbook ~ % kubectl get nodeclaims
No resources found

Check yourself

8 questions on what happened in this lab. Pick an answer for each, then check.

01Why do Karpenter's own pods run on the two managed m5.large nodes and not on nodes Karpenter launches?
02Five inflate pods each request 1 CPU and go Pending with "0/2 nodes are available: 2 Insufficient cpu". What does Karpenter react to?
03Karpenter launched a single c6a.2xlarge for all five pods instead of five small instances. Which statement is correct?
04The NodePool has limits: cpu: 8, and the c6a.2xlarge it launched has 8 vCPUs. If you now scaled inflate to 12 pods, what would happen?
05The launch logs read: found provisionable 11:07:53, launched 11:07:56, registered 11:08:12, initialized 11:08:23. What does this timeline show?
06You deploy pods that require arm64 nodes. They stay Pending forever and Karpenter launches nothing. What is the most likely reason with this NodePool?
07After the deployment is deleted, the logs show "disrupting node(s)", then "tainted node", then "deleted node" and "deleted nodeclaim" about 30 seconds later. Why taint the node first?
08A team complains that consolidation keeps restarting their pods. Which response is best?
Go deeper

Take this further

The Field Guides cover the whole topic: more labs, a troubleshooting playbook, interview questions and a 30-day plan.

Field Guide · 97 pages · 12 labs

Terraform on AWS: From First Resource to Production Modules

Go from your first S3 bucket to tested, versioned AWS modules shipped by an OIDC pipeline, with state, refactoring and drift handled properly.

$39 CADView guide
Field Guide · 87 pages · 10 labs

End-to-End DevOps Project: Terraform, EKS and Argo CD

Build a resume-ready AWS delivery platform end to end: Terraform, EKS, GitHub Actions with OIDC, Argo CD GitOps, then tear it all down.

$39 CADView guide
Field Guide · 79 pages · 11 labs

Kubernetes Autoscaling and Cost

Make Kubernetes autoscaling boring and your bill smaller: requests, HPA, VPA, KEDA, Karpenter and FinOps, with hands-on labs.

$39 CADView guide

Or get every guide with All-Access.

Get the next lab first

One real lab every week, straight to your inbox.

One email a week, real labs only. Unsubscribe any time.