Labs / Kubernetes Incident Lab: Production Is Down
Lab · Intermediate · 45–60 minKubernetes Incident Lab: Production Is Down
Break a real checkout service with a bad release, then debug it like an on-call SRE: CrashLoopBackOff, OOMKilled (exit 137), a rollback that doesn't fix it, a service selector trap, a capacity crunch and a fix-forward. Plus Kubernetes rapid-fire scenarios and the classic Linux “disk full but du shows nothing” problem.
The output on this page is from our actual run, not edited. Your pod names, IPs and timings will differ; the behaviour won't.
What you'll learn
- Read pod state, events and previous-container logs to find a root cause
- Why exit code 137 means OOMKilled, and how limits cause it
- Why
rollout undodidn't bring traffic back (service selectors) - Requests vs real usage, and how bad requests block scale-ups
- PodDisruptionBudgets, QoS classes, cluster DNS and ndots
- Finding disk space held by deleted files with lsof
Before you start
- Docker Desktop (or any Docker engine)
- kind v0.27+ and kubectl
- ~4 GB free RAM
Setup
Create a 3-node cluster
cat > kind.yaml <<'EOF' kind: Cluster apiVersion: kind.x-k8s.io/v1alpha4 nodes: - role: control-plane - role: worker - role: worker EOF kind create cluster --name prod-east --config kind.yaml --wait 180s
Install metrics-server (for kubectl top)
kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.7.2/components.yaml
kubectl -n kube-system patch deploy metrics-server --type=json \
-p '[{"op":"add","path":"/spec/template/spec/containers/0/args/-","value":"--kubelet-insecure-tls"}]'
kubectl -n kube-system rollout status deploy/metrics-serverDeploy the shop (storefront + checkout v1)
kubectl apply -f manifests.yaml kubectl apply -f checkout-v1.yaml kubectl -n shop rollout status deploy/checkout kubectl -n shop wait --for=condition=Ready pod/storefront
Lab files
The lab
01The healthy baseline
Before breaking anything, look at what normal looks like. Three nodes, three checkout pods, and the storefront sending 40 requests that all succeed. Every dot is a successful request.
randy@macbook ~ % kubectl get nodes NAME STATUS ROLES AGE VERSION prod-east-control-plane Ready control-plane 63s v1.32.2 prod-east-worker Ready <none> 48s v1.32.2 prod-east-worker2 Ready <none> 48s v1.32.2
randy@macbook ~ % kubectl -n shop get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES checkout-5dd967cbb-2pqgt 1/1 Running 0 5s 10.244.1.4 prod-east-worker <none> <none> checkout-5dd967cbb-jk6bl 1/1 Running 0 5s 10.244.1.3 prod-east-worker <none> <none> checkout-5dd967cbb-sj752 1/1 Running 0 5s 10.244.2.4 prod-east-worker2 <none> <none> storefront 1/1 Running 0 5s 10.244.1.2 prod-east-worker <none> <none>
randy@macbook ~ % kubectl -n shop exec storefront -- python /app/hit.py 40 requests: ........................................ 200 OK: 40 errors: 0 error rate: 0%
02The bad release
Release 2.0 ships a new checkout image and the pipeline also points the service selector at version=2. This is the change that takes checkout down.
randy@macbook ~ % kubectl apply -f checkout-v2.yaml deployment.apps/checkout configured
randy@macbook ~ % kubectl -n shop patch svc checkout -p '{"spec":{"selector":{"app":"checkout","version":"2"}}}' service/checkout patched
03Triage: what changed?
Every request now fails. Start with the question that matters most at 3 a.m.: what changed? Then read the evidence in order: pod status, rollout history, the pod's last state, the previous container's logs, and events. Exit code 137 plus OOMKilled tells you the new cache needs more memory than the 64Mi limit.
randy@macbook ~ % kubectl -n shop exec storefront -- python /app/hit.py 40 requests: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 200 OK: 0 errors: 40 error rate: 100%
randy@macbook ~ % kubectl -n shop get pods NAME READY STATUS RESTARTS AGE checkout-5dd967cbb-2pqgt 1/1 Running 0 82s checkout-5dd967cbb-jk6bl 1/1 Running 0 82s checkout-5dd967cbb-sj752 1/1 Running 0 82s checkout-8b854d58f-jpsjt 0/1 CrashLoopBackOff 3 (30s ago) 74s storefront 1/1 Running 0 82s
randy@macbook ~ % kubectl -n shop rollout history deploy/checkout deployment.apps/checkout REVISION CHANGE-CAUSE 1 release 1.0 - baseline 2 release 2.0 - add product cache warm-up
randy@macbook ~ % kubectl -n shop rollout status deploy/checkout --timeout=5s Waiting for deployment "checkout" rollout to finish: 1 out of 3 new replicas have been updated... error: timed out waiting for the condition
randy@macbook ~ % kubectl -n shop describe pod checkout-8b854d58f-jpsjt Name: checkout-8b854d58f-jpsjt Namespace: shop Priority: 0 Service Account: default Node: prod-east-worker2/172.18.0.2 Start Time: Mon, 26 Oct 2026 23:44:08 +0000 Labels: app=checkout pod-template-hash=8b854d58f version=2 Annotations: <none> Status: Running IP: 10.244.2.5 IPs: IP: 10.244.2.5 Controlled By: ReplicaSet/checkout-8b854d58f Containers: checkout: Container ID: containerd://ed853fdb9dd2025bafb5951d52cf0de12c35df69b27741df2cb4b1e9fb495d79 Image: python:3.12-alpine Image ID: sha256:38fd5a425aa93b06d367e2e47e2da6a9bc0cabdaf5788fb1707d143fd1416e97 Port: <none> Host Port: <none> Command: python -u /app/app.py State: Waiting Reason: CrashLoopBackOff Last State: Terminated Reason: OOMKilled Exit Code: 137 Started: Mon, 26 Oct 2026 23:44:51 +0000 Finished: Mon, 26 Oct 2026 23:44:52 +0000 Ready: False Restart Count: 3 Limits: memory: 64Mi Requests: cpu: 1500m memory: 48Mi Readiness: http-get http://:8080/ delay=0s timeout=1s period=3s #success=1 #failure=3 Environment: VERSION: 2 WARM_CACHE: 1 Mounts: /app from code (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-kv24w (ro) Conditions: Type Status PodReadyToStartContainers True Initialized True Ready False ContainersReady False PodScheduled True Volumes: code: Type: ConfigMap (a volume populated by a ConfigMap) Name: checkout-code Optional: false kube-api-access-kv24w: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: <nil> DownwardAPI: true QoS Class: Burstable Node-Selectors: <none> Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 79s default-scheduler Successfully assigned shop/checkout-8b854d58f-jpsjt to prod-east-worker2 Normal Pulled 36s (x4 over 79s) kubelet Container image "python:3.12-alpine" already present on machine Normal Created 36s (x4 over 79s) kubelet Created container: checkout Normal Started 36s (x4 over 79s) kubelet Started container checkout Warning Unhealthy 35s (x5 over 78s) kubelet Readiness probe failed: Get "http://10.244.2.5:8080/": dial tcp 10.244.2.5:8080: connect: connection refused Warning BackOff 10s (x9 over 74s) kubelet Back-off restarting failed container checkout in pod checkout-8b854d58f-jpsjt_shop(a85506e3-c395-4a64-a5a5-f8a0025b7b50)
randy@macbook ~ % kubectl -n shop logs checkout-8b854d58f-jpsjt --previous checkout v2: warming product cache (80 MB)... cache loaded 20 MB cache loaded 40 MB
randy@macbook ~ % kubectl -n shop get events --sort-by=.lastTimestamp | tail -14 87s Normal SuccessfulCreate replicaset/checkout-5dd967cbb Created pod: checkout-5dd967cbb-2pqgt 87s Normal Pulled pod/storefront Container image "python:3.12-alpine" already present on machine 87s Normal ScalingReplicaSet deployment/checkout Scaled up replica set checkout-5dd967cbb from 0 to 3 86s Normal Started pod/checkout-5dd967cbb-jk6bl Started container checkout 86s Warning Unhealthy pod/checkout-5dd967cbb-sj752 Readiness probe failed: Get "http://10.244.2.4:8080/": dial tcp 10.244.2.4:8080: connect: connection refused 86s Normal Started pod/checkout-5dd967cbb-2pqgt Started container checkout 79s Normal SuccessfulCreate replicaset/checkout-8b854d58f Created pod: checkout-8b854d58f-jpsjt 79s Normal ScalingReplicaSet deployment/checkout Scaled up replica set checkout-8b854d58f from 0 to 1 79s Normal Scheduled pod/checkout-8b854d58f-jpsjt Successfully assigned shop/checkout-8b854d58f-jpsjt to prod-east-worker2 36s Normal Started pod/checkout-8b854d58f-jpsjt Started container checkout 36s Normal Created pod/checkout-8b854d58f-jpsjt Created container: checkout 36s Normal Pulled pod/checkout-8b854d58f-jpsjt Container image "python:3.12-alpine" already present on machine 35s Warning Unhealthy pod/checkout-8b854d58f-jpsjt Readiness probe failed: Get "http://10.244.2.5:8080/": dial tcp 10.244.2.5:8080: connect: connection refused 10s Warning BackOff pod/checkout-8b854d58f-jpsjt Back-off restarting failed container checkout in pod checkout-8b854d58f-jpsjt_shop(a85506e3-c395-4a64-a5a5-f8a0025b7b50)
randy@macbook ~ % kubectl -n shop top pods NAME CPU(cores) MEMORY(bytes) checkout-5dd967cbb-2pqgt 1m 11Mi checkout-5dd967cbb-jk6bl 1m 11Mi checkout-5dd967cbb-sj752 1m 11Mi storefront 4m 11Mi
04The rollback that didn't fix it
Rolling back the deployment brings v1 pods back, but traffic still fails. The service is still selecting version=2, so it has zero endpoints. Follow the request: service → endpoints → pod labels.
randy@macbook ~ % kubectl -n shop rollout undo deploy/checkout deployment.apps/checkout rolled back
randy@macbook ~ % kubectl -n shop rollout status deploy/checkout --timeout=120s deployment "checkout" successfully rolled out
randy@macbook ~ % kubectl -n shop get pods NAME READY STATUS RESTARTS AGE checkout-5dd967cbb-2pqgt 1/1 Running 0 88s checkout-5dd967cbb-jk6bl 1/1 Running 0 88s checkout-5dd967cbb-sj752 1/1 Running 0 88s checkout-8b854d58f-jpsjt 0/1 Terminating 3 80s storefront 1/1 Running 0 88s
randy@macbook ~ % kubectl -n shop exec storefront -- python /app/hit.py 40 requests: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 200 OK: 0 errors: 40 error rate: 100%
randy@macbook ~ % kubectl -n shop get endpoints checkout NAME ENDPOINTS AGE checkout <none> 2m6s
randy@macbook ~ % kubectl -n shop get svc checkout -o wide NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR checkout ClusterIP 10.96.31.165 <none> 8080/TCP 2m6s app=checkout,version=2
randy@macbook ~ % kubectl -n shop get pods --show-labels NAME READY STATUS RESTARTS AGE LABELS checkout-5dd967cbb-2pqgt 1/1 Running 0 2m6s app=checkout,pod-template-hash=5dd967cbb,version=1 checkout-5dd967cbb-jk6bl 1/1 Running 0 2m6s app=checkout,pod-template-hash=5dd967cbb,version=1 checkout-5dd967cbb-sj752 1/1 Running 0 2m6s app=checkout,pod-template-hash=5dd967cbb,version=1 storefront 1/1 Running 0 2m6s app=storefront
05The real fix
Remove version from the service selector. Endpoints come back and every request succeeds. Action item for the postmortem: never put a version label in a service selector.
randy@macbook ~ % kubectl -n shop patch svc checkout --type=json -p '[{"op":"remove","path":"/spec/selector/version"}]' service/checkout patched
randy@macbook ~ % kubectl -n shop get endpoints checkout NAME ENDPOINTS AGE checkout 10.244.1.3:8080,10.244.1.4:8080,10.244.2.4:8080 2m6s
randy@macbook ~ % kubectl -n shop exec storefront -- python /app/hit.py 40 requests: ........................................ 200 OK: 40 errors: 0 error rate: 0%
06Traffic spike: capacity
Marketing sends a big email and you scale to 8 replicas. Some pods stay Pending with Insufficient cpu, even though real CPU usage is tiny. The scheduler only looks at requests. Right-size the requests from real usage and everything fits.
randy@macbook ~ % kubectl -n shop scale deploy/checkout --replicas=8 deployment.apps/checkout scaled
randy@macbook ~ % kubectl -n shop get pods NAME READY STATUS RESTARTS AGE checkout-5dd967cbb-2pqgt 1/1 Running 0 2m26s checkout-5dd967cbb-58s52 0/1 Pending 0 15s checkout-5dd967cbb-hl7bg 1/1 Running 0 15s checkout-5dd967cbb-jk6bl 1/1 Running 0 2m26s checkout-5dd967cbb-r5pxc 0/1 Pending 0 15s checkout-5dd967cbb-sj752 1/1 Running 0 2m26s checkout-5dd967cbb-ssdhd 0/1 Pending 0 15s checkout-5dd967cbb-v5v4d 0/1 Pending 0 15s storefront 1/1 Running 0 2m26s
randy@macbook ~ % kubectl -n shop describe pod checkout-5dd967cbb-58s52 | tail -6 Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedScheduling 15s default-scheduler 0/3 nodes are available: 1 node(s) had untolerated taint {node-role.kubernetes.io/control-plane: }, 2 Insufficient cpu. preemption: 0/3 nodes are available: 1 Preemption is not helpful for scheduling, 2 No preemption victims found for incoming pod.
randy@macbook ~ % for n in $(kubectl get nodes -o name); do echo "== ${n#node/}"; kubectl describe $n | grep -A5 'Allocated resources' | grep cpu; done == prod-east-control-plane cpu 950m (23%) 100m (2%) == prod-east-worker cpu 3100m (77%) 100m (2%) == prod-east-worker2 cpu 3200m (80%) 100m (2%)
randy@macbook ~ % kubectl -n shop top pods NAME CPU(cores) MEMORY(bytes) checkout-5dd967cbb-2pqgt 1m 11Mi checkout-5dd967cbb-jk6bl 1m 11Mi checkout-5dd967cbb-sj752 2m 11Mi storefront 1m 12Mi
randy@macbook ~ % kubectl -n shop set resources deploy/checkout -c checkout --requests=cpu=100m,memory=48Mi deployment.apps/checkout resource requirements updated
randy@macbook ~ % kubectl -n shop rollout status deploy/checkout --timeout=180s Waiting for deployment "checkout" rollout to finish: 4 out of 8 new replicas have been updated... Waiting for deployment "checkout" rollout to finish: 4 out of 8 new replicas have been updated... Waiting for deployment "checkout" rollout to finish: 4 out of 8 new replicas have been updated... Waiting for deployment "checkout" rollout to finish: 4 out of 8 new replicas have been updated... Waiting for deployment "checkout" rollout to finish: 5 out of 8 new replicas have been updated... Waiting for deployment "checkout" rollout to finish: 5 out of 8 new replicas have been updated... Waiting for deployment "checkout" rollout to finish: 6 out of 8 new replicas have been updated... Waiting for deployment "checkout" rollout to finish: 6 out of 8 new replicas have been updated... Waiting for deployment "checkout" rollout to finish: 6 out of 8 new replicas have been updated... Waiting for deployment "checkout" rollout to finish: 6 out of 8 new replicas have been updated... Waiting for deployment "checkout" rollout to finish: 7 out of 8 new replicas have been updated... Waiting for deployment "checkout" rollout to finish: 7 out of 8 new replicas have been updated... Waiting for deployment "checkout" rollout to finish: 2 old replicas are pending termination... Waiting for deployment "checkout" rollout to finish: 2 old replicas are pending termination... Waiting for deployment "checkout" rollout to finish: 2 old replicas are pending termination... Waiting for deployment "checkout" rollout to finish: 1 old replicas are pending termination... Waiting for deployment "checkout" rollout to finish: 1 old replicas are pending termination... Waiting for deployment "checkout" rollout to finish: 6 of 8 updated replicas are available... Waiting for deployment "checkout" rollout to finish: 7 of 8 updated replicas are available... deployment "checkout" successfully rolled out
randy@macbook ~ % kubectl -n shop get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES checkout-5dd967cbb-2pqgt 1/1 Terminating 0 2m31s 10.244.1.4 prod-east-worker <none> <none> checkout-5dd967cbb-hl7bg 1/1 Terminating 0 20s 10.244.2.6 prod-east-worker2 <none> <none> checkout-5dd967cbb-jk6bl 1/1 Terminating 0 2m31s 10.244.1.3 prod-east-worker <none> <none> checkout-5dd967cbb-sj752 1/1 Terminating 0 2m31s 10.244.2.4 prod-east-worker2 <none> <none> checkout-7d54675988-22kc9 1/1 Running 0 4s 10.244.2.7 prod-east-worker2 <none> <none> checkout-7d54675988-4qh4w 1/1 Running 0 4s 10.244.1.5 prod-east-worker <none> <none> checkout-7d54675988-7ppqv 1/1 Running 0 2s 10.244.2.10 prod-east-worker2 <none> <none> checkout-7d54675988-9qp94 1/1 Running 0 4s 10.244.1.6 prod-east-worker <none> <none> checkout-7d54675988-jcz5v 1/1 Running 0 2s 10.244.1.8 prod-east-worker <none> <none> checkout-7d54675988-k9wq6 1/1 Running 0 4s 10.244.2.8 prod-east-worker2 <none> <none> checkout-7d54675988-kwzgx 1/1 Running 0 2s 10.244.2.9 prod-east-worker2 <none> <none> checkout-7d54675988-vslt2 1/1 Running 0 2s 10.244.1.7 prod-east-worker <none> <none> storefront 1/1 Running 0 2m31s 10.244.1.2 prod-east-worker <none> <none>
randy@macbook ~ % kubectl -n shop exec storefront -- python /app/hit.py 40 requests: ........................................ 200 OK: 40 errors: 0 error rate: 0%
07Fix forward
Ship v2 again with a memory limit that fits its cache and a selector without a version. Watch it roll out cleanly and serve traffic.
randy@macbook ~ % kubectl apply -f checkout-v2-fixed.yaml deployment.apps/checkout configured
randy@macbook ~ % kubectl -n shop rollout status deploy/checkout --timeout=240s Waiting for deployment "checkout" rollout to finish: 1 out of 3 new replicas have been updated... Waiting for deployment "checkout" rollout to finish: 1 out of 3 new replicas have been updated... Waiting for deployment "checkout" rollout to finish: 1 out of 3 new replicas have been updated... Waiting for deployment "checkout" rollout to finish: 2 out of 3 new replicas have been updated... Waiting for deployment "checkout" rollout to finish: 2 out of 3 new replicas have been updated... Waiting for deployment "checkout" rollout to finish: 2 out of 3 new replicas have been updated... Waiting for deployment "checkout" rollout to finish: 1 old replicas are pending termination... Waiting for deployment "checkout" rollout to finish: 1 old replicas are pending termination... deployment "checkout" successfully rolled out
randy@macbook ~ % kubectl -n shop get pods NAME READY STATUS RESTARTS AGE checkout-5dd967cbb-2pqgt 1/1 Terminating 0 2m43s checkout-5dd967cbb-hl7bg 1/1 Terminating 0 32s checkout-5dd967cbb-jk6bl 1/1 Terminating 0 2m43s checkout-5dd967cbb-sj752 1/1 Terminating 0 2m43s checkout-7d54675988-22kc9 1/1 Terminating 0 16s checkout-7d54675988-4qh4w 1/1 Terminating 0 16s checkout-7d54675988-7ppqv 1/1 Terminating 0 14s checkout-7d54675988-9qp94 1/1 Terminating 0 16s checkout-7d54675988-jcz5v 1/1 Terminating 0 14s checkout-7d54675988-k9wq6 1/1 Terminating 0 16s checkout-7d54675988-kwzgx 1/1 Terminating 0 14s checkout-7d54675988-vslt2 1/1 Terminating 0 14s checkout-bbcf8fbcb-26wb2 1/1 Running 0 4s checkout-bbcf8fbcb-2p9c2 1/1 Running 0 7s checkout-bbcf8fbcb-4x9c7 1/1 Running 0 2s storefront 1/1 Running 0 2m43s
randy@macbook ~ % kubectl -n shop top pods NAME CPU(cores) MEMORY(bytes) checkout-7d54675988-22kc9 49m 11Mi checkout-7d54675988-4qh4w 62m 11Mi checkout-7d54675988-7ppqv 1m 11Mi checkout-7d54675988-9qp94 1m 11Mi checkout-7d54675988-jcz5v 46m 11Mi checkout-7d54675988-k9wq6 1m 11Mi checkout-7d54675988-kwzgx 45m 11Mi checkout-7d54675988-vslt2 1m 11Mi checkout-bbcf8fbcb-2p9c2 41m 92Mi storefront 5m 2Mi
randy@macbook ~ % kubectl -n shop rollout history deploy/checkout deployment.apps/checkout REVISION CHANGE-CAUSE 2 release 2.0 - add product cache warm-up 3 release 1.0 - baseline 4 release 1.0 - baseline 5 release 2.1 - cache warm-up with right-sized memory
randy@macbook ~ % kubectl -n shop exec storefront -- python /app/hit.py 40 requests: ........................................ 200 OK: 40 errors: 0 error rate: 0%
randy@macbook ~ % kubectl -n shop logs deploy/checkout | head -8 checkout v1 listening on :8080 Found 11 pods, using pod/checkout-7d54675988-22kc9
08Kubernetes rapid fire
QoS classes, how service DNS resolves (and the ndots setting), a PodDisruptionBudget blocking a node drain, and the most common image pull mistake.
randy@macbook ~ % kubectl -n shop get pods -l app=checkout -o custom-columns=NAME:.metadata.name,QOS:.status.qosClass,NODE:.spec.nodeName | head -5 NAME QOS NODE checkout-bbcf8fbcb-nnh9g Burstable prod-east-worker2 checkout-bbcf8fbcb-vhpxb Burstable prod-east-worker checkout-bbcf8fbcb-w4gdg Burstable prod-east-worker2
randy@macbook ~ % kubectl -n shop exec storefront -- nslookup checkout.shop Server: 10.96.0.10 Address: 10.96.0.10:53 Non-authoritative answer: Name: checkout.shop Address: 2606:4700:3033::ac43:a79f Name: checkout.shop Address: 2606:4700:3037::6815:4312 Non-authoritative answer: Name: checkout.shop Address: 172.67.167.159 Name: checkout.shop Address: 104.21.67.18
randy@macbook ~ % kubectl -n shop exec storefront -- cat /etc/resolv.conf search shop.svc.cluster.local svc.cluster.local cluster.local us-east-2.compute.internal nameserver 10.96.0.10 options ndots:5
randy@macbook ~ % kubectl apply -f pdb.yaml poddisruptionbudget.policy/checkout-pdb created
randy@macbook ~ % kubectl -n shop get pdb NAME MIN AVAILABLE MAX UNAVAILABLE ALLOWED DISRUPTIONS AGE checkout-pdb N/A 0 0 1s
randy@macbook ~ % kubectl drain prod-east-worker --ignore-daemonsets --delete-emptydir-data --timeout=25s node/prod-east-worker cordoned evicting pod shop/checkout-bbcf8fbcb-vhpxb evicting pod kube-system/metrics-server-8467fcc7b7-6pmbk pod/metrics-server-8467fcc7b7-6pmbk evicted evicting pod shop/checkout-bbcf8fbcb-vhpxb evicting pod shop/checkout-bbcf8fbcb-vhpxb evicting pod shop/checkout-bbcf8fbcb-vhpxb evicting pod shop/checkout-bbcf8fbcb-vhpxb evicting pod shop/checkout-bbcf8fbcb-vhpxb Warning: ignoring DaemonSet-managed Pods: kube-system/kindnet-ns8tg, kube-system/kube-proxy-8htc4 error when evicting pods/"checkout-bbcf8fbcb-vhpxb" -n "shop" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. error when evicting pods/"checkout-bbcf8fbcb-vhpxb" -n "shop" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. error when evicting pods/"checkout-bbcf8fbcb-vhpxb" -n "shop" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. error when evicting pods/"checkout-bbcf8fbcb-vhpxb" -n "shop" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. error when evicting pods/"checkout-bbcf8fbcb-vhpxb" -n "shop" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. There are pending pods in node "prod-east-worker" when an error occurred: error when evicting pods/"checkout-bbcf8fbcb-vhpxb" -n "shop": global timeout reached: 25s pod/checkout-bbcf8fbcb-vhpxb error: unable to drain node "prod-east-worker" due to error: error when evicting pods/"checkout-bbcf8fbcb-vhpxb" -n "shop": global timeout reached: 25s, continuing command... There are pending nodes to be drained: prod-east-worker error when evicting pods/"checkout-bbcf8fbcb-vhpxb" -n "shop": global timeout reached: 25s
randy@macbook ~ % kubectl get nodes NAME STATUS ROLES AGE VERSION prod-east-control-plane Ready control-plane 86s v1.32.2 prod-east-worker Ready,SchedulingDisabled <none> 76s v1.32.2 prod-east-worker2 Ready <none> 76s v1.32.2
randy@macbook ~ % kubectl uncordon prod-east-worker && kubectl delete pdb checkout-pdb -n shop node/prod-east-worker uncordoned poddisruptionbudget.policy "checkout-pdb" deleted
randy@macbook ~ % kubectl -n shop run typo --image=pyhton:3.12-alpine --restart=Never -- sleep 60 pod/typo created
randy@macbook ~ % kubectl -n shop get pod typo NAME READY STATUS RESTARTS AGE typo 0/1 ImagePullBackOff 0 25s
randy@macbook ~ % kubectl -n shop describe pod typo | tail -7 ---- ------ ---- ---- ------- Normal Scheduled 25s default-scheduler Successfully assigned shop/typo to prod-east-worker Normal BackOff 24s kubelet Back-off pulling image "pyhton:3.12-alpine" Warning Failed 24s kubelet Error: ImagePullBackOff Normal Pulling 9s (x2 over 25s) kubelet Pulling image "pyhton:3.12-alpine" Warning Failed 9s (x2 over 25s) kubelet Failed to pull image "pyhton:3.12-alpine": failed to pull and unpack image "docker.io/library/pyhton:3.12-alpine": failed to resolve reference "docker.io/library/pyhton:3.12-alpine": pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed Warning Failed 9s (x2 over 25s) kubelet Error: ErrImagePull
09Linux on-call: disk full, but du finds nothing
df says the volume is almost full, du says the files are small. A process still holds a deleted log file open. lsof +L1 finds it, and truncating the file through /proc frees the space without restarting the process. (To reproduce this part, run it on any Linux VM: write a large file from a long-running process, then delete it.)
randy@macbook ~ % df -h /var/log/checkout Filesystem Size Used Avail Use% Mounted on tmpfs 512M 471M 42M 92% /var/log/checkout
randy@macbook ~ % du -sh /var/log/checkout/* 4.0K /var/log/checkout/status.txt
randy@macbook ~ % ls -la /var/log/checkout total 20 drwxrwxrwt. 2 root root 60 Oct 26 23:49 . drwxr-xr-x. 10 root root 16384 Oct 26 23:49 .. -rw-r--r--. 1 root root 23 Oct 26 23:49 status.txt
randy@macbook ~ % lsof +L1 2>/dev/null | head -5 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NLINK NODE NAME dbus-brok 2058 dbus 12u REG 0,1 2097152 0 2 /memfd:dbus-broker-log (deleted) python3 38182 root 3w REG 0,316 492830720 0 2 /var/log/checkout/app.log (deleted)
randy@macbook ~ % PID=$(lsof +L1 2>/dev/null | awk '/app.log/{print $2; exit}'); ls -l /proc/$PID/fd | grep deleted l-wx------. 1 root root 64 Oct 26 23:49 3 -> /var/log/checkout/app.log (deleted)
randy@macbook ~ % : > /proc/38182/fd/3 && df -h /var/log/checkout Filesystem Size Used Avail Use% Mounted on tmpfs 512M 4.0K 512M 1% /var/log/checkout
randy@macbook ~ % uptime 23:49:28 up 3 min, 0 users, load average: 0.86, 0.67, 0.28
randy@macbook ~ % free -m total used free shared buff/cache available Mem: 15619 1623 9040 28 4955 13638 Swap: 0 0 0
randy@macbook ~ % df -i / | head -3 Filesystem Inodes IUsed IFree IUse% Mounted on /dev/nvme0n1p1 15722480 96912 15625568 1% /
Check yourself
8 questions on what happened in this lab. Pick an answer for each, then check.
Go deeper
Field Guide · 97 pages · 12 labs
Field Guide · 70 pages · 11 labs
Take this further
The Field Guides cover the whole topic: more labs, a troubleshooting playbook, interview questions and a 30-day plan.
Kubernetes Full Course 2026: Zero to Production
Learn Kubernetes the way production uses it: the control loop, every core workload type, and a 3-tier app deployed properly on kind.
$39 CADView guide
Kubernetes Networking Deep Dive
Trace every packet in a Kubernetes cluster, from pod IP to Service, DNS, Gateway and policy, and fix it when it breaks.
$39 CADView guide
Or get every guide with All-Access.
Get the next lab first
One real lab every week, straight to your inbox.
One email a week, real labs only. Unsubscribe any time.