Most CrashLoopBackOff guides show you a list of commands. That's useful, but it isn't what an incident feels like. In a real incident you run a command, read the output, form a guess, and run the next command because of what you just saw.
So we built one. A three-node Kubernetes cluster (kind, v1.32), a small online shop with a storefront pod and a checkout deployment behind a service, and a release that takes checkout down. Every command and every line of output below came from that cluster. We'll walk through it in the order we debugged it, including the part where the obvious fix didn't work.
The healthy baseline
Before you break anything, know what normal looks like. Three checkout pods, one storefront, and a small test script in the storefront that sends 40 requests to checkout and prints a dot for every success.
$ kubectl -n shop exec storefront -- python /app/hit.py 40 requests: ........................................ 200 OK: 40 errors: 0 error rate: 0%
Then release 2.0 ships. Two things happen in the pipeline. A new deployment manifest goes out, and the service selector gets patched to point at the new version:
$ kubectl apply -f checkout-v2.yaml
deployment.apps/checkout configured
$ kubectl -n shop patch svc checkout -p '{"spec":{"selector":{"app":"checkout","version":"2"}}}'
service/checkout patchedRemember that second command. It's the whole story.
Step 1: Confirm the impact and ask what changed
The first thing we check is the thing customers feel:
$ kubectl -n shop exec storefront -- python /app/hit.py 40 requests: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 200 OK: 0 errors: 40 error rate: 100%
Zero out of forty. That isn't degraded, that's down. Now the pods:
$ kubectl -n shop get pods NAME READY STATUS RESTARTS AGE checkout-5dd967cbb-2pqgt 1/1 Running 0 82s checkout-5dd967cbb-jk6bl 1/1 Running 0 82s checkout-5dd967cbb-sj752 1/1 Running 0 82s checkout-8b854d58f-jpsjt 0/1 CrashLoopBackOff 3 (30s ago) 74s storefront 1/1 Running 0 82s
Three old pods are running, and one new pod is in CrashLoopBackOff. The new pod has a different hash in its name (8b854d58f), so it belongs to a new ReplicaSet. Something got deployed. The rollout history confirms it:
$ kubectl -n shop rollout history deploy/checkout deployment.apps/checkout REVISION CHANGE-CAUSE 1 release 1.0 - baseline 2 release 2.0 - add product cache warm-up
This is why we always set kubernetes.io/change-cause on deployments. At 3 a.m. "what changed?" is the most useful question you can ask, and a readable history answers it in one command.
Notice something odd, though. Three v1 pods are still running, so why is every request failing? Hold that thought. It comes back later.
Step 2: Read the pod's last state
CrashLoopBackOff isn't a cause. It just means the container keeps exiting and the kubelet is backing off between restarts. The cause is in the container's last state. kubectl describe pod is long, so here are the lines that matter:
$ kubectl -n shop describe pod checkout-8b854d58f-jpsjt
...
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: OOMKilled
Exit Code: 137
...
Restart Count: 3
Limits:
memory: 64Mi
Requests:
cpu: 1500m
memory: 48Mi
...
QoS Class: BurstableOOMKilled with exit code 137. The container went over its memory limit and the kernel killed it.
Why exit code 137 means OOMKilled
This one comes up in almost every SRE and Kubernetes interview. When a process dies from a signal, its exit code is 128 plus the signal number. SIGKILL is signal 9, so 128 + 9 = 137.
SIGKILL can't be caught or handled. The app gets no chance to log an error or clean up. It just disappears. That's why the application logs often look like they stop mid-sentence, and why you should look for 137 in the pod's last state rather than in the app's own output.
One caution: 137 means "killed with SIGKILL", and the OOM killer is the most common sender, but not the only one. The Reason: OOMKilled line is what confirms it was memory.
Step 3: Read the previous container's logs
Plain kubectl logs on a crash-looping pod often shows you the current attempt, which may have just started. You want the container that died. That's what --previous is for:
$ kubectl -n shop logs checkout-8b854d58f-jpsjt --previous checkout v2: warming product cache (80 MB)... cache loaded 20 MB cache loaded 40 MB
The new version warms an 80 MB product cache on startup. The memory limit is 64Mi. It loads 40 MB, keeps going, and gets killed. Version 2 can never start. It isn't flaky, it's mathematically dead.
The events tell the same story from the kubelet's side: the readiness probe failing with connection refused (the app never got far enough to listen on 8080), then Back-off restarting failed container.
Step 4: Roll back now, fix forward later
Customers are still failing. We have a clear suspect and enough evidence for the postmortem. Time to undo the release:
$ kubectl -n shop rollout undo deploy/checkout deployment.apps/checkout rolled back $ kubectl -n shop rollout status deploy/checkout --timeout=120s deployment "checkout" successfully rolled out
The crashing pod is terminating and three v1 pods are running. Run the test again:
$ kubectl -n shop exec storefront -- python /app/hit.py 40 requests: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 200 OK: 0 errors: 40 error rate: 100%
Still 40 out of 40 failing.
Why rollout undo didn't fix it: the service selector trap
This is the moment that separates people who follow a runbook from people who debug. The rollback worked. The pods are healthy. Customers are still down.
Healthy pods plus failing requests means traffic isn't reaching the pods. So we stop looking at pods and follow the request instead. The storefront calls the checkout service. Does that service have any endpoints?
$ kubectl -n shop get endpoints checkout NAME ENDPOINTS AGE checkout <none> 2m6s
None. The service is pointing at nothing, which means its selector matches no pods. Let's see the selector and the pod labels side by side:
$ kubectl -n shop get svc checkout -o wide NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR checkout ClusterIP 10.96.31.165 <none> 8080/TCP 2m6s app=checkout,version=2 $ kubectl -n shop get pods --show-labels NAME READY STATUS RESTARTS AGE LABELS checkout-5dd967cbb-2pqgt 1/1 Running 0 2m6s app=checkout,pod-template-hash=5dd967cbb,version=1 checkout-5dd967cbb-jk6bl 1/1 Running 0 2m6s app=checkout,pod-template-hash=5dd967cbb,version=1 checkout-5dd967cbb-sj752 1/1 Running 0 2m6s app=checkout,pod-template-hash=5dd967cbb,version=1 ...
The service selects version=2. Every pod is labelled version=1. The release pipeline didn't just change the deployment, it also patched the service. When we rolled back the deployment, the service didn't roll back with it.
That also explains the odd thing from step 1. Three v1 pods were running the whole time, but the moment the service was patched to version=2, nothing matched except a pod that never became ready.
A rollback only rolls back the thing you rolled back.
The fix: take version out of the service selector
Versioning belongs to the deployment, not the service. The service should select app=checkout and nothing else:
$ kubectl -n shop patch svc checkout --type=json -p '[{"op":"remove","path":"/spec/selector/version"}]'
service/checkout patched
$ kubectl -n shop get endpoints checkout
NAME ENDPOINTS AGE
checkout 10.244.1.3:8080,10.244.1.4:8080,10.244.2.4:8080 2m6s
$ kubectl -n shop exec storefront -- python /app/hit.py 40
requests: ........................................
200 OK: 40 errors: 0 error rate: 0%Three endpoints, forty for forty. We're back.
Fixing forward: right-size the memory limit
Rolling back stops the bleeding, but the product team still wants their cache. So we measure first and ship 2.1 with a memory limit that fits: the request stays at 48Mi and the limit goes from 64Mi to 160Mi. After rolling it out, the new v2 pod sits comfortably under its limit:
$ kubectl -n shop top pods NAME CPU(cores) MEMORY(bytes) ... checkout-bbcf8fbcb-2p9c2 41m 92Mi
92Mi used against a 160Mi limit. Healthy headroom.
One more trap showed up here. Before the fix-forward we had scaled checkout to 8 replicas for a traffic spike. The 2.1 manifest still said replicas: 3, so applying it quietly scaled us back down to three. If an autoscaler owns the replica count, take replicas out of the manifest, or every deploy will fight it.
What we'd tell an on-call engineer
Here's the order that worked, as a checklist you can keep:
- Confirm the user impact (error rate, not CPU).
- Ask what changed:
kubectl rollout history. - Read the pod's Last State, reason and exit code:
kubectl describe pod. - Read the dead container's logs:
kubectl logs --previous. - Roll back if you have a clear suspect.
- If it's still broken, follow the request: service, endpoints, pod labels.
- Fix forward with measured numbers, not guesses.
And one thing the terminal won't remind you of: before you start debugging, declare the incident and post a status update. Even if you're the only person working it. Support is getting complaints and has no idea why. Then update every 15 to 30 minutes, even if the update is "still investigating".
Postmortem action items
A blameless postmortem for this incident lists three contributing factors: no memory test of the new image under its real limits, a service selector tied to a version label, and CPU requests set far higher than real usage. The action items that would stop it happening again:
- Remove
versionfrom service selectors everywhere. - Add a pre-production smoke test that runs the image with production limits.
- Right-size requests from measured usage.
- Alert when a service's endpoints drop to zero. That one alert would have pointed straight at the selector.
Go deeper
You can run this exact incident yourself on a laptop with kind. The [Kubernetes Incident Lab](/labs/kubernetes-incident/) has the manifests, every step and the real output, plus a capacity crunch, a PodDisruptionBudget that blocks a node drain, and the classic "disk full but du finds nothing" Linux problem. To watch it play out as a live interview, with scoring and follow-up questions, see the [SRE Mock Interview](/episodes/sre-mock-interview/) episode.