Look around
kubectl get pods -A -o wide | Every pod, its node and IP |
kubectl get pods -n shop --field-selector=status.phase!=Running | Only the pods that aren't healthy |
kubectl get events -n shop --sort-by=.lastTimestamp | What happened, newest last |
kubectl get deploy,rs,svc,ep -n shop | The whole chain from deployment to endpoints |
kubectl api-resources --namespaced=true | Everything you can get in a namespace |
Debug a pod
kubectl describe pod <pod> -n shop | State, last state, exit code, events |
kubectl logs <pod> -n shop --previous | Logs from the container that crashed |
kubectl logs deploy/checkout -n shop -f --since=10m | Follow logs for a deployment |
kubectl get pod <pod> -o jsonpath='{.status.containerStatuses[0].lastState}' | Why the last container died (OOMKilled, 137) |
kubectl exec -it <pod> -n shop -- sh | Shell into a running container |
kubectl debug -it <pod> --image=busybox --target=app | Ephemeral debug container in the pod |
Services and networking
kubectl get endpointslices -n shop -l kubernetes.io/service-name=checkout | Which pod IPs the service actually routes to |
kubectl get pods -n shop --show-labels | Compare pod labels with the service selector |
kubectl run tmp --rm -it --image=busybox -- wget -qO- http://checkout.shop | Test a service from inside the cluster |
kubectl port-forward svc/checkout 8080:80 -n shop | Reach a service from your laptop |
Rollouts
kubectl rollout status deploy/checkout -n shop | Wait for a rollout to finish |
kubectl rollout history deploy/checkout -n shop | Revisions and change causes |
kubectl rollout undo deploy/checkout -n shop | Back to the previous ReplicaSet |
kubectl set image deploy/checkout app=shop/checkout:v3 -n shop | Fix forward with a new image |
kubectl rollout restart deploy/checkout -n shop | Recreate pods without changing the spec |
Capacity and resources
kubectl top nodes / kubectl top pods -n shop | Real usage (needs metrics-server) |
kubectl describe node <node> | grep -A8 Allocated | Requests vs allocatable on a node |
kubectl get pods -A -o custom-columns=NS:.metadata.namespace,POD:.metadata.name,QOS:.status.qosClass | QoS class of every pod |
kubectl get pdb -A | PodDisruptionBudgets that can block drains |
kubectl cordon <node> && kubectl drain <node> --ignore-daemonsets --delete-emptydir-data | Take a node out safely |
Remember
- Exit code 137 = SIGKILL. With reason OOMKilled, the container hit its memory limit.
- rollout undo doesn't help if the service selector no longer matches the pods. Check endpoints first.
- Pending pods? Read the scheduler event: Insufficient cpu usually means requests, not real usage.