We ran a full SRE mock interview against a live Kubernetes cluster: fundamentals, a real production incident, Kubernetes rapid fire, Linux on-call, observability, system design, a postmortem and behavioural questions. Every answer was scored.
This post collects the questions that matter most, with the answers that scored well and, just as useful, the follow-ups where points were lost. Where a question was tested on the cluster, we've included the real output.
SRE fundamentals
What's the difference between an SLI, an SLO and an SLA?
An SLI is the thing you measure. For checkout, maybe the percentage of payment requests that succeed in under 500 ms. The SLO is the internal target for that number, like 99.9% over 30 days. The SLA is the promise to customers, usually with money attached.
The line that lands: the SLA is always looser than the SLO. You want to find out you're in trouble long before the lawyers do.
How much downtime is 99.9% over 30 days?
Thirty days is 43,200 minutes. 0.1% of that is 43.2 minutes. That's the error budget.
Then say what you do with it. You spend it. If the budget is barely touched, ship faster and run riskier migrations. If it's burned, slow down and spend time on reliability until it recovers. The answer that separates people who read about SRE from people who've lived it: the error budget policy must be agreed and signed off by engineering and product before anything breaks. Otherwise every release freeze becomes an argument at the worst moment.
What is toil?
Work that's manual, repetitive, automatable, has no lasting value, and grows with the service. Classic example: someone restarting a leaky job every morning. Google's SRE book suggests keeping toil under 50% of an SRE's time. The bonus point is measuring it: tag toil tickets so you can show the trend and justify the automation work.
It's 3 a.m. and CPU on one node is at 95%. Do you wake up?
This is a trap. That alert shouldn't page anyone. High CPU is a cause, not a symptom. Customers feel errors and slow pages, not CPU. Page on symptoms (error rate, latency, SLO burn) and keep CPU on a dashboard or as a morning ticket. If high CPU really hurts users, the symptom alert will catch it. A pager that fires for a batch job doing its job teaches people to ignore the pager, and a pager nobody trusts is worse than none.
The live incident questions
In the incident round, a release took checkout down. One new pod was in CrashLoopBackOff, and its describe output showed this:
Last State: Terminated
Reason: OOMKilled
Exit Code: 137Why exit code 137?
128 plus 9. When a process dies from a signal, the exit code is 128 plus the signal number, and 9 is SIGKILL. SIGKILL can't be caught, so the app never gets to log anything. It just disappears. Here the previous container's logs showed it warming an 80 MB cache under a 64Mi memory limit, so it could never start.
The rollback worked, the pods are healthy, and customers are still down. What now?
This is the question that decides the round. kubectl rollout undo brought back three healthy v1 pods, and the test still failed 40 out of 40.
Healthy pods plus failing requests means traffic isn't reaching the pods. Stop looking at pods and follow the request: service, endpoints, pod labels.
$ kubectl -n shop get endpoints checkout NAME ENDPOINTS AGE checkout <none> 2m6s $ kubectl -n shop get svc checkout -o wide NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR checkout ClusterIP 10.96.31.165 <none> 8080/TCP 2m6s app=checkout,version=2
The pipeline had patched the service selector to version=2, and the rollback only rolled back the deployment. The fix is to remove version from the selector. Versioning belongs to the deployment, not the service. We wrote this incident up step by step in [CrashLoopBackOff and OOMKilled in a real incident](/blog/kubernetes-crashloopbackoff-oomkilled-real-incident/).
Where did the candidate lose a point?
Communication. The moment every request was failing, nobody was told. In a real incident someone declares it, someone runs comms and someone fixes, even if all three are you. Declare, post a status, then debug, and update every 15 to 30 minutes even if the update is "still investigating". That's the gap between mid-level and senior.
Kubernetes questions
Requests vs limits?
Requests are what the scheduler reserves. Limits are the ceiling. Go over the memory limit and you're OOMKilled. Go over a CPU limit and you're throttled, which is sneaky because the pod looks healthy and is just slow. The scheduler only looks at requests, never real usage. In the lab, pods requesting 1500m CPU and using 1m left new replicas Pending with Insufficient cpu.
What are the QoS classes, and why do they matter?
Guaranteed (requests equal limits for every container), Burstable (requests set but not equal to limits), BestEffort (nothing set). When a node runs low on memory, BestEffort pods are evicted first, then Burstable pods using more than they asked for, then Guaranteed. Tie QoS to eviction order without being asked; the interviewer noticed when it was missing.
How does a service name become an IP?
CoreDNS resolves service.namespace.svc.cluster.local to the ClusterIP, and kube-proxy rules on the node send traffic to a healthy endpoint. The bonus point is /etc/resolv.conf:
$ kubectl -n shop exec storefront -- cat /etc/resolv.conf search shop.svc.cluster.local svc.cluster.local cluster.local us-east-2.compute.internal nameserver 10.96.0.10 options ndots:5
With ndots:5, any name with fewer than five dots is tried against every search domain first, so a call to an external API can trigger several lookups before the real one.
You're draining a node to patch it. What can go wrong?
A PodDisruptionBudget can block it. We set maxUnavailable: 0 and tried:
$ kubectl drain prod-east-worker --ignore-daemonsets --delete-emptydir-data --timeout=25s node/prod-east-worker cordoned ... error when evicting pods/"checkout-bbcf8fbcb-vhpxb" -n "shop" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. ... error: unable to drain node "prod-east-worker" due to error: error when evicting pods/"checkout-bbcf8fbcb-vhpxb" -n "shop": global timeout reached: 25s, continuing command...
The strong answer: don't delete the PDB yourself at 2 a.m. Talk to the owning team, and fix the budget to something sane like maxUnavailable: 1 with enough replicas. Max unavailable zero isn't safety. It means you can never patch.
A pod is in ImagePullBackOff. First step?
Read the event. In the lab it said Failed to pull image "pyhton:3.12-alpine". A typo. After that, check registry credentials, the pull secret and whether the tag exists.
Linux on-call
df says the disk is 92% full, but du finds almost nothing. Why?
A deleted file that a process still has open. Deleting removes the name, not the data, until the last process closes it.
$ lsof +L1 2>/dev/null | head -5 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NLINK NODE NAME ... python3 38182 root 3w REG 0,316 492830720 0 2 /var/log/checkout/app.log (deleted) $ : > /proc/38182/fd/3 && df -h /var/log/checkout Filesystem Size Used Avail Use% Mounted on tmpfs 512M 4.0K 512M 1% /var/log/checkout
Truncating through /proc/<pid>/fd frees the space without restarting the process. Then the real fix: log rotation with copytruncate or a signal to reopen the log, and a disk alert at 80%, not 100%. "I'd just reboot it" is the answer that loses points.
Quick follow-ups worth knowing: load average is compared against the CPU count (and on Linux includes processes waiting on disk), free should be read from the available column, and a disk can be "full" from inodes with df -i.
Observability and alerting
What are the first four things you'd monitor on a new service?
The four golden signals: latency, traffic, errors and saturation. Split latency by success and failure, because failed requests are often fast and make averages look great. And use percentiles, not averages. If 99 requests take 100 ms and one takes 10 seconds, the average looks fine, but p99 shows the user having a terrible day, often one of your biggest customers.
What is a burn rate?
How fast you're spending the error budget compared to the steady pace. A burn rate of 1 uses the whole budget in 30 days. A burn rate of 14.4 uses it in about two days. The approach from Google's SRE workbook is multi-window alerting: page when burning at 14.4x over the last hour and still burning over the last five minutes, so the alert doesn't fire after the problem has gone. In PromQL terms, compare the ratio of 5xx to all requests against 14.4 times the allowed 0.1% for both windows, and use recording rules in real life.
Logs, metrics or traces?
Metrics tell you something is wrong. Traces tell you where. Logs tell you why. Tie it back to a real incident if you can; the interviewer rewarded that directly.
System design
Design checkout for 99.99%.
Start with whether it's possible. Four nines is about 4 minutes 20 seconds a month, less than one bad deploy. Then dependency math: if checkout hard-depends on three services at 99.9% each, 0.999 cubed is about 99.7%. You can't be more reliable than what you hard-depend on.
So stop hard-depending: accept orders and reconcile inventory later, fail over between two payment providers, and put orders on a durable queue. Add three availability zones with enough spare capacity to lose one, synchronous database replication with automatic failover, and timeouts, retries with exponential backoff and jitter, and circuit breakers on every network call.
The point that was missed: push back on product. Does all of checkout need four nines, or just payment authorisation?
What's the danger of retrying a payment?
Charging someone twice. Every payment request carries an idempotency key, so the provider returns the first result for a repeated key instead of charging again. This is the answer that gets people hired at payment companies.
Postmortems and on-call
Walk me through the postmortem.
Blameless first: not who ran the pipeline, but why the system let a bad release take checkout down. Then summary, impact, a timeline with timestamps, contributing factors, and action items, each with an owner and a date. For our incident the action items were: no version labels in service selectors, a smoke test with production limits, right-sized requests, and an alert when a service's endpoints drop to zero.
Your team gets paged 40 times a week. What do you do in month one?
Measure first. Group a month of pages by alert; usually five alerts cause 80% of the noise. For each one, delete it, turn it into a ticket, or fix the real problem. Every page needs a runbook link or it doesn't page. Then the humans: a rotation of six to eight people, time off after a bad night, and a weekly handoff. Burnout is a reliability problem.
Tell me about a time you pushed back on a release.
Bring data and options, not a "no". The scoring answer: a load test showed the database failing at 60% of expected traffic, so the candidate proposed launching to 10% of users behind a feature flag or slipping two days. To make it a senior answer, add what you changed afterwards so it doesn't happen again.
What actually earns the offer
Three habits scored the points: asking what changed before touching anything, following the request when the obvious fix failed, and backing every decision with evidence. In platform interviews the same pattern holds; the strongest answers in our [Platform Engineer Mock Interview](/episodes/platform-engineer-mock-interview/) also started with finding evidence before guessing.
Go deeper
Practise these on a real cluster with the [Kubernetes Incident Lab](/labs/kubernetes-incident/), which includes the incident, the PDB drain, DNS, ImagePullBackOff and the deleted-file disk problem with real output. Then watch the full scored interview in the [SRE Mock Interview](/episodes/sre-mock-interview/) episode.