SRE Mock Interview: Production Is Down. Can You Fix It?

Sure. An SLI is the thing you measure. A service level indicator. For checkout, that might be the percentage of payment requests that succeed in under five hundred milliseconds.

The SLO is the target we set for that number. Say, ninety-nine point nine percent of requests succeed, measured over thirty days. That's internal. It's the line the engineering team holds itself to.

And the SLA is the promise we make to customers, usually with money attached. If we break it, we pay credits. So the SLA is always looser than the SLO. You want to find out you're in trouble long before the lawyers do.

Thirty days is forty-three thousand two hundred minutes. Point one percent of that is forty-three point two minutes. So roughly forty-three minutes of total failure a month. That's your error budget.

You spend it. That's the part people miss. If we've barely touched it, we can ship faster, run riskier experiments, do that migration. If we've burned through it, we slow down. Feature releases pause, and the team spends its time on reliability until the budget recovers.

It should be agreed before anything breaks. An error budget policy, signed off by engineering and product. Otherwise every freeze turns into an argument at the worst possible moment.

Toil is work that's manual, repetitive, can be automated, and doesn't make the system better over time. And it grows as the service grows. The classic one is a person restarting a stuck job every morning because it leaks memory.

Google's SRE book suggests keeping toil under fifty percent of an SRE's time. If you're above that, you're an operator, not an engineer. My move is to track it. Tickets tagged as toil, so you can show the trend and justify the automation work.

Honestly? That alert shouldn't wake anyone up. CPU at ninety-five percent is a cause, not a symptom. Customers don't feel CPU. They feel errors and slow pages.

So I'd page on symptoms. Error rate, latency, the SLO burning too fast. And keep CPU as a dashboard signal or a ticket for the morning. If high CPU actually hurts users, the symptom alert will catch it.

I've been that person. You get paged for a batch job that's supposed to use all the CPU. Twice. Then you stop trusting the pager.

One twenty-eight plus nine. Nine is SIGKILL. When a process dies from a signal, the exit code is one twenty-eight plus the signal number. And SIGKILL can't be caught, so the app never gets to log anything. It just disappears.

It's warming an eighty megabyte product cache, loads forty, and then... nothing. And the memory limit on this deployment is sixty-four megabytes. So version two can never start. It's not flaky. It's mathematically dead.

Roll back now, fix forward later. Undo the deployment to the previous revision.

Three version-one pods running, and the crashing pod is terminating. Let's run the test again.

Wait. Still forty out of forty failing.

Okay. Healthy pods but failing requests means the traffic isn't reaching them. So I stop looking at the pods and follow the request. Storefront calls the checkout service. Does the service have any endpoints?

None. The service is pointing at nothing. So the selector doesn't match the pods. Show me the service with its selector.

The selector is app checkout, version two. And we just rolled every pod back to version one.

Yep. Every pod is labelled version one. So the release pipeline didn't just change the deployment. It also patched the service to point at version two. When we rolled back the deployment, the service didn't roll back with it.

So the fix is to stop the service caring about version at all. It should select app checkout. Versioning belongs to the deployment, not the service.

Three endpoints. Now the test.

Okay. We're back.

...No one. I went straight into the terminal.

That's fair. Declare, post a status, then debug. And update every fifteen to thirty minutes even if the update is, still investigating.

Requests are what the scheduler reserves for you. Limits are the ceiling. Go over the memory limit, you get OOMKilled, like we just saw. Go over a CPU limit, you don't die, you get throttled. Which is sneaky, because the pod looks healthy and it's just slow.

Three of them. Guaranteed, when requests equal limits for everything. Burstable, when you set requests but they're not equal to limits. BestEffort, when you set nothing. When a node runs out of memory, BestEffort pods get evicted first, then Burstable. Our checkout pods should be Burstable right now.

Burstable. Memory request forty-eight, limit one sixty. If checkout was critical enough, I'd make it Guaranteed.

CoreDNS. Every service gets a DNS name, service dot namespace dot svc dot cluster dot local, and it resolves to the service's cluster IP. Then kube-proxy rules on the node send that traffic to one of the healthy pod endpoints. Let's look from inside the storefront pod.

Wait. Those aren't cluster IPs. That's the public internet. Dot shop is a real top-level domain, and this nslookup sent the name exactly as typed. Our app still works because the normal resolver tries the search domains first. Let's look at them.

Search domains, starting with shop dot svc dot cluster dot local. And ndots five. That ndots line is a famous performance trap. Any name with fewer than five dots gets tried against every search domain first. So calling an external API like api dot stripe dot com can fire off several DNS lookups before the real one.

A PodDisruptionBudget can block it. That's the point of a PDB. It says how many pods of an app are allowed to be down during voluntary disruptions, like drains and upgrades. If someone sets max unavailable to zero, the drain can never finish.

There it is. Cannot evict pod, as it would violate the pod's disruption budget. The drain is stuck, and the node's already cordoned, so nothing new lands on it either.

I don't delete the PDB on my own. It's there for a reason. I'd talk to the owning team, and the fix is a sane budget, like max unavailable one, with enough replicas that losing one is fine. Max unavailable zero isn't safety. It just means you can never patch.

Failed to pull image. It can't even resolve docker dot io, library, P-Y-H-T-O-N. Look at the name. It's a typo. Otherwise I'd check registry credentials, the image pull secret, and whether the tag exists. But first, always read the event. It usually tells you.

Because deleting a file only removes its name. The space isn't freed until the last process closes it. Classic cause: someone runs rm on a big log file, but the app is still writing to it. The file's gone from ls, but not from the disk. lsof plus L one lists open files with zero links.

There it is. A python process holding app dot log, marked deleted. Around four hundred and seventy megabytes.

That works, but it's checkout. I don't want a restart if I can avoid it. You can truncate the file through the process's file descriptor in proc. That frees the space instantly and the process keeps running.

Back to almost nothing used. Then the real fix: proper log rotation, with copytruncate or a signal so the app reopens its log, and an alert at eighty percent disk instead of a hundred.

The four golden signals. Latency, traffic, errors, and saturation.

Latency, but split by success and failure, because failed requests are often fast and they make your averages look great. Traffic, so you know the load. Errors, both explicit five hundreds and wrong answers. And saturation, how full the most constrained resource is.

Because averages hide pain. If ninety-nine requests take a hundred milliseconds and one takes ten seconds, the average looks fine. But that one user is having a terrible day. I'd look at the ninety-ninth percentile. And the users stuck in p99 are often your biggest customers, the ones with the most data.

Burn rate is how fast you're spending the error budget compared to the steady pace. A burn rate of one means you'll use exactly the whole budget in thirty days. A burn rate of fourteen point four means you'd use the whole month's budget in about two days.

The approach from Google's SRE workbook is multi-window alerts. Page if you're burning at fourteen point four times over the last hour, and still burning over the last five minutes. That second window stops the alert from paging after the problem's already gone.

Roughly, the error ratio is the rate of five hundreds over the rate of all requests. Then you compare it to fourteen point four times the allowed error, which is point one percent. And you do the same for the five-minute window and require both.

Metrics tell you something's wrong. Traces tell you where. Logs tell you why. In the incident we just did, the error rate told us something was wrong, following the request found the broken hop, and the pod logs showed the cache warm-up dying.

First, four nines is about four minutes and twenty seconds of downtime a month. That's less than one bad deploy. So it changes how you ship, not just how you build.

Second, the math of dependencies. Checkout calls the payment provider, inventory, and the database. If each one is ninety-nine point nine, and checkout needs all of them, you multiply. Point nine nine nine cubed is about ninety-nine point seven. You can't be more reliable than the things you hard-depend on.

You stop hard-depending on them. Inventory: if it's slow, take the order and reconcile after. The payment provider: two providers, with failover. And put the order on a durable queue, so if a downstream service is down, the order waits instead of failing.

Then redundancy. At least three availability zones, with enough capacity that losing one zone doesn't overload the other two. Database with synchronous replication across zones and automatic failover.

And every network call gets a timeout, retries with exponential backoff and jitter, and a circuit breaker. Without jitter, every client retries at the same moment and you get a thundering herd that keeps the service down.

Charging someone twice. If the request times out, you don't know if the charge happened. So every payment request carries an idempotency key. The provider sees the same key twice and returns the first result instead of charging again.

Small and slow. Canary to one percent, watch the SLO burn rate automatically, then ramp. Auto-rollback if the burn rate spikes. Feature flags so you can turn off code without a deploy. And everything we learned today: no version in service selectors, and test the image under real limits.

First, I don't fix anything. I measure. Every page for the last month, grouped by alert. Usually it's five alerts causing eighty percent of the noise.

Then for each noisy alert, one of three things. Delete it if nobody acts on it. Turn it into a ticket if it can wait until morning. Or fix the real problem if it's a genuine symptom. And I'd make a rule: every page needs a runbook link, or it doesn't page.

A healthy rotation. At least six to eight people so you're not on call every other week. Time off after a bad night. And a weekly on-call handoff where we look at what paged and why. Burnout is a reliability problem too. Tired people make the three a.m. mistakes.

We had a launch date fixed by marketing, and the load test showed the database falling over at about sixty percent of the expected traffic. I didn't just say no. I brought the graph, the error budget, and two options: launch to ten percent of users with a feature flag, or slip two days. They picked the flag. We fixed the query during the ramp, and nobody outside the team ever knew.