Incident Response and Postmortems

This is for you if

• You carry a pager, or you're about to, and want a calm, repeatable way to handle the 3 a.m. page
• You're the person who fixes everything and want to turn that into a process your whole team can run
• You lead or build an on-call program and need severity levels, rotations and postmortems that work
• You're preparing for SRE, platform or DevOps interviews where incident stories are guaranteed

What's inside

• A severity matrix tied to SLO burn rates, plus on-call rotation design that people can sustain
• The incident commander, ops, comms and scribe roles, with handover and sync scripts
• Ready-to-use status page, stakeholder update and handoff templates
• A first-five-minutes and first-hour playbook, including evidence capture before you mitigate
• Mitigation levers practised on a live cluster: rollback, feature flag kill switch, failover and load shedding
• A structured method for debugging under pressure, drilled against four hidden faults
• A blameless postmortem template, contributing factors instead of root cause, and the trouble with five whys
• Accurate breakdowns of the Cloudflare 2019, Facebook 2021 and CrowdStrike 2024 outages and what to copy from them
• Game days with Chaos Mesh and manual fault injection, runbooks, honest incident metrics and 20 interview Q&As

After this guide you can

• Declare, command and communicate an incident from page to resolution with clear roles
• End user impact quickly with rollback, flags, failover or load shedding before you know the cause
• Debug production faults on Kubernetes with a structured, hypothesis-driven loop
• Write blameless postmortems with action items that have owners, dates and a real chance of getting done
• Plan and run a game day or a full simulated incident with your team
• Answer senior-level incident response interview questions with judgment, not heroics

Contents

1. The mental model: what an incident really isp. 3
2. Severity levels and on-call that doesn't burn people outp. 12
3. Incident command and communicationp. 17
4. The first five minutes and the first hourp. 22
5. Mitigate first: rollback, flags, failover and load sheddingp. 27
6. Debugging under pressurep. 35
7. Blameless postmortems and action items that get donep. 39
8. Learning from public postmortemsp. 45
9. Game days, chaos engineering and runbooksp. 50
10. Troubleshooting playbookp. 59
11. Production checklist, metrics and design decisionsp. 62
12. Interview questionsp. 65
13. Cheat sheet and 30-day practice planp. 72