First 5 minutes
| Declare it | Open the incident channel. Say what's broken and who's affected. |
| Assign roles | Incident commander, ops lead, comms. One person can't do all three. |
| Check the blast radius | Which users, regions, services? Is it getting worse? |
| Ask: what changed? | Deploys, config, flags, infra, traffic, dependencies in the last hour. |
Stabilise first
| Roll back or disable | Revert the last change, turn off the flag, fail over. Fix the cause later. |
| Verify with users' eyes | Error rate and latency from the edge, not just green pods. |
| Protect the rest | Shed load, rate limit, scale out, pause batch jobs. |
| Write it down | Timeline in the channel as you go: time, action, result. |
Communicate
| Status update every 30 min | Impact, what we're doing, next update time. Even if nothing changed. |
| One voice outward | Comms lead owns the status page and stakeholders. |
| Hand over cleanly | New IC gets the summary, open threads and current hypothesis. |
After
| Confirm recovery | Watch for at least one traffic peak before closing. |
| Blameless postmortem within 5 days | Timeline, impact, contributing factors, what went well. |
| Action items with owners and dates | Detect faster, mitigate faster, prevent recurrence. |
| Burn the error budget honestly | Feed it into SLO reviews and release decisions. |
Remember
- Mitigation beats diagnosis. Users don't care why while it's still down.
- "Human error" is never a root cause. Ask why the system let it happen.
- If you're not sure whether to declare, declare. Closing a small one is cheap.