Incident Reviews That Change the Code
I have sat in postmortems that produced beautiful timelines and zero safer systems. The doc got shared. The Slack channel went quiet. Two sprints later, the same class of failure returned with a different costume.
A review only counts if the code, config, or process that allowed the incident is different when you leave the room.
Blameless is not consequence-free
Blameless culture means you do not hunt people. It does not mean you skip ownership of fixes.
In regulated crypto SRE, the useful posture was:
- Assume competent people hit a bad system shape
- Name the missing guardrail without naming a villain
- Assign a single owner and a merge-by date for each fix
- Track those fixes like customer bugs, not "nice to haves"
If the only outcome is "we should be more careful," the review failed.
Write the story for the next on-call
A good incident review answers four questions in order:
- What did the user experience? Not the internal exception name — the broken promise.
- What was the detection gap? How long until a human or alert knew?
- What made recovery hard? Missing runbook, wrong dashboard, unsafe rollback.
- What systemic change prevents the class? Not the exact line that broke.
Timelines matter for compliance. Engineers need the causal chain more than the minute-by-minute Slack archaeology.
Prefer mechanical prevention
Training and "remember to check X" rot. Prefer fixes that fail closed:
| Soft fix | Harder, better fix |
|---|---|
| "Double-check the flag" | Flag defaults to safe; progressive rollout required |
| "Monitor the queue depth" | Alert + auto-shed at a threshold |
| "Don't deploy on Fridays" | Change freeze window enforced in CI |
| "Be careful with retries" | Idempotency keys + bounded backoff in the client |
From SRE lessons in crypto: ambiguity is the enemy. Reviews should remove ambiguity from the next incident, not add a checklist item.
Cap the action items
Five unfinished action items are worse than two shipped ones. My rule:
- At most three must-fix items from a single review
- Each has an owner, a PR or ticket link, and a date
- Anything else goes in a backlog labeled
incident-followup— visible, not fictional
Close the incident when the must-fixes merge, not when the doc is approved.
Close the loop in public (internally)
Send a short "what changed" note after the fixes land: alert added, retry bounded, runbook updated. People trust the process when they see the system move.
Externally, the same habit shows up here as writing in public — but for incidents, the audience is your future self at 3 a.m.
The standard
Ask one question at the end of every review: If this happened tomorrow, would recovery be faster for a reason we can point to in the repo?
If you cannot point, you held a meeting. You did not improve reliability.