Incident Postmortem

An incident postmortem is a structured, written analysis produced after an outage or serious failure — documenting the timeline, root causes, impact, and corrective actions — with the goal of preventing recurrence rather than assigning blame.

The practice is a load-bearing ritual of modern reliability engineering, codified publicly in Google's SRE book and practiced at every serious engineering organization. Its central discipline is blamelessness: the analysis assumes the humans involved acted reasonably given what they knew, and asks what about the system — tooling, alerts, defaults, documentation, process — allowed a reasonable person's action to become an outage.

How it works

A strong postmortem contains:

• Summary and impact: what broke, for how long, how many users or dollars affected.
• Timeline: a minute-by-minute reconstruction — when the bad change shipped, when alerts fired, when humans engaged, each diagnostic turn, when mitigation landed. Honest timelines are where the real lessons hide: the 40 minutes lost to a misleading dashboard matters more than the one-line fix.
• Root cause analysis: not the trigger but the conditions. Techniques like the five whys push past "an engineer ran the wrong command" to "the destructive command and the safe one differ by one flag and nothing asks for confirmation." A named human is never a root cause.
• Action items with owners and deadlines: the mechanism by which the organization actually learns. A postmortem without tracked follow-through is theater.

Blamelessness is not softness — it is epistemics. People who fear punishment sanitize timelines, and sanitized timelines teach nothing. The organizations with the most honest postmortems have the fewest repeat incidents.

Why it matters

Incidents are the most expensive education an engineering organization ever buys; postmortems are how you keep the receipts. Written well and shared widely, they compound into institutional memory — new engineers absorb a decade of hard-won failure knowledge without paying the outage price again. Publicly published postmortems from companies like Cloudflare and GitLab extend that gift industry-wide: a real, documented outage is a richer engineering text than most textbooks, because it shows how systems fail in combination, under time pressure, with incomplete information.

A worked example

A payments outage postmortem, condensed:

Impact: Checkout failed for 43 minutes; ~2,100 orders lost.

Trigger: A config change reduced the payment service's

connection pool from 100 to 10 — a typo.

Timeline: 14:02 change deployed. 14:09 error-rate alert fired.

14:31 root cause found (misleading dashboard suggested

the database was at fault). 14:45 rollback complete.

Roots: Config changes skip code review; no validation rejects

a 10x pool reduction; the latency dashboard aggregates

in a way that pointed at the wrong tier.

Actions: Config changes enter PR review (owner: platform, 2 wks);

add bounds validation; fix dashboard aggregation.

Note what is absent: the name of whoever made the typo. It is irrelevant to preventing the next one.

How miatz uses it

Real postmortems are miatz's War Room curriculum. Learners drill inside incidents reconstructed from publicly documented outages — reading the same misleading signals, feeling the same time pressure, then writing their own blameless postmortem and comparing it against the published one. Diagnosis under pressure is a skill, and like every skill miatz trains, it is built through reps, not lectures.