vault

Blameless Postmortems

15 min read

Blameless Postmortems

A postmortem is the written record of an incident — what happened, its impact, the actions taken to mitigate it, the contributing causes, and the follow-up work to keep it from recurring (Google SRE Book, ch. 15). A postmortem is blameless when it treats the failure as a property of the system rather than the fault of a person: it assumes everyone involved acted with good intentions and made the best decision they could with the information they had at the time, and it asks not “who broke it?” but “what about our system let a well-intentioned, competent engineer cause this?” The blameless stance is not softness or an excuse — it is an engineering decision grounded in decades of safety science (John Allspaw at Etsy, 2012; Sidney Dekker’s just-culture work). Punish the human and you teach the whole organization to hide the next failure; understand the system and you actually get safer. This note covers both halves: the culture that makes honest accounts possible, and the document that captures the learning.

This is an operational-practice note. It sits downstream of running the incident itself — see Incident Command and the Incident Lifecycle for the roles and the detect-triage-mitigate-resolve loop, and Incident Severity and Escalation for what makes an event postmortem-worthy. The quantitative levers a postmortem often tries to improve are covered in Mean Time to Detect and Recover.

Mental Model — Blame Drives Information Underground

The single most important idea is a feedback loop about information. An incident is an opportunity to learn how the system really behaves, but almost all of the useful knowledge lives inside the heads of the people who were at the “sharp end” — the engineer who ran the command, the on-call who made the judgment call at 3 a.m. That knowledge only becomes organizational learning if those people volunteer a detailed, honest account. Whether they do is decided almost entirely by what they expect to happen to them afterward.

flowchart TD
    INC["Incident happens"] --> Q{"What does the org<br/>do with the humans<br/>at the sharp end?"}

    Q -->|"Blame &amp; punish<br/>(retributive)"| P1["Engineers fear reprisal"]
    P1 --> P2["Give guarded, defensive<br/>'cover-your-ass' accounts"]
    P2 --> P3["Systemic causes stay hidden"]
    P3 --> P4["Same class of failure recurs"]
    P4 --> INC

    Q -->|"Understand &amp; learn<br/>(blameless / restorative)"| B1["Engineers give full,<br/>honest accounts"]
    B1 --> B2["Second story surfaces:<br/>why it made sense at the time"]
    B2 --> B3["Systemic fixes with owners"]
    B3 --> B4["That class of failure<br/>gets harder to repeat"]
    B4 --> INC

What it shows and the insight to take: both paths loop back to another incident — failure is inevitable in any complex system. The difference is what each loop does with the failure. The retributive loop optimizes for finding someone to hold responsible, which teaches people to protect themselves by withholding detail, so the organization never learns and the loop repeats unchanged. The blameless loop optimizes for extracting the maximum learning, which requires that telling the truth be safe. Blamelessness is therefore instrumental, not sentimental: it is the precondition for the accurate information without which no real fix is possible (Allspaw, Etsy, 2012).

Why “Human Error” Is a Symptom, Not a Cause

The intellectual foundation of the blameless movement comes from human-factors and safety research — Erik Hollnagel, David Woods, Richard Cook, James Reason, and Sidney Dekker — imported into software operations by John Allspaw’s 2012 essay “Blameless PostMortems and a Just Culture” while he was CTO of Etsy. The field distinguishes two ways of explaining an accident, which Woods and colleagues named the first story and the second story in Behind Human Error.

The first story (the “old view” or “bad apple theory”) says: the system was basically fine, a careless person broke it, so the remedy is to reprimand that person and remind everyone to be more careful. The second story (the “new view”) says: human error is not a cause but a symptom — a signal pointing at deeper, systemic vulnerabilities in the tools, processes, defaults, and pressures that surrounded the person. The load-bearing claim, in Allspaw’s paraphrase of Hollnagel, is that the action made sense to the person at the time they took it — because if it had not made sense, they would not have taken it (Allspaw). The job of the postmortem is to reconstruct that local rationality: to understand what the world looked like from inside the situation, with the information, tooling, and time pressure actually available, such that the action taken was the reasonable one. When you can explain why it made sense, you have found the real leverage — the misleading dashboard, the missing guardrail, the deploy tool with no confirmation step, the runbook that was subtly wrong.

This reframes accidents entirely. People at the “sharp end” — the operators touching the live system — are not the cause of failure; they are the org’s early-warning sensors, relied upon to inform everyone else where the safety boundary actually lies. Accidents don’t happen because people gamble and lose; they happen because the person believed the bad outcome wasn’t possible, or believed their action had no connection to it, or judged the reward worth the risk given what they knew (Allspaw, drawing on Hollnagel).

The Cognitive Traps: Hindsight Bias and Counterfactuals

Two well-documented cognitive biases make blame feel justified even when it is not, and a disciplined postmortem actively defends against both.

Hindsight bias is the tendency, once you know how a story ended, to see that ending as having been obvious and predictable all along. After an outage we know exactly which signal mattered, so the engineer’s failure to act on it looks like negligence — but in the moment that signal was one of dozens, most of them noise. Hindsight bias makes ordinary, reasonable decisions look like obvious errors, and thereby makes punishment feel deserved when it rarely is (PagerDuty). The countermeasure is a discipline of foresight: build the timeline forward from a point before the incident, reconstructing what was known at each step, instead of reasoning backward from the resolution. Analyze the decision using only the information the person had at that moment, not what you learned later.

Counterfactual reasoning — “if only they had checked the graph,” “the system would have been fine had they not deployed” — is seductive and nearly useless. It describes a fictional world that did not happen instead of explaining the real one that did. Counterfactuals smuggle blame in through the back door by implying the correct action was obvious. The antidote is to keep asking what and how rather than why: “what did you think was happening?”, “how did the tooling present the state to you?” — grounding the account in the concrete situation instead of demanding the person justify themselves (PagerDuty).

Just Culture: Retributive versus Restorative

Sidney Dekker’s framing of just culture gives the sharpest language for what “blameless” is trying to achieve. Dekker contrasts two kinds of justice an organization can pursue after harm (Dekker, Restorative Just Culture Checklist):

  • Retributive just culture asks: which rule was broken? how bad is the breach? what should the consequences be? Accountability here is backward-looking — it locates a person to blame and imposes a proportional sanction. In Dekker’s phrase, an account is something you settle or pay.
  • Restorative just culture asks: who is hurt? what do they need? whose obligation is it to meet that need? Accountability here is forward-looking — the parties together work out what must be done and who should do it. An account is something you tell and learn from.

Dekker’s argument against the retributive path is not that accountability is bad but that retribution is ineffective: it turns into a blunt HR instrument for removing people, it excludes the voices of everyone except “offender” and employer, it correlates with hiding incidents and unwillingness to report, and — decisively for reliability — a retributive response never identifies the systemic contributions, so it invites the failure to repeat (Dekker). Restorative practice also recognizes the second victim: the practitioner involved in the incident is themselves harmed by it — guilt, humiliation, loss of confidence — and a healthy culture reintegrates them rather than scapegoating them. Dekker’s blunt test: if you fire someone, what have you actually fixed? The expensive lesson that engineer just learned walks out the door with them.

This resolves the objection that always arises — “doesn’t blameless mean nobody is accountable?” No. Blameless does not mean consequence-free; it means the accountability is redirected from paying for the past to improving the future. The engineer who “caused” an outage is treated as the person who now understands that failure mode best, and is made accountable for helping the whole organization avoid it — writing the postmortem, driving the fixes, teaching others. Allspaw’s observation was that engineers given this treatment are not merely willing to be accountable, they are enthusiastic about it (Allspaw). J. Paul Reed’s practical refinement, adopted by PagerDuty, is that perfect blamelessness is unattainable because the impulse to blame is deeply wired in humans; the workable goal is to be blame-aware — to notice the blame reflex when it surfaces in a review and consciously set it aside — rather than to pretend it never appears (PagerDuty).

The Postmortem Document — Structure and Contents

The culture makes honest accounts possible; the document captures them. A good postmortem is a durable, searchable artifact, not a compliance checkbox. Drawing on the Google SRE Book’s template and PagerDuty’s process, a complete postmortem contains:

  1. Metadata — title, date, authors/owner, incident commander, responders, status (draft / in-review / final), and severity. Structured fields make the corpus aggregable for trend analysis across incidents.
  2. Summary — a few sentences a non-expert can read: what broke, who was affected, for how long, and the eventual cause. Written last, read first.
  3. Impact — the quantified user-facing damage: duration, requests failed or degraded, revenue or SLA effect, number of customers affected, data lost. This is what justifies the effort and calibrates the action items.
  4. Timeline — a timestamped, factual reconstruction of the event: when it began, when it was detected, key status changes, and the decisive actions responders took (PagerDuty process). Build it forward, in the participants’ own words, to fight hindsight bias.
  5. Root cause(s) / contributing factors — the honest, systemic analysis. Note the plural: complex-system failures almost never have a single root cause but a confluence of contributing conditions (a bad default, a missing check, an unusual load, a stale runbook) that lined up. This is where the second story lives.
  6. Resolution and recovery — how service was actually restored (the mitigation) versus how the underlying condition was eliminated (the fix) — often two different things at two different times.
  7. What went well / what went poorly / where we got lucky — the reflective section. “Where we got lucky” is the sleeper: naming the near-miss (“the failover happened to be tested last week”, “the on-call happened to recognize the symptom”) surfaces latent risk that did not bite this time but will next time.
  8. Action items — concrete, prioritized follow-ups, each with an explicit owner and a due date, tracked in the normal work-tracking system (Jira, etc.) so they don’t evaporate. The discipline is to file only the items that genuinely matter (P0/P1) — a flood of low-value tickets is how action-item lists become ignored (PagerDuty process).

The language of the document matters as much as its structure. The Google SRE Book contrasts blaming phrasing — swearing to personally rewrite “the entire complicated backend system” — with the same substance reframed constructively as an action item to improve that system so the pages stop (Google SRE Book). One useful de-personalizing tactic from PagerDuty is to write the account in terms of a generic responder (“anyone on-call could have made this call”) rather than naming the individual, which keeps the analysis on the conditions (PagerDuty).

Triggers, Review, and Sharing

When to write one. The org should define postmortem triggers before incidents, so writing one is never read as an accusation. Typical triggers (Google SRE Book): user-visible downtime or degradation past a threshold, any data loss, any on-call intervention (a rollback, a traffic reroute, a manual failover), resolution time exceeding a threshold, or a monitoring miss (the incident had to be found by hand). Crucially, any stakeholder may request a postmortem even when no automatic trigger fired. Writing a postmortem is a routine, expected consequence of a significant event — not a punishment.

Review. The best-practice standard is no postmortem left unreviewed (Google SRE Book). A senior reviewer checks that the data was collected while it was fresh, that impact and timeline are complete, that the root-cause analysis is deep rather than superficial, and that the action items are appropriately prioritized. PagerDuty runs a short (15–30 minute) review meeting — for a SEV-1 within a few days of the incident — where the responders agree on the timeline, discuss detection and prevention opportunities, and confirm ownership of each action item (PagerDuty process). Collaborative drafting tools (real-time editing, inline comments, notifications) let a wide set of people contribute detail and catch gaps.

Share widely. A postmortem read only by the team that had the outage wastes most of its value. Publish to the widest audience that could learn from it (while protecting user privacy through access controls). Reinforcement rituals keep the culture alive: featuring an exemplary postmortem in a newsletter, postmortem reading groups, and — the SRE classic — the Wheel of Misfortune, a role-play drill that replays a past incident so on-call engineers build muscle memory before the real page (Google SRE Book). Google’s practice even includes surveying the team on whether the postmortem process itself is helping or has become toil, so the process improves too.

Failure Modes and Common Misunderstandings

  • “Blameless” mistaken for “no accountability.” The most common misread. Blameless redirects accountability toward fixing the future; it does not remove it (see the just-culture section above). Repeatedly ignoring known risks is accountable behavior worth addressing — but as a systemic and managerial question, not a scapegoating one.
  • Root-cause tunnel vision. Forcing every incident down to a single “root cause” (often via a mechanical Five Whys) reintroduces blame when that chain terminates at a person, and it hides the multi-factor reality of complex-system failure. Prefer contributing factors, plural.
  • Blaming language leaking back in. Even in a nominally blameless review, “why did you…” questions and counterfactuals (“if only you had…”) smuggle blame in. Being blame-aware — catching and reframing these live — is the practical defense.
  • Action items with no owner or no due date. An unowned action item is a wish. The single most common reason the same incident recurs is that last time’s fixes were filed and never done. Owners, due dates, and normal tracking are non-negotiable.
  • The postmortem as ritual theater. A postmortem that changes nothing — perfectly blameless, dutifully filed, action items quietly abandoned — is worse than none, because it consumes trust while delivering no learning. The measure of a postmortem program is whether the class of failure gets rarer, not whether the documents look tidy.
  • Overhead / postmortem fatigue. If every trivial blip demands a full postmortem, the process becomes toil and quality collapses. Reserve the full treatment for genuinely postmortem-worthy incidents; lighter events get lighter records.

The blameless postmortem is one instance of a broader family of retrospective learning practices. In safety-critical industries it echoes the NTSB’s aviation-accident investigations and hospital morbidity-and-mortality conferences — both explicitly non-punitive fact-finding. In Agile software teams the sprint retrospective serves a similar reflective role at lower stakes. Some organizations deliberately drop the word “postmortem” (with its death connotation) for learning review or incident analysis, emphasizing that the goal is understanding rather than autopsy. Chaos-engineering GameDays and the Wheel of Misfortune are the proactive mirror image: manufacture the incident on purpose, in controlled conditions, and run the same learning loop before the failure happens for real (see The Wheel of Misfortune and Incident Preparedness).

Where postmortems reach for numbers, they run into a genuine caution: the incident-metrics literature (notably the VOID’s critique of MTTR) argues that shallow duration metrics tell you little about socio-technical systems, and that rich narrative analysis — exactly what a good postmortem produces — is more valuable than reducing an incident to a single number (InfoQ / VOID). See Mean Time to Detect and Recover for that argument in full.

Uncertain

Verify: the direct quotations and specific claims attributed to John Allspaw’s 2012 Etsy essay “Blameless PostMortems and a Just Culture.” Reason: the canonical source at etsy.com/codeascraft/blameless-postmortems returned HTTP 403 during this research, so Allspaw’s exact wording was reconstructed from a text mirror (jaytaylor.com) and secondary summaries rather than the original. The substance (second story, human error as symptom, accountability redirected to learning) is corroborated across the Google SRE Book, PagerDuty, and Dekker; only the exact phrasing/attribution is second-hand. To resolve: fetch the original Etsy post from an unblocked source and confirm wording. #uncertain

See Also

Graph View