How do you run an incident postmortem?

Agree which incidents require one, assign an owner, collect evidence and draft a timeline before meeting, then document impact in user terms, contributing causes, what helped and hindered response, and a small number of specific actions with named owners and dates tracked in the normal backlog.

What should an incident postmortem include?

A plain summary, user-facing impact, an evidence-based timeline including detection and mitigation, contributing causes rather than a single culprit, what went well, what made response harder, owned and dated actions, and a consistent cause code for later pattern analysis. Keep it readable by people outside the team.

What does blameless postmortem mean?

Investigating why actions were reasonable given the information and tooling available at the time, instead of attributing failure to individuals. It is not consequence-free or vague: it focuses on conditions and system design, and the account is explicitly not used in performance management.

Why do postmortem action items never get done?

Usually because there are too many, they are sized as single lines when they are really projects, they live in a separate tracker that nobody prioritises, or nobody reviews them again. Fewer actions, in the normal backlog, with a standing review, is what makes them happen.

How to Run an Incident Postmortem

Run a postmortem by agreeing which incidents require one, writing an evidence-based timeline, stating user impact precisely, identifying contributing causes rather than a single culprit, and producing specific actions with named owners and dates. Keep it blameless, publish it where others can read it, and review outstanding actions until they are done.

Before the meeting

Agree the trigger. Decide as a team which incidents require a postmortem: severity above a threshold, any customer-visible outage over a duration, any data loss, any security incident, or any near miss the responders found alarming. A written rule prevents the question becoming political.

Assign an owner. One person responsible for producing the document, usually the incident commander or a responder, not necessarily the most senior person present.

Collect evidence first. Alerts and their timestamps, deployment and configuration change records, logs and traces, dashboards as they appeared at the time, the incident chat, and the ticket history. Screenshot dashboards, since retention may be shorter than you expect.

Draft the timeline before the meeting. Meetings spent reconstructing the sequence run out of time before reaching causes. Circulate a draft and use the meeting to correct and interpret it.

Invite the right people. Responders, service owners, anyone who made a relevant change, and someone from support if customers were affected. Keep it small enough for discussion.

Timing

Hold the postmortem soon enough that details are fresh, usually within a few days, but not while responders are still exhausted or the incident is ongoing. If mitigation is temporary, note the permanent fix as an action rather than delaying the review until it lands.

What the document contains

Summary. A short paragraph anyone in the company can understand: what broke, who was affected, for how long, and what has changed since.

Impact. In user terms: how many customers, which functionality, what they experienced, and any financial, contractual, or regulatory consequences. Vague impact statements make prioritising follow-up work impossible.

Timeline. Times, events, and observations, including detection, escalation, mitigation, and resolution. Mark the gaps explicitly, particularly between failure and detection.

Contributing causes. The trigger, the conditions that allowed it, and the systemic reasons those conditions existed. Several causes are normal.

What went well. Genuinely useful, both for morale and because working controls should be protected in future changes.

What made it harder. Missing runbooks, unclear ownership, noisy alerts, absent access. These are usually the most actionable findings.

Actions. Specific, owned, dated, and tracked in the same system as other engineering work. Distinguish prevention, detection, and mitigation improvements; a fix that shortens the next outage counts.

Cause codes. A consistent classification on the incident record so patterns across many incidents are visible later.

Keeping it blameless

Blameless does not mean consequence-free or vague. It means investigating why an action was reasonable to the person taking it given the information and tooling they had. Describe roles rather than names in the narrative, focus on conditions rather than choices, and make it explicit that the account is not used in performance management.

Making the actions happen

Limit the number. Thirty actions from one incident means none get done. Pick the few that most reduce recurrence or detection time, and record the rest as observations.

Size them like work. An action that is really a project needs to be scoped as one, with a plan, rather than sitting on a list as a single line for a year.

Put them in the normal backlog. Separate incident action trackers become graveyards. Actions that compete with other work get prioritised or explicitly dropped, which is honest either way.

Review them regularly. A standing review of open incident actions, ideally at the start of the next postmortem meeting. Ageing actions are data: they usually mean the action was too large, unowned, or not actually important.

Publish the document. Internally readable postmortems spread knowledge to teams who have not hit the problem yet, and make it harder for the same class of failure to surprise three services in a row.

Aggregate periodically. Every quarter, read the cause codes together. Individual incidents feel unique; in aggregate they usually cluster around a few systemic issues such as configuration changes, dependency failures, or capacity limits.

Consider external write-ups. For customer-affecting outages, a public summary with impact, cause, and changes made rebuilds trust faster than silence, provided it is accurate and avoids blaming named third parties without evidence.

Frequently asked questions

How do you run an incident postmortem?: Agree which incidents require one, assign an owner, collect evidence and draft a timeline before meeting, then document impact in user terms, contributing causes, what helped and hindered response, and a small number of specific actions with named owners and dates tracked in the normal backlog.
What should an incident postmortem include?: A plain summary, user-facing impact, an evidence-based timeline including detection and mitigation, contributing causes rather than a single culprit, what went well, what made response harder, owned and dated actions, and a consistent cause code for later pattern analysis. Keep it readable by people outside the team.
What does blameless postmortem mean?: Investigating why actions were reasonable given the information and tooling available at the time, instead of attributing failure to individuals. It is not consequence-free or vague: it focuses on conditions and system design, and the account is explicitly not used in performance management.
Why do postmortem action items never get done?: Usually because there are too many, they are sized as single lines when they are really projects, they live in a separate tracker that nobody prioritises, or nobody reviews them again. Fewer actions, in the normal backlog, with a standing review, is what makes them happen.