Postmortem

A postmortem is a document teams write after something goes wrong in production: a site outage, data loss, a payment system that double-charged customers, a deploy that took the API down for 40 minutes. It captures the full story so the organization can learn from it instead of repeating it.

Borrowing the word from medicine, the term means an examination after death, and in engineering it became popular through Google's Site Reliability Engineering practice and Etsy's writing on the topic. The point is not to assign blame but to understand cause. The phrase you will hear constantly is blameless postmortem, which means the document assumes everyone acted with good intentions given the information they had at the time.

A typical postmortem includes a short summary, a timeline of events with timestamps, the customer and business impact, the root cause and contributing factors, what went well and what went poorly during the response, and a list of action items with owners and due dates. That last part matters most. A postmortem with a great narrative but no tracked follow-ups is just a story.

The process starts the moment an incident is declared. During the incident, responders post timestamped updates in a chat channel or incident tool. Those messages become the raw material for the timeline, which is why teams insist on writing things down as they happen rather than reconstructing them from memory later.

After the incident is resolved, an owner is assigned, usually the incident commander or a senior engineer who was close to the response. They gather logs, metrics, deploy records, and chat transcripts, then write the timeline and dig for the actual cause. A useful technique is the Five Whys: keep asking why until you reach a systemic issue rather than stopping at the surface symptom. The disk did not fill up because someone forgot to clean it, it filled up because there was no alert at 80 percent and no automatic log rotation.

The team then holds a review meeting to discuss the draft, challenge the conclusions, and agree on action items. Each action item gets an owner and a deadline and is filed as a real ticket in the issue tracker. Many companies trigger a mandatory postmortem automatically whenever an incident crosses a severity threshold, for example any SEV1 or SEV2.

You do not write a postmortem for every blip. Most teams set a trigger: any incident above a certain severity, any event that breached an SLO, any customer-visible outage, or any near miss that could easily have been catastrophic. Near misses are worth writing up precisely because you got lucky and the next time you might not.

The main trade-off is cost. A thorough postmortem takes hours to write and a meeting to review, so writing one for trivial issues wastes time and trains people to treat the format as bureaucracy. The opposite failure is skipping them for real incidents, which means the same outage recurs in three months because nobody fixed the underlying gap.

The other risk is the blame trap. If a postmortem turns into finding who to fire, engineers stop reporting incidents honestly and start hiding mistakes, which makes systems less safe. This is why blamelessness is a hard rule and not a nicety. The hardest discipline of all is closing action items. It is easy to generate twelve follow-ups in the meeting and let them rot in the backlog, so mature teams track action item completion rate as a metric and review stale items regularly.

Suppose a checkout service goes down for 25 minutes during a Friday evening sale. Customers see errors, and roughly 4,000 orders fail. The on-call engineer pages a teammate, they trace it to a database connection pool that hit its limit of 100 connections, and they recover by restarting the service and raising the pool size.

The postmortem timeline shows the deploy that introduced a slow query at 18:02, the first alert at 18:14, the page at 18:16, and recovery at 18:39. The root cause is not the engineer who shipped the query. It is that the new query was not load tested, there was no alert on connection pool saturation, and the pool limit was set years ago and never revisited.

The action items become: add a connection pool saturation alert at 80 percent, add load testing to the deploy pipeline for queries touching the orders table, and document a runbook for pool exhaustion. Each gets an owner and a due date within two weeks. Three months later, when a similar slow query ships, the alert fires early and the runbook makes recovery a five-minute task instead of a 25-minute outage. That difference is the entire return on writing the document.