Incident Response Doesn't End at Recovery — A Postmortem (Recurrence Prevention) Template for Small Development Teams

“We’re seeing the same error you fixed last month, just on a different screen this time.” Anyone who maintains a system has probably been on the receiving end of that sentence, and found themselves at a loss for what to say.

Incident response itself — detect, pinpoint the cause, fix, release, apologize — is something most teams already do properly. The problem is what comes after. The moment recovery happens, everyone goes back to their regular work, and the record of the incident is left to fade away, scattered across someone’s inbox and chat logs. Six months later, a structurally identical failure recurs somewhere else, and the same investigation gets redone from scratch. Treating incident response as “fix it, apologize, done” is a way of operating that pays the same tuition over and over.

On this blog, we’ve already covered the technical side of incident investigation — how to actually get to the root cause — in “An Introduction to Collecting Windows Crash Dumps” and “Where Should catch and Logging Go in Exception Handling?”. This article covers the step that comes next: how to run the retrospective (the postmortem) that makes sure the same incident never happens again, once you already know the cause. Rather than large-scale web services, we focus narrowly on a template that a small team of two to five people maintaining business apps and Windows software can actually keep up over time.

1. The Bottom Line First

• Recovery, root-cause investigation, and recurrence prevention are separate jobs. Recovery means “getting today’s operations back”; root-cause investigation means “being able to explain why it happened”; recurrence prevention means “changing the system.” Mix them together and all three end up half-done.
• Not hunting for a culprit (being blameless) is a matter of practicality, not ethics. In a setting where individuals get blamed, information stops surfacing and you never reach the root cause. Assuming that everyone involved acted correctly given the information they had at the time, and then looking for the hole in the system — that’s the blameless postmortem principle established in SRE practice.¹
• Recurrence-prevention measures should land in a system (code, tests, monitoring, procedures), not in “being careful.” Measures that depend on human attentiveness vanish the moment the person in charge changes. Chapter 6 gives a decision table for judging the strength of a given measure.
• Split the cause into a “direct cause” and “contributing factors.” What you can actually fix is usually the contributing factors, and the trick to the Five Whys is not stopping at a person’s action but digging all the way down to the system.
• Don’t run a full postmortem on every incident. Triage by impact times likelihood of recurrence (Chapter 7), and for minor incidents just leave a one-paragraph record. Keeping the workload at a sustainable level is the single biggest condition for keeping the practice alive.
• Size the postmortem so it can be written in an hour. Chapter 4 gives a minimal template. Leaving a rough record every single time is worth more than writing zero polished documents a month.
• In contract development, keep the purpose separate from the customer-facing incident report while repurposing its content from the postmortem (Chapter 8). Not mixing the accountability document with the recurrence-prevention document protects the quality of both.

2. Why the Same Failures Keep Recurring

Incidents that recur follow an operational pattern, not a technical one.

The moment recovery happens, it counts as “done.” Incident response is an emergency interrupt, so once things are back up, everyone returns to the backlog of normal work that piled up. Nobody’s calendar has time blocked for the retrospective, and “once things calm down” never actually arrives.

The report ends with “we will be more careful going forward.” The recurrence-prevention field of a report to a customer or manager gets closed out with wording like “we will thoroughly verify” or “we will double-check.” That wording changes nothing about the system, so a few months later, once attentiveness has worn off, everyone falls into the same hole again.

Pin it on an individual and never fix the structure. Close the case with “because that person skipped the test,” and the structure that allowed the test to be skipped — no test gate in the release procedure, omissions tacitly tolerated under deadline pressure — stays exactly as it was. Next time, someone else makes the same omission.

No record survives, and the same hole gets fallen into years later. Small-team maintenance runs fine even without records, for a while, precisely because the same person keeps handling everything. But that person’s memory fades over the years, and disappears entirely when they leave or hand off the role. “I feel like I’ve seen this error before, but I can’t recall what we did about it” turns an investigation that a record would have made a 10-minute job into a full day’s work.

None of this is a matter of competence — it’s a consequence of the fact that the retrospective has never been defined as part of the job. That’s exactly why it’s worth deciding on a template (a document format and criteria for when to use it) up front.

3. What a Postmortem Is — Translating the SRE Template for Small-Scale Maintenance

A postmortem is a retrospective template for documenting an incident record — impact, root cause, the timeline of the response, and recurrence-prevention actions — and it became widely established through Google’s Site Reliability Engineering (SRE) practice. At its core is the principle of being blameless. “A blameless postmortem assumes that everyone involved acted in good faith, doing the right thing given the information they had at the time” — instead of punishing people, you fix the system that got in the way of the right action.¹

This idea isn’t unique to Google. Microsoft’s architecture guidance (the Azure Well-Architected Framework) likewise defines a postmortem as “a structured, blameless review involving all the teams concerned,” and recommends feeding the results of root cause analysis (RCA) back into the system in the form of improvements to the response process, strengthened detection (observability), and design improvements.²

It’s easy to assume “we’re not a large-scale web service, so this doesn’t apply to us,” but I think the opposite is true. The postmortem earns its keep precisely in a setup where a small team, without anyone stationed on-site at the customer, maintains a desktop application. There are three reasons for this.

1. The same person keeps handling everything, so without records, knowledge becomes completely trapped in that one head. In a large organization, someone else usually remembers; on a team of two or three, “the person who remembers” is the only database there is. The postmortem becomes that external memory.
2. Incidents are spaced far apart. Unlike a web service, a business application’s serious incidents number only a few per year. The next one tends to arrive right around when memory of the last response has already faded, which makes the value of a written record relatively higher.
3. Because you can’t go on-site, evidence and records become the lifeline. In remote maintenance, everything hinges on the ability to reconstruct after the fact “what was actually happening at that moment,” and that’s directly continuous with a postmortem’s timeline record.

As a rough guideline for what warrants a postmortem, the SRE book cites examples such as user-visible outages, data loss, or cases that required on-call intervention.¹ We revisit criteria suited to a small team in Chapter 7.

4. A Minimal Postmortem Template

The single most important condition for keeping this up is the amount of work involved. Here’s a Markdown template designed with the ceiling of being able to write the first draft in an hour. I’d recommend keeping it as a dated file under something like docs/postmortem/ in the repository, version-controlled in the same place as the code.

# Postmortem: Order List Shows Duplicate Rows Across a Date Boundary (2026-07-15)

- Status: Complete / Actions in progress / Filed only
- Author: Komura
- Severity: Medium (business continued, but required manual reconciliation)

## Summary (3 lines or fewer)
Opening the order list while the monthly closing job was running showed
the previous day's slips duplicated. Display-only issue; the data in
the database was correct.

## Impact (who / what / how much)
- Affected: 3 sales-office staff
- What happened: Duplicate rows on the list screen. 1 case where a
  shipment was almost sent out twice by mistake
- Duration: roughly 09:10-11:40 on 7/15 (about 2.5 hours)

## Timeline
- 09:10 User calls in: "the same slip shows up as two rows" (detected)
- 09:30 Remote screen share; confirm the reproduction conditions
- 10:15 Interim fix: ask users not to open the list while closing is running
- 11:40 Fixed version deployed and recovery confirmed

## Direct cause
The list query was doing a UNION ALL between the main table and the
rows the closing job had copied into a temporary table (with no
locking).

## Contributing factors
- Concurrent execution of the closing job and the screen query was
  not in any test scenario
- The existence of the temporary table was never written into the
  design document, so it wasn't considered when the screen was
  modified
- There was no mechanism to detect the duplicate display; discovery
  depended entirely on the user noticing

## What went well
- The user had noted the time in their own operation log, which made
  pinning down the repro conditions fast
- The deployment pipeline was automated, so the fix could go out the
  same day

## Recurrence-prevention actions (owner and deadline)
- [ ] Add a duplicate-detection assertion to the list query (Komura, 7/22)
- [ ] Add a test for concurrent closing-job/query execution (Komura, 7/29)
- [ ] Retire the temporary-table approach and consider snapshot isolation (Komura, policy decision by end of August)

A few notes on how to write this well.

• Keep the “Summary” to three lines or fewer. The person who goes looking for this document later is your future self. Being able to understand it in three lines when you find it in a search matters more than an elegant structure.
• The timeline should contain only facts, each with a timestamp. Keep interpretation (“it should have been…”, “we should have…”) separated out into the cause sections. Mix interpretation into the timeline and you lose the ability to reconstruct what actually happened when you read it later.
• Always write down “what went well.” This keeps the retrospective from turning into a session of self-flagellation, and gives you an entry point for promoting things that happened to work out (a log that just happened to be there, say) into an actual mechanism.
• Every action must have an owner and a deadline. An action with neither never gets done. A postmortem’s real effectiveness is determined by whether it gets reviewed and whether its actions get tracked.¹

5. The Practice of Root Cause Analysis — Separating Direct Causes From Contributing Factors

There’s a reason the template splits the cause field into two.

The direct cause is the technical event that directly triggered the incident — “an exception from a missing NULL check,” “a UNION ALL with no locking.” This is what the fix patch actually corrects.

Contributing factors are the conditions that allowed the direct cause to slip in, go unnoticed, or spread damage further — “there was no test for that case,” “it wasn’t written in the design document,” “detection depended on the user noticing.” This is the main battleground for recurrence-prevention measures, and there are usually several of them.

The reason for separating them is simple: fix only the direct cause, and if the contributing factors remain, a different direct cause will come in through the same route. The “same error on a different screen” from the opening of this article is exactly that.

5.1 Don’t Let the Five Whys Stop at “a Person’s Action”

The Five Whys is an effective tool for digging into a cause, but dig in the wrong direction and it turns into a tool for hunting down a culprit. The typical failure is stopping at “why? — because the person in charge forgot to check.” Don’t stop there; dig one level further.

• Why was it possible to forget the check? → Because the check existed in neither a procedure document nor a checklist, and depended purely on memory
• Why did it depend on memory? → Because the release procedure was never documented, and was assembled on the spot every single time

When a person’s action shows up as the answer, that’s not the end point — it’s the entry to the next question, which asks about the system. Once you accept the premise that people will always make mistakes, the question worth digging into shifts from “why did they make the mistake” to “why did the mistake reach production unchanged.”

5.2 Without Evidence, Analysis Never Gets Started

The quality of a root-cause analysis is capped by the quality of the evidence left behind at the time the incident occurred. A postmortem that can’t reconstruct the timeline turns into a piece of speculative fiction. For maintaining a Windows app, here are the minimum three things you need in place.

• Crash dumps. Set up WER LocalDumps and a dump gets left behind whenever the app terminates abnormally, without asking the user to do anything. Setup is covered in “An Introduction to Collecting Windows Crash Dumps”, and how to read one in “Reading Crash Dumps with WinDbg + SOS”.
• Application logs. Timestamps, a correlatable ID, and synchronous writes for critical events. See “Designing Windows Apps to Leave Logs and Dumps When They Crash” for how to make sure logging survives a crash.
• The event log. Even if your own logging breaks, the OS’s standard event log still keeps a record of the app crash (Application Error). How to make use of it is covered in “An Introduction to Windows Event Log and ETW”.

If you sit down to write the postmortem and can’t fill in the timeline, that inability is itself a contributing factor — “the detection and logging mechanisms are insufficient” — and a candidate for a recurrence-prevention action.

6. The Quality of Recurrence-Prevention Measures — A Strength Decision Table

Once you’ve written out recurrence-prevention actions, judge their strength. The axis for judging is how much the measure depends on human attentiveness.

The realistic rule here is not “ban weak measures” but “don’t let a weak measure be the final word.” Raising awareness has value as an interim response you can put in place the same day, but what belongs in the permanent-fix field is at least Medium, and Strong where possible.

When only weak measures come to mind, re-examine them with these questions.

• “If a brand-new hire were put in the exact same situation, would this incident still not happen?” If the answer is no, it isn’t a system yet.
• “Can this mistake be caught by the compiler, a test, or CI?” For example, if the issue was “we were swallowing an exception on shutdown,” the strong measure isn’t raising awareness — it’s implementing, as the app’s shared exception handler, a policy along the lines laid out in “A Decision Table for Whether to Exit or Continue After an Unexpected Exception.”
• “Can we make the mistake impossible to make? Can we make it possible to notice quickly?” If prevention is expensive, leaning on detection instead (monitoring, alerts, a reconciliation batch job) is also a perfectly legitimate strong measure. Feeding the lessons of an incident back into stronger detection and improved design is the same structure Microsoft’s guidance recommends.²

Because strong measures take real effort, the practical approach is to put both “a medium measure to do this week” and “a strong measure to do next month” on the action list, and manage both by deadline.

7. Which Incidents Get One — Triaging Implementation

Mandate a full postmortem for every single incident, and within three months nobody will be writing them anymore. Decide the level of effort by impact times likelihood of recurrence.

There are three keys to running this.

• A recurrence of the same type of incident gets bumped up one rank, regardless of impact. The very fact that it recurred is proof that the previous measure never became a real system.
• Even for something minor, always leave at least a record. One paragraph is fine. With just four points — date, symptom, cause, response — your future self gets rescued by a search years later. Once enough small-incident records accumulate, structural imbalances also become visible, like “this one screen alone accounts for most of the incidents.”
• When in doubt, err on the side of writing something, but scale down the amount. If you’re spending time debating full versus not, you’ll finish faster just starting the light version.

8. Handling This in Contract Development — The Relationship With Customer-Facing Incident Reports

In contract or maintenance work, customers ask for an “incident report” after an incident. Sorting out the relationship between the postmortem and that report ahead of time saves you from doing the work twice.

About 80% of the content can be repurposed directly. The summary, impact, timeline, direct cause, and recurrence-prevention measures are exactly the components of a customer-facing report. Write the internal postmortem first, then edit it down for the customer, and you never end up writing prose purely for the sake of the report.

That said, keep in mind that these are two documents with different purposes, and keep them separate.

The biggest reason to keep them separate is that mixing the liability/compensation discussion with the recurrence-prevention discussion distorts both. In a document tangled up with a question of liability, everyone involved inevitably writes defensively. Contributing factors vanish from a defensive document, and recurrence prevention degrades into “we will be careful.” Conversely, hand a candidly written internal postmortem straight to a customer, and passages stripped of context can take on a life of their own. The safe approach is to keep “the internal document, written frankly” and “the external document, communicated accurately” separate, and make it a one-way street: build the latter from the former.

One more thing: once a planned recurrence-prevention measure is written into a customer-facing report, it becomes a promise to that customer. Manage its deadline the same way you manage the postmortem’s action items, and report back once it’s done. That one round trip is what turns an incident into an opportunity to actually build trust.

9. Summary

• Incident response doesn’t end at recovery. Treat recovery, root-cause investigation, and recurrence prevention as separate jobs, and build the retrospective (the postmortem) into your regular work.
• Blamelessness is the postmortem’s guiding principle. Blame people, and information stops surfacing, and you never reach the root cause. Don’t stop the analysis at a person’s action — dig all the way down to the hole in the system (the contributing factors).¹
• A minimal template you can write in an hour — summary, impact, timeline, direct cause, contributing factors, what went well, and actions with an owner and a deadline — is enough. As a prerequisite for filling in the timeline, set up evidence preservation ahead of time: crash dumps, logs, and the event log.
• Raise recurrence-prevention measures from “being careful” (weak) to a procedure document (medium), and where possible up to mechanically preventing it with tests, assertions, monitoring, or a design change (strong). Feeding lessons back into detection and design is a structure common to both SRE and Microsoft’s guidance.¹²
• Don’t run the full process on every incident. Triage by impact times likelihood of recurrence, and even for minor ones always leave at least a one-paragraph record.
• In contract development, write the internal postmortem first, and edit the customer-facing incident report from it. Separating the liability/compensation document from the recurrence-prevention document protects the quality of both.

Related Articles

• An Introduction to Collecting Windows Crash Dumps - WER/ProcDump/WinDbg
• Reading Crash Dumps with WinDbg + SOS — A Practical Guide to Analysis After Collection
• Designing Windows Apps to Leave Logs and Dumps When They Crash
• An Introduction to Windows Event Log and ETW — Putting Your Business App’s Logs on the OS’s Standard Mechanisms
• Where Should catch and Logging Go in Exception Handling?
• A Decision Table for Whether to Exit or Continue After an Unexpected Exception

Related Consulting Areas

Komura Software LLC handles everything from investigating and diagnosing hard-to-reproduce bugs, to building mechanisms for preserving evidence (logging and dump collection), to organizing a maintenance and operations setup that includes recurrence prevention. We also welcome inquiries at the stage of “the same incident keeps recurring” or “the recurrence-prevention measures in our incident reports have become a formality.”

• Bug Investigation & Root Cause Analysis
• Maintenance & Modernization of Existing Windows Software
• Technical Consulting & Design Review
• Contact

References

Related Articles

Recent articles sharing the same tags. Deepen your understanding with closely related topics.

Related Topics

These topic pages place the article in a broader service and decision context.

Where This Topic Connects

This article connects naturally to the following service pages.

Frequently Asked Questions

Common questions about the topic of this article.

What is a postmortem? How is it different from an incident report?

A postmortem is a structured retrospective document and activity carried out after an incident is recovered from — a technique that became established in the field of SRE (Site Reliability Engineering). Working from the premise of being blameless (not blaming individuals), it records the timeline, the direct cause, the contributing factors, and the recurrence-prevention actions. Where a customer-facing incident report is a document that fulfills accountability — 'what happened and how we responded' — a postmortem is an internal document for deciding 'how to change the system so this never happens again.' Since most of the content overlaps, though, it's efficient to write the postmortem first and then edit a customer-facing report out of it.

My recurrence-prevention measures keep coming out as nothing but 'we will be more careful going forward.' What should I do?

'Be careful' and 'raise awareness' depend on human memory and good intentions, so their effect is guaranteed to fade as time passes and people change roles. When you're working out a measure, ask yourself again: 'if a brand-new hire were put in the exact same situation, would this incident still not happen?' If the answer is no, it isn't a system yet. A strong measure is one that gets mechanically detected or prevented without anyone having to be careful — a test, an assertion, a constraint built into the type system or design, a monitoring alert. If you can't put a strong measure in place right away, use a checklist or a written procedure as an interim step, and record the permanent fix as an action with a deadline.

We're a small contract-development shop with no room to write a postmortem for every single incident.

You don't need a full postmortem for every incident, and if you tried, the practice itself wouldn't survive. Triage which level of effort to apply by impact and likelihood of recurrence: run the full process for incidents involving a business stoppage or data corruption, or when the same type of incident recurs, and for minor ones just leave a one-paragraph entry in the incident log. What matters is that you always leave at least a record, even for something minor — whether you can search up a past record when the same phenomenon happens again years later makes a huge difference to how long the investigation takes.

Isn't 'not hunting for a culprit' (being blameless) just a way of leaving accountability vague?

No. Blamelessness is a principle for shifting the focus away from 'who made the mistake' and onto 'why the system allowed that person to make it.' If an operator made a mistake with an operation, there's a contributing factor in the UI or procedure that made the mistake possible. In an organization that punishes individuals, information gets hidden the next time something goes wrong, and you never reach the root cause. Accountability toward the customer — what happened, and how it will be compensated for — should be handled as a separate document and a separate process, and that doesn't conflict with a blameless retrospective. Keeping the accountability document and the recurrence-prevention document separate is the practical key.

Author Profile

Profile page for the article author.

Go Komura

Representative of KomuraSoft LLC

Focused on Windows software development, technical consulting, and investigations into failures that are difficult to reproduce.