Incident Post Mortem Template & Writing Guide
When a major production system crashes, fixing the immediate bug is only the first step. The true test of a mature engineering team is how they handle the post-incident phase. Writing a blameless post-mortem report helps your team analyze what went wrong, identify system vulnerabilities, and build trust with your users.
Below is our SRE-verified writing guide and a copy-pasteable Markdown template you can use to document outages.
🛠️SRE Best Practices for Writing Post-Mortems
1. Maintain a Blameless Culture
A blameless post-mortem assumes that engineers make choices based on the information they have at the time. Focus on system design, automated checks, and alerts rather than pointing fingers at individual developers who ran a command or pushed a commit.
2. Establish a Clear Timeline
Detail exactly when the outage began, when alerts triggered, when team members acknowledged the incident, and when resolution steps were completed. This helps identify delays in incident detection and response.
3. Identify Action Items Natively
Every post-mortem must result in actionable, scheduled tickets to prevent similar failures. If a database crashed due to lack of disk space, an action item should be to configure automated disk usage threshold checks.
📄 Copy-Pasteable Markdown Template
Copy the Markdown template below and save it as incident-YYYY-MM-DD-description.md in your team repository:
# Incident Post-Mortem: [Incident Title / Outage Description] * **Date of Incident**: YYYY-MM-DD * **Author / Lead Investigator**: [Name/Role] * **Severity Level**: [P1 / P2 / P3] * **Total Downtime**: [X Minutes / Hours] --- ## 🚨 Executive Summary Provide a brief, high-level summary of the outage, the user impact (e.g., "50% of checkout requests failed"), and the resolution. --- ## 📈 Timeline * **HH:MM UTC** - Outage begins. Background task execution fails. * **HH:MM UTC** - Pingzo alerts trigger via WhatsApp / Slack notifications. * **HH:MM UTC** - On-call engineer acknowledges the alert and begins investigation. * **HH:MM UTC** - Root cause identified as a database connection leak. * **HH:MM UTC** - Server pool restarted; connections return to normal. * **HH:MM UTC** - Uptime monitors return to operational state. --- ## 🔍 Root Cause Analysis Explain the underlying issue in technical detail. Why did this happen? What failed in our safety checks or configurations? --- ## 🛠️ Resolution and Recovery Detail the steps taken to stabilize the system and return services to normal operation. Note if temporary workarounds were applied. --- ## 🛡️ Preventative Action Items Detail specific, owner-assigned tasks to ensure this failure does not happen again: * [ ] **Task 1**: Configure SQL connection pool warnings in Pingzo. (Assignee: [Name]) * [ ] **Task 2**: Add automated CPU utilization alert webhooks. (Assignee: [Name]) * [ ] **Task 3**: Document manual database failover runbook. (Assignee: [Name])