Learning from Historic Infrastructure Outages: Post-Mortem Best Practices
Writing post-mortems after production incidents is essential for improving platform reliability. However, many post-mortems focus only on immediate triggers, such as developer typos or hardware failures, while ignoring the systemic contributing factors that allowed the error to propagate.
To prevent repeat incidents, SRE teams use structured timeline analysis, blast radius metrics, and engineering controls. This guide details how to calculate incident blast radius, differentiate causal factors, and construct post-mortem action items.
1. Blast Radius and System Capacity Mathematics
When compiling an incident report, calculate the exact customer impact rather than relying on average container resource charts. We define the transactional blast radius ((\text{Blast Radius})) as:
[\text{Blast Radius} = \frac{\text{Affected Customer Requests}}{\text{Total Customer Requests}} \cdot 100]
SREs use this metric to weigh the severity of outages. During high-traffic events, client retry behavior often escalates the original outage. We model this retry amplification ((R_{\text{effective}})) across (n) retries as:
[R_{\text{effective}} = R_{\text{clients}} \cdot (1 + r + r^2 + \dots + r^n)]
Where (r) represents the client retry rate. If clients retry immediately on failure without delay, (R_{\text{effective}}) will spike, saturating the origin.
To prevent resource saturation, calculate the required platform capacity ((\text{Required Capacity})) using:
[\text{Required Capacity} = \frac{\text{Peak Load} \cdot (1 + \text{Growth Margin})}{\text{Target Utilization}}]
2. Strong Engineering Controls vs. Weak Actions
A post-mortem is only as effective as the action items it produces. SRE teams replace administrative request-based tasks with technical guardrails:
| Incident Finding | Weak Action | Strong Engineering Control |
|---|---|---|
| TLS Certificate Expiry | "Be more careful during rotation windows" | Implement Let's Encrypt automated renewal with expiry monitoring |
| DNS Configuration Error | "Get changes reviewed by another teammate" | Add DNS record validation checks inside CI/CD linting pipelines |
| Database Saturation | "Monitor the CPU and connections closely" | Configure database admission control and query timeout limits |
| Client Retry Storms | "Reduce the default retry values" | Deploy client-side exponential backoff loops with random jitter |
| Unstable Deployment | "Add extra staging manual test cycles" | Implement canary testing routes with automated rollback triggers |
3. Incident Evidence Diagnostic Commands
Collect network-level evidence to reconstruct the event timeline during post-mortem audits:
# Verify DNS resolution metrics and record TTLs
dig +noall +answer pingzoapp.com
# Trace nameserver delegation chains globally
dig +trace pingzoapp.com
# Inspect TLS configuration details and certificates
openssl s_client -connect pingzoapp.com:443 -servername pingzoapp.com -alpn h2 -showcerts </dev/null
# Measure exact HTTP connect and transit timings
curl -sS -o /dev/null \
-w 'dns=%{time_namelookup} connect=%{time_connect} tls=%{time_appconnect} ttfb=%{time_starttransfer} total=%{time_total}\n' \
https://pingzoapp.com/health
[!NOTE] SRE Error Budget Tip: Use the SLA Calculator to translate historic outage downtime into error budget consumption percentages. Tracking how many minutes of your monthly budget were consumed by a single incident helps prioritize remediation tickets.
4. Reconstructing and Troubleshooting Post-Mortem Outlines
When investigating a past outage to write a post-mortem, follow this step-by-step troubleshooting checklist:
- Construct a unified timeline: Collate logs, traces, and metrics using UTC timestamps. Match client-side observations with server logs.
- Calculate the blast radius: Count the number of unique user sessions that received HTTP 5xx errors or experienced response times above your SLO.
- Trace the propagation path: Map how the initial error moved through downstream microservices, queues, and caches.
- Audit deployment events: Review recent git commits, configuration updates, and feature flag changes.
- Examine connection pool configurations: Check if network socket timeouts were configured shorter than application deadlines to prevent thread exhaustion:
{ "incident": { "id": "INC-2026-042", "service": "checkout", "primary_failure": "database_connection_exhaustion" } } - Evaluate cache eviction metrics: Confirm if expiration storms or thundering herd behavior triggered origin spikes.
- Identify the detection gap: Measure the delay between failure onset and alert generation to find observability gaps.
- Formulate causal hypotheses: Run a "Five Whys" analysis to identify missing system guardrails instead of blaming operator error.
- Schedule verification tests: Design load tests or staging chaos game days to confirm that your new engineering controls prevent the failure mode from recurring.