When production systems crash, confusion is the SRE team's worst enemy. If your developers do not know who is on call, how to communicate with customers, or where recovery scripts are stored, a minor bug can escalate into a major outage.
To prevent chaos, growing engineering teams rely on an Incident Response Playbook. This guide explains how to structure your team's playbook and provides a copy-pasteable Markdown template to get started.
🛠️ The Four Phases of DevOps Incident Triage
A robust incident response process consists of four distinct phases:
Phase 1: Detection and Alerting
The moment a system fails, the monitoring stack must dispatch high-priority notifications. Rather than relying on email alerts (which are easily missed in cluttered inboxes), teams should use native alert channels like WhatsApp, SMS, or PagerDuty to bypass silent mode configurations.
Phase 2: Triaging and Escalation
Once the on-call engineer receives the alert, they must assess the severity using a predefined matrix. If the issue is a P1 (Critical Outage), they must instantly update the status page and open an incident triage room.
Phase 3: Mitigation and Stabilization
The primary responder's goal is to stabilize the system, not necessarily write the perfect permanent code fix. If restarting a service pool or rolling back a deployment restores uptime, do that first. Permanent fixes belong in post-outage ticket backlogs.
Phase 4: Resolution and Post-Mortem
Once checks return to green status, update your users, close the incident room, and schedule a blameless post-mortem meeting within 48 hours to prevent recurrence.
📄 Incident Response Playbook Template
Copy this Markdown template and add it to your team's internal wiki or wiki repository:
# Incident Response Playbook
## 1. Incident Severity Definitions
* **P1 - Critical**: Core service down for > 50% of users. Target response: < 5 minutes.
* **P2 - Major**: Core features degraded. Target response: < 15 minutes.
* **P3 - Minor**: Non-critical bugs. Target response: Next business day.
---
## 2. On-Call Communication Protocol
* **Primary Alert Channel**: WhatsApp / Pager
* **Slack Triage Channel**: `#incident-war-room`
* **Public Status Page**: `https://status.yourdomain.com`
---
## 3. Incident Triage Steps (P1 & P2)
### Step 1: Acknowledge the Alert
Acknowledge the page in your monitoring dashboard within 5 minutes. This stops the warning from escalating to secondary responders.
### Step 2: Open the Incident Room
Create a triage channel and pin the active incident dashboard.
### Step 3: Update the Status Page
Update the status page within 10 minutes to notify stakeholders and reduce incoming support ticket volumes.
### Step 4: Run Initial Diagnostics
1. Check the active HTTP latency charts.
2. Verify database connection counts.
3. Review recent code deployment logs.
### Step 5: Stabilize and Resolve
* If recent deployment failed, run the rollback script: `./scripts/rollback.sh`
* If database connections leaked, trigger server group pool recycle.
* Once checks turn green, update the status page to **Operational** and log the post-mortem.