DevOps Incident Runbook: How to Investigate a Production Server Outage
When a production system drops offline, every minute of downtime directly impacts revenue and customer trust. Under the pressure of an active outage, SRE and DevOps teams need a standardized troubleshooting runbook to isolate the root cause, mitigate the failure, and restore services.
Triage follows a strict sequence: Isolate Externally -> Audit Host Health -> Review Logs -> Mitigate -> Verify -> Conduct Post-Mortem.
This runbook provides the step-by-step diagnostic commands and recovery procedures needed to resolve server outages.
1. Phase 1: External Isolation and Connectivity
Before logging into your infrastructure host, verify the scope of the outage from an external network vantage point to isolate network-level failures.
Verify DNS Resolution
Check if the domain resolves to the correct IP address:
dig api.yourdomain.com
Audit HTTP Status and TLS
Send a verbose head request to check if the server is dropping connection handshakes:
curl -Iv https://yourdomain.com
Test SSH Connectivity
If the web server is unresponsive, check if the SSH daemon is responding:
ssh -o ConnectTimeout=5 -vvv user@server-ip
If the connection times out, the server may be completely offline due to hypervisor failure or firewall restrictions. If it returns Connection refused, the server is running, but the SSH service has crashed or was disabled.
2. Phase 2: Host Resource Audit
Once you establish a terminal session on the host, audit resource allocations to locate CPU, memory, or disk constraints.
Run this diagnostic command block to retrieve system health:
# Check load average and active uptimes
uptime
# Check physical memory allocation and swap space
free -h
# Check storage utilization on mounted partitions
df -h
# Check directory inode utilization
df -i
The Inode Exhaustion Trap
Many DevOps engineers check physical disk space (df -h) but overlook inode exhaustion (df -i). An inode is a data structure on Linux filesystems that stores file metadata. Even if you have gigabytes of free disk capacity, if the inode usage reaches 100%, the filesystem cannot write new files:
[\text{Inode Utilization %} = \frac{\text{Used Inodes}}{\text{Total Inodes}} \times 100]
When this threshold is reached, databases will fail to execute writes, session files cannot be created, and Nginx reverse proxy buffers will throw errors.
3. Phase 3: Service Status and Log Audits
If system resources are stable, verify the status of the container or systemd services.
Audit Systemd Services
List all systemd services that have failed to load:
systemctl --failed
Check the status of your web server and application unit:
systemctl status nginx
systemctl status my-app-service
Triage Container Performance
If you are running Docker, check container resource utilization and status:
# List all running and stopped containers
docker ps -a
# View live container resource telemetry
docker stats --no-stream
Review System Logs
Review the system logs for kernel errors or OOM terminations:
# Inspect the last 200 system events
dmesg -T | tail -n 200
# View errors logged in systemd journal
journalctl -p err..alert --since "1 hour ago" --no-pager
4. Incident Triage Reference Matrix
Refer to the matrix below to identify common outage categories and their recovery steps:
| Outage Category | Verification Command | Mitigation Step |
|---|---|---|
| CPU Starvation | top -b -n 1 | head -n 20 | Kill offending PID (kill -9 <PID>). |
| Inode Depletion | df -i | Delete temporary files / old sessions. |
| Nginx 502/504 | tail -n 50 /var/log/nginx/error.log | Restart backend service. |
| TLS Outage | openssl s_client -connect host:443 | Force renew Let's Encrypt certificate. |
| Docker Outage | docker ps -a | Restart Docker daemon (systemctl restart docker). |
5. Mitigation and Verification
Once the cause is isolated, mitigate the issue using the fastest path to recovery. Save deep code investigations for the post-mortem.
- Rollback Deployments: If the outage started immediately following a code release, rollback to the last known-good commit.
- Service Restarts: If a service has locked up, execute a restart:
sudo systemctl restart my-app-service - Storage Clearing: If the disk is full, clear out log files or temporary sessions to free up write access:
sudo find /var/log -type f -name "*.log" -delete
Verify the recovery using external curls before closing the incident room.
6. Bypassing Host-Level Observability with Pingzo
Local server monitoring tools can fail if the server itself is locked. If your CPU spikes to 100% or your disk fills up, local agents often crash, leaving you without alerts.
Pingzo provides external observation:
- Independent Probes: It tests your server endpoints from outside your hosting network.
- Outage Logs: It logs the HTTP response codes and headers during an outage, preserving evidence even if your server logs are wiped.
- WhatsApp Escalation: Alerts are sent directly to your team on WhatsApp, ensuring rapid escalation to minimize MTTR.