Website Outage Detection: How SREs Confirm Real Outages vs. Local Blips

Website Outage Detection: How SREs Confirm Real Outages vs. Local Blips

When an engineer receives a message stating "the site is down", the operational risk is premature escalation. Waking up an entire secondary on-call rotation for a localized ISP routing anomaly, a laptop-specific DNS resolver cache issue, or a corporate VPN certificate inspection error wastes critical engineering hours and induces alert fatigue. Conversely, dismissing real customer-facing incidents as transient blips burns through Service Level Objective (SLO) error budgets.

Site Reliability Engineers (SREs) establish deterministic diagnostic protocols to rapidly distinguish localized client-side glitches from systemic production outages. This guide breaks down protocol-level verification across DNS, TCP, TLS, and HTTP layers, decision matrices, diagnostic shell scripts, and structured incident runbooks.

1. What Defines a True Production Outage?

A service failure exists across a spectrum of reachability, availability, and semantic correctness:

┌─────────────────────────────────────────────────────────────┐
│ 1. Local / Device Failure (Local DNS, VPN, Browser Cache)   │
├─────────────────────────────────────────────────────────────┤
│ 2. Edge / Transit Failure (ISP Peering, CDN Node, Anycast)  │
├─────────────────────────────────────────────────────────────┤
│ 3. Protocol Handshake Failure (DNSSEC, TCP Drops, TLS Cert) │
├─────────────────────────────────────────────────────────────┤
│ 4. HTTP / Gateway Failure (502 Bad Gateway, 504 Gateway TO) │
├─────────────────────────────────────────────────────────────┤
│ 5. Semantic / Application Failure (HTTP 200 with Error JSON)│
└─────────────────────────────────────────────────────────────┘

An SRE defines a confirmed production incident only when independent external probes reach consensus across multiple autonomous network domains (ASNs) and geographical regions.

2. The First 60 Seconds: Protocol-Level Isolation Stack

To prevent misdiagnosis, isolate the exact failure layer from the wire upwards:

Client Request
  │
  ├─► 1. DNS Resolution (UDP/TCP Port 53) ──► Error: NXDOMAIN / SERVFAIL / Timeout
  │
  ├─► 2. TCP SYN Handshake (Port 443)    ──► Error: Connection Refused (RST) / Timeout
  │
  ├─► 3. TLS 1.3 Key Exchange            ──► Error: Expired Cert / SNI Mismatch / Cipher Alert
  │
  ├─► 4. HTTP Request / Headers          ──► Error: 500 / 502 / 503 / 504 Status
  │
  └─► 5. DOM & Application Semantics     ──► Error: Uncaught JS Exception / Empty Body

3. SRE Decision Matrix: Real Outage vs. Local Blip

Evaluate anomalous telemetry against this multi-dimensional correlation matrix:

4. Layer-by-Layer CLI Diagnostic Playbook

Execute these copy-pasteable commands in sequence to isolate the exact failure layer:

Layer 1: Validate DNS Across Independent Public Resolvers

# Query system resolver, Cloudflare (1.1.1.1), and Google (8.8.8.8)
dig example.com A +short
dig @1.1.1.1 example.com A +stats
dig @8.8.8.8 example.com A +stats

# Trace full DNS authoritative hierarchy
dig +trace example.com

# Verify DNSSEC signature chains
dig +dnssec example.com

Layer 2: Test TCP Connectivity Directly by IP (Bypassing DNS)

# Verify raw TCP socket reachability on port 443
nc -zv -w 5 203.0.113.10 443

# Force curl to target specific IP while preserving SNI and HTTP Host header
curl -v --resolve example.com:443:203.0.113.10 https://example.com/

Layer 3: Inspect TLS Certificate & Handshake Negotiation

# Check TLS handshake, protocol version, cipher suite, and certificate dates
openssl s_client -connect example.com:443 -servername example.com -showcerts </dev/null 2>/dev/null | openssl x509 -noout -subject -issuer -dates

Layer 4: Measure HTTP Timings & Status Codes

# Deconstruct high-resolution timing metrics
curl -sS -o /dev/null \
  -w 'DNS: %{time_namelookup}s\nTCP: %{time_connect}s\nTLS: %{time_appconnect}s\nTTFB: %{time_starttransfer}s\nTotal: %{time_total}s\nHTTP Code: %{http_code}\nRemote IP: %{remote_ip}\n' \
  https://example.com/health

Test your live endpoint reachability, DNS records, and SSL health instantly using our free tools: Website Uptime Checker, DNS Lookup Tool, and SSL Inspector.

5. Multi-Vantage Independence Matrix

When a monitoring probe detects an anomaly, construct an Independence Matrix across distinct geographic networks before triggering paging alerts:

Vantage Point       DNS Status    TCP Connect   TLS Handshake   HTTP Status
───────────────────────────────────────────────────────────────────────────
1. US East (AWS)        ✓             ✓              ✓            200 OK
2. EU West (GCP)        ✓             ✓              ✓            200 OK
3. AP South (Mumbai)    ✗ (Timeout)   ✗              ✗            No Reply
4. Local Workstation    ✓             ✓              ✓            200 OK

• Interpretation: The failure is confined to a regional ISP or transit route into ap-south-1. This is a Regional Degradation, not a total service outage.

6. Mathematical Outage Probability & Quorum Verification

SRE alerting engines employ quorum-based verification to calculate true outage probability:

[P(\text{Outage} \mid k \text{ probe failures}) = 1 - \prod_{i=1}^{k} P(\text{False Positive on Probe } i)]

If a single probe has an independent false positive rate of (p = 0.02) ((2%)), requiring (k = 3) independent vantage point failures yields:

[P(\text{Outage}) = 1 - (0.02)^3 = 1 - 0.000008 = 99.9992% \text{ Confidence}]

Connecting Outage Detection to Service Error Budgets

Calculate permissible downtime before declaring an SLA breach:

[E = 1 - S]

For a (99.95%) availability objective, total permissible downtime across a 30-day billing cycle is (21.9\text{ minutes}).

Model your downtime budgets and calculate outage financial exposure. Use our SLA Calculator to evaluate allowable downtime minutes and translate outage costs using the Downtime Calculator.

7. Production Shell Script: Rapid Outage Verifier

Deploy this lightweight diagnostic script across your operations fleet to standardize incident verification:

#!/usr/bin/env bash
# outage-verifier.sh - Rapid multi-protocol outage confirmation
set -euo pipefail

TARGET_HOST="${1:-example.com}"
TARGET_URL="https://${TARGET_HOST}/"

echo "=================================================="
echo " Starting Diagnostic for: ${TARGET_HOST}"
echo " Timestamp: $(date -u +"%Y-%m-%dT%H:%M:%SZ")"
echo "=================================================="

# 1. DNS Check
echo -n "[1/4] Testing DNS Resolution... "
RESOLVED_IP=$(dig +short A "${TARGET_HOST}" | head -n 1)
if [[ -z "${RESOLVED_IP}" ]]; then
  echo "FAILED (No A record found)"
else
  echo "OK (Resolved: ${RESOLVED_IP})"
fi

# 2. TLS Certificate Expiry Check
echo -n "[2/4] Testing TLS Handshake & Certificate... "
CERT_EXPIRY=$(openssl s_client -connect "${TARGET_HOST}:443" -servername "${TARGET_HOST}" </dev/null 2>/dev/null | openssl x509 -noout -enddate 2>/dev/null | cut -d= -f2 || echo "")
if [[ -z "${CERT_EXPIRY}" ]]; then
  echo "FAILED (TLS Handshake Error)"
else
  echo "OK (Expires: ${CERT_EXPIRY})"
fi

# 3. HTTP Protocol Timing
echo "[3/4] Testing HTTP Response over IPv4..."
curl -4 -sS -o /dev/null \
  -w "  --> HTTP Status: %{http_code} | Total Time: %{time_total}s | Remote IP: %{remote_ip}\n" \
  "${TARGET_URL}"

# 4. HTTP Protocol Timing over IPv6 (if available)
echo "[4/4] Testing HTTP Response over IPv6..."
curl -6 -sS -o /dev/null \
  -w "  --> IPv6 Status: %{http_code} | Total Time: %{time_total}s\n" \
  "${TARGET_URL}" 2>/dev/null || echo "  --> IPv6 Not Configured / Unreachable"

echo "=================================================="
echo " Diagnostic Completed."

8. SRE 12-Step Production Outage Runbook

When an outage report arrives, execute this structured operational procedure:

1. Verify whether the alert reflects multi-region probe consensus or an isolated single-device report.
2. Execute dig across multiple external resolvers (1.1.1.1, 8.8.8.8) to confirm authoritative DNS health.
3. Compare IPv4 and IPv6 endpoint reachability to rule out dual-stack routing black holes.
4. Test direct TCP socket connectivity on port 443 using nc or curl --resolve.
5. Inspect SSL/TLS certificate validity dates and SNI negotiation with OpenSSL.
6. Evaluate HTTP response status codes (500, 502, 503, 504) and response headers (cf-ray, x-amz-cf-id).
7. Review edge CDN and Cloud WAF block rates to confirm valid customer traffic is not being filtered.
8. Correlate observed failures with recent production deployments, feature flag toggles, or infrastructure changes.
9. Declare an active incident if independent verification confirms global or severe regional user impact.
10. Mitigate immediately via CDN traffic failover, rollback, or secondary region DNS shifting.
11. Verify customer traffic recovery across all geographical regions using synthetic monitors.
12. Preserve captured diagnostic telemetry and conduct a blameless post-mortem review.

9. 15-Point SRE Outage Verification Checklist

Maintain this operational checklist in your incident response repository:

• DNS tested against at least 3 distinct public recursive resolvers.
• IPv4 and IPv6 connectivity verified independently.
• Direct TCP connection tested via --resolve to bypass local DNS caches.
• TLS certificate expiration and intermediate certificate chain validated.
• High-resolution HTTP timing breakdown (DNS, TCP, TLS, TTFB) captured.
• HTTP status codes and CDN edge response headers recorded.
• Multi-region synthetic probe consensus verified before paging on-call.
• Internal APM and load balancer error metrics cross-referenced.
• ISP / transit peering issues ruled out via MTR path tracing.
• Status page updated to communicate incident progress to end users.
• Error budgets calculated using the SLA Calculator.
• Automated multi-region synthetic uptime monitoring configured in Pingzo to guarantee zero false positives.