Root Cause Analysis with DNS and Traceroute: Isolating Anycast BGP and Nameserver Failures

Root Cause Analysis with DNS and Traceroute: Isolating Anycast BGP and Nameserver Failures

When external synthetic probes report a sudden global drop in availability while internal server health checks show 100% CPU and memory headroom, the failure almost invariably resides in the network control plane: DNS delegation errors, Anycast BGP routing black-holes, or nameserver zone synchronization divergence.

An Anycast IP address does not represent a single physical server; it represents an identical IP prefix announced over BGP across dozens of geographically distributed Points of Presence (PoPs). A localized fiber cut, BGP route flap, or DNSSEC expiration can make a domain completely unreachable for 30% of global users while remaining perfectly functional for the rest.

Site Reliability Engineers must isolate root causes by correlating recursive resolver observations, authoritative nameserver health, and transport-level traceroute paths.

Client Application
   │
   ├── 1. Query Local Recursive Resolver (UDP/TCP 53)
   │      │
   │      ├── [If Resolver Cache Hit] ──► Return Cached A/AAAA Record
   │      └── [If Cache Miss] ───────► Recursive Resolution (Root ──► TLD ──► Authoritative)
   │
   ▼
BGP Anycast Routing Layer (Autonomous System Path)
   │
   ├── [BGP Route Optimal] ──► Nearest Healthy Anycast PoP (London / Tokyo / New York)
   └── [BGP Route Flap / Leak] ──► Dropped Packets / Black-Hole Routing Loop
   │
   ▼
Authoritative Nameserver Daemon (BIND / PowerDNS / NSD / Knot)
   │
   ├── Check Zone Serial Consistency (SOA Serial Match Across All NS)
   ├── Validate DNSSEC Keys (DNSKEY / DS / RRSIG Signatures)
   └── Return Authoritative A/AAAA Resource Records

1. The DNS and Anycast Failure Domains

During an incident, determine which architectural layer failed before making changes:

• Resolver Layer: Recursive resolvers (e.g., Google 8.8.8.8, Cloudflare 1.1.1.1, ISP resolvers) failing to resolve upstream due to rate-limiting or stale caching.
• Delegation Layer: Parent TLD nameservers (e.g., .com root servers) returning stale or missing NS and DS records.
• BGP Anycast Control Plane: BGP peers withdrawing routes or advertising sub-optimal AS_PATH prefixes, sending traffic to dead PoPs.
• Authoritative DNS Layer: Individual nameservers returning SERVFAIL due to expired DNSSEC signatures or out-of-sync zone serials (SOA).
• Transport Layer: Intermediate firewalls blocking TCP/53 fallback or MTU limits causing packet fragmentation drops under EDNS(0).

Testing your live domain records, nameservers, and TTL propagation? Run an external query across global resolvers with our interactive DNS Lookup Tool.

2. Isolating Authoritative Nameservers with dig

Never rely solely on your local workstation resolver. Always query the authoritative nameservers directly to eliminate recursive caching bias.

2.1 Full Delegation Trace

Trace the complete hierarchical resolution chain from root hints down to your authoritative nameservers:

dig +trace +nodnssec example.com

2.2 Auditing All Authoritative Nameservers for Zone Consistency

Run this loop to verify that every authoritative nameserver serves the exact same SOA serial and A records:

for ns in $(dig +short NS example.com); do
  echo "=================================================="
  echo " Querying Authoritative NS: $ns"
  echo "=================================================="
  dig @"$ns" example.com A +noall +answer +authority +stats
  dig @"$ns" example.com SOA +noall +answer
done

• SOA Serial Mismatch: If ns1.example.com returns serial 2026090801 but ns2.example.com returns 2026090701, secondary nameservers are failing AXFR zone transfers, serving stale or missing records to a percentage of visitors.
• SERVFAIL on Specific NS: Indicates DNSSEC validation failure (expired RRSIG or mismatched DS record at the registrar).

3. Dissecting Anycast BGP Routing Paths with Traceroute

Traceroute identifies where network transit breaks down between the client and the Anycast edge.

IPv4 Traceroute Mechanics:
Packet 1 (TTL=1) ──► Router 1 (Decrements TTL to 0) ──► Returns ICMP Type 11 (Time Exceeded)
Packet 2 (TTL=2) ──► Router 2 (Decrements TTL to 0) ──► Returns ICMP Type 11 (Time Exceeded)
Packet N (TTL=N) ──► Anycast Edge Server ───────────► Returns ICMP Echo Reply / TCP SYN-ACK

3.1 UDP vs. ICMP vs. TCP Traceroute

Standard UDP traceroute is frequently rate-limited or dropped by enterprise firewalls. Always test across all three transport modes:

# 1. Standard ICMP Traceroute
traceroute -I example.com

# 2. TCP Traceroute to Port 53 (Tests DNS TCP listener path)
traceroute -T -p 53 ns1.example.com

# 3. TCP Traceroute to Port 443 (Tests Web HTTPS path)
traceroute -T -p 443 example.com

3.2 Interpreting Asterisks (* * *)

Three consecutive asterisks (* * *) at a specific hop do not inherently mean packet loss:

• If subsequent hops respond normally with low latency, intermediate router interfaces are simply configured to ignore ICMP generation to protect control-plane CPUs.
• If asterisks continue indefinitely until timeout (30 hops max), the packet encountered an Anycast black-hole, an ACL block, or a broken BGP routing loop.

4. SRE Diagnostic Matrix: Correlating DNS and Traceroute

5. Automated Production DNS & Anycast RCA Script

Deploy this diagnostic script during incidents to capture a comprehensive RCA snapshot across public resolvers, authoritative nameservers, and network transport hops:

#!/usr/bin/env bash
# Production SRE DNS & Anycast Root Cause Analysis Probe
set -euo pipefail

TARGET_DOMAIN="${1:-example.com}"
echo "======================================================================"
echo " DNS & ANYCAST RCA DIAGNOSTIC: $TARGET_DOMAIN"
echo " Timestamp: $(date -u '+%Y-%m-%d %H:%M:%SZ')"
echo "======================================================================"

echo -e "\n[1/5] DELEGATION & NAMESERVERS (Parent TLD View)"
dig "$TARGET_DOMAIN" NS +noall +answer

echo -e "\n[2/5] AUTHORITATIVE NAMESERVER COMPARISON"
for ns in $(dig +short NS "$TARGET_DOMAIN"); do
  NS_IP=$(dig +short "$ns" A | head -n1)
  echo "--- NS: $ns ($NS_IP) ---"
  dig @"$ns" "$TARGET_DOMAIN" A +noall +answer +stats | grep -E 'ANSWER:|Query time:|SERVER:' || true
  dig @"$ns" "$TARGET_DOMAIN" SOA +noall +answer || true
done

echo -e "\n[3/5] PUBLIC RECURSIVE RESOLVER COMPARISON"
for resolver in "1.1.1.1" "8.8.8.8" "9.9.9.9" "208.67.222.222"; do
  echo "--- Resolver: $resolver ---"
  dig @"$resolver" "$TARGET_DOMAIN" A +noall +answer +stats | grep -E 'ANSWER:|Query time:|SERVER:' || true
done

echo -e "\n[4/5] TRANSPORT LAYER REACHABILITY (UDP vs TCP Port 53)"
FIRST_NS=$(dig +short NS "$TARGET_DOMAIN" | head -n1)
FIRST_NS_IP=$(dig +short "$FIRST_NS" A | head -n1)
echo "Testing $FIRST_NS ($FIRST_NS_IP)..."
if nc -z -v -w3 "$FIRST_NS_IP" 53 2>&1 | grep -q 'succeeded'; then
  echo "TCP/53 Reachable: YES"
else
  echo "TCP/53 Reachable: NO (Firewall / Listener issue!)"
fi

echo -e "\n[5/5] TRACEROUTE TO AUTHORITATIVE NAMESERVER"
traceroute -n -w 2 -m 15 "$FIRST_NS_IP" || true

echo -e "\n======================================================================"
echo " RCA SNAPSHOT COMPLETE"
echo "======================================================================"

6. SRE Operational Threshold Matrix for DNS Health

Need to translate DNS downtime into permissible error budget loss? Use our interactive SLA Calculator to calculate downtime thresholds across monthly and annual windows.

7. Troubleshooting Runbook: Resolving DNS & Anycast Outages

When external monitoring detects domain resolution failures, follow this ten-step operational triage:

1. Query the domain from multiple public Anycast resolvers (1.1.1.1, 8.8.8.8, 9.9.9.9) to distinguish localized resolver outages from authoritative failures.
2. Execute dig +trace <domain> to verify that TLD root servers return valid delegation NS records.
3. Query every authoritative nameserver individually to compare SOA serial numbers and detect replication lag.
4. Test both UDP/53 and TCP/53 connectivity to ensure firewall rules are not dropping truncated DNS packets.
5. Inspect DNSSEC status using dig +dnssec <domain>; verify that RRSIG inception and expiration timestamps are valid.
6. Execute traceroute -T -p 53 <nameserver_ip> from affected regions to identify the exact autonomous system (AS) where traffic stalls.
7. Audit BGP route announcements using public route collectors (e.g., RIPE RIS or Route Views) to detect route withdrawals.
8. Withdraw BGP prefix announcements from a degraded Anycast PoP to force traffic onto healthy adjacent PoPs.
9. Verify that A and AAAA records resolve consistently to prevent IPv6 split-brain failures.
10. Calculate error budget impact using the Downtime Calculator.

8. Engineering Implementation Checklist

• Deploy Multi-Provider DNS: Utilize two independent authoritative DNS providers (e.g., Route53 + Cloudflare) with synchronized zone serials.
• Enforce Anycast DNS Routing: Ensure authoritative nameservers advertise prefixes globally over BGP Anycast.
• Automate DNSSEC Monitoring: Set alerts on RRSIG expiration 14 days prior to signature lapse.
• Open Both UDP and TCP Port 53: Ensure security groups and firewalls accept incoming connections on TCP/53 for responses exceeding 1232 bytes.
• Set Conservative TTLs: Maintain a 300-second TTL on critical A/AAAA records to enable rapid traffic shifting during outages.
• Monitor SOA Serial Consistency: Alert immediately if authoritative nameservers exhibit divergent SOA serials for $> 5\text{ minutes}$.
• Track BGP Route Visibility: Monitor global prefix reachability across tier-1 transit providers.

Related Network & SRE Guides

When conducting network-level root cause investigations:

• To verify whether DNS delegation failures are disrupting TLS verification, see SSL/TLS certificate and handshake monitoring.
• To determine whether connectivity failures stem from DNS routing or origin daemon crashes, review web server availability.
• To isolate network transit degradation from host resource exhaustion, consult server resource and saturation monitoring.