Web Server Availability Monitoring: Nginx, Apache, and Caddy Uptime Best Practices

Web Server Availability Monitoring: Nginx, Apache, and Caddy Uptime Best Practices

Executing systemctl is-active nginx or verifying that port 80 accepts a TCP socket connection does not prove that a web server is available to end users. A web server process can be active while worker thread pools are saturated, upstream reverse-proxy sockets are timing out with HTTP 504 errors, TLS certificates are expired, or Anycast DNS routes are black-holed.

Site Reliability Engineers must monitor web server availability across multiple operational layers: process lifecycle, socket connection queues, TLS negotiation, HTTP semantics, upstream application health, and external multi-region reachability.

External Multi-Region Synthetic Probes (Quorum Verification)
   │
   ├── 1. Authoritative DNS Lookup (A / AAAA Dual-Stack)
   │
   ├── 2. TCP Handshake (:80 / :443 SYN-ACK Timing)
   │
   ├── 3. TLS 1.3 Handshake (SNI & ALPN Negotiation)
   │
   ▼
Web Server Edge Layer (Nginx / Apache / Caddy)
   │
   ├── Process & Worker Pool Saturation (stub_status / mod_status / metrics)
   │
   ▼
Reverse Proxy Upstream Layer (HTTP 502 / 504 Detection)
   │
   ├── Upstream Connect Time (Unix Socket / Private VPC IP)
   ├── Upstream Response Time (Node.js / Python / Go / PHP-FPM)
   │
   ▼
Deep Health Check Endpoint (/healthz with JSON Schema Validation)

1. Defining True Web Server Availability

In high-throughput environments, availability is a continuum rather than a binary up/down state. SREs classify availability across five distinct operational boundaries:

1. Process Availability   ──► systemd unit running (systemctl is-active caddy == active)
2. Socket Availability    ──► Kernel accepts TCP SYN on port 443 without backlog drops
3. TLS Availability       ──► Valid X.509 certificate served with negotiated ALPN (h2 / http/1.1)
4. HTTP Gateway Health    ──► Web server returns valid HTTP status (excludes 502/503/504)
5. Application Correctness──► Response body matches expected semantic schema within latency budget

The Availability Calculation Formula

Calculate request-based availability over defined observation windows to determine error budget consumption:

$$ \text{Availability}_{\text{req}} = \frac{\sum \text{Valid 2xx / 3xx Responses}}{\sum \text{Total Ingress Requests} - \text{Client Errors (4xx)}} \times 100 $$

Target SLO: 99.95% Availability
Monthly Error Budget: 0.05% of Total Requests (Approx. 21m 36s of Allowable Downtime)

Need to calculate your error budget or convert uptime nines into allowable downtime? Use our interactive SLA Calculator to model monthly and annual availability targets.

2. Deep Web Server Health Check Design

A production health check must validate end-to-end functionality without placing synchronous query load on backend databases.

GET /healthz ──► Lightweight In-Memory Check (Process Alive & Upstream Socket Reachable)
GET /readyz  ──► Deep Dependency Check (Cache / Database Read-Only Ping)

Comprehensive cURL Diagnostic Command

Use cURL with formatted timing variables to dissect DNS resolution, TCP connect, TLS handshake, and Time to First Byte (TTFB):

curl --fail \
  --silent \
  --show-error \
  --location \
  --connect-timeout 3 \
  --max-time 10 \
  --write-out '\n--- Timing & Status Breakdown ---\nHTTP Status:        %{http_code}\nDNS Lookup:         %{time_namelookup}s\nTCP Connect:        %{time_connect}s\nTLS Handshake:      %{time_appconnect}s\nPre-Transfer:       %{time_pretransfer}s\nStart Transfer:     %{time_starttransfer}s (TTFB)\nTotal Duration:     %{time_total}s\n' \
  https://example.com/healthz

3. Nginx Availability and Saturation Monitoring

Nginx uses an event-driven, non-blocking asynchronous architecture. When Nginx fails, it typically stems from worker connection exhaustion or upstream gateway timeouts.

3.1 Enabling Nginx stub_status

Configure an internal-only status endpoint on 127.0.0.1:

# /etc/nginx/conf.d/stub_status.conf
server {
    listen 127.0.0.1:8080;
    server_name 127.0.0.1;

    location = /nginx_status {
        stub_status;
        allow 127.0.0.1;
        deny all;
        access_log off;
    }
}

# Query Nginx internal connection counters
curl -s http://127.0.0.1:8080/nginx_status

Active connections: 290 
server accepts handled requests
 142095 142095 385012 
Reading: 0 Writing: 14 Waiting: 276 

• Active connections: Current open client connections.
• Waiting: Keep-alive connections waiting for subsequent requests. High Waiting with low Writing indicates idle keep-alive connections consuming memory.
• Handled < Accepts: Indicates Nginx dropped incoming TCP connections due to worker_connections or file-descriptor limits.

3.2 High-Value Nginx Upstream Logging

Add upstream timing variables to the Nginx log_format in /etc/nginx/nginx.conf:

log_format upstream_timed '$remote_addr - $remote_user [$time_local] '
                          '"$request" $status $body_bytes_sent '
                          '"$http_referer" "$http_user_agent" '
                          'rt=$request_time uct="$upstream_connect_time" '
                          'uht="$upstream_header_time" urt="$upstream_response_time"';

• uct (upstream_connect_time): Time taken to establish a TCP/Unix socket connection to the backend application worker.
• urt (upstream_response_time): Time taken for the backend application to generate and return the response body.

4. Apache HTTP Server Availability Monitoring

Apache relies on Multi-Processing Modules (MPMs): event, worker, or legacy prefork.

4.1 Enabling Apache mod_status

Enable mod_status with extended metrics in /etc/apache2/mods-available/status.conf:

<IfModule mod_status.c>
    ExtendedStatus On
    <Location /server-status>
        SetHandler server-status
        Require ip 127.0.0.1 ::1
    </Location>
</IfModule>

# Fetch machine-readable Apache metrics
curl -s "http://127.0.0.1/server-status?auto"

Total Accesses: 492019
Total kBytes: 1048576
BusyWorkers: 42
IdleWorkers: 86
Scoreboard: __________________WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW...

• BusyWorkers vs. IdleWorkers: When IdleWorkers == 0, Apache has reached MaxRequestWorkers. Incoming TCP connections will queue in the OS socket backlog until timeout.

5. Caddy Availability Monitoring

Caddy provides native Prometheus metrics and a dynamic JSON Admin API over a local Unix domain socket or loopback TCP port (127.0.0.1:2019).

5.1 Querying Caddy Native Metrics

# Scrape Caddy native Prometheus telemetry directly
curl -s http://127.0.0.1:2019/metrics | grep -E 'caddy_http_requests_total|caddy_http_request_duration_seconds'

5.2 Automatic HTTPS & ACME Renewal Monitoring

Caddy manages Let's Encrypt / ZeroSSL certificate issuance automatically. Monitor certificate renewal status by checking Caddy's internal storage and logs:

# Check Caddy certificate expiration and internal ACME state
journalctl -u caddy --since "1 hour ago" | grep -Ei 'certificate|acme|renew|error'

Verifying SSL/TLS certificates and expiration on your live endpoints? Use our SSL Inspector Tool to validate certificate chains and expiration dates.

6. Web Server Comparison: Availability & Saturation

7. Prometheus Alerting Rules for Web Servers

Implement PromQL alerting rules to detect gateway errors, latency spikes, and socket drops before user impact escalates.

7.1 Web Server High 5xx Gateway Error Rate (> 1%)

(
  sum(rate(http_requests_total{status=~"5.."}[5m]))
  /
  sum(rate(http_requests_total[5m]))
) * 100 > 1.0

7.2 Nginx Worker Connection Saturation (> 85%)

(
  nginx_connections_active
  /
  (nginx_worker_processes * nginx_worker_connections)
) * 100 > 85.0

7.3 Apache MPM Worker Pool Starvation (Zero Idle Workers)

apache_workers{state="idle"} == 0

8. SRE Web Server Health Threshold Matrix

9. Troubleshooting Runbook: Resolving Web Server Outages

When external monitoring reports an outage or HTTP 5xx errors spike, execute this twelve-step diagnostic sequence:

1. Verify reachability from at least three external geographic vantage points to rule out local transit issues.
2. Inspect DNS resolution across IPv4 and IPv6 using dig +short A example.com and dig +short AAAA example.com.
3. Test direct port reachability using curl -Iv --connect-timeout 3 https://example.com/healthz.
4. Validate SSL/TLS certificate chains and cipher negotiation using openssl s_client -connect example.com:443 -servername example.com.
5. Check web server process status and recent crash logs using systemctl status nginx (or apache2, caddy) and journalctl -xeu <service>.
6. Query internal telemetry (/nginx_status, /server-status?auto, /metrics) to check active worker saturation.
7. Examine access and error logs for HTTP 502 (Bad Gateway) vs HTTP 504 (Gateway Timeout) signatures.
8. Inspect upstream application sockets; verify whether backend workers (PHP-FPM, Gunicorn, Puma, Node.js) are alive and listening.
9. Audit Linux system file-descriptor limits using cat /proc/sys/fs/file-nr and ulimit -n.
10. Check socket listen overflow drops using nstat -az | grep -i listen.
11. Restart or gracefully reload the web server configuration (nginx -s reload / systemctl reload apache2 / caddy reload).
12. Calculate error budget consumption using the Downtime Calculator to determine SLA compliance.

10. Engineering Implementation Checklist

• Configure Deep Health Endpoints: Expose /healthz (liveness) and /readyz (readiness) with JSON semantic output.
• Enable Status Modules: Securely configure stub_status (Nginx), mod_status (Apache), or /metrics (Caddy) on loopback interfaces only.
• Log Upstream Response Times: Capture upstream_connect_time and upstream_response_time in access logs.
• Tune OS Socket Backlogs: Set /proc/sys/net/core/somaxconn to at least 4096 and match Nginx listen 443 backlog=4096.
• Deploy Multi-Region Synthetic Probes: Run external black-box availability probes every 30–60 seconds across multiple cloud regions.
• Implement Rate-Based 5xx Alerts: Configure Prometheus alerts on 5xx error rates (> 1%) rather than static counts.
• Automate TLS Expiry Monitoring: Alert on certificate expiration with warning at 14 days and critical at 72 hours.
• Verify Dual-Stack IPv4 / IPv6: Test that web servers answer identically on both A and AAAA records.

Related Infrastructure & SRE Runbooks

When troubleshooting web server downtime and upstream proxy degradation:

• To identify whether CPU, memory, or socket starvation is freezing worker pools, read Linux server resource saturation.
• For step-by-step triage of upstream 502/504 errors, follow our HTTP 5xx server error diagnosis runbook.
• To prevent certificate expiry from breaking HTTPS virtual hosts, configure automated SSL/TLS certificate monitoring.