Diagnosing NGINX 110 Connection Timed Out: Upstream Timeouts, Keepalive Pools, and Proxy Buffer Tuning

Few production errors cause as much confusion during a Sev-1 incident as the NGINX upstream timeout:

2026/09/15 03:12:45 [error] 28941#28941: *894021 upstream timed out (110: Connection timed out) 
while reading response header from upstream, client: 198.51.100.42, server: api.pingzoapp.com, 
request: "POST /v1/telemetry/events HTTP/1.1", upstream: "http://10.0.4.12:8080/v1/telemetry/events", 
host: "api.pingzoapp.com"

To the downstream user or mobile client, this failure manifests as an immediate HTTP 504 Gateway Timeout or dropped TCP connection. To the operations team, it triggers a cascade of automated client retries, thread pool saturation, socket buffer exhaustion, and severe upstream overload.

Resolving error 110 requires looking beyond the naive reaction of arbitrarily inflating proxy_read_timeout to 300 seconds. In this comprehensive guide, we dissect the Linux kernel and NGINX socket mechanics behind ETIMEDOUT (errno 110), trace the exact timeout boundaries of the proxy lifecycle, tune upstream keepalive pools and memory buffers, and establish production-grade SRE runbooks.

1. What NGINX Error 110 Actually Means

In Linux POSIX error semantics, 110 corresponds directly to ETIMEDOUT (Connection timed out). When NGINX outputs this error in its error log, it indicates that a non-blocking asynchronous I/O timer registered with the kernel event notification facility (epoll on Linux or kqueue on BSD) expired before the required socket event occurred.

However, error 110 is not a single uniform failure. It occurs across five distinct phases of the network request lifecycle:

+---------------------------------------------------------------------------------------------+
|                                NGINX REQUEST TIMEOUT MATRIX                                 |
+----------------------+--------------------+---------------------+---------------------------+
| Error Log String     | Kernel / OS Event  | Timeout Directive   | Root Failure Mechanism    |
+----------------------+--------------------+---------------------+---------------------------+
| connect() failed     | TCP SYN unack'd    | proxy_connect_timeout| Security group drop,      |
| (110: Connection     | (No SYN-ACK recv)  |                     | dead host, route blackhole|
| timed out)           |                    |                     |                           |
+----------------------+--------------------+---------------------+---------------------------+
| while SSL            | TLS Handshake stall| proxy_connect_timeout| Upstream TLS engine stall,|
| handshaking          | (ClientHello unack)|                     | entropy exhaustion, cipher|
+----------------------+--------------------+---------------------+---------------------------+
| while sending        | TCP Send window    | proxy_send_timeout  | Upstream TCP receive      |
| request to upstream  | full / stalled     |                     | window clamped to 0       |
+----------------------+--------------------+---------------------+---------------------------+
| while reading        | Application idle   | proxy_read_timeout  | Slow database query, lock |
| response header      | (0 bytes returned) |                     | contention, thread stall  |
+----------------------+--------------------+---------------------+---------------------------+
| while reading        | Stream stall       | proxy_read_timeout  | Mid-payload generation    |
| upstream body        | between chunks     |                     | stall, slow gRPC/SSE stream|
+----------------------+--------------------+---------------------+---------------------------+

Why HTTP 504 Accompanies Error 110

When NGINX fails to receive an upstream response within the configured timeout window, it terminates the upstream socket connection (sending a TCP RST or FIN) and synthesizes an internal 504 Gateway Timeout response downstream to the requesting client.

If the downstream client closes the connection before NGINX's internal timer expires, NGINX logs an HTTP 499 (Client Closed Request) instead.

2. NGINX Request Lifecycle and Timeout Boundaries

Understanding where each timeout directive applies is critical to avoiding misconfigurations.

Client                          NGINX Edge                     Upstream Application
  |                                 |                                   |
  |--- TCP 3-Way Handshake -------->|                                   |
  |    (Client <-> NGINX)           |                                   |
  |                                 |                                   |
  |--- HTTP Request Payload ------->|                                   |
  |    (client_body_timeout)        |                                   |
  |                                 |--- DNS Query (resolver_timeout) ->|
  |                                 |<-- DNS IP Response --------------|
  |                                 |                                   |
  |                                 |--- TCP SYN (proxy_connect_timeout)|
  |                                 |<-- TCP SYN-ACK -------------------|
  |                                 |--- TCP ACK ---------------------->|
  |                                 |                                   |
  |                                 |--- TLS Handshake (if HTTPS) ----->|
  |                                 |<-- TLS Finished ------------------|
  |                                 |                                   |
  |                                 |--- Forward Request Header/Body -->|
  |                                 |    (proxy_send_timeout)           |
  |                                 |                                   |
  |                                 |    [App Processing / DB Query]    |
  |                                 |                                   |
  |                                 |<-- 1st Response Byte (Header) ----|
  |                                 |    (proxy_read_timeout)           |
  |                                 |                                   |
  |                                 |<-- Response Body Data Chunks -----|
  |                                 |    (proxy_read_timeout gap timer) |
  |                                 |                                   |
  |<-- Stream Response to Client ---|                                   |
       (send_timeout)

The Critical "Gap Timer" Nuance of proxy_read_timeout

A pervasive misconception is that proxy_read_timeout defines the maximum total execution time for an upstream request.

It does not.

proxy_read_timeout sets a timeout between two successive read operations, not for the transmission of the entire response. If the upstream service sends at least one byte of data every 20 seconds, a request governed by proxy_read_timeout 30s; can run continuously for two hours without ever triggering a 110 timeout.

3. TCP-Level Diagnostics for Error 110

When connect() failed (110: Connection timed out) occurs, the problem is situated in the Layer 3/Layer 4 transport stack.

   NGINX Host                                           Upstream Host
   (10.0.1.50)                                           (10.0.4.12)
        |                                                     |
        |--- TCP SYN (Seq=0) -------------------------------->| [Dropped by AWS Security Group /
        |    [Timer: 1s]                                      |  iptables / Listen Queue Full]
        |--- Retransmit TCP SYN (Seq=0, 2s backoff) --------->| (No ACK sent)
        |    [Timer: 2s]                                      |
        |--- Retransmit TCP SYN (Seq=0, 4s backoff) --------->|
        |                                                     |
        X [proxy_connect_timeout (5s) expires]                |
        |                                                     |
        |--> NGINX logs: "connect() failed (110)"             |

Linux Socket State Inspection

To determine if sockets are accumulating in TIME_WAIT, SYN_SENT, or experiencing queue drops, execute:

# Inspect total socket counts and memory allocations
ss -s

# Check connections in SYN-SENT state (unacknowledged handshakes)
ss -tan state syn-sent

# Inspect listening queue backlog depth on upstream port
ss -ltn '( sport = :8080 )'

In the output of ss -ltn:

• Send-Q: Indicates the maximum listener backlog (backlog parameter in listen, bounded by net.core.somaxconn).
• Recv-Q: Indicates the number of fully established TCP connections currently waiting in the kernel queue to be accept()'ed by the application. If Recv-Q > Send-Q, the application is CPU-saturated or event-blocked, and incoming SYNs are dropped silently.

4. Separating Connection Timeouts from Response Timeouts

When diagnosing error 110 in production, follow this diagnostic decision table:

5. Instrument NGINX Before Changing Configuration

Never tune timeout parameters blindly without telemetry. Add microsecond-precision timing variables to your log_format in /etc/nginx/nginx.conf:

http {
    log_format upstream_perf 
        '[$time_iso8601] client=$remote_addr request_id=$request_id '
        'status=$status upstream_status=$upstream_status '
        'request_time=$request_time '
        'upstream_connect_time=$upstream_connect_time '
        'upstream_header_time=$upstream_header_time '
        'upstream_response_time=$upstream_response_time '
        'upstream_addr="$upstream_addr" '
        'bytes_sent=$bytes_sent '
        'upstream_bytes_received=$upstream_bytes_received '
        'uri="$request_uri" host="$host"';

    access_log /var/log/nginx/access_perf.log upstream_perf;
}

Decoding the Metrics:

• $request_time: Total elapsed time in seconds (with millisecond resolution) from receiving the first byte from the client until the last byte was sent.
• $upstream_connect_time: Time spent establishing the TCP (and TLS) connection to the upstream server. If this equals your proxy_connect_timeout (e.g. 5.000), your network/firewall is dropping SYNs.
• $upstream_header_time: Time taken between initiating the connection and receiving the first byte of HTTP response headers from the upstream.
• $upstream_response_time: Total time spent communicating with the upstream server to complete the entire response.

6. Upstream Timeout Configuration & Concurrency Math

The default NGINX configuration values are:

location / {
    proxy_pass http://backend_pool;
    proxy_connect_timeout 60s; # Default is 60s (excessive for internal networks)
    proxy_send_timeout    60s; # Default is 60s
    proxy_read_timeout    60s; # Default is 60s
    send_timeout          60s; # Client-side transmit timeout
}

The Danger of Timeout Inflation (Little's Law)

When faced with 110 errors, engineers often increase proxy_read_timeout to 300s. This frequently converts an isolated microservice slowdown into a total infrastructure outage due to concurrency explosion:

$$ C approx lambda imes W $$

Where:

• $C$ = Number of concurrent connections held open in NGINX.
• $lambda$ = Incoming request arrival rate (requests/sec).
• $W$ = Average request holding time (seconds).

SCENARIO A: Normal Operation
Arrival Rate (λ): 1,000 req/sec
Average Duration (W): 0.050s (50ms)
Concurrent Sockets (C): 1,000 * 0.05 = 50 concurrent connections

SCENARIO B: Upstream Stalled with proxy_read_timeout = 300s
Arrival Rate (λ): 1,000 req/sec
Average Duration (W): 300s (stalled requests accumulating)
Concurrent Sockets (C): 1,000 * 300 = 300,000 concurrent connections!

At 300,000 concurrent sockets, NGINX exhausts its worker_connections, Linux exhausts its ephemeral port range and file descriptor limits (nofile), and the entire server stops accepting new traffic.

[!TIP] Use Pingzo's interactive SLA Calculator and Downtime Calculator to evaluate how your latency budgets, error rates, and endpoint timeout thresholds impact your team's overall system availability and monthly SLA commitments.

7. Keepalive Pools and Connection Reuse

Opening a new TCP connection for every upstream HTTP request introduces significant latency:

• Plain HTTP: 1 round-trip time (RTT) for TCP 3-way handshake.
• HTTPS: 1 RTT for TCP + 1 to 2 RTTs for TLS 1.3/1.2 handshake.

Under 2,000 req/sec, opening fresh connections burns ephemeral ports, generates thousands of TIME_WAIT sockets, and strains upstream accept loops.

Hardened Upstream Keepalive Configuration

http {
    upstream backend_api {
        server 10.0.4.10:8080 max_fails=3 fail_timeout=10s;
        server 10.0.4.11:8080 max_fails=3 fail_timeout=10s;

        # Maintain up to 128 idle keepalive connections per NGINX worker process
        keepalive 128;

        # Bound maximum requests per cached connection to avoid stale socket drift
        keepalive_requests 10000;

        # Close idle pooled connections after 60 seconds
        keepalive_timeout 60s;
    }

    server {
        listen 80;
        server_name api.pingzoapp.com;

        location / {
            proxy_pass http://backend_api;

            # MANDATORY: HTTP/1.1 is required for upstream keepalive
            proxy_http_version 1.1;

            # MANDATORY: Clear the "Connection: close" header sent by HTTP/1.0 defaults
            proxy_set_header Connection "";

            # Forward client host headers
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;

            # Calibrated timeout budgets
            proxy_connect_timeout 3s;
            proxy_send_timeout 10s;
            proxy_read_timeout 15s;
        }
    }
}

8. Diagnosing Stale or Exhausted Keepalive Pools

A subtle cause of intermittent 110 errors is a keepalive timeout mismatch between NGINX and the upstream application:

NGINX                                                 Upstream (e.g. Node/Gunicorn)
keepalive_timeout: 65s                                keepalive_timeout: 60s
  |                                                              |
  |--- Reuses pooled connection at t=61s ----------------------->| [Upstream closed socket at t=60s!]
  |    (Sends HTTP Request)                                      |
  |                                                              |
  |<-- TCP RST (Connection Reset) -------------------------------|
  |                                                              |
  |--> NGINX: Connection reset by peer / upstream timed out      |

The Golden Rule of Keepalive Timeouts

Upstream Application Keepalive Timeout MUST be greater than NGINX's keepalive_timeout:

$$\text{Upstream Timeout (e.g., Gunicorn)} > \text{NGINX keepalive_timeout} > \text{Edge Cloud LB Timeout}$$

• Set NGINX keepalive_timeout 60s;
• Set Gunicorn/Uvicorn/Tomcat keepalive timeout to 65s or 75s.

9. Proxy Buffer Architecture and Disk I/O Bottlenecks

When NGINX proxies a response from the upstream, it buffers the response in memory before transmitting it to the client. This decouples fast upstream servers from slow mobile clients.

Upstream Socket
      |
      v
+-------------------------------------------------------------+
| NGINX Memory Ring Buffer                                    |
| [proxy_buffer_size]          -> Stores initial HTTP headers  |
| [proxy_buffers 8 16k]        -> Stores response body blocks |
+-------------------------------------------------------------+
      |
      |-- (If body fits in proxy_buffers) -------> Stream to Client Socket
      |
      +-- (If body EXCEEDS proxy_buffers) -------> Spools to Disk Temp File
                                                   (/var/lib/nginx/proxy_temp)

If response bodies exceed memory buffers, NGINX writes the excess payload to temporary files on disk (proxy_temp_path). Under high concurrency, disk I/O saturation blocks NGINX worker event loops, causing downstream read operations to stall and trigger error 110.

10. Proxy Buffer Tuning Without Creating Memory Pressure

Calculate total proxy buffer memory consumption using the formula:

$$ M_{\text{proxy}} \approx N_{\text{active}} \times \left( B_{\text{header}} + (n \times B_{\text{body}}) \right) $$

Where:

• $N_{\text{active}}$ = Number of concurrent active buffered requests.
• $B_{\text{header}}$ = proxy_buffer_size (e.g. 16KB).
• $n$ = Number of buffers in proxy_buffers (e.g. 8).
• $B_{\text{body}}$ = Buffer size in proxy_buffers (e.g. 16KB).

$$\text{Per Request Memory} = 16\text{ KB} + (8 \times 16\text{ KB}) = 144\text{ KB}$$

$$\text{For 5,000 Concurrent Requests} = 5,000 \times 144\text{ KB} \approx 720\text{ MB RAM}$$

Production Buffer Configurations

1. High-Throughput REST / JSON API (Payloads < 64KB)

location /v1/api/ {
    proxy_buffering on;
    proxy_buffer_size 16k;
    proxy_buffers 8 16k;             # Up to 128KB in memory
    proxy_busy_buffers_size 32k;
    proxy_max_temp_file_size 0;      # Disable disk spooling entirely for pure memory speed
}

2. Streaming API, Server-Sent Events (SSE), or WebSockets

location /v1/stream/ {
    # Disable buffering so every chunk is immediately flushed downstream
    proxy_buffering off;
    proxy_cache off;
    proxy_set_header Connection '';
    proxy_http_version 1.1;
    chunked_transfer_encoding off;
    
    # Use continuous keepalive timers
    proxy_read_timeout 3600s;
    proxy_send_timeout 3600s;
}

11. Production-Safe Diagnostic Toolkit

Run these diagnostic commands directly from the NGINX instance during an active timeout event:

# 1. Test Layer 4 TCP connectivity and latency directly to upstream
nc -zv -w 3 10.0.4.12 8080

# 2. Trace exact HTTP latency breakdown from the NGINX shell
curl -s -o /dev/null -w "\
DNS Lookup:        %{time_namelookup}s\n\
TCP Connect:       %{time_connect}s\n\
TLS Handshake:     %{time_appconnect}s\n\
Pre-Transfer:      %{time_pretransfer}s\n\
Start Transfer:    %{time_starttransfer}s\n\
Total Time:        %{time_total}s\n\
HTTP Code:         %{http_code}\n" \
http://10.0.4.12:8080/health

# 3. Capture TCP SYN retransmissions or reset flags on the upstream interface
sudo tcpdump -nn -i eth0 'host 10.0.4.12 and (tcp-syn != 0 or tcp-rst != 0)'

# 4. Inspect kernel TCP retransmission and drop counters
nstat -az | grep -Ei 'TcpRetransSegs|TcpExtListenDrops|TcpExtListenOverflows'

# 5. Check whether NGINX is writing temporary proxy files to disk
ls -lh /var/lib/nginx/proxy_temp/

# 6. Validate NGINX syntax and reload without dropping active traffic
sudo nginx -t && sudo nginx -s reload

12. Retries, Queueing, and Cascading Failure Control

By default, NGINX will automatically retry a failed request on the next upstream server if proxy_next_upstream conditions are met.

However, uncontrolled retries create a Retry Storm:

$$ R_{\text{effective}} = R_{\text{incoming}} \times (1 + \text{Retries}) $$

If 1,000 req/sec encounter a slow upstream and NGINX retries twice on remaining nodes, traffic surges to 3,000 req/sec, instantly knocking over healthy instances.

Hardening proxy_next_upstream

location /api/ {
    proxy_pass http://backend_pool;
    
    # Only retry on network-level connection failures or unaccepted connections
    # NEVER retry non-idempotent operations on 504 / timeout unless idempotency is guaranteed
    proxy_next_upstream error timeout invalid_header http_502 http_503;
    
    # Cap maximum retry attempts to 2 (including initial try)
    proxy_next_upstream_tries 2;
    
    # Maximum time allocated across ALL retry attempts
    proxy_next_upstream_timeout 10s;
}

13. Hardened Reference Configurations

High-Performance Microservice Reverse Proxy with SSL Termination

upstream microservice_cluster {
    zone upstream_dynamic 64k;
    least_conn;

    server 10.0.4.101:8080 max_fails=3 fail_timeout=10s weight=10;
    server 10.0.4.102:8080 max_fails=3 fail_timeout=10s weight=10;
    server 10.0.4.103:8080 backup;

    keepalive 256;
    keepalive_requests 20000;
    keepalive_timeout 60s;
}

server {
    listen 443 ssl http2;
    server_name api.pingzoapp.com;

    ssl_certificate /etc/letsencrypt/live/api.pingzoapp.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/api.pingzoapp.com/privkey.pem;
    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256;

    # Performance Tuning
    client_body_buffer_size 128k;
    client_max_body_size 10m;

    location / {
        proxy_pass http://microservice_cluster;
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        # Upstream Timeouts (Calibrated to SLO)
        proxy_connect_timeout 2s;
        proxy_send_timeout 10s;
        proxy_read_timeout 15s;

        # Memory Buffers
        proxy_buffering on;
        proxy_buffer_size 16k;
        proxy_buffers 16 16k;
        proxy_busy_buffers_size 64k;

        # Bounded Retries
        proxy_next_upstream error timeout http_502 http_503;
        proxy_next_upstream_tries 2;
        proxy_next_upstream_timeout 5s;
    }
}

14. Production SRE Runbook & Incident Decision Tree

When NGINX 110 errors spike on your monitoring dashboards, follow this systematic resolution tree:

                                [ NGINX 110 ERROR DETECTED ]
                                              |
                       Check $upstream_connect_time in Access Logs
                                              |
                   +--------------------------+--------------------------+
                   |                                                     |
         Connect Time >= Timeout (e.g. 5.0s)                  Connect Time < 0.05s
                   |                                                     |
       [ LAYER 4 NETWORK / ROUTING ]                         [ LAYER 7 APPLICATION ]
                   |                                                     |
  - Check AWS Security Groups / Firewall NACLs          - Check $upstream_header_time
  - Inspect SYN backlog: ss -ltn                        - Inspect Database slow query log
  - Test connectivity: nc -zv 10.0.4.12 8080            - Inspect application CPU & Thread pool
  - Check kernel conntrack table saturation             - Verify Keepalive timeout alignment

Production Checklist Before Exiting the Incident:

1. Verify Metrics: Confirm $upstream_connect_time < 10ms and $upstream_header_time is within p95 SLO.
2. Audit Connection Pools: Verify ss -tan state time-wait is stable and not surging past net.ipv4.tcp_max_tw_buckets.
3. Validate Keepalives: Ensure proxy_set_header Connection ""; and proxy_http_version 1.1; are set on all active proxy locations.
4. Tune Retry Limits: Confirm proxy_next_upstream_tries is explicitly capped to prevent cascading failover loops.
5. Set Automated Alerting: Ensure Pingzo monitors your public edge endpoints every 60 seconds with instant WhatsApp and webhook escalations.