When a 504 alarm fires at 3 AM, the instinct is to grep the backend logs, restart the service, and bump the timeout. I've watched this loop close the ticket without fixing anything. The cleaner mental model is that a 504 tells you almost nothing about the backend by itself — it only tells you one thing: the upstream never returned a single byte before proxy_read_timeout expired.

That single fact splits the diagnosis into six root causes that occur in order along the request path. Debug them in that order and you stop guessing.

What a 504 Actually Is

Nginx is almost never the problem. It is a messenger. The worker writes the request into the upstream socket and then enters a read loop. Each event-loop cycle it checks whether proxy_read_timeout has been exceeded; when it has, it closes the upstream connection and returns 504, logging:

upstream timed out (110: Connection timed out) while reading response header from upstream 

The phrase "timed out" trips everyone up. This is a read timeout, not a connect timeout. A connect timeout is a 502 with connect() failed. In nginx's own source they are different error codes: NGX_HTTP_BAD_GATEWAY for 502, NGX_HTTP_GATEWAY_TIME_OUT for 504. Keeping that distinction straight prevents most of the cross-wiring between the two.

Six Root Causes, In Request Order

They are not parallel branches. They happen in sequence, so that is the order you walk them:

nginx worker receives request [5] DNS resolution slow TCP connect + [4] SSL handshake [3] connection-pool acquisition [2] backend CPU saturation / event loop blocked [6] slow query or slow I/O (DB, ES, disk) [1] proxy_read_timeout too short -> return 504 

Roughly, in my experience: misconfigured timeout about 40%, backend CPU about 25%, slow query about 20%, connection pool about 8%, SSL handshake about 5%, DNS about 2%. Your mix will differ, but the priority order holds.

The Quick Discriminator

The access-log upstream_response_time field is the fastest way to separate the cases:

  • 502: upstream_response_time is small (under a second) — bytes arrived and were rejected, or the connection RST'd.
  • 504 pinned by a timeout: upstream_response_time equals proxy_read_timeout to the millisecond.

If every 504 row in your access log shows the same upstream_response_time value, you are pinned by a timeout configuration, not by a dead backend.

The Timeout Anti-Pattern

The most common 504 "fix" is to crank proxy_read_timeout to 300s and declare victory. You have not fixed anything. You have just turned a 504 into a P99 of 99 seconds — the user waits just as long, only the error-rate dashboard looks better. Set the timeout from the real P99 (roughly 2-3x), watch the P99 curve after the change, and remember that enlarging a timeout is a painkiller, not a cure. One customer bumped 60s to 300s and the 504s "vanished"; a week later P99 climbed from 3s to 45s because the real cause — a missing index behind a slow query — was still there.

A related trap: with proxy_next_upstream_tries 2 and a proxy_read_timeout of 20s, a user perceives up to 40s. If your client gives up at 30s, you start seeing 499s instead of the 504 you actually wanted to diagnose. Keep the sum of all nginx timeouts below the client timeout.

Two Branches Worth Their Own Mention

Slow DNS. When proxy_pass uses a hostname, nginx resolves it via system DNS. A stuck resolver poisons every new upstream connection (cold start, pool flush) and the symptom is 504s lasting 3-10 seconds that only appear on fresh connections. The fix is to give nginx its own resolver and force dynamic resolution with the variable form — set $backend host; proxy_pass http://$backend; — because a static proxy_pass http://host resolves once at startup and never again.

Connection pool exhaustion. A depleted upstream pool makes nginx wait too long for a free connection and it trips proxy_read_timeout — which surfaces as a 504, not the 502 you might expect. ss -tanp showing ESTAB connections well beyond worker_processes x keepalive is the tell.

A Note on the Source

I keep running into 504 variants on my own infrastructure and have been recording the branches, the exact error.log keywords, and the 30-command scan order I use when the alarm fires. The longer writeup — with the per-branch diagnostic commands and the timeout-derivation script — is in my ops notes:

The decision-tree framing — walk the request path in order instead of guessing — is what actually gets a 504 closed in one session rather than three.