HTTP 504 Gateway Timeout — Fix the Upstream That Responded Too Slowly

A 504 Gateway Timeout is not a single error with a single fix — it is a category of error produced by different systems at different time thresholds. Understanding which layer is timing out determines both the diagnosis and the correct fix.

nginx has a configurable proxy_read_timeout directive that defaults to 60 seconds. This is the amount of time nginx waits for the upstream to send any data. If the upstream does not send the first byte of the response within 60 seconds, nginx closes the connection and returns 504. The limit applies per read operation, not to the total response time, so streaming responses that send chunks can survive longer.

AWS API Gateway imposes a hard 29-second integration timeout on all Lambda proxy integrations. This limit cannot be increased through configuration — it is enforced at the API Gateway service level regardless of your Lambda timeout setting. Even if your Lambda is configured with a 15-minute timeout, API Gateway will close the connection after 29 seconds. This is the most common source of 504 errors in serverless architectures and the one most frequently misunderstood.

AWS ALB has a configurable idle timeout that defaults to 60 seconds. Unlike API Gateway, this can be raised through load balancer attributes in the AWS console or CLI. However, raising it treats the symptom rather than the cause. ALB also deregisters targets that fail health checks, and a slow upstream that is not responding to health checks may be removed from the pool entirely.

Cloudflare enforces a 100-second response timeout from the origin server. Requests that take longer than 100 seconds receive a 524 error (Cloudflare-specific) rather than a generic 504. Enterprise Cloudflare plans allow custom timeout values, but most projects operate under the 100-second limit. Because Cloudflare's limit is higher than most origin timeouts, a 524 usually means the origin nginx or ALB already gave up and Cloudflare is presenting that failure.

All of these limits exist for legitimate reasons: a proxy that waits indefinitely for slow upstreams accumulates open connections, exhausts file descriptors, and eventually fails entirely. The architectural response to hard limits is decoupling slow operations from the request-response cycle using queues, background workers, and polling endpoints.

The precision of the timeout value is your first diagnostic clue. If requests fail at exactly 60 seconds, nginx proxy_read_timeout is the culprit. If failures happen at exactly 29 seconds, AWS API Gateway is timing out. If failures cluster around 100 seconds, Cloudflare is the terminating layer. Exact thresholds indicate proxy timeouts rather than variable upstream slowness.

Open the browser DevTools Network tab and click the failing request. Look at the Timing tab — the TTFB (Time To First Byte) section shows how long the browser waited before receiving any response. A TTFB equal to the timeout value confirms the proxy gave up. The Response tab will show the 504 page HTML, which often includes identifying markers: nginx version, Cloudflare ray ID, or AWS error codes.

For database-related timeouts, enable the slow query log on your database server. In PostgreSQL, set log_min_duration_statement = 5000 in postgresql.conf to log all queries taking more than 5 seconds. In MySQL, set slow_query_log = 1 and long_query_time = 5. Then reproduce the 504 and check the slow query log immediately — the offending query will appear with its execution time and query plan. Run EXPLAIN ANALYZE on the query to find missing indexes or expensive full-table scans.

For Node.js applications, add timing instrumentation at the handler level. Log the time at the start and end of each async operation: database query, external HTTP call, Redis command, and file I/O. Even a simple Date.now() before and after each await call is enough to identify which operation is consuming the most time. The operation whose duration approaches or exceeds the proxy timeout is the one to optimize.

For Lambda, open CloudWatch Logs and filter for 'Task timed out' or review the Duration field in log entries. Lambda logs the billed duration of each invocation. If requests are failing at 29 seconds on API Gateway but Lambda logs show them completing in 35 seconds, the function is completing successfully but API Gateway already returned 504 before Lambda finished. Use /tools/http-request-builder to measure round-trip response times from outside the AWS network and compare against CloudWatch timing data.

The right fix depends on which layer is timing out and why the upstream is slow. Never start by raising the proxy timeout — that is the last resort after all other options are exhausted.

For slow database queries, the fix is query optimization. Run EXPLAIN ANALYZE and look for sequential scans on large tables. Add indexes on the columns in WHERE, JOIN, and ORDER BY clauses. For aggregation queries that touch millions of rows, precompute the result on a schedule and serve it from a cache. Use materialized views in PostgreSQL for aggregations that are expensive to compute but acceptable to refresh periodically.

For nginx proxy_read_timeout adjustments when raising is genuinely appropriate — for example, a legitimate long-running export endpoint — increase it only for that specific location block rather than globally. Use 'location /api/exports { proxy_read_timeout 120s; }' rather than setting it in the main http block. Document the reason for the higher timeout adjacent to the configuration so future engineers understand why it exists.

For AWS API Gateway with Lambda, the architectural fix is always async decoupling. Accept the request in a fast Lambda that validates input and publishes to SQS, EventBridge, or SNS. Return 202 Accepted with a job ID. Implement a polling endpoint that checks job status from DynamoDB or Redis. Process the actual work in a separate Lambda subscribed to the queue. This pattern has no timeout ceiling and scales horizontally through queue depth.

For FastAPI, Django, or Flask applications where synchronous database calls block async handlers, migrate heavy operations to background tasks using Celery, ARQ, or FastAPI's BackgroundTasks. The request handler should do only lightweight work: validate input, enqueue the job, return a response. The worker processes the job asynchronously. Use Redis as the message broker for simple setups or RabbitMQ for more complex routing requirements.

For Cloudflare 524 errors specifically, the fix is usually in your origin server configuration. Cloudflare's 100-second limit is generous — if you are hitting it, the origin is genuinely too slow. Profile the slowest requests in your application using APM tools like Datadog, New Relic, or the open-source OpenTelemetry stack. Identify the p99 response time for each endpoint and set a target of keeping p99 below 10 seconds for interactive endpoints.

Several scenarios produce 504 errors that are difficult to reproduce locally because they depend on production load patterns or distributed system behavior.

Cascading timeouts are the most damaging pattern. Service A calls Service B with a 30-second timeout. Service B calls Service C with a 30-second timeout. If Service C is slow, both B and A accumulate open connections waiting for responses. The cascade amplifies load on every upstream service. The fix is to use context propagation with deadline reduction: subtract the time already spent from the remaining timeout budget at each hop, and set a global request deadline at the edge. In Go, the context package supports this natively. In Node.js, use AbortController with a shared signal passed through async call chains.

Event loop blocking in Node.js produces intermittent 504 errors that appear random. If a synchronous CPU-intensive operation — JSON parsing of a large payload, bcrypt with high rounds, regex on long strings — executes in the request handler, it blocks the event loop and prevents other requests from progressing. Under load, multiple requests pile up behind the blocked event loop and eventually hit the proxy timeout. Profile Node.js with 'node --prof' and look for synchronous operations taking more than a few milliseconds per invocation.

Database connection pool exhaustion creates a timeout pattern that worsens under load. When all pool connections are in use, new requests queue waiting for a connection. The queue grows faster than connections are released, and requests eventually wait long enough to trigger the proxy timeout. Increase the pool size, but also audit whether connections are being released promptly — especially in error handling paths where a thrown exception might skip the connection release code.

Redis CLUSTER operations with WAIT commands can block for extended periods during network partitions between cluster nodes. If your application uses WAIT to ensure replication durability, a partition event can cause arbitrary delays. Set a reasonable timeout on WAIT commands and handle the timeout as a degraded but acceptable state rather than blocking indefinitely. This is especially important in multi-region setups where cross-region replication latency is variable.

Raising the proxy timeout globally is the most common band-aid applied to 504 errors. Setting proxy_read_timeout to 300 or 600 seconds in the nginx http block affects every upstream, including ones that should fail fast. A legitimate connection to a crashed upstream will now hold open connections for 5 to 10 minutes instead of 60 seconds, consuming file descriptors and memory on the proxy. Always scope timeout increases to the specific location or upstream that legitimately needs more time.

Setting Lambda timeout to 15 minutes without addressing API Gateway is a common serverless misunderstanding. Lambda has a configurable 15-minute timeout, but if the Lambda is invoked through API Gateway, the 29-second API Gateway limit applies regardless of the Lambda setting. Engineers who raise the Lambda timeout expecting it to fix 504 errors from API Gateway are surprised when nothing changes. The Lambda timeout only applies to direct invocations and queued invocations — not to API Gateway proxy integrations.

Not distinguishing between a 504 and a 502 when choosing a fix leads to wasted effort. A 504 means the upstream is alive but slow — fix the slowness or use async processing. A 502 means the upstream is not responding at all — fix the crash or connection issue. The nginx error log message distinguishes them: 'upstream timed out' is a 504 cause, while 'upstream prematurely closed connection' or 'connect() failed' is a 502 cause.

Ignoring cold start latency in serverless 504 troubleshooting creates a recurring problem. Lambda cold starts for large functions can add 1 to 5 seconds or more to first invocations after idle periods. For functions with heavy dependencies — large node_modules, compiled binaries — cold starts can exceed 10 seconds. Combined with actual processing time, the total easily approaches the API Gateway limit. Use Provisioned Concurrency for latency-sensitive endpoints, or restructure the deployment to use Lambda@Edge for cacheable responses.

Not having a health check endpoint that responds in under 500 milliseconds makes the load balancer remove healthy instances when they are momentarily slow. The health check must be a fast, shallow check, not a deep application probe. Many teams add database queries to health checks thinking this makes them more informative, but a slow health check during a database overload event removes the instance from the pool at exactly the wrong time.

The most effective way to prevent 504 errors is to design every API endpoint to complete within a predictable, short time budget. For interactive endpoints — those called from a browser or mobile app — target a p99 response time under 2 seconds. For internal service-to-service calls, target under 500 milliseconds. Any operation that cannot reliably meet these targets should be decoupled into an async flow with a polling or webhook notification mechanism.

Add response time SLOs to every endpoint during design, not after incidents. Define the maximum acceptable response time and instrument the code to measure it. Alert when p95 or p99 exceeds 50 percent of the timeout threshold — this gives you time to fix performance regressions before they start producing 504 errors in production.

Use circuit breakers between services. When Service A calls Service B and B becomes slow, a circuit breaker detects the degraded response times and short-circuits calls to B, returning a cached response or a meaningful error immediately. Libraries like Resilience4j for JVM, Polly for .NET, and opossum for Node.js implement circuit breaker patterns. This prevents slow upstreams from causing cascading timeouts across the entire dependency graph.

Every health check endpoint must respond in under 500 milliseconds unconditionally. A health endpoint that itself runs a slow database query defeats its purpose — a 504 health check causes the load balancer to remove the target from the pool, reducing capacity exactly when you need it most. The health endpoint should check database reachability with a fast SELECT 1 or a cached connection ping, not run application logic.

Use /tools/http-request-builder and /tools/cors-tester to test endpoint response times from multiple geographic locations and network conditions. Observing response times from outside your infrastructure reveals proxy timeout behaviors and CDN edge caching effects that are invisible from within the application. Monitor p50, p95, and p99 latencies continuously — a drift in p99 is the first sign of an emerging timeout problem before it affects a significant fraction of users.