APISIX Gateway 504s After Exactly 15 Seconds: Tracing It to tcp_syn_retries

Posted on Views: Word count in article: 4.2k Reading time ≈ 4 mins.
An API gateway kept returning 504s with request time pinned at exactly 15 seconds. This postmortem walks through timeout tracing, kernel parameter analysis, DNS elimination and packet capture to root-cause it as a SYN retransmission timeout.

Our production API gateway (API7 Enterprise, a commercial distribution of Apache APISIX) started returning intermittent 504s. Every single one of those requests took exactly 15 seconds. Not 14, not 16 — 15.

The plugin timeout was configured at 600 seconds, so the application layer had no reason to give up that fast. Where does 15 seconds come from? This post walks through how we narrowed it down to tcp_syn_retries=3 and a SYN retransmission timeout — and why we deliberately did not ship the obvious fix.

Following the timeout through the code

First step: verify that the 600-second timeout actually reaches the bottom of the stack. Tracing the call chain from the plugin configuration, through the Lua layer, through the FFI boundary, into the C module:

1
2
3
4
5
6
7
8
9
-- ai-transport/http.lua:145
httpc:set_timeout(timeout)

-- resty/ngx_http_ffi_client.lua
client.set_timeouts(self, 600000, 600000, 600000)

-- ngx_http_ffi_client_request.c:2238
op->connect_timeout = 600000
ngx_add_timer(c->write, op->connect_timeout)

The 600s value is passed down correctly, and the C module really does arm a 600s timer. So the timeout is not coming from the application. Something else owns the 15-second deadline.

Suspecting the kernel

15 seconds is a suspicious number. It smells like a kernel TCP constant rather than an application default. Checking the kernel parameters on the node running the gateway pod:

1
2
sysctl net.ipv4.tcp_syn_retries
# output: net.ipv4.tcp_syn_retries = 3

The kernel computes the total SYN timeout as:

1
timeout = ((2 << boundary) - 1) * rto_base

With tcp_syn_retries=3 and a 1-second base RTO:

1
timeout = ((2 << 3) - 1) * 1 = 15 seconds

An exact match. The retransmission timeline:

1
2
3
4
1st SYN:        sent t=0s,  times out t=1s   (cumulative 1s)
1st retransmit: sent t=1s,  times out t=3s   (cumulative 3s)
2nd retransmit: sent t=3s,  times out t=7s   (cumulative 7s)
3rd retransmit: sent t=7s,  times out t=15s  (cumulative 15s) <- kernel gives up

There is no fifth attempt. tcp_syn_retries=3 means at most 3 retransmissions, 4 attempts total, and then the kernel declares the connection dead.

Eliminating DNS

Could this be DNS resolution failing slowly? Looking at APISIX’s DNS resolution code for the error message formats:

1
2
3
4
5
-- resolver.lua:83
log.error("failed to parse domain: ", host, ", error: ", err)

-- resty/ngx_http_ffi_client.lua
return nil, host .. " could not be resolved (" .. (err or "no address") .. ")"

Searching the gateway logs for these strings did turn up DNS errors:

1
2
3
2026/09/07 08:59:15 [warn] base.lua:255: failed to send request to LLM server:
connect: failed to parse domain: failed to query the DNS server:
dns server error: 2 server failure

But when we pulled the request_ids of those DNS errors from our log store and compared the two populations, a clear difference emerged:

DNS errors SYN timeouts
count 21 750
avg duration 2.25s 15.38s
status code 500 504

DNS failures return immediately — they never sit for 15 seconds. Two independent problems that merely look similar.

Packet capture

Theory is nice; packets are better. Capturing traffic on the gateway node and filtering for handshakes:

1
2
3
4
5
6
# find SYN packets
tcpdump -r capture.pcap 'tcp[tcpflags] & (tcp-syn) != 0' -nn

# find SYN retransmissions (multiple SYNs from the same port)
tcpdump -r capture.pcap 'tcp[tcpflags] & (tcp-syn) != 0' -nn | \
  awk '{print $3, $5}' | sort | uniq -c | sort -rn

We captured a complete retransmission sequence from a single connection:

1
2
3
4
5
6
10.200.76.55.18494 -> 39.96.198.249.443:
  initial SYN:  15:27:20.096
  retransmit 1: 15:27:21.117 (gap 1.0s)
  retransmit 2: 15:27:23.133 (gap 3.0s)
  retransmit 3: 15:27:27.229 (gap 7.1s)
  -> kernel gives up, connection fails

Distribution of the gaps across all captured failures:

gap count theory
1.0s 2 ✓
3.0s 2 ✓
7.1s 2 ✓

The wire-level evidence matches the kernel parameter calculation exactly.

Three layers of evidence

layer evidence conclusion
application 717 requests with request_time = 15s timeout is pinned at exactly 15s, not at 600s
code source trace: 600s propagates correctly not an application timeout
network PCAP: SYN retransmit gaps of 1s, 3s, 7s kernel SYN retransmission timeout

All three layers point to the same root cause: tcp_syn_retries=3 combined with SYNs that never got answered.

The fix we did not ship

The tempting fix — bumping tcp_syn_retries to 5 — was rejected on purpose.

It treats the symptom, not the disease:

  1. The root cause stays. Why are the SYNs not getting through, or not getting answers? Something at the network level is broken. Retrying more often does not repair it.
  2. Failure just takes longer. tcp_syn_retries=5 stretches the total timeout from 15s to 63s. Users now wait over a minute to learn the same thing. That is a worse experience, not a better one.
  3. It hides the real problem. A longer timeout makes the symptom recede from dashboards and alerts while the underlying connectivity issue keeps getting worse.

The correct directions: fix connectivity at the network layer, or implement retry/fallback in the application layer where you control backoff, budgets and error semantics — rather than depending on kernel-level retries you cannot observe or tune per-request.

We followed up by pulling in the corporate network team for a deeper investigation of the network layer. That is where the actual fix lives.

Takeaways

  • Odd, round timeout values are a fingerprint. 15s, 31s, 63s are almost never application configuration. They are the kernel telling you its retry budget expired. tcp_syn_retries=N yields total timeouts of 1s, 3s, 7s, 15s, 31s, 63s for N = 0…5.
  • Cross-validate across layers. Logs alone suggested DNS. Source code alone exonerated the application. Packet capture alone showed retransmissions but not their business impact. Only the combination — request-duration histograms from logs, source-level timeout tracing, and PCAP — pinned the root cause.
  • Similar-looking errors can be unrelated. The DNS failures and the SYN timeouts appeared in the same gateway, in the same period, but had completely different duration profiles. Always compare distributions, not anecdotes.
  • A retry knob is not a fix. If you cannot explain why packets are being dropped, more retries just convert a fast failure into a slow one.

The gateway in this incident is API7 Enterprise, a commercial distribution of Apache APISIX. The fix discussed above was deliberately not applied; connectivity was escalated to the network team instead.

This article is the English edition of a post originally published in Chinese.