Runtime Behavior Blind Spots: Postmortem of CPU Saturation Bugs in an API7 AI Gateway

Posted on Edited on Views: Word count in article: 19k Reading time ≈ 18 mins.
Postmortem of repeated CPU saturation in an API7 AI gateway: cancellation propagation, scheduling fairness, per-request decode cost, and streaming backpressure — four distinct failure modes with separate fixes.

Correct code logic doesn’t mean correct runtime behavior. This postmortem covers the recurring CPU saturation problem in an API7 AI gateway — and the blind spots in runtime behavior that code review alone cannot see.

Background

Recently, while using the API7 AI gateway, I repeatedly ran into workers with CPU pegged at 100%. API7 is a commercial distribution of Apache APISIX, and the fixes ultimately went into upstream APISIX, spread across several PRs:

Time Fix Core problem addressed
– PR #13254 Terminate the upstream stream after the downstream client disconnects, via error propagation from a synchronous flush
– PR #13255 Reported Add explicit scheduling points to the streaming loop so a single request can’t hold a worker for a long time
– PR #13356 Reported Cache post_arg.* request body parse results to avoid repeated decodes
– PR #13377 Reported Cache the JSON parse result of AI request bodies to eliminate repeated decodes
– PR #13391 Optimize SSE parsing and the flush strategy, plus add protections for streaming requests

At first I tried to find a single “root cause” that explained everything. I later realized that this way of attributing the problem was itself the mistake. These PRs actually address four distinct classes of problems:

  1. Cancellation propagation: the downstream is already gone — does upstream work keep running?
  2. Scheduling fairness: can one coroutine hold a worker for a long time?
  3. Cost per unit of work: how much parsing, copying, and allocation happens per request, per chunk?
  4. Flush and backpressure: how much syscall and scheduling cost does per-token real-time output demand?

They amplify each other, but they are not the same problem, and no single fix resolves all of them.

The most important conclusion of this article: yields fix fairness; caching and batching reduce CPU cost; cancellation propagation reclaims wasted work; backpressure bounds the speed gap between producers and consumers.

Why static code reading didn’t reveal it

At the time, when reading the code, I mostly checked:

  • whether the branches were correct;
  • whether errors were handled;
  • whether data would be forwarded completely;
  • whether the loops had exit conditions.

From that angle, there’s nothing visibly wrong with this streaming loop:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
while true do
    local chunk, err = body_reader()
    if not chunk then
        break
    end

    local ok, flush_err = send_to_downstream(chunk)
    if not ok then
        close_upstream(flush_err)
        return
    end

    ngx.sleep(0)
end

But a runtime review has to keep asking:

  • Will body_reader() wait this round, or return immediately?
  • Will send_to_downstream() wait, or just put data into a buffer?
  • How many temporary strings and Lua objects does each loop iteration create?
  • How many iterations can run per second?
  • After the downstream disconnects, who is responsible for terminating the upstream?
  • If the upstream keeps producing at high speed, is there any time, byte, or rate budget?

So the real blind spot wasn’t “failing to understand the while loop” — it was relying on several unverified runtime assumptions:

What the code looks like Relation What’s not actually guaranteed
Called a network API ≠ An I/O wait definitely happened this round
Called flush ≠ The client application has received the data
The code contains a yield ≠ Total CPU consumption necessarily goes down
The client has disconnected ≠ Upstream work stops automatically
One pass of the loop is cheap ≠ Total cost stays low when it runs at high frequency

Problem 1: why the upstream keeps running after the downstream cancels

3.1 The upstream and downstream connections are two independent chains

An AI gateway sits between the client and the LLM:

1
client  <-- downstream -->  API7/APISIX  <-- upstream -->  LLM

Reading responses from the upstream doesn’t mean the gateway also knows the current state of the downstream connection.

If the client closes the browser, cancels the request, or sends a TCP RST, the gateway may keep:

  1. Reading data from the LLM;
  2. Parsing SSE;
  3. Running response filters;
  4. Attempting to write to the downstream.

Only when the downstream close event is observed by Nginx/OpenResty and propagated into the current Lua request processing logic does the code get a chance to stop the upstream work.

3.2 What ngx.flush(true) actually guarantees

OpenResty’s definition of ngx.flush(wait?) is:

  • ngx.flush(false): starts an asynchronous flush; doesn’t wait for the output to be written into the system send buffer;
  • ngx.flush(true): waits until the output is written into the system send buffer, or until send_timeout/an error occurs.

A common misconception must be avoided here:

ngx.flush(true) waits for data to enter the system send buffer. That is not the same as the client application having received or consumed the data.

The value of a synchronous flush is that it advances downstream output in a way the caller can wait on. When the write path has already observed a connection error, the call can return a failure, and APISIX then closes the upstream response, avoiding continued processing of data that nobody is receiving anymore.

This is the core problem PR #13254 addresses.

The more accurate timeline is:

1
2
3
4
5
6
7
8
9
T0  client cancels the request
    ↓
T1  the kernel/Nginx observes the connection close at some point
    ↓
T2  the gateway advances downstream output or receives the connection close event
    ↓
T3  flush/write returns an error
    ↓
T4  APISIX terminates the loop and closes the upstream connection

This is neither “the Lua code knows instantly when the client disconnects” nor “every ngx.flush(true) can detect client liveness in real time, 100% of the time”.

OpenResty also provides lua_check_client_abort on and ngx.on_abort for monitoring early downstream closure. That mechanism is off by default and has its own applicability conditions and runtime cost. In practice you have to choose based on the request model:

  • Terminating the upstream through output error propagation;
  • Explicitly listening for client aborts;
  • Or combining both.

3.3 The nature of the problem

This is first and foremost a cancellation propagation and resource lifecycle management problem, not a pure scheduling problem:

1
2
3
4
5
6
7
the downstream request no longer has value
        ↓
the upstream request isn't canceled in time
        ↓
reading, parsing, filtering, and flushing continue
        ↓
wasted CPU, network, and connection consumption

Code review should treat a bidirectional proxy as two resources with different lifecycles, not as a single request whose two directions naturally end together.

Problem 2: cooperative scheduling and ngx.sleep(0)

4.1 A cosocket call doesn’t necessarily yield

An OpenResty worker typically runs as a single-threaded event loop. A Lua coroutine only hands control back to the Nginx scheduler when it reaches an operation that can yield.

The cosocket API is non-blocking, but “called a cosocket” doesn’t mean “this call definitely yields”:

1
2
3
4
5
6
7
8
socket has no data
  → register a read event
  → the current coroutine yields
  → the worker can process other events

socket buffer already has data
  → receive() returns immediately
  → the current coroutine keeps executing

When the upstream LLM returns many small chunks in bursts, several consecutive body_reader() calls may all complete immediately. If the loop has no other explicit scheduling point, a single request can hold the worker for a relatively long time.

4.2 ngx.sleep(0) is a real scheduling yield

PR #13255 adds the following after each streaming pass:

1
ngx.sleep(0)

It hands control back to the Nginx scheduler through a zero-delay timer. So it’s not “a yield in form but not in substance” — it’s a real scheduling point.

What it does solve:

  • Preventing one request from running continuously without bound;
  • Giving health checks, timers, and other requests a chance to run;
  • Improving tail latency and scheduling fairness within the same worker.

What it doesn’t solve:

  • Reducing the number of JSON/SSE parses;
  • Reducing string allocations per chunk;
  • Limiting how many chunks per second a single stream processes;
  • Limiting total upstream response bytes;
  • Applying backpressure to a fast upstream;
  • Guaranteeing that worker CPU goes down.

So Issue #13256 and the follow-up code reasonably call it a workaround: it prevents a single request from monopolizing a worker, but doesn’t limit how much CPU a single request consumes.

4.3 Why CPU can still be at 100% after adding yields

Suppose a stream produces many small chunks per second, each needing parsing, transformation, and output:

1
2
3
4
5
6
7
process chunk A
  → ngx.sleep(0), yield scheduling
  → process other ready tasks
  → schedule the current request again
process chunk B
  → ngx.sleep(0)
  → ...

Scheduling is fair, but as long as there are enough ready tasks, or the current stream keeps having work, the worker can remain in a runnable state continuously, and CPU utilization can naturally approach 100%.

At this point two metrics need to be distinguished:

Metric Question it answers
Scheduling fairness Do other requests get a chance to run? Is tail latency out of control?
CPU utilization How much work do all requests together need to execute?

ngx.sleep(0) mainly improves the former; it doesn’t directly reduce the latter.

If there are no other ready coroutines at the time of the yield, the current request running again right after is normal; nothing is being “starved” at that moment. The explicit yield provides a scheduling opportunity exactly when other tasks become ready.

Problem 3: repeated JSON decode amplifies per-request cost

5.1 Repeated parsing of post_arg.*

When the expression system accesses post_arg.* multiple times, if each access re-reads and re-parses the request body, the same work runs repeatedly.

PR #13356 caches the parse results so the same request isn’t decoded multiple times during expression matching.

The characteristics of this problem:

  • A single call looks acceptably cheap;
  • The cost scales linearly as the number of rules grows;
  • The larger the request body, the more expensive the repeated parsing;
  • It happens in the request-handling phase — a different problem from the streaming response loop.

5.2 Three decodes on the AI request path

PR #13377 shows that the APISIX AI request path used to decode the same JSON request body three times. Caching the parsed Lua object eliminates the repeated parsing.

“Wasted work” needs to be described accurately here:

  • Under the current design, at least one JSON decode is usually necessary;
  • What can be eliminated are the subsequent repeated decodes;
  • A JSON encode may be necessary when sending the modified request upstream; without evidence, it shouldn’t be lumped in with repeated decodes as wasted overhead.

So the more rigorous conclusion is:

Repeated JSON decode is a CPU amplifier already verified by source code and PRs. Whether it’s the sole root cause of some production CPU saturation still needs to be confirmed with production profiles, request body sizes, and before/after comparison data.

5.3 Why code review easily misses it

Developers usually judge complexity inside each function, but rarely notice the same piece of data being converted repeatedly across layers:

1
2
3
4
5
6
raw request body
  → decoded by the expression system
  → decoded by the AI provider
  → decoded again by a plugin
  → the object is modified
  → encoded and sent upstream

Each step can locally look “reasonable,” but combined they become duplicated work.

A runtime review needs to trace the data lifecycle:

  • Where are the raw bytes read?
  • Where does the first structured parse happen?
  • Can the parse result be placed in the request context for reuse?
  • Which layers only read, and which layers actually modify?
  • Does the encode happen only once, at the final boundary?

Problem 4: the per-chunk cost of streaming responses

6.1 Don’t assume body_reader() must block

lua-resty-http‘s chunked body reader maintains HTTP chunk framing state and calls the cosocket receive() to read chunk sizes and content.

It’s not a simple self.buffer Lua string cache, so without pinning the exact dependency version and source, you can’t assume “the first read goes into self.buffer, and later reads never touch the cosocket”.

But the core risk still holds:

Even if the code calls sock:receive(), as long as the cosocket/Nginx receive buffer already has enough data, the call can still return immediately, without any I/O wait.

This is also why, when evaluating a loop, you can’t just count “how many network API calls there are” — you have to judge whether those calls actually wait under the target load.

6.2 Why many small chunks are expensive

When the upstream returns the same total number of bytes, more small chunks means the fixed cost runs more times:

1
2
3
4
5
6
read chunk
  → concatenate or scan SSE boundaries
  → JSON parse/transform
  → run response filters
  → write downstream
  → flush

For example, for the same 1 MiB of data:

  • 1,024 chunks of 1 KiB;
  • 16,384 chunks of 64 B;

The latter runs more loops, function calls, boundary scans, temporary object allocations, and flushes. Even if each pass is fast, the total cost can rise significantly.

6.3 The flush trade-off between real-time output and throughput

The upside of synchronous flush per chunk is low first-token and inter-token latency, and downstream write errors surface sooner; the downside is that the number of flushes scales directly with the number of chunks.

Periodic flushing combines multiple chunks within a short window into one output:

1
2
per-chunk flush: chunk → flush → chunk → flush → chunk → flush
periodic flush:  chunk → chunk → chunk → flush every 10ms

This reduces flush and scheduling frequency, at the cost of a controllable extra latency window.

PR #13391 doesn’t simply remove flushes; it separates the mechanisms:

  • Optimizes SSE framing to reduce string scanning and allocations;
  • Adds streaming_flush_interval_ms, batching flushes on a 10ms period by default;
  • With a positive interval, a background light thread performs periodic asynchronous flushes;
  • When configured as 0, the original synchronous flush per chunk is preserved;
  • Performs a final synchronous flush at the end of the stream;
  • Propagates asynchronous flush errors;
  • Adds maximum stream duration and maximum response bytes protections;
  • Keeps ngx.sleep(0) as a fairness protection.

This design illustrates exactly:

1
2
3
4
ngx.sleep(0)        → scheduling fairness
SSE parser tuning   → lower per-chunk CPU cost
periodic flush      → lower high-frequency output cost
duration/byte limit → upper bound on resource use for abnormal streams

The four solve different problems.

6.4 Periodic flushing still isn’t full backpressure

The goal of backpressure is: when the downstream can’t consume as fast as the upstream produces, slow the producer down or bound the growth of intermediate buffers.

Periodic flushing mainly reduces output frequency; it doesn’t necessarily slow the upstream. Full streaming resource governance also needs to consider:

  • Maximum stream duration;
  • Maximum response bytes;
  • Maximum unsent buffer amount;
  • Upstream read pacing;
  • Downstream write speed;
  • The cancellation and connection-close policy once a limit is exceeded.

So “adding a flush interval” shouldn’t be equated with “backpressure has been implemented”.

Reassembling the incident’s causal chain

Once split apart, the problems fit into a more accurate resource model:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
repeated JSON decode of the request body
  → increases fixed per-request CPU cost

upstream produces many small chunks at high speed
  → amplifies SSE parsing, string allocations, and flush count
  → increases streaming-phase CPU cost

streaming loop lacks explicit scheduling points
  → a single request can hold the worker continuously
  → tail latency of other requests and timers degrades

downstream cancellation isn't propagated in time
  → upstream streams that have lost business value keep running
  → all of the above CPU and network cost keeps accruing

These four chains amplify each other. For example:

1
2
3
4
5
downstream already canceled
  + upstream still producing at high speed
  + high per-chunk processing cost
  + no resource upper bounds
  = wasted work keeps occupying the worker

But we still need to avoid letting one line like “no yield caused the CPU saturation” cover the whole story.

A more accurate incident classification:

Dimension Symptom Primary fix
Lifecycle upstream keeps running after downstream cancels abort propagation, closing the upstream
Fairness a single request holds the worker for a long time explicit yields
Efficiency repeated decodes, SSE allocations, high-frequency flushes caching, parser tuning, batched flushes
Resource bounds abnormally long streams or large responses consume resources without limit duration/bytes/buffer limits
Backpressure upstream persistently faster than downstream rate limiting, pausing reads, or bounded-buffer policy

Runtime code review checklist

Loops and scheduling

  • How much CPU work does one loop pass do in the worst case?
  • Could the loop’s I/O return immediately because the buffer already has data?
  • Is there an explicit, verifiable yield point?
  • Is the yield for fairness, or is it mistakenly being used as rate limiting?
  • Could a single request continuously process a large amount of ready data?
  • Are worker event loop latency and request tail latency monitored?

Cancellation and resource lifecycle

  • After the downstream disconnects, how is the upstream told to stop?
  • Are upstream connections, light threads, and timers all cleaned up?
  • Are flush/write errors fully propagated?
  • Do we need lua_check_client_abort / ngx.on_abort?
  • After the client cancels, how much more upstream data can still be read?
  • Is there a metric for cancellation propagation latency?

Buffers and backpressure

  • What buffer layers exist on the upstream and downstream sides?
  • When upstream is faster than downstream, where does data accumulate?
  • Do the buffers have upper bounds?
  • Can upstream reading be paused or slowed down?
  • Is the flush strategy per chunk, per byte, or per time window?
  • Is the real-time target an explicit number, rather than a default of “the faster the better”?

Encode/decode and memory allocation

  • How many times is the same request body decoded?
  • Can parse results be reused in the request context?
  • Is .. concatenation in loops creating large numbers of temporary strings?
  • Does SSE boundary scanning repeatedly traverse the same data?
  • Does the encode happen only once, at the final boundary?
  • Have we benchmarked by request body size and chunk count?

Resource bounds

  • Is there a maximum duration per stream?
  • Is there a maximum response bytes limit?
  • Is there a maximum event count or unsent buffer limit?
  • When a limit is exceeded, is the upstream canceled and the reason logged?
  • Can timeouts, cancellations, and upstream errors be distinguished in metrics?

Evidence and observability

  • Are the exact versions and commits recorded?
  • Are there per-worker CPU and latency metrics?
  • Have we captured flame graphs / CPU profiles?
  • Are request body size, response bytes, chunk count, and stream duration recorded?
  • Is there a same-load before/after comparison experiment for the fix?
  • Does every “root cause” in the write-up have corresponding evidence?

Takeaways

What this incident forced me to build wasn’t just knowledge of some OpenResty API — it was a method of runtime analysis.

Static correctness is only a starting point

Code that exits correctly and forwards data correctly isn’t necessarily still efficient under high-frequency, bursty, cancellation, or slow-client scenarios.

Don’t blend four classes of problems into one cause

  • Cancellation propagation determines when wasted work stops;
  • Yields determine whether scheduling is fair;
  • Caching and batching determine the per-unit-of-work cost;
  • Backpressure and resource bounds determine how much resource an abnormal stream can consume.

ngx.sleep(0) works, but its limits must be stated clearly

It genuinely hands scheduling control back and can mitigate a single request monopolizing a worker; it doesn’t reduce total parsing volume, doesn’t rate-limit, and isn’t backpressure.

Network APIs don’t necessarily wait

The runtime behavior of cosocket receive(), flush, and similar APIs depends on the buffer, socket readiness, and call arguments. Seeing an I/O call isn’t evidence that a yield has happened.

AI expands the search space, evidence converges on the conclusion

AI can generate hypotheses, explain mechanisms, and design experiments quickly, but each conclusion still has to be checked against the exact version’s source code, the official documentation, profiles, and controlled experiments. A plausible-sounding runtime story is not necessarily the real call chain.

The skill worth cultivating isn’t reciting “where an ngx.sleep(0) should be added”, but proactively asking of any event-driven system: when is control yielded back? Where is data buffered? How many times does duplicated work happen? How does cancellation propagate? Where are the resource bounds?

Disclosure: AI assistance in this postmortem

A note on method: AI assistance was used in this investigation, and the honest boundary of that help matters.

I first brought the problem to Claude Opus, hoping the AI could help analyze:

  • Why reading the code didn’t reveal the problem;
  • What OpenResty and cooperative-scheduling knowledge I was missing;
  • How to turn the findings into a reusable code review checklist.

The value of AI was first and foremost rapidly expanding the hypothesis space:

1
2
3
4
5
6
7
8
CPU saturation
  ├─ repeated JSON encode/decode?
  ├─ streaming loop without yields?
  ├─ cosocket repeatedly returning immediately?
  ├─ high-frequency flush?
  ├─ SSE parser over-allocating?
  ├─ downstream cancellation not propagated?
  └─ no stream duration or byte limits?

But AI explanations can’t be taken directly as incident conclusions. They can:

  • Hallucinate a self.buffer that doesn’t exist in the current version, based on common library implementations;
  • Exaggerate “the call might return immediately” into “it absolutely never yields”;
  • Misread the limitations of ngx.sleep(0) as “it doesn’t really yield”;
  • Treat an optimization PR’s benchmark as proof of a production incident’s root cause;
  • Conflate the system send buffer with what the client actually received.

So the more reliable AI-assisted postmortem workflow I settled on is:

Step 1: Have the AI generate candidate hypotheses

Ask it to be exhaustive: scheduling, I/O, buffering, serialization, lifecycle, timeouts, rate limiting, and observability.

Step 2: Pin the exact versions

Record and provide:

  • The API7/APISIX commit;
  • The OpenResty version;
  • The lua-resty-http version;
  • Configuration files;
  • The actual call chain.

Source analysis without a pinned version easily ends up describing a different implementation.

Step 3: Establish evidence levels

Level Example What it can show
Symptom worker CPU at 100%, rising latency a problem definitely occurred
Source code the same request body decoded three times duplicated work exists
Profile JSON decode occupying the main CPU share the duplicated work relates to production CPU
Controlled experiment same traffic, CPU clearly drops after caching establishes a strong causal relationship
Production verification metrics recover after release and the problem doesn’t recur the fix was effective for the incident

PRs, source code, and benchmarks can prove a problem exists, but to call it “the root cause” of some production incident, you still want profiles and controlled experiments.

Step 4: Proactively verify the AI’s most confident conclusions

Prioritize checking answers containing these words:

  • “definitely”;
  • “absolutely”;
  • “necessarily”;
  • “equivalent to nothing”;
  • “the sole root cause”.

The more confident the phrasing, the more it’s worth going back to the official documentation and the source code of the exact version to verify.

Step 5: Let experiments, not narrative, do the attribution

Recommended: use fixed traffic replay, controlling one variable at a time:

Experiment Metric observed Goal
keep/remove ngx.sleep(0) other requests’ P99, worker event loop latency verify fairness
cache/repeat JSON decode CPU profile, per-request CPU time verify parsing cost
per-chunk/periodic flush CPU, first-token latency, inter-token latency quantify the real-time vs. throughput trade-off
large/small chunks, same total bytes loop iterations, allocations, CPU verify fixed per-chunk cost
client cancels midway upstream close latency, bytes after cancellation verify cancellation propagation
fast upstream, slow downstream buffer amount, memory, read rate verify backpressure and resource bounds

AI is best suited to help design these experiments, interpret results, and spot omissions — not to replace them.

References

This article is the English edition of a post originally published in Chinese.