How to Judge an AI Gateway: Protocol Compatibility, Traffic Governance, and a Performance Scorecard

Posted on Edited on Views: Word count in article: 23k Reading time ≈ 21 mins.
A practical framework for judging AI gateways: L7 fundamentals, AI traffic handling, provable runtime performance, production maturity, five veto lines, and a weighted scorecard.

This article doesn’t rank specific products. It answers a single question: how do you tell whether an AI gateway is any good?

My conclusion: a good AI gateway is not the one with the longest feature list. It’s the one that keeps running stably inside your production boundaries for the long haul. At minimum, it has to do four things at once:

  1. Solid L7 gateway fundamentals: proxying, routing, load balancing, timeouts, TLS, rate limiting, authentication, and observability — none of them obviously weak;
  2. Real understanding of AI traffic: model routing, body rewriting, token governance, SSE streaming, protocol conversion, and agent / tool call handling;
  3. Provable performance and runtime behavior: with AI plugins enabled, CPU, memory, GC, p99, streaming backpressure, and cancellation handling all have to hold up under load testing;
  4. Enough production maturity: core capabilities that are usable, operable, easy to roll back, and observable, with enough production references behind them.

There is no universally best AI gateway — only the one that fits your team’s current boundaries. When your model platform, logging, alerting, and security sidecar are all mature, the gateway should be “thin and fast.” When those capabilities are still incomplete, the gateway can be “thick and full-featured” — but you have to accept a heavier data plane and prove with load tests that the complexity is controllable.

This article is meant for AI gateway selection, for evaluating an existing gateway, or for breaking down the capabilities of a homegrown gateway. It is not an installation tutorial for any particular product, and it doesn’t cover personal relay proxies.

Overview: The Conclusion First

The one-line model

1
good AI gateway = solid L7 gateway + AI traffic capabilities + provable performance + production maturity

All four are required:

Layer Keywords What to judge
L7 foundation proxying, routing, LB, timeouts, TLS, rate limiting, auth, observability Whether the basic gateway fundamentals are solid
AI capabilities body routing / rewriting, token governance, SSE, protocol conversion, agents Whether it actually understands LLM / agent traffic
Performance & runtime safety JSON codec, SSE flush, backpressure, yield, GC, p99 Whether the gateway itself becomes a bottleneck once AI features are turned on
Production maturity GA, production references, usable core capabilities, operability Whether it can run long-term on critical production paths

Responsibility boundaries come before feature comparison

Before evaluating anything, answer one question: which capabilities must live in the gateway, and which belong on the model side or in sidecar systems?

Model-side / sidecar maturity Suitable gateway shape The gateway’s core responsibilities
Thick: mature model platform, logging, alerting, and security sidecar “Thin” gateway Routing, auth, basic rate limiting, lightweight observability, low overhead
Thin: bare vLLM or mostly third-party APIs “Thick” gateway Availability fallback, log enrichment, content moderation, protocol conversion, cost governance

The core trade-off: you can’t simultaneously maximize stability, flexibility, and performance. The thicker the gateway, the more capable it is — but also the heavier the data plane and the larger the failure surface. Capabilities that can live on the model side or in a sidecar shouldn’t be crammed into the gateway just because the gateway can do them.

A three-step evaluation process

Step What you do Purpose
① Draw boundaries Define what the gateway, model platform, logging system, and security system each own Avoid piling all the complexity onto the gateway
② Apply the veto lines Screen candidates first with the one-vote veto lines in §5 Weed out options that don’t belong on critical production paths
③ Score Score against your P0 / P1 requirements using §7 Find the best fit, not the longest feature list

1. What Actually Makes AI Traffic Special

Most ordinary APIs are stateless, small, and structurally fixed. AI traffic, by contrast, combines semantic request bodies, responses that may stream for a long time, cost metered per token, backends that are heterogeneous and unreliable, and agents that can trigger side-effecting tool calls.

AI traffic trait New requirement on the gateway
Request / response bodies are semantic Must be able to read and rewrite fields like model, messages, max_tokens, and stream
Responses are often SSE streams Rate limiting, moderation, observability, protocol conversion, and disconnect handling must all work on the stream
Cost is metered per token QPS alone isn’t enough — you need TPM, TTFT / TPOT, token attribution, and budgets
Backends are heterogeneous and prone to rate limiting / failures Multi-provider protocol adaptation, fallback, health checks, and traffic demotion
Agents trigger tool calls Traces, permissions, audit, idempotency, and cost must be correlated across model calls and tool calls

So a gateway that only rewrites URLs and headers while passing the body through as a black box is, in essence, just an ordinary gateway. A real AI gateway must understand the runtime semantics of the body, the stream, tokens, providers, and tool calls.

2. The AI Gateway Capability Map

Once you lay out the capabilities, they fall into roughly ten categories. Note: this is a capability map — it does not mean the gateway must implement every one of them.

Capability area Problem it solves Typical capabilities
Management & operations plane Can it be managed systematically REST API, CRD, GitOps, automatic certificate renewal, hot config reload, hot plugin loading
Core gateway data plane Is it a competent L7 gateway HTTP proxy, unified entry point, connection reuse, timeouts, TLS, path rewriting
Performance & runtime safety Stable under high concurrency, large bodies, and streaming Extra CPU, JSON decode / encode counts, SSE flush, backpressure, yield, fairness, memory and GC
AI routing & load balancing Where should a request go model routing, body-based conditional routing, session affinity, weight + priority, multi-provider mixing
Fallback & high availability Can backend failures be absorbed 429 / 5xx / timeout fallback, health checks, automatic ejection / recovery, alert-driven demotion
Request / response rewriting & protocol adaptation Shielding backend differences Model name mapping, credential injection, body field injection, max_tokens enforcement, OpenAI / Anthropic / Gemini / Bedrock conversion
Traffic governance & cost Governing tokens, concurrency, and money RPM, TPM, composite rate limits, instance-level limits, token accounting, budget quotas, semantic caching
Observability & audit Can you diagnose what went wrong OTel, structured logs, Prometheus, TTFT / TPOT, trace, upstream, risk level
Security & content governance Are requests and content safe Consumer authentication, credential vaulting, request body limits, bidirectional moderation, multimodal moderation, PII handling
Extensibility & ecosystem Can it keep growing Plugin mechanism, execution ordering, external moderation services, AlertManager, Kafka / ES / HTTP sinks

A few points that are easy to get wrong:

  • Performance isn’t just hello-world QPS: you need to look at the extra CPU with AI plugins enabled, memory allocations, GC, JSON encode/decode counts — plus runtime edge cases like small SSE chunks, fast upstreams, slow downstreams, and client disconnects.
  • Streaming capability isn’t just SSE passthrough: the real question is whether rate limiting, moderation, observability, fallback, protocol conversion, and disconnect handling are all still available on a streaming request.
  • Protocol conversion is advanced rewriting: converting non-streaming requests isn’t enough — you also have to handle SSE, error codes, tool calls, usage, and provider-specific fields.
  • Observability doesn’t mean logging bodies without limits: the more complete the body logs, the higher the compliance risk. Redaction, sampling, sensitive-field filtering, and expiration cleanup are mandatory.
  • Multi-region is not automatically high availability: deploying multiple regions isn’t enough — routing, probing, failover, logging, alerting, and recovery have to form a closed loop.

3. Core Capability Areas: Minimum Bar / Excellence Bar

3.1 Routing and load balancing

Minimum bar: routes by model, supports weighted load balancing and basic fallback; plain HTTP requests pass through too, without being forcibly parsed as Chat Completions.

Excellence bar: conditional routing based on body content — requests with images go to a multimodal instance, long prompts go to a large-context model; weight + priority, session affinity, and fallback configured separately for 429, 5xx, and timeouts.

Watch the fallback boundary in streaming scenarios: switching before the first token is relatively safe. Once output has started flowing to the client, switching backends can cause duplicate output, broken context, or double billing. Tool calls also require distinguishing whether they have side effects.

3.2 Performance and runtime safety

Performance is the most underestimated capability of an AI gateway. Plenty of products will show you plain HTTP forwarding QPS, but once you’re really in production, the bottleneck usually comes from the extra work after the AI plugins are turned on: reading the body, parsing JSON, rewriting fields, streaming parsing, content moderation, log enrichment, token accounting, protocol conversion.

Minimum bar: you can produce load test results aimed at AI scenarios, not just plain HTTP hello-world QPS. The test matrix should cover at least:

  • body routing / rewriting;
  • request bodies of different sizes: 32 KB, 128 KB, 1 MB;
  • SSE streaming forwarding;
  • an upstream that returns small chunks quickly;
  • a slow downstream consumer;
  • a client that disconnects mid-request;
  • plugin combinations such as content moderation, logging, and token accounting.

Metrics should at least include worker CPU, RSS, Lua / runtime GC, p95 / p99 latency, TTFT, TPOT, token throughput, and flame graphs.

Excellence bar: performance engineering becomes part of the product. On key hot paths, you know exactly how many JSON decode / encode operations each request performs; reading only a few fields like model and stream goes through a raw get fast path; patching only a few fields like model, temperature, and stream goes through a raw patch fast path; complex protocol conversion, RAG, and deep rewriting fall back to full decode / encode. The SSE loop has explicit backpressure, flush, cancellation, and yield policies, and every version upgrade runs performance regression tests.

Two traps deserve special attention here.

First: “looks like I/O” doesn’t mean the CPU actually gets yielded. In cooperative scheduling models like OpenResty / Nginx, if the upstream socket buffer always has data available, body_reader() can degenerate into something close to a memory copy; and if ngx.flush() never creates real backpressure, the loop can go a long time without yielding — a single SSE request can saturate a worker. Problems like this are nearly impossible to spot by reading static logic; you have to put “fast upstream + slow or disconnecting downstream + small chunks with frequent flush” into your load test matrix. See the earlier postmortem: Runtime behavior blind spots: a postmortem on CPU saturation bugs in the API7 AI gateway.

Second: body JSON processing is not free. AI gateways frequently full-decode the entire messages payload — sometimes even full-encode it again — just to route on post_arg.model, detect the protocol, or override a handful of fields like model/temperature/stream. With large prompts, multi-turn messages, and multimodal bodies, this turns directly into worker CPU, memory allocation, and GC pressure. A better strategy: raw get / raw patch on the hot path, full decode / encode reserved for complex cases, and flame graphs to verify whether cjson.decode and encode are really hotspots.

An actionable performance acceptance bar:

Scenario Minimum acceptance question Suggested metrics
post_arg.model routing When reading a single field, does it still full-decode the whole body Visibly lower worker CPU on 32 KB+ bodies
AI passthrough + few-field overrides Does it avoid decoding + encoding the entire request body Clearly lower decode / encode share on the flame graph
SSE fast small chunks Does the loop go a long time without yielding when buffers are ready A single request must not monopolize a worker; p99 must not drag down other requests on the same worker
Client cancels the request After the downstream disconnects, does it stop reading upstream and parsing JSON ASAP Wasted CPU, upstream cancellation latency, 499 / disconnect logs
Content moderation / logging Is the extra overhead explainable once the plugin is on CPU, RSS, GC, and external call latency reported separately

The core of performance is not “what language the gateway is written in” — it’s: how much extra work the data plane does once you enable the AI capabilities you actually need, whether it can lose scheduling fairness in the worst case, and whether those costs can be proven with load tests and flame graphs.

3.3 Request / response rewriting and protocol adaptation

Minimum bar: path rewriting, header injection, backend credential injection, and model name mapping are all configurable.

Excellence bar: body field injection can distinguish “backfill only when missing” from “forceful override”; field order is preserved where possible when modifying the body; protocols like OpenAI, Anthropic, Gemini, and Bedrock convert into each other, correctly handling tool calls, error codes, usage, and streaming formats.

A simple test: body rewriting should ideally be doable through configuration alone. If every scenario requires writing a plugin, the product’s abstraction isn’t mature enough.

3.4 Traffic governance and cost

Minimum bar: consumer-level RPM, concurrency limits, and SSE passthrough.

Excellence bar: token-based rate limiting (TPM) supporting composite dimensions such as consumer × model, tenant × model, route level, and instance level; cost attribution, budget quotas, and over-limit alerting or circuit breaking.

Semantic caching can save money and reduce latency, but you have to handle cache hits on the wrong data, permission isolation, freshness, and audit. For enterprise use cases, a wrong cache hit is more dangerous than a cache miss — you can’t just turn it on blindly.

3.5 Observability

Minimum bar: OpenTelemetry integration, structured access logs, body logging switchable per tenant / route.

Excellence bar: latency metrics like TTFT and TPOT exposed as histograms with P90 / P99 percentiles; token totals attributable by model, consumer, and project; logs covering key fields like model mapping, token, upstream, trace, and risk level, shippable to Kafka, ES, HTTP, and OTLP.

Full body logging must support redaction, sampling, sensitive-field filtering, and expiration cleanup — otherwise the stronger your observability, the higher your compliance risk.

3.6 Health checking and high availability

Minimum bar: passive health checks with automatic ejection after consecutive failures.

Excellence bar: active health checks that POST real inference requests, not just GET /health. The common failure mode of LLM backends is “the process is alive but inference is unavailable” — shallow probes can’t detect that. Going further: alerting wired to demotion and recovery — when an alert fires, the instance gets demoted; after recovery, traffic flows back automatically.

3.7 Security: authentication, authorization, and content moderation

Minimum bar: consumer authentication, API key management, backend credential vaulting, request body size limits.

Excellence bar: JWT / OAuth, RBAC, multi-tenant isolation; consumer identity carried through to rate limiting and observability; content moderation covering both request and response, effective on streams, configurable per consumer or route, and able to call external services.

3.8 Agent / tool call traffic

If you only serve ordinary Chat Completions, this is a bonus item. If you need to support agents, it’s mandatory.

A single user request to an agent can trigger multiple model calls and tool calls. The gateway must at minimum correlate conversation id, tool call id, and trace id — and enforce permissions, audit, idempotency, and cost attribution across tool calls. A tool call that sends an email, places an order, or changes configuration can become an incident if it gets retried just once.

3.9 Extensibility and operability

Minimum bar: plugin mechanism, hot config reload, and complete management interfaces (REST API, CRD, or GitOps).

Excellence bar: extension point ordering is controllable, plugins and policies hot-loadable, certificates auto-renewed, both declarative and imperative management supported; critical config changes are auditable, canary-deployable, and easy to roll back.

4. Thick Gateway or Thin Gateway: Which One Do You Need

Most of these arguments aren’t really about “which gateway is better” — they start because nobody established the responsibility boundary first.

When a thin gateway fits

If you already have a mature model platform, unified logging, alerting, security review, cost attribution, and scheduling systems, the gateway doesn’t need to carry much AI logic. Priorities then:

  1. A stable data plane with low overhead;
  2. Reliable routing, auth, basic rate limiting, and lightweight observability;
  3. Clean boundaries with existing platforms;
  4. No complex protocol conversion, deep moderation, or heavy log processing on the request hot path.

When a thick gateway fits

If your model side and sidecar systems are still thin — you mostly rely on third-party APIs, run bare vLLM, or don’t yet have a unified LLMOps platform — the gateway can take on more:

  1. Multi-provider protocol adaptation;
  2. Fallback, circuit breaking, demotion, active health checks;
  3. Request / response content moderation;
  4. Token accounting, budgets, cost attribution;
  5. Log enrichment and alerting integration.

But a thick gateway must pay the performance verification cost. The more logic moves into the data plane, the more you have to prove that it won’t become a new source of failures under large bodies, small streaming chunks, client disconnects, content moderation, and high QPS.

5. Five One-Vote Veto Lines

Some weaknesses aren’t points deducted — they disqualify a gateway from critical production paths outright.

  1. Weak L7 foundation: unstable at plain HTTP proxying, routing, LB, timeouts, TLS, rate limiting, auth, and observability, yet claiming to be an AI gateway — treat such candidates with caution.
  2. Unprovable data plane performance or unsafe runtime behavior: mature data planes like Rust / Go / Envoy / Nginx are steadier, but the language and framework are no get-out-of-jail-free card. You must prove controllable overhead with real AI workloads: does body routing / rewriting repeatedly decode and encode JSON, do small SSE chunks busy-wait, does the CPU keep burning after a client disconnects, and are worker CPU, RSS, GC, and p99 still acceptable with content moderation / logging turned on?
  3. Insufficient maturity: pre-1.0, no production references, too small a community, an unclear upgrade path, or no transparent record of fixing serious past issues — all warrant a heavy discount.
  4. Core capabilities unavailable: if body conditional routing, model-level fallback, TPM rate limiting, and semantic moderation are all locked behind the enterprise edition, the open-source edition has limited value; if the enterprise edition is also missing key capabilities, it’s even less suitable for critical paths.
  5. No trustworthy active health check: passive ejection only, or GET /health, will rarely surface “the service is alive but inference is unavailable.” AI backends need real inference probes — at minimum, detecting anomalies by model, route, and instance, and wiring that into demotion / recovery.

6. Good vs. Mediocre: A Quick Comparison

To quickly judge whether a gateway has obvious weaknesses, start with these high-risk points.

Quick check Mediocre approach Good approach
L7 foundation Can only forward AI requests; thin ordinary HTTP capability Complete L7 gateway capability; AI features enhance rather than replace it
body capability Treats the body as a black box; only changes headers / paths Routes, rewrites, and injects fields based on the body, preserving order where possible
performance evidence Only shows plain HTTP QPS or touts language / framework Provides CPU, RSS, GC, flame graphs, and p99 data with AI plugins enabled
JSON hot path Repeatedly full-decodes / full-encodes to read / modify a few fields raw get / patch on hot fields, falling back for complex cases
streaming path Only the non-streaming path is complete Rate limiting, moderation, observability, protocol conversion, and disconnect handling all run on the stream
streaming runtime Behavior unclear when the upstream is fast and the downstream is slow or disconnected Explicit backpressure, flush, yield, fairness, and cancellation policies
fallback Retries only by status code Distinguishes failure types, before/after the first token, instance rate limits, and tool call side effects
metrics and logs Only average latency and request counts TTFT / TPOT by percentile; token, model, consumer, upstream, and trace all attributable
health checks Passive ejection or GET /health Active POST real inference probes, with automatic ejection / recovery
Agent / tool call Forwards them as ordinary HTTP requests Trace, permissions, audit, idempotency, and cost all correlated
production maturity pre-1.0, performance unprovable GA, production references, key capabilities usable, overhead verifiable under load tests

7. The Scorecard (Weights Total 100%)

The weights below are generic — adjust them for your team’s P0 / P1 priorities. If you already have a mature security sidecar, lower the weight on “content governance”; if you rely heavily on SSE and long multi-turn contexts, raise the weight on “performance and runtime safety.”

A. Core gateway capabilities (10%)

  • Plain HTTP proxying / routing works too, not just binding to AI providers
  • Path rewriting, header injection, backend credential injection
  • Connection reuse, route-level timeouts, controllable TLS verification
  • Rate limiting, authentication, and basic observability are complete

B. Performance and runtime safety (15%)

  • AI-scenario load tests exist, not just plain HTTP QPS
  • You know how many JSON decode / encode operations each request performs on hot paths, provable with flame graphs
  • A fast path — or an explicit optimization plan — exists for reading / modifying only a few fields
  • No busy-waiting or worker monopolization under small SSE chunks, client disconnects, or slow downstream consumption
  • Observability covers worker CPU, RSS, GC, p95 / p99, TTFT / TPOT, and token throughput

C. Routing and load balancing (12%)

  • Routing on the model field, weight + priority load balancing
  • Conditional routing based on body content
  • Session affinity, instance-level rate limiting, and multi-provider mixing
  • Fallback distinguishes failure types, streaming phases, and tool call side effects

D. Request / response rewriting and protocol adaptation (12%)

  • Field injection clearly distinguishes backfill from override
  • Field order preserved where possible, avoiding pointless reordering that breaks signatures / caches / audit
  • Model name mapping and multi-provider protocol adaptation are usable
  • Error codes, streaming formats, tool call fields, and usage convert correctly

E. Traffic governance and cost (13%)

  • TPM rate limiting supporting arbitrary dimension combinations
  • Cost attribution and budget quotas, with alerting / circuit breaking on overage
  • SSE supported end to end, not only non-streaming requests
  • Semantic cache has permission isolation, freshness, and audit boundaries

F. Observability (13%)

  • TTFT / TPOT as histograms, viewable at P90 / P99
  • Tokens attributable by model, consumer, project, upstream instance
  • Structured log fields are complete enough and can feed multiple sinks
  • Body logs support redaction, sampling, per-tenant / route switches, and expiration cleanup

G. Health checking and high availability (10%)

  • Both passive outlier detection and active health checks
  • Active probes can POST real inference requests
  • Ejection and recovery by model, route, and instance
  • Alerting integration, with instances demoted and restored automatically

H. Security and compliance (10%)

  • API Key / JWT / OAuth / RBAC, backend credential vaulting
  • Bidirectional content moderation for requests and responses, streams included
  • PII, prompt injection, and multimodal moderation can integrate with external services
  • Multi-tenant isolation, audit, and controllable key lifecycle

I. Extensibility and operability (5%)

  • Plugin mechanism, extension point ordering, and hot policy reload are controllable
  • At least one of REST API / CRD / GitOps is pleasant to use
  • Certificates, config, and secret lifecycles can be automated
  • Critical changes are auditable, canary-deployable, and easy to roll back

Veto items: one-vote vetoes — not scored, but mandatory

  • Solid L7 foundation
  • Provable data plane performance, with safe streaming runtime behavior
  • Sufficient maturity: GA, production references, active community or reliable commercial support
  • Core capabilities usable — or the enterprise edition cost explicitly accepted
  • Trustworthy active health checks that can prove a model instance can actually run inference

8. Final Thoughts

Don’t start judging an AI gateway by comparing feature-list length. The right order is:

  1. Draw boundaries first: which capabilities go in the gateway, which go to the model platform, logging system, and security system;
  2. Then pass the veto lines: L7 foundation, performance, maturity, core capabilities, active health checking;
  3. Finally, score: adjust weights against your own P0 / P1 requirements.

If the model side, logging, alerting, and security sidecar are all strong, the gateway should be thinner, with stability and performance first. If the model side is thin, the gateway has to take on more protocol adaptation, availability fallback, content governance, and cost control — while accepting a more complex data plane.

The thicker the AI gateway, the more you must treat performance as a first-class capability: not only forwarding requests, but proving you won’t become the new bottleneck under large bodies, small SSE chunks, client disconnects, content moderation, and high QPS.

In one line: a good AI gateway is neither the thickest nor the flashiest — it’s the one that closes the loop on features, performance, and production stability within a clear set of responsibility boundaries.

For how the model platform on the other side of the boundary manages models — the life of a model from selection and access approval, through tokens, invocation, and billing, to decommissioning — see From Model Management to an AI Gateway: Tying Assets, Traffic, and Cost into a Closed Loop.

2026-07-11 | Based on: verification against mainstream AI gateway source code and documentation, plus hands-on production experience

This article is the English edition of a post originally published in Chinese.