From Model Management to an AI Gateway: Tying Assets, Traffic and Cost into a Closed Loop

Posted on Edited on Views: Word count in article: 17k Reading time ≈ 16 mins.
Model management decides who can use what and how much; the AI gateway enforces it on every request. A tour of the three planes, the model lifecycle, and the request pipeline.

Model management decides who can use what and how much; the AI gateway enforces those decisions on every request; and the outcome of each call flows back to become the basis for cost allocation, scale-out, and retirement. This article follows two lines — the life of a model, and the life of a request — to see how the loop turns.

1. Two kinds of incidents

Anyone who has built a model platform has probably seen both of these incidents:

  • A model has been retired, yet traffic is still hitting it.
  • An approval has gone through, yet the call simply does not work.

Both stem from the same root: the system that manages assets and the system that manages traffic are two separate systems, each holding half the picture. Approvals, permissions and quotas stay inside the business system and cannot constrain real requests; the gateway can forward traffic, but it does not know why a model was listed, who is allowed to use it, or whose cost the call books to.

The core object of traditional API management is the interface; the core object of AI platform management is the model. Both appear to handle HTTP requests, but what they care about differs:

What interface management cares about What model management additionally cares about
Address, protocol, authentication, routing Model listing, selection, permissions, quotas, unit price, retirement
Whether the request succeeded How many input and output tokens were consumed
Whether the upstream is healthy Whether instances are healthy, whether quota remains, whether the provider needs to be switched
Who the caller is Which application, user and cost center the call belongs to

The real relationship between the two is a loop:

1
2
3
Model listing → selection and application → authorization and quota generation
        → gateway authentication, rate limiting, routing, rewriting
        → usage and status flow back → cost allocation, scale-out, migration and retirement

The loop runs in two directions: decisions flow down — models, permissions, quotas and routing rules enter the gateway through the configuration plane; results flow up — token usage, callers and call outcomes return from the gateway to the business side.

Three planes keep the loop turning: the business plane makes decisions and owns model assets and usage rules; the configuration plane translates and delivers those decisions, turning business rules into router, instance and plugin configuration the gateway can execute; the data plane executes decisions and reports results back — built on a mature API gateway, it is responsible only for authentication, rate limiting, routing, rewriting, forwarding and recording along the request path. Each plane keeps to its boundary: the business plane does not touch inference traffic, the data plane does not hold asset rules, and the configuration plane only translates and delivers.

Figure 2 · The three planes

The hard part is right here: the same model, token and caller must point to one and the same thing in both the asset system and the traffic system. Next, two lines show how this loop turns — first the life of a model, then the life of a request.

2. The life of a model

One running example throughout: the customer support team wants to connect a new inference model to their low-code agent platform. From arrival to exit, the model passes through six gates:

Figure 3 · Model lifecycle

2.1 Selection: verify first, then add to the asset catalog

They first tried several candidates side by side in the model marketplace: tuning Temperature, Top_p, Max_tokens, and also re-running with different models at identical parameters to compare the results.

Selection is not just picking a model name; it also means judging whether reasoning capability is needed, how high the quality bar is, and whether there are security risks. Security risk is especially decisive — it directly determines whether to go with a private deployment or a cloud service, and the later application and routing approach follow from it.

2.2 Application: establish ownership first, then open production access

The application form must state the usage mode (low-code agent platform, microservice calls, or personal office use) and the environment (test or production).

Production access must be bound to an explicit scenario. Without a scenario, calls cannot be attributed to an application, and auditing, cost allocation and problem diagnosis become impossible. So what this step really establishes is the three-way relationship between model, caller, and cost ownership.

2.3 Tokens: the carrier of permission, not an isolated key

One token can carry permissions for multiple models — you do not need a separate key per model. The deletion boundary for tokens is deliberately tight: delete the last model permission and the token is invalidated with it; if an agent still references the token, deletion is refused. A token that is “still valid but whose user is unknown” must never exist in the system.

Once the application is approved, the rules continue down the chain. The enterprise/application path issues a token, creates a Consumer (the gateway-side object representing the caller, holding its credentials and plugin configuration), writes model permissions and quota, and adds the route to a whitelist; the personal path only issues a token, with no Consumer — authentication and quota checks happen in real time on the business side. Until this step is complete, “authorized” in the business system cannot become “may call” in the request path.

2.4 Invocation: asset relationships carried into the request

Once real calls begin, the SDK documentation is only the surface; underneath it, model management rules are expressed on the request side:

  • Exceeding either RPM or TPM causes a rejection, protecting request frequency, model throughput and the cost budget;
  • Keeping X-Session-ID constant within a conversation lets consistent hashing land requests on the same instance as much as possible, reusing the KV cache;
  • Passing user information via baggage: sentry-user (baggage is the standard request header OpenTelemetry uses to propagate key-value context along a trace) lets the call be correlated with the log field caller_user, enabling end-to-end attribution.

These request headers are not business decoration; they are the carriers through which session, user and cost ownership enter the data plane.

2.5 Billing: usage must flow back to the asset relationship

A single user interaction can trigger several model calls; counting by interactions or requests alone cannot explain real consumption, and capacity and cost have to be measured in tokens.

Token-based measurement supports three things: load tests give the average token consumption per interaction; cost allocation aggregates by token to the application or sub-product; ROI estimates daily cost as DAU × interactions per user per day × average tokens per interaction × unit price.

The gateway is the feedback entry point here: it parses upstream usage, records input and output tokens, model instance, caller and trace information, and hands them to quota settlement, detail queries and cost allocation.

2.6 Retirement: remove references first, then delete the asset

A model cannot be retired by deleting it from a list. Published routes, issued tokens, agent references and in-flight migrations must all be handled in sync. The system uses a migration protection list to block new routes being created against the old model, and route creation also validates service template compatibility. The correct order is: switch the calling relationships first, then reclaim tokens and routes, and only then remove the model asset. In reverse, you get the two incidents from the opening.

The six stages do not really end here. The two dashed lines in Figure 3 are the feedback flow: the slot freed by retirement is filled by a replacement model, entering a new round of selection; usage and status continuously correct quotas, scale-out plans and provider trade-offs.

3. The life of a request

The life of a model establishes and changes asset relationships; the life of a request is those relationships taking effect along a millisecond-to-minute path.

Figure 4 · Request execution pipeline

Once the low-code agent platform is officially connected, every request on the enterprise path enters the gateway, and the gateway must answer five questions in order:

  1. Who are you: verify the enterprise credentials.
  2. May you call this model: check model permissions and the routing whitelist.
  3. Where does this request go: filter out unhealthy or quota-exhausted candidate instances, then complete routing.
  4. May this request be admitted: run the token pre-check and RPM rate limit against the selected instance.
  5. How much was actually used: parse the response usage, write logs, and report back to the business side.

Identity and permission come first, instances must be selected before forwarding, and actual usage can only be settled after the response. Implementations may vary; this order cannot.

Behind all five questions are the decisions from Section 2: who you are comes from the ownership established at application time; whether you may call comes from token permissions; where it goes comes from the instance list determined by selection and retirement; whether it is admitted comes from the quota verdict; how much was used returns to the billing basis. The two lines mesh right here.

3.1 Two kinds of callers, two paths

The low-code agent platform takes the enterprise path; personal office use takes the personal path. The two kinds of callers have different data sources and do not fit into one authentication model:

Figure 5 · Different scenario domains map to two invocation paths

  • Enterprise/application path: the token is bound to a fixed caller, and the gateway completes identity, model permission and quota checks locally.
  • Personal path: no fixed consumer object is created; the business side validates token and quota in real time, and actual usage is reclaimed after the request ends.

The two kinds of calls use different scenario domains. By the time a request reaches the gateway, the domain has already determined whether it hits the enterprise route or the personal route — it is not a choice made by route labels after entry.

3.2 Usage is parsed exactly once

The gateway parses upstream response usage exactly once, and hands the same result to token quota settlement, personal usage reporting, logs and metrics, avoiding the inconsistent accounting that arises when each of them computes it separately.

This step turns “the request succeeded” into operational information: not only whether the call completed, but which instance served it, how many tokens it burned, and whose account it books to.

Request and response bodies may contain prompts, business data, and even personal information; “can be recorded” does not mean “retained in full by default”. In production, sampling decisions must also be made per scenario, together with the corresponding masking, access control and retention periods.

3.3 A real request

Figure 8 · A complete request: from entry to settlement

In the figure, instance C has already been removed from the candidates after failing an active health check. This request first reached instance A, which returned 429 — A is excluded only for this request and is not written into health-check state. Fallback is enabled, the time since the last failure is still within the retry window, and retries remain, so the router then selected instance B. The low-code agent platform notices nothing: no retry of its own, no need to change address or credentials.

If instance A keeps failing and triggers an alert, alert integration sets its priority to 0, subsequent traffic no longer selects it, and the priority is restored once the alert clears. Request-level exclusion, health-check removal, and alert-based demotion — three mechanisms that do not interfere with one another. How they are wired together is the topic of the next section.

4. Three scenarios where the loop turns

Quota, availability, configuration — every scenario is the same loop: decisions down, results up.

4.1 Quota: computed on the business side, executed on the data plane

The business side holds model permissions, plans and cost rules, so it is the right place to answer “how much should be granted”; the data plane sees every real request, so it is the right place to answer “how much has been used”. The two are not interchangeable.

The enterprise/application path connects them with a four-step chain:

Figure 6 · The four-step chain of quota generation and execution

The core boundary in one sentence: on the enterprise path, the gateway does not hold quota rules — only the computed verdict. The personal path goes further: authentication and quota checks are all done in real time on the business side.

Token rate limiting has two stages: check remaining headroom before forwarding, then book against real usage after the response. That way a call can be rejected before it happens and settled accurately afterward. The cost is that concurrent in-flight requests have no reserved quota, so actual usage can briefly exceed the threshold.

To users, this implementation ultimately surfaces as just two kinds of information: the rate-limit dimensions are RPM / TPM, and the performance tiers include standard mode / high-performance mode, with exact values on the model marketplace detail page. Time-window rate limiting works the same way: the time window is a variable read when the request arrives, not a scheduled daily batch edit of live configuration — this capability is already live, with busy/off-peak quotas switching at runtime under the same rules.

4.2 Availability: the instance pool automatically avoids failures

One model may map to self-hosted instances, a cloud service and a backup provider at the same time. Before forwarding, health status, remaining quota and alert-based demotion jointly decide whether an instance may enter the candidate set; after forwarding, an upstream 429/5xx triggers candidate switching according to the retry conditions. These conditions must converge into a single routing-and-retry chain, otherwise an instance just removed by health checks may be re-selected by another logic path, and a high-priority instance that has exhausted its quota may keep receiving traffic.

Figure 7 · Fallback's filtering and retry chain

The request in the previous section traveled exactly this path: although the three exclusion mechanisms are independent of each other, they all act on the same candidate pool — with high-priority A unavailable, the request landed on B.

4.3 Configuration: the database is the source of truth; delivery is just delivery

Model permissions, quotas and instance state all change, but changes cannot be made by overwriting the gateway configuration wholesale — a single business-side delivery could wipe out manual operations adjustments.

So the configuration stage enforces exactly two boundaries. Source-of-truth boundary: managed plugins treat the database as the single source of truth; unmanaged plugins keep their current data-plane state; the gateway is called only after the database transaction commits, and external IO stays out of the transaction. Failure boundary: a failed delivery records status, alerts and retries — a database rollback is never used to paper over “not delivered”; if a rollback is genuinely needed, restore the gateway first, then roll back the database.

The configuration plane only changes target fields and reads back to confirm after delivery. The goal is not that a given call never fails, but that after a failure you still know where the source of truth is and the data plane can re-converge.

5. The capability map: how the boundaries were drawn

Looking back at the boundaries of this system, the starting point was a list of 13 capability categories:

1
2
Management · Traffic governance · Connections and timeouts · Routing and load balancing · Request rewriting · Observability
Health checks · Model management · Security · Multi-region · Protocol conversion · Extensibility · High availability

Observability and request rewriting had the most entries — the inventory was never really about forwarding itself. Rather than listing items one by one from a product menu, I grouped them into 8 domains by where the capability finally executes:

Figure 1 · Full capability map (8 domains)

A–G are gateway-side capabilities covering the execution of a request and its operational guarantees; H is the asset chain, answering where a model comes from, who it is for, and when it exits. The details are all in Figure 1 and are not repeated here.

Cross-referencing the existing code and configuration material shows that the main path already covers model routing, rate limiting across four scopes, SSE, request rewriting, health checks, fallback, baseline logging and metrics, and multi-region access. A few boundaries are still not in place:

Capability Current boundary
Content moderation Integration point reserved, but consumer-level, bidirectional request/response moderation is not yet complete
Protocol conversion Partial directional conversion exists; transparent two-way conversion between OpenAI and Anthropic still needs validation
Observability Tokens, latency and TTFT are in place; output TPS metrics are still to be built, and some business fields need custom configuration
Automatic certificate renewal A target capability; no implementation evidence located in the current material

The most important thing about this map is not the number of features but the boundaries: A–G decide how a request executes; H decides why it executes that way. Traditional API gateways mainly cover the former; to make models first-class citizens, the latter must be supplied by model management and the business control plane.

When placing items, I used a single rule: every capability must have an explicit executor, input and output; anything without a clear responsibility boundary is recorded as a gap first, rather than turned into a menu item.

6. How feedback becomes decisions

Operational results flow back from three places:

Feedback trigger Who catches it What decision it changes
Persistent upstream 429 / 5xx triggering an alert Data-plane alert integration demotes priority The instance’s routing priority, directly affecting every subsequent routing decision
Token usage and call details Quota and analytics layer Cost allocation aggregates to the application or cost center, and supports the capacity and cost basis (2.5)
Instance health status and usage growth trends Operations and model operations Scale out instances, degrade, or add providers — eventually entering the retirement and migration of 2.6

The first row needs no human: once the state changes, routing priority changes with it. The latter two need a human decision: data first becomes bills and dashboards, and only then does someone decide to scale out, switch providers, or retire.

Back to the two incidents from the opening — a retired model still receiving traffic, an approved call that does not work. Both stem from asset relationships drifting apart from real traffic: decisions do not get through, and results do not come back. Only once both lines run does “who can use what and how much” stop being a ledger locked in by a single approval, and become a decision continuously corrected by operational results.

Conclusion

Model management decides who can use what and how much; the AI gateway is responsible for enforcing those decisions on every request; and after each request, usage, callers and outcomes flow back into model management to support cost allocation, scale-out, migration and retirement.

This is also where the two main lines intersect: the life of a model establishes and changes asset relationships, and the life of a request executes and verifies them. Without the first, the gateway does not know why it routes the way it does; without the second, permissions and quotas in model management are just rules on paper.

The value of the capability map is not how many features it lists, but that every capability can be found with an execution location, input, output and feedback path. Connect these things together, and model assets, traffic governance and cost management finally form a genuine closed loop.

A note from the author

I drove this system end to end — from the capability inventory and three-plane architecture design, through custom plugins, the full model lifecycle, rate-limiting migration, and the operations console. Putting implementations scattered across different systems onto one diagram made one thing clear: those who manage assets, those who compute quotas, and those who count requests must each hold their own boundary, while closing the loop with one another through two chains — control and feedback.

If you are also working on model platforms, AI gateways or LLMOps, I would be glad to compare notes.

Further reading

This article is the English edition of a post originally published in Chinese.