DEV Community: Yilia

TLS Key Exchange for API Gateway Origins: Cut Handshake Retries Safely

Yilia — Thu, 29 Oct 2026 06:15:26 +0000

By Yilia Lin. Originally published on API7.ai.

Disclosure: This syndicated edition was prepared with AI assistance.

TLS latency is not only a client-to-edge concern. An API gateway also acts as a TLS client when it opens an HTTPS connection to an upstream service. If the gateway offers a key share that the origin cannot use, TLS 1.3 may require a HelloRetryRequest and another network round trip before application data can flow.

That detail became a developer-infrastructure topic in September 2026 after Cloudflare reported that its origin-facing Automatic Key Exchange rollout reduced HelloRetryRequest rates from roughly 52% to 3.7% for a scanned cohort. The accompanying Hacker News discussion focused on the practical result: better key-share selection can improve post-quantum adoption and remove avoidable connection latency.

An API gateway team should not copy Cloudflare's system or production numbers into its own plan. It should copy the engineering question: how do we make gateway-to-origin TLS secure, compatible, observable, and cheap enough that handshakes do not become an invisible tax?

Key Takeaways

Client-to-gateway and gateway-to-origin TLS are separate handshakes with separate certificates, protocol support, and failure modes.
In TLS 1.3, an unsupported initial key share can trigger a HelloRetryRequest, adding a round trip before the handshake completes.
Post-quantum policy cannot create support on an origin; incompatible strict policy can make every connection fail.
Connection reuse often matters more than micro-optimizing a single handshake, because reused keep-alive connections do not repeat it.
Measure handshake outcomes by upstream and deployment version before changing algorithms or compliance policy.
API7 Gateway and Apache APISIX can terminate TLS at the edge and establish protected upstream connections, but automatic origin capability probing should not be assumed unless the deployed product documents it.

Understand the Two TLS Boundaries

In a common reverse-proxy path, one request can cross two independent encrypted connections:

On the first connection, the gateway is the server. It selects a certificate based on SNI, negotiates a protocol and cipher suite, and may verify a client certificate. On the second, the gateway is the client. It sends the origin's hostname, offers supported algorithms, verifies the server according to configured policy, and may present its own certificate for upstream mTLS.

These boundaries should be operated separately. Rotating the public certificate does not rotate an upstream client certificate. Enabling TLS 1.3 for incoming clients does not prove every upstream load balancer accepts the same key exchanges. A dashboard that combines both directions into one “TLS errors” counter hides which trust boundary failed.

The API7 Gateway SSL certificate guide explains edge termination and upstream re-encryption. The upstream mTLS guide covers the gateway's client certificate and upstream certificate verification. Apache APISIX also documents its TLS protocol configuration and upstream TLS fields.

Why `HelloRetryRequest` Changes Latency

TLS 1.3 normally completes a full handshake in one round trip. The client includes a key share in ClientHello so the server can immediately continue with a compatible choice. If the server supports another offered group but not the supplied key share, it can return HelloRetryRequest and ask for a new one. The client sends another ClientHello, adding a network round trip.

This mechanism is part of RFC 8446. It preserves interoperability, but it is not free. The extra delay is especially visible when the gateway and origin are in different regions, a cache miss opens a new connection, or a burst exceeds the existing connection pool.

Cloudflare's Automatic Key Exchange report describes an out-of-band scanner that learns an origin's supported preference before leading with a key share. For its scanned origins, avoiding the retry reduced p90 handshake latency by more than 150 ms. That is evidence for Cloudflare's topology, not a universal benchmark. A private gateway in the same Kubernetes cluster may have sub-millisecond network latency; a hybrid-cloud origin can have a far larger round trip.

Post-Quantum Readiness Is a Compatibility Program

The post-quantum part of this story matters because Cloudflare's preferred X25519MLKEM768 group combines X25519 with ML-KEM-768. NIST's FIPS 203 standardizes ML-KEM, including ML-KEM-768. RFC 10024 also standardizes hybrid groups based on P-256 and P-384, so gateway teams should inventory the exact groups supported by each TLS stack. A hybrid TLS group can protect against “harvest now, decrypt later” risk while retaining a classical component.

Yet a policy switch cannot upgrade an origin. The gateway's TLS library, origin server, load balancer, service mesh, firewall, and any middlebox on the path must handle the larger ClientHello and the selected group correctly. If policy permits only a hybrid group and the origin does not support it, there is no mutually acceptable key exchange and the connection fails.

Treat rollout as capability discovery plus controlled enforcement:

Inventory TLS libraries and termination points on both sides.
Test a representative origin from the actual gateway network path.
Record negotiated protocol, group, handshake duration, retries, and failures.
Enable compatible preferences in a small canary pool.
Keep a rollback path to the last approved classical policy.
Enforce a strict post-quantum-only requirement only where policy and origin capability both demand it.

Do not label an entire API “post-quantum protected” after testing only client-to-gateway TLS. If the requirement covers data to the origin, verify the second connection too.

Connection Reuse Is the First Performance Control

The fastest handshake is the one the gateway does not repeat. A keep-alive pool lets many requests reuse an established upstream connection, subject to protocol, concurrency, idle timeout, and maximum-request settings. Cloudflare explicitly notes that requests on existing keep-alive connections were unaffected by its handshake optimization.

This changes the order of operations for API gateway teams:

verify that upstream connection reuse is actually occurring;
align idle timeouts across gateway, load balancer, and origin;
avoid connection churn during deployments and endpoint updates;
separate new-connection latency from total upstream response latency;
then optimize key-share selection for the connections that remain.

Apache APISIX upstream resources expose a configurable keepalive_pool, along with connection, send, and read timeouts. Those controls are described in the APISIX Admin API reference. They do not remove the need to observe whether an upstream or network appliance closes connections earlier than expected.

Build an Origin TLS Dashboard

A useful dashboard groups measurements by upstream service, region, gateway version, and TLS policy. Avoid high-cardinality labels such as raw client IDs or request paths.

Track at least:

new upstream TLS connections per second;
handshake success and failure counts;
handshake duration percentiles;
negotiated TLS version and key-exchange group where available;
HelloRetryRequest or retry indicators where the TLS stack exposes them;
connection reuse ratio and pool misses;
certificate validation, SNI, and mTLS errors;
end-to-end upstream latency for reused versus new connections.

A spike in handshake failures after an origin deployment suggests compatibility. A normal handshake rate with worsening application latency points elsewhere. A rising new-connection rate with stable traffic suggests pool churn rather than cryptography.

The gateway's request metrics still matter. API7.ai's API observability guide shows how metrics, logs, traces, and SLOs complement one another. Correlate an affected request with gateway error logs and origin deployment events; do not place sensitive certificate material or keys into telemetry.

A Safe Rollout Checklist

Before changing gateway-to-origin TLS policy, confirm:

every origin hostname and SNI is explicit;
server-certificate verification and trusted CAs match the environment;
upstream client certificates are scoped and rotated independently;
TLS 1.3 support is verified on the real path, not inferred from a version string;
allowed key exchanges meet policy without excluding required origins;
keep-alive reuse is measured and timeout settings agree;
canaries include long-distance, legacy, and multi-origin paths;
dashboards separate handshake failures from HTTP errors;
rollback does not require distributing new client configuration.

Conclusion

Cloudflare's result makes an obscure handshake message operationally concrete: a wrong initial guess can cost a round trip at enormous scale. For most API gateway deployments, the lesson is not to build a global origin scanner. It is to treat upstream TLS as a measurable connection system rather than a certificate checkbox.

Start with correct trust and SNI, maximize safe connection reuse, measure negotiation by origin, and introduce post-quantum policy through compatibility-tested canaries. Explore API7 Gateway when you need a governed place to manage client-facing and upstream API traffic while keeping their TLS responsibilities explicit.

Originally published at API7.ai by Yilia Lin.

HTTP Connection Reuse: The Hidden API Latency Multiplier

Yilia — Thu, 22 Oct 2026 09:57:22 +0000

By Yilia Lin. Originally published on API7.ai.

Disclosure: This syndicated edition was prepared with AI assistance.

An API can be fast at the origin and still feel slow because a client repeatedly pays to establish TCP and TLS connections. That cost is easy to miss: application traces show a short handler, the gateway reports healthy backends, and no single request looks broken. The lost time sits between requests, where a reusable connection should have been.

Go made this failure mode unusually visible in October 2026. Its engineering team explained how Go 1.27 improves HTTP/1 connection reuse when a caller closes a response without reading its body to the end. In the post's local example, 100 sequential requests fell from about 710 milliseconds to about 150 milliseconds. That is an illustration from one machine and workload, not a general five-times performance promise.

The Hacker News discussion focused on the runtime fix, but the operational lesson is larger. API teams need to know which hop owns each connection, why reuse was lost, and whether a proposed fix trades handshake cost for extra body reads, idle sockets, or retry risk.

Key Takeaways

Draw the full path: client → API gateway → origin. Each hop has its own protocol, pool, limits, and failure modes.
A successful HTTP response does not prove that its connection returned to the reusable pool.
Go 1.27's automatic drain is bounded to 50 milliseconds or 256 KiB and applies to unread HTTP/1 response bodies.
HTTP/2 multiplexing changes the mechanics; do not copy an HTTP/1 body-drain workaround into every protocol.
Measure handshake rate, connection churn, request latency, and origin latency together before changing pool sizes.
Treat gateway keep-alive tuning and client-library behavior as related evidence, not interchangeable fixes.

Why an Unread Body Can Discard a Healthy Connection

HTTP/1 responses share a byte stream. Before a client can parse the next response on the same connection, it must know where the current response ends. Content-Length, chunked transfer framing, or connection closure supplies that boundary, but the client still has to consume the framed body.

Through Go 1.26, closing resp.Body with unread bytes generally prevented net/http.Transport from reusing that connection. A program that wanted only a status code could therefore open another connection for its next request. Even JSON decoding could leave one trailing newline unread after the decoder found the closing object delimiter.

Starting in Go 1.27, Transport asynchronously drains an unread HTTP/1 response body for up to 50 milliseconds or 256 KiB, according to the Go 1.27 release notes. It skips work when reuse cannot help, such as a response with Connection: close. The bounds matter: downloading an unlimited or slow body merely to save a socket could cost more than establishing another connection.

This behavior does not make body ownership optional. Applications still need to close bodies, stream large responses deliberately, and set request deadlines. It also does not change an API gateway written on another network stack. It removes one sharp edge in one Go client path.

Map the Two Connection Pools

Teams often say “the gateway connection” as if there were one socket. A reverse proxy terminates one downstream connection and opens or reuses a separate upstream connection.

The two hops can use different protocols. A client may speak HTTP/2 to the gateway while the gateway uses HTTP/1.1 to an origin. A mobile client may create a new downstream connection on every network transition while the gateway maintains a warm upstream pool. Conversely, downstream reuse can look healthy while an origin sends Connection: close or a gateway pool is undersized.

Ask four questions for each hop:

Question	Client → gateway	Gateway → origin
Which protocol is negotiated?	HTTP/1.1, HTTP/2, or HTTP/3	Depends on gateway and upstream configuration
Who owns the pool?	Client library, SDK, browser, or sidecar	Gateway worker and upstream configuration
What prevents reuse?	Unread body, close header, timeout, network change	Close header, pool pressure, idle expiry, origin behavior
Which evidence exists?	Client trace, DNS/TCP/TLS timings	Gateway latency, origin logs, socket and handshake telemetry

This map prevents a common mistake: changing an upstream gateway pool to fix a downstream mobile-client problem.

Diagnose Churn Before Tuning Keep-Alive

Start with a controlled comparison, not a production-wide configuration change.

Choose one route and one client version. Mixes of SDKs and protocols blur the result.
Record total and origin latency. Apache APISIX's Prometheus plugin exposes request, upstream, and gateway-side latency views. Those signals do not identify every handshake by themselves, but their divergence helps locate the slow hop.
Count new connections and handshakes. Use client tracing, origin server metrics, load-balancer logs, or a bounded packet capture in a test environment.
Check response framing and completion. Look for unread bodies, early cancellation, streaming responses, Connection: close, and timeouts.
Repeat with connection reuse disabled. If latency barely changes, the suspected pool may not be the bottleneck. If it changes sharply, confirm why before increasing limits.

Correlate the result with the broader API observability signals for errors, saturation, and route-level latency. A lower handshake count is useful only if tail latency, error rate, memory, and file-descriptor pressure remain acceptable.

Configure the Gateway Side Deliberately

Apache APISIX lets an Upstream use a separate keep-alive pool. The current APISIX Admin API documentation exposes pool size, idle timeout, and requests-per-connection controls. This article references the APISIX 3.19 documentation available on October 20, 2026; verify fields and defaults against the version you deploy.

There is no universal correct pool size. A larger pool can reduce connection creation, but it also retains more idle sockets per worker and per upstream. Long idle timeouts may preserve warm connections while increasing the chance that an intermediary or origin has already discarded them. A high requests-per-connection value reduces churn but may conflict with an origin's lifecycle policy.

Tune from the path's capacity model:

gateway workers × upstreams × idle connections per pool;
expected concurrency and request duration;
TLS handshake cost and network round-trip time;
origin limits, load-balancer idle timeout, and deployment churn;
long-lived streams versus short request-response traffic.

Keep retries separate from reuse. Retrying an idempotent request after a stale pooled connection fails may be safe under a defined policy. Retrying a payment or another non-idempotent write can duplicate a side effect. Connection tuning must not quietly broaden retry semantics.

Design API Responses for Predictable Consumption

Clients and gateways cannot optimize around an API contract that sends unnecessary or unbounded response bodies.

For status-oriented operations, return the smallest representation that still communicates the result. A 204 No Content response may fit an operation with no representation, while a structured error body is valuable when a call fails. Do not remove useful diagnostics merely to save a connection; make success and error shapes intentional.

For large exports and streams, document whether callers may stop early, whether byte ranges or pagination exist, and how cancellation reaches the origin. For ordinary JSON APIs, keep framing correct and avoid middleware that appends unexpected bytes. Test with the actual SDKs your consumers use.

The same principle appears in TLS key exchange planning for API gateway origins: handshake choices, connection lifetime, and compatibility belong to one measured path. Optimizing one component without the path model can move cost rather than remove it.

A Safe Rollout Checklist

Before changing client or gateway behavior, capture a baseline and a rollback trigger:

protocol on both sides of the gateway;
new connections and TLS handshakes per request;
total, upstream, and gateway-attributed latency percentiles;
client cancellations, resets, and timeout errors;
gateway and origin socket counts;
response-body size distribution;
retry count and method distribution;
memory and file-descriptor headroom.

Upgrade one client cohort or adjust one Upstream at a time. Confirm that fewer handshakes correspond to better tail latency without increasing stale-connection errors or resource pressure. For Go applications, test on Go 1.27 with the real response sizes and cancellation patterns; do not infer production gains from the blog's small sequential benchmark.

Conclusion

Connection reuse is a path property, not a switch labeled “keep-alive.” Go 1.27 fixes a specific HTTP/1 client sharp edge, but API teams still need to distinguish downstream from upstream pools, protocol behavior from configuration, and lower handshake counts from actual user value.

Use an API gateway as the observable boundary: compare end-to-end and upstream latency, keep pool settings explicit, and roll changes out by route and client cohort. If you operate Apache APISIX or API7 Gateway, start with one high-volume API, draw both connection hops, and test the smallest change that explains the evidence.

Originally published at API7.ai by Yilia Lin.

Bot Traffic Moves: Build API Defenses That Follow Behavior

Yilia — Wed, 21 Oct 2026 09:11:00 +0000

Blocking one source network can make a dashboard look safer while the automation simply moves. The operator changes cloud providers, the source IPs and autonomous system number (ASN) change, and the same request sequence resumes against the same API paths.

That pattern became visible in an October 2026 Cloudflare analysis of the Yandex network disruption. Cloudflare observed one targeted customer's matching bot traffic disappear from Yandex.Cloud and appear from other Russian hosting ASNs with the same JA4 TLS fingerprint, uncommon paths and headers, and custom user-agent strings. Across the described week, the relevant Russian bot traffic for that account remained roughly flat. This is one network's observation of specific traffic; it does not identify the operator or prove that every bot migrates.

The Hacker News discussion made the operational question timely: if hostile or unwanted automation can change infrastructure faster than a blocklist review, which controls should remain effective after the move?

Key Takeaways

Use source IP and ASN as signals, not as durable identities.
Authenticate valuable API operations and rate-limit by a trustworthy consumer or tenant identity where possible.
Combine identity, route, method, request cost, concurrency, and behavior; no single fingerprint is permanent.
Keep global abuse controls separate from per-replica counters and document degradation behavior.
Measure allowed, challenged, rate-limited, and blocked traffic by policy reason, not only by source network.
A gateway can enforce API policy, but upstream DDoS absorption, device reputation, and application fraud controls remain separate layers.

What the Yandex Event Actually Showed

Cloudflare separated two Yandex networks with different traffic profiles. The Yandex Search ASN primarily carried verified search crawlers. Yandex.Cloud carried third-party workloads and a different mix of automation. During the disruption, requests from both networks fell, but their meaning was not the same.

For one gambling customer, Cloudflare saw identical request behavior move from Yandex.Cloud to Radist and Selectel. The evidence was not merely “traffic increased elsewhere.” It included the same TLS client fingerprint, unusual paths and headers, and custom user-agent strings. That correlation made workload migration a stronger explanation than unrelated traffic.

Two cautions belong next to that conclusion:

A source network does not prove who operates a request. Cloud platforms host many unrelated tenants.
A fingerprint describes observed behavior at a point in time. A capable client can change its TLS stack, headers, timing, and hosting provider.

An API defense should therefore use source attributes as part of a decision, not equate them with identity.

Build Controls That Survive an IP Change

The most durable control is a principal or credential the API can validate and bind to a consumer or tenant. For an authenticated API, that may be a validated access token, service identity, consumer credential, or tenant key. Rate limits and authorization can then follow the caller across networks.

That identity is only as strong as its verification and lifecycle. A raw header such as X-Tenant-ID is caller-controlled unless a trusted authentication layer binds it to a verified principal. A leaked API key lets automation carry the same identity to a new provider. Pair the identifier with expiration, rotation, revocation, scope, and anomaly detection.

For public or anonymous endpoints, combine weaker signals:

Signal	Useful for	Limitation
Source IP or ASN	Burst control and emergency containment	Shared, rotating, proxied, or easy to move
Authenticated consumer	Quotas and attribution	Credential theft or sharing changes the actor behind it
Route and method	Protecting expensive operations	Does not distinguish callers on the same operation
Request shape	Detecting scanners and repeated abuse sequences	Legitimate clients can share shapes; bots can mutate them
TLS or client fingerprint	Correlating toolchains across networks	Can change and may group unrelated clients
Concurrency and cost	Protecting capacity	Needs a defensible cost model and safe thresholds

The goal is not a magical bot score. It is a policy whose important limits still hold when one signal changes.

Rate-Limit by the Resource You Need to Protect

A request-per-second limit is useful when request cost is reasonably uniform. Many APIs are not uniform. A cached GET and a report export may consume very different database, CPU, and bandwidth capacity. Login attempts, search queries, uploads, and bulk writes also need different windows and keys.

Design limits around the protected resource:

per verified consumer for contractual quotas;
per tenant and route for shared-credential applications;
per IP for unauthenticated bursts, with proxy trust configured correctly;
per operation for expensive endpoints;
concurrency limits for long-running requests;
a global safety ceiling for origin capacity.

Apache APISIX provides request-rate, count, and concurrency controls. Its current limit-count documentation supports variable and variable-combination keys. This article references Apache APISIX 3.19 documentation available on October 20, 2026; confirm behavior in your deployed release.

Do not compose a limit key from an unverified header and call it tenant-aware. Authenticate first, or explicitly treat the value as a weak partition that an attacker can change.

Understand Local and Shared Counter Boundaries

Distributed gateways introduce another scope problem. With the default local policy documented for APISIX limit-count, each gateway instance maintains its own counters. If traffic reaches several instances, the effective aggregate allowance is approximately the configured limit multiplied by the number of instances.

That may be acceptable for coarse load shedding. It is usually wrong for a contractual tenant quota that must apply across the fleet. APISIX also documents Redis-backed policies for shared counters. The trade-off is a runtime dependency: latency, availability, synchronization, and failure behavior now affect enforcement.

Write the intended semantics next to the configuration:

Is the limit per instance or fleet-wide?
What happens when the shared counter store is slow or unavailable?
Is temporary over-admission acceptable?
Which status and Retry-After behavior does the client receive?
Can an operator disable or bypass the rule, and is that action audited?

The HTTP specification defines 429 Too Many Requests and permits a Retry-After header. A gateway may be configured to return another status, so test the actual route rather than assuming the default communicates the policy you intended.

Use Behavior for Detection, Identity for Enforcement

Behavioral signals help discover that traffic moved. Identity and resource controls decide what it is allowed to do.

Suppose an API sees a sequence of requests for rarely used administration paths. After one ASN is blocked, the same sequence returns from several providers. A useful response is not an endless list of ASNs. It is to verify that administrative routes require authentication, reduce anonymous reachability, rate-limit repeated failures, and alert on the sequence across source changes.

At the gateway, record the smallest useful evidence:

verified consumer or anonymous classification;
route and method;
status and policy outcome;
rate-limit key category, without logging the secret itself;
source network and trusted proxy chain;
request-size and latency bands;
bounded behavioral or fingerprint labels;
origin outcome and gateway-generated outcome separately.

Avoid logging full credentials, request bodies, or personal data merely to improve correlation. The API gateway logging best practices apply here: evidence should support response without creating a second sensitive-data store.

Separate Gateway Policy from the Other Defense Layers

An API gateway is well placed to authenticate, authorize, apply route-aware limits, normalize selected requests, and produce consistent decision telemetry. It is not every part of bot defense.

Volumetric attacks may need upstream network or DDoS protection before traffic reaches the gateway. Browser automation may need challenge, device, or session signals. Credential stuffing requires identity-provider and account-risk controls. Business abuse—such as inventory hoarding or promotion fraud—needs application context the gateway may not have.

The OWASP API Security Top 10 calls out unrestricted resource consumption and sensitive business flows as distinct risks. Treat them that way. One capacity limit should not stand in for transaction authorization, and a WAF rule should not stand in for account recovery controls.

For scraping-specific patterns, use the narrower guidance in How to Protect Your APIs from Scrapers. The lesson from migrating traffic is to make those controls portable across source changes.

Test the Migration Counterexample

Before rollout, run a small adversarial test:

Send the same allowed and denied request sequences from two source networks.
Rotate the source IP while keeping a valid consumer credential.
Keep the source fixed while changing the credential.
Remove the shared counter backend in a test environment.
Send an expensive operation below the simple request-rate threshold.
Confirm which layer blocks, which status the caller sees, and which evidence operators receive.

The counterexample is the real gate: if changing an IP resets every meaningful control, the design still depends on location rather than caller or cost. If losing Redis silently removes a fleet-wide quota, the failure policy is incomplete. If the rule blocks legitimate crawler or integration traffic without an exception process, it is not operationally ready.

Conclusion

Bot infrastructure is replaceable. Durable API defenses therefore need to follow verified identity, protected resources, and repeated behavior—not a single IP range or ASN.

Use network blocks for tactical containment, then move the lasting policy to the gateway boundary: authenticate valuable operations, scope limits to the right identity and route, make distributed-counter behavior explicit, and preserve evidence that correlates a workload across source changes. Apache APISIX and API7 Gateway can provide that policy and observation point, while DDoS, identity, fraud, and application teams keep ownership of the layers the gateway cannot replace.

Originally published by Yilia Lin on API7.ai.

Disclosure: AI tools were used to draft portions of this article.

Release Apache APISIX Ingress Controller 2.0

Yilia — Mon, 22 Dec 2025 06:50:22 +0000

Apache APISIX Ingress Controller 2.0 is officially released. It delivers comprehensive Gateway API support, flexible multi-data-plane deployment, and etcd-free operation for robust, scalable Kubernetes traffic management.

Built on the high-performance API gateway Apache APISIX, APISIX Ingress Controller has undergone multiple iterations and validations, and is now capable of handling large-scale traffic management demands. The Apache APISIX community is pleased to announce the official release of APISIX Ingress Controller 2.0. This release delivers substantial enhancements across three foundational pillars—comprehensive compatibility, adaptable architecture, and enterprise-grade stability—empowering users to migrate their technology stacks smoothly and reliably.

Highlights of Apache APISIX Ingress Controller 2.0

Support Gateway API

This release achieves a significant milestone in Gateway API coverage with the addition of TCPRoute, UDPRoute, GRPCRoute, and TLSRoute. These extensions provide native, protocol-aware routing for a wide range of traffic types—from traditional HTTP and TCP/UDP to modern gRPC and TLS passthrough/termination. This unified support allows organizations to manage diverse ingress requirements within a consistent, future-ready configuration model, simplifying multi-protocol deployment and easing the transition to full Gateway API adoption.

Introduce Gateway API Extensions

Building upon adherence to the Gateway API design principles, APISIX Ingress Controller 2.0 introduces a set of API extensions under apisix.apache.org/v1alpha1 based on the Gateway API. These extensions provide additional capabilities not currently directly covered by the standard Gateway API, while maintaining the core semantics and usage patterns of the standard resources. They are designed to meet more complex and diverse real-world usage scenarios.

GatewayProxy: It defines the connection between the APISIX Ingress Controller and the APISIX, including auth, endpoints, and global plugins. It is referenced via parametersRef in Gateway, GatewayClass, or IngressClass resources.
BackendTrafficPolicy: It is for fine-grained traffic management of backend services, including load balancing, timeouts, retries, and host header handling in the APISIX Ingress Controller.
Consumer: It defines API consumers and their credentials, enabling authentication and plugin configuration for controlling access to API endpoints.
PluginConfig: It defines reusable plugin configurations that can be referenced by other resources like HTTPRoute, enabling separation of routing logic and plugin settings for better reusability and manageability.
HTTPRoutePolicy: It configures advanced traffic management and routing policies for HTTPRoute or Ingress resources, enhancing functionality without modifying the original resources.

These extensions offer a standardized, vendor-supported path to leverage advanced APISIX features directly within the Gateway API ecosystem.

Support APISIX Standalone API-Driven Mode

APISIX Ingress Controller 2.0 offers a lightweight, etcd-free deployment option through its Standalone API-Driven Mode.

This deployment paradigm stores routing configurations entirely in memory rather than in a configuration file. Updates are performed through a dedicated Standalone Admin API, which replaces the full configuration in a single operation and takes effect immediately via hot reloading, without requiring a restart.

This mode is designed specifically for the APISIX Ingress Controller and is primarily intended for integration with ADC (API Declarative CLI).

Support Multi-Data-Plane Deployment Mode

This release introduces flexible deployment options supporting multiple data plane modes, enabling a single ingress controller to manage several independent APISIX instances. This approach is ideal for environments requiring strict isolation—such as multi-tenancy, staging vs. production, or region-based routing—while maintaining centralized control.

Admin API Mode

In the traditional deployment approach, APISIX uses etcd as its configuration center, allowing administrators to dynamically manage routes, upstreams, and other resources through RESTful APIs. It supports distributed cluster deployments with real-time configuration synchronization.

Standalone Mode

APISIX can also run independently without relying on etcd, which is especially well‑suited for Kubernetes and single‑node deployments. It stores configurations in memory and manages them through the dedicated /apisix/admin/configs endpoint.

This mode is particularly suitable for Kubernetes environments and single-node deployments, where the API-driven memory management approach combines the convenience of traditional Admin API with the simplicity of Standalone mode.

This multi-mode strategy empowers organizations to tailor their ingress architecture to diverse requirements without sacrificing manageability or control.

Conclusion

Apache APISIX Ingress Controller 2.0 represents a significant evolution in Kubernetes ingress management, delivering a robust platform built for the complexity of modern, multi-protocol applications. By uniting comprehensive Gateway API support, extensible configuration through official API extensions, a lightweight standalone deployment mode, and versatile multi-data-plane management, this release provides a cohesive and powerful foundation for dynamic cloud environments.

Whether you are standardizing ingress across diverse workloads, seeking greater architectural flexibility, or requiring enterprise-grade stability at scale, APISIX Ingress Controller 2.0 offers a forward-looking solution that simplifies operations without compromising capability. It stands as a testament to the community-driven innovation within the Apache APISIX ecosystem, designed to meet today's demands while adapting to tomorrow's challenges.

For a complete list of features and changes, please refer to the Release Changelog.

Building an AI Agent Traffic Management Platform: APISIX AI Gateway in Practice

Yilia — Thu, 20 Nov 2025 08:12:49 +0000

Introduction: The Turning Point from Dispersed Traffic to Intelligent Governance

Since early 2025, within a leading global appliance giant, multiple business lines have introduced numerous large language models (LLMs). The R&D department needed coding assistants to improve efficiency, the marketing team focused on content generation, and the smart product team aimed to integrate conversational capabilities into home appliances. The variety of models rapidly expanded to include both self-built solutions like DeepSeek and Qwen, as well as proprietary models from multiple cloud service providers.

However, this rapid expansion soon exposed new bottlenecks: fragmented inference traffic, chaotic scheduling, rising operational costs, and uncontrollable stability issues.

The infrastructure team realized they needed a central system capable of unified control and dynamic scheduling at the traffic layer—a gateway born for AI.

Thus, the enterprise began collaborating with the API7 team to jointly build an enterprise-grade AI Agent traffic management and scheduling platform. This was not just an upgrade in gateway technology, but a comprehensive architectural transformation for the AI era.

Challenges: The Complexity of Multi-Model, Multi-Tenant, Hybrid Cloud

In this appliance giant's AI practice, challenges are primarily focused on three levels:

1. Stability Assurance

With rapid model iterations and service diversification, how to ensure stable proxying and quick recovery for each request?
How to achieve zero-interruption switching between different vendors' LLM services?

2. Multi-tenant Isolation

Each business department operated independent AI Agents. When tasks from one tenant spiraled out of control, resource and fault isolation became essential to prevent chain reactions.

3. Intelligent Scheduling

The hybrid cloud architecture coexisted with self-built models and cloud models. Facing dynamic loads, the system lacked real-time health awareness and automatic routing optimization.

These problems collectively pointed to a core requirement: AI traffic must be uniformly governed, visually monitored, and intelligently scheduled.

System Design: Core Architecture of the AI Gateway

The enterprise chose to build AI gateway capabilities on top of its existing API gateway, transforming it into a unified intelligent traffic hub.

From an overall perspective, the system comprises three core layers:

Access Layer: Provides unified entry points, handling protocol conversion, authentication, and rate limiting.
Governance Layer: Implements dynamic routing, circuit breaking, fault detection, and content filtering through a plugin mechanism.
Scheduling Layer: Combines health checks with real-time load information to enable automatic switching between self-built and cloud models.

On the AI gateway, some AI models undergo rapid version iterations with stability risks. For example, improper request formats might trigger model loops, persistent abnormal outputs, or generate unreasonable content. Therefore, the internal technical team leveraged APISIX AI Gateway's plugin extension mechanism. Through custom plugins for request rewriting and defense, along with flexible configuration, they implemented intervention and filtering of request and response content to ensure service reliability and output quality.

Key Selection Criteria for AI Gateways

In the process of building AI capability platforms, gateway selection significantly impacts the overall architecture. The enterprise evaluated solutions based on several core dimensions:

Production-Grade Stability: Stability is paramount. Ensuring service stability for users, enabling business operations to continue uninterrupted even during model fluctuations, is the most critical requirement.
Continuously Evolving Technical Capabilities: With AI technology iterating rapidly, the AI gateway must maintain fast update cycles to promptly adapt to new model protocols and interaction patterns. The chosen AI gateway needs to keep pace with technological trends, avoiding becoming a bottleneck for business innovation.
Standardized, Reusable Architecture: Mature, reusable architecture is another key point. Providing standard API management and extension interfaces that comply with mainstream technical standards and best practices. APISIX AI Gateway's extensibility stood out as a highlight, directly determining integration costs with existing technology stacks and the smoothness of future integration into broader AI ecosystems.

Fine-Grained AI Traffic Governance and Multi-tenant Isolation

Scenario 1: Automatic Fallback for Hybrid Models

In actual usage, this leading appliance enterprise adopted a hybrid deployment model for critical models (Model A): part of the service was self-built in private data centers, served as the main carrier for core traffic; simultaneously, using this model on public cloud with pay-as-you-go pricing served as Plan B.

All requests were initially directed to self-built services by default. When self-built services encountered performance bottlenecks or became unavailable due to sudden traffic spikes or peaks, the gateway—based on preset token rate limiting policies and real-time health checks—automatically and seamlessly switched requests to cloud services, achieving smooth fallback. Once self-built services recovered, traffic automatically reverted.

curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d '{
    "id": "ai-proxy-multi-route",
    "uri": "/anything",
    "methods": ["POST"],
    "plugins": {
      "ai-proxy-multi": {
          "balancer": {
            "algorithm": "roundrobin",
            "hash_on": "vars"
          },
          "fallback_strategy": "instance_health_and_rate_limiting",
          "instances": [
            {
              "auth": {
                "header": {
                  "Authorization": "Bearer {ALIYUN_API_KEY}"
                }
              },
              "name": "qwen2.5-32b-instruct-ali-bailian",
              "options": {
                "model": "qwen2.5-32b-instruct"
              },
              "override": {
                "
                ": "https://dashscope.aliyuncs.com/compatible-mode/v1/chat/completions"
              },
              "priority": 1,
              "provider": "openai-compatible",
              "weight": 100
            },
            {
              "auth": {
                "header": {
                  "Authorization": "Bearer {CUSTOM_API_KEY}"
                }
              },
              "checks": {
                "active": {
                  "concurrency": 10,
                  "healthy": {
                    "http_statuses": [
                      200,
                      302
                    ],
                    "interval": 30,
                    "successes": 1
                  },
                  "host": "{CUSTOM_HOST_1}:{CUSTOM_PORT_1}",
                  "http_method": "POST",
                  "http_path": "/v1/chat/completions",
                  "http_req_body": "{\"model\":\"Qwen/Qwen2.5-32B-Instruct\",\"messages\":[{\"role\":\"user\",\"content\":\"0\"}],\"stream\":false,\"max_tokens\":1}",
                  "https_verify_certificate": false,
                  "req_headers": [
                    "Content-Type: application/json"
                  ],
                  "request_body": "",
                  "timeout": 2,
                  "type": "http",
                  "unhealthy": {
                    "http_failures": 1,
                    "http_statuses": [
                      404,
                      429,
                      500,
                      501,
                      502,
                      503,
                      504,
                      505
                    ],
                    "interval": 30,
                    "tcp_failures": 2,
                    "timeouts": 2
                  }
                }
              },
              "name": "qwen2.5-32b-instruct-b",
              "options": {
                "model": "Qwen/Qwen2.5-32B-Instruct"
              },
              "override": {
                "endpoint": "http://{CUSTOM_HOST_1}:{CUSTOM_PORT_1}/v1/chat/completions"
              },
              "priority": 5,
              "provider": "openai-compatible",
              "weight": 100
            },
            {
              "auth": {
                "header": {
                  "Authorization": "Bearer {NLB_API_KEY}"
                }
              },
              "checks": {
                "active": {
                  "concurrency": 10,
                  "healthy": {
                    "http_statuses": [
                      200,
                      302
                    ],
                    "interval": 30,
                    "successes": 1
                  },
                  "host": "{CUSTOM_NLB_HOST}:{CUSTOM_NLB_PORT}",
                  "http_method": "POST",
                  "http_path": "/v1/chat/completions",
                  "http_req_body": "{\"model\":\"Qwen/Qwen2.5-32B-Instruct\",\"messages\":[{\"role\":\"user\",\"content\":\"0\"}],\"stream\":false,\"max_tokens\":1}",
                  "https_verify_certificate": false,
                  "req_headers": [
                    "Content-Type: application/json"
                  ],
                  "request_body": "",
                  "timeout": 3,
                  "type": "http",
                  "unhealthy": {
                    "http_failures": 2,
                    "http_statuses": [
                      404,
                      429,
                      500,
                      501,
                      502,
                      503,
                      504,
                      505
                    ],
                    "interval": 30,
                    "tcp_failures": 2,
                    "timeouts": 3
                  }
                }
              },
              "name": "qwen2.5-32b-instruct-c",
              "options": {
                "model": "Qwen/Qwen2.5-32B-Instruct"
              },
              "override": {
                "endpoint": "http://{CUSTOM_NLB_HOST}:{CUSTOM_NLB_PORT}/v1/chat/completions"
              },
              "priority": 10,
              "provider": "openai-compatible",
              "weight": 100
            }
          ],
          "keepalive": true,
          "keepalive_pool": 30,
          "keepalive_timeout": 4000,
          "ssl_verify": false,
          "timeout": 600000
        }

This mechanism operated fully automated, ensuring business continuity. Operations teams only became aware of state transitions through alerts, requiring no manual intervention. This capability not only significantly enhanced business continuity but also greatly reduced operational complexity, becoming key infrastructure for ensuring AI service high availability.

Scenario 2: Token-Based Rate Limiting

In this enterprise's AI service multi-tenant architecture, reasonable resource allocation and isolation between different users were the most core requirements. Since token costs varied significantly across different AI models, traditional request-based rate limiting couldn't accurately measure real resource consumption. Therefore, it was essential to introduce fine-grained quota management and traffic control mechanisms based on token volume, thereby truly reflecting resource consumption and ensuring reasonable scheduling and cost control between users.

In this mechanism, different consumers had independent rate-limiting quotas, while different LLMs had separate token limits. Both took effect simultaneously, with consumer quotas having higher priority than LLM quotas. Once quotas were exhausted, consumers were prohibited from continuing to call LLM services.

For example, for LLM A, consumers A, B, and C had quotas of 10,000, 20,000, and 5,000 tokens, respectively, while LLM A overall had a global limit of 50,000 tokens. When consumers sent requests, the gateway would sequentially check both quotas: first verifying whether individual consumer quotas were sufficient, then confirming whether global LLM quotas were adequate. Only when both conditions were met would requests be forwarded to LLM A; insufficient quotas in either category would immediately return 429 errors and reject requests.

In practical configuration, first enable the ai-proxy-multi and ai-rate-limiting plugins to set up rate limiting for the LLM.

curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d '{
    "id": "ai-proxy-multi-route",
    "uri": "/anything",
    "methods": ["POST"],
    "plugins": {
      "key-auth": {},
      "ai-proxy-multi": {
        "instances": [
          {
            "name": "qwen2.5-32b-instruct-ali-bailian",
            "options": {
              "model": "qwen2.5-32b-instruct"
             },
            "auth": {
              "header": {
                "Authorization": "Bearer {NLB_API_KEY}"
              }
            },
            "override": {
              "endpoint": "https://dashscope.aliyuncs.com/compatible-mode/v1/chat/completions"
            },
            "priority": 1,
            "provider": "openai-compatible",
            "weight": 100
          },
          {
            "name": "qwen2.5-32b-instruct-b",
            "options": {
              "model": "Qwen/Qwen2.5-32B-Instruct"
            },
            "auth": {
              "header": {
                "Authorization": "Bearer {NLB_API_KEY}"
              }
            },
            "override": {
              "endpoint": "http://{CUSTOM_HOST_1}:{CUSTOM_PORT_1}/v1/chat/completions"
            },
            "priority": 5,
            "provider": "openai-compatible",
            "weight": 100
          }
        ]
      },
      "ai-rate-limiting": {
        "instances": [
          {
            "name": "qwen2.5-32b-instruct-ali-bailian",
            "limit": 50000,
            "time_window": 3600
          },
          {
            "name": "qwen2.5-32b-instruct-b",
            "limit": 50000,
            "time_window": 3600
          }
        ],
        "rejected_code": 429,
        "limit_strategy": "total_tokens"
      }
    }

Then, create three consumers and configure corresponding rate limiting for each. The ai-consumer-rate-limiting plugin is specifically used to enforce rate limits on consumers. Taking Consumer A as an example, the configuration is as follows:

curl "http://127.0.0.1:9180/apisix/admin/consumers" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d '{
    "username": "consumer_a",
    "plugins": {
      "key-auth": {
        "key": "consumer_a_key"
      },
      "ai-consumer-rate-limiting": {
        "instances": [
          {
            "name": "qwen2.5-32b-instruct-ali-bailian",
            "limit_strategy": "total_tokens",
            "limit": 10000,
            "time_window": 3600
          },
          {
            "name": "qwen2.5-32b-instruct-b",
            "limit_strategy": "total_tokens", 
            "limit": 10000,
            "time_window": 3600
          }
        ],
        "rejected_code": 429,
        "rejected_msg": "Insufficient token, try in one hour"
      }
    }
  }'

This solution effectively prevents individual consumers from excessive consumption, affecting other users, protects backend LLM instances from being overwhelmed by sudden traffic spikes, manages quotas based on actual token consumption, and provides differentiated services for different user levels.

Value Delivered by APISIX AI Gateway

By building a unified AI gateway and consolidating AI traffic entry points, the technical team significantly improved the overall usage efficiency and management capability of model services. Main achievements include the following aspects:

1. Simplified Large Model Access, Lowering Usage Barriers

The AI gateway provides unified access addresses and keys for all model services. Users don't need to concern themselves with backend model deployment and operational details—they can flexibly call various model resources through fixed entry points, greatly reducing the barrier to using AI capabilities.

2. Achieved Centralized Resource Management with Service Stability

Without a unified AI gateway, various business units would need to build and maintain model services independently. Particularly when facing high resource consumption scenarios like large models, this would lead to duplicated GPU investments and waste. Through unified management and scheduling, efficient resource utilization was achieved, with service stability centrally guaranteed at the gateway level.

3. Unified Control with Traffic Security Assurance

As the unified consolidation point for all AI traffic, the AI gateway became the critical node for implementing common capabilities. At this node, identity authentication, access auditing, content security review, abnormal request protection, and output content filtering could be centrally implemented, systematically enhancing overall platform controllability and security.

AI Gateway Evolution Direction and Outlook

As AI integrates into all aspects of R&D, manufacturing, and sales, this industry benchmark enterprise's goal is shifting from "connecting models" to "building a unified AI platform." In this process, the AI gateway is no longer just a traffic distribution node but is gradually evolving into the scheduling core of the entire AI capability system. In the future, it will carry new capabilities, including MCP (Model Context Protocol) and Agent2Agent (A2A) protocol, evolving into the enterprise's AI operating system kernel.

For this appliance enterprise, the current phase focuses on building foundations: making every request observable, schedulable, and governable.

While deeply applying APISIX AI Gateway in business scenarios, both parties are also jointly exploring evolution directions for next-generation AI infrastructure. As AI-native workloads like large model inference gradually become core business traffic, the team observed in practice that AI traffic exhibits significant differences from traditional web traffic in scheduling sensitivity, response patterns, and service governance dimensions. This presents new propositions for the gateway's continuous evolution:

More Intelligent Traffic Scheduling: Current load balancing strategies excel at handling high-concurrency, fast-response traditional traffic. For AI services, we hope to introduce metrics like GPU load, inference queue depth, and single-request latency to achieve intelligent distribution based on real-time service capabilities, making resource utilization more efficient and responses more stable.
Backend Service State Awareness: When model services experience slowed responses or queue buildup, the gateway should detect and switch faster. We're exploring how to implement dynamic routing based on real-time service states, such as inference performance and queue length, to ensure smooth user experiences.
Completing Observability Data: The plugin architecture provides flexibility for traffic governance. Next, the technical team hopes to further enhance the gateway's fine-grained metric collection capabilities, such as upstream service status codes and precise response latency, making it more naturally integrated into existing monitoring and logging systems, providing solid support for fault localization and system optimization.

In an era where AI traffic becomes an enterprise-critical workload, API7 and this globally leading multinational appliance giant have jointly explored an evolution path of "gateway intelligence." It represents both a technological upgrade and an organizational capability transformation—making AI truly become an enterprise's underlying operational capability, rather than a passive tool.

Load Balancing AI/ML API with Apache APISIX

Yilia — Thu, 31 Jul 2025 09:13:52 +0000

This blog provides a step-by-step guide to configure Apache APISIX for AI traffic splitting and load balancing between API versions, covering security setup, canary testing, and deployment monitoring.

Overview

AI/ML API is a one-stop, OpenAI-compatible endpoint that is trusted by 150,000+ developers to 300+ state-of-the-art models—chat, vision, image/video/music generation, embeddings, OCR, and more—from Google, Meta, OpenAI, Anthropic, Mistral, and others.

Apache APISIX is a dynamic, real-time, high-performance API Gateway. APISIX API Gateway provides rich traffic management features and can serve as an AI Gateway through its flexible plugin system.

Modern AI workloads often require smooth version migrations, A/B testing, and rolling updates. This guide shows you how to:

Install Apache APISIX with Docker quickstart.
Secure the Admin API with keys and IP whitelisting.
Define separate routes for API versions v1 and v2.
Implement weighted traffic splitting (50/50) via the traffic-split plugin.
Verify the newly created split endpoint functionality.
Load test and monitor distribution accuracy.

To perform authenticated requests, you'll need an AI/ML API key. You can get one at https://aimlapi.com/app/keys/ and use it as a Bearer token in your Authorization headers.

Quickstart Installation

# 1. Download and run the quickstart script (includes etcd + APISIX)
curl -sL https://run.api7.ai/apisix/quickstart | sh

# 2. Confirm APISIX is up and running
curl -I http://127.0.0.1:9080 | grep Server
# ➜ Server: APISIX/3.13.0

Tip: If you encounter port conflicts, adjust Docker host networking or map to different ports in the quickstart script.

Secure the Admin API

By default, quickstart bypasses Admin API authentication. For any non-development environment, enforce security:

1. Set an Admin Key

Edit conf/config.yaml inside the APISIX container or local install directory, replacing the example key with your own API key obtained from the link above:

apisix:
  enable_admin: true            # Enable Admin API
  admin_key_required: true      # Reject unauthenticated Admin requests
  admin_key:
    - name: admin
      key: YOUR_ADMIN_KEY_HERE  # Generated admin key - you can replace this with a secure key as you wish
      role: admin

Security Best Practice: Use at least 32 characters, mix letters/numbers/symbols, and rotate keys quarterly.

2. Whitelist Management IPs (allow_admin)

Add your management or local networks under the admin: section:

admin:
  allow_admin:
    - 127.0.0.0/24   # Localhost & host network
    - 0.0.0.0/0      # Allow all (temporary/testing only)

Warning: 0.0.0.0/0 opens Admin API to the world! Lock this down to specific subnets in production.

3. Restart APISIX

docker restart apisix-quickstart

Check Logs: docker logs apisix-quickstart --tail 50 to ensure no errors about admin authentication.

Define Basic Routes for v1 and v2

Before splitting traffic, ensure each version works individually.

1. Route for v1

curl -i http://127.0.0.1:9180/apisix/admin/routes/test-v1 \
  -X PUT \
  -H "X-API-KEY: YOUR_ADMIN_KEY_HERE" \
  -d '{
    "uri": "/test/v1",
    "upstream": {
      "type": "roundrobin",
      "nodes": {"api.aimlapi.com:443": 1},
      "scheme": "https",
      "pass_host": "node"
    }
  }'

Tip: Use id fields if you want to manage or delete routes easily later.

2. Route for v2

curl -i http://127.0.0.1:9180/apisix/admin/routes/test-v2 \
  -X PUT \
  -H "X-API-KEY: YOUR_ADMIN_KEY_HERE" \
  -d '{
    "uri": "/test/v2",
    "upstream": {
      "type": "roundrobin",
      "nodes": {"api.aimlapi.com:443": 1},
      "scheme": "https",
      "pass_host": "node"
    }
  }'

Implement Traffic Splitting (50/50)

Use the traffic-split plugin for controlled distribution between v1 and v2. In the admin request below, replace YOUR_ADMIN_KEY_HERE with your actual key.

curl -i http://127.0.0.1:9180/apisix/admin/routes/aimlapi-split \
  -X PUT \
  -H "X-API-KEY: YOUR_ADMIN_KEY_HERE" \
  -d '{
    "id": "aimlapi-split",
    "uri": "/chat/completions",
    "upstream": {
      "type": "roundrobin",
      "nodes": {"api.aimlapi.com:443": 1},
      "scheme": "https",
      "pass_host": "node"
    },
    "plugins": {
      "traffic-split": {
        "rules": [
          {
            "weight": 50,
            "upstream": {"type":"roundrobin","nodes":{"api.aimlapi.com:443":1},"scheme":"https","pass_host":"node"},
            "rewrite": {"uri":"/v1/chat/completions"}
          },
          {
            "weight": 50,
            "upstream": {"type":"roundrobin","nodes":{"api.aimlapi.com:443":1},"scheme":"https","pass_host":"node"},
            "rewrite": {"uri":"/v2/chat/completions"}
          }
        ]
      }
    }
  }'

Tip: Adjust the weight values to shift traffic ratios (e.g., 80/20 for canary).

Note: rewrite must match the internal API path exactly.

Verify Split Endpoint Functionality

Test the /chat/completions endpoint you just created. Replace <AIML_API_KEY> with the key obtained earlier and use it as a Bearer token:

curl -v -X POST http://127.0.0.1:9080/chat/completions \
  -H "Authorization: Bearer <AIML_API_KEY>" \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-4","messages":[{"role":"user","content":"ping"}]}'

Expected Output:

{"content":"Pong! How can I assist you today?"}

Tip: Use -v for verbose output to troubleshoot headers or TLS issues.

Load Test & Distribution Validation

After configuring the split route, use the following commands to validate distribution. Replace <AIML_API_KEY> with your Bearer token.

# 1. Send 100 test requests
time seq 100 | xargs -I {} curl -s -o /dev/null -X POST http://127.0.0.1:9080/chat/completions \
  -H "Authorization: Bearer <AIML_API_KEY>" \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-4","messages":[{"role":"user","content":"ping"}]}'
# 2. Check APISIX logs for upstream hits (replace IPs with actual resolved IPs)
echo "v1 hits: $(docker logs apisix-quickstart --since 5m | grep -c '188.114.97.3:443')"
echo "v2 hits: $(docker logs apisix-quickstart --since 5m | grep -c '188.114.96.3:443')"

Expected: Approximately 50 requests to each upstream.

Tip: Use Prometheus or OpenTelemetry plugins for real‑time metrics instead of manual log parsing.

Best Practices & Next Steps

Rate Limiting & Quotas: Add limit-count plugin to protect your upstream from spikes.
Authentication: Layer on the key-auth plugin for consumer management.
Circuit Breaker: Prevent cascading failures with the api-breaker plugin.
Observability: Integrate Prometheus, Skywalking, or Loki for dashboards and alerts.
Infrastructure as Code: Consider managing APISIX config via Kubernetes CRDs or ADC for reproducibility.

References

Announcing APISIX Integration with AI/ML API

Yilia — Wed, 30 Jul 2025 08:56:39 +0000

We're thrilled to announce that AI/ML API has become a supported provider to the ai-proxy, ai-proxy-multi, and ai-request-rewrite plugins in Apache APISIX. All the AI/ML APIs will be supported in the next APISIX version.

Introduction

AI/ML API is a single endpoint that gives you access to more than 300 ready-to-use AI models—large language models, embeddings, image and audio tools—through one standard REST interface. It is used by over 150,000 developers and organizations as a centralized LLM API gateway.

We're thrilled to announce that AI/ML API has become a supported provider to the ai-proxy, ai-proxy-multi, and ai-request-rewrite plugins in Apache APISIX.

AI/ML API provides a unified OpenAI-compatible API with access to 300+ LLMs such as GPT-4, Claude, Gemini, DeepSeek, and others. This integration bridges the gap between your API infrastructure and leading AI services, enabling you to deploy intelligent features—like chatbots, real-time translations, and data analysis—faster than ever.

Proxy to OpenAI via AI/ML API

Prerequisites

Install APISIX.
Generate your API key on AI/ML API dashboard.

Configure the Route

Create a route and configure the ai-proxy plugin as such:

curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d '{
    "id": "ai-proxy-route",
    "uri": "/anything",
    "methods": ["POST"],
    "plugins": {
      "ai-proxy": {
        "provider": "aimlapi",
        "auth": {
          "header": {
            "Authorization": "Bearer '"$OPENAI_API_KEY"'" # Generated openai key from AI/ML API dashboard
          }
        },
        "options":{
          "model": "gpt-4"
        }
      }
    }
  }'

Test the Integration

Send a POST request to the route with a system prompt and a sample user question in the request body:

curl "http://127.0.0.1:9080/anything" -X POST \
  -H "Content-Type: application/json" \
  -H "Host: api.openai.com" \
  -d '{
    "messages": [
      { "role": "system", "content": "You are a mathematician" },
      { "role": "user", "content": "What is 1+1?" }
    ]
  }'

Verify Response

You should receive a response similar to the following:

{
  ...,
  "choices": [
    {
      "index": 0,
      "finish_reason": "stop",
      "logprobs": null,
      "message": {
        "role": "assistant",
        "content": "1 + 1 equals 2.",
        "refusal": null,
        "annotations": []
      }
    }
  ],
  "created": 1753845968,
  "model": "gpt-4-0613",
  "usage": {
    "prompt_tokens": 1449,
    "completion_tokens": 1008,
    "total_tokens": 2457
  ...
}

Core Use Cases

1.Unified AI Service Management

Multi-Model Proxy and Load Balancing: Replace hardcoded vendor endpoints with a single APISIX interface, dynamically routing requests to models from OpenAI, Claude, DeepSeek, Gemini, Mistral, etc., based on cost, latency, or performance needs.
Vendor-Agnostic Workflows: Seamlessly switch between models (e.g., GPT-4 for creative tasks, Claude for document analysis) without code changes.

2.Cost-Optimized Token Governance

Token-Based Budget Enforcement: Set per-team/monthly spending limits; auto-throttle requests when thresholds are exceeded.
Caching & Fallbacks: Cache frequent LLM responses (e.g., FAQ answers) or reroute to cheaper models during provider outages.

3.Real-Time AI Application Scaling

Chatbots & Virtual Agents: Power low-latency conversational interfaces with streaming support for token-by-token responses.
Data Enrichment Pipelines: Augment APIs with AI—e.g., auto-summarize user reviews or translate product descriptions on-the-fly.

4.Hybrid/Multi-Cloud AI Deployment

Unified Control Plane: Manage on-prem LLMs (e.g., Llama 3) alongside cloud APIs (OpenAI, Azure) with consistent policy enforcement.
High Availability & Fault Tolerance: Built-in health-checks, automatic retries and failover; if one LLM fails, traffic is rerouted within seconds to keep services alive.

5.Enterprise AI Security & Compliance

Data Security and Compliance: Prompt Guard, content moderation, PII redaction, and full audit logs in a single place.
One Auth Layer for 300+ LLMs: Unified authentication (JWT/OAuth2/OIDC) and authorization for 300+ LLM keys and policies.

Conclusion

With AI/ML API now natively supported in Apache APISIX, you no longer have to choose between speed, security, or scale—you get all three.

One line of YAML turns your gateway into a 300-model AI powerhouse.
Zero code changes let you hot-swap GPT-4 for Claude, or route 10 % of traffic to a cheaper model for instant cost savings.
Built-in guardrails (PII redaction, token budgets, content moderation) keep compliance teams happy while your product team ships faster.

More Resources

Related APISIX AI Plugins
AI/ML API Community

GraphQL vs REST API: Which is Better for Your Project in 2025?

Yilia — Mon, 21 Jul 2025 08:57:32 +0000

Key Takeaways

REST APIs excel in simplicity, caching, and microservices architecture, with widespread adoption and mature tooling ecosystem
GraphQL provides precise data fetching, reduces over-fetching, and offers superior flexibility for complex data relationships
Performance varies by use case: REST wins for simple CRUD operations and caching scenarios, while GraphQL shines in mobile apps and complex queries
API Gateway integration is crucial for managing both approaches effectively, providing unified security, monitoring, and transformation capabilities
No universal winner: The choice depends on project requirements, team expertise, and specific technical constraints rather than inherent superiority

Understanding REST APIs and GraphQL: The Foundation of Modern API Architecture

When evaluating modern API architectures, developers frequently encounter the question: "What is a RESTful API, and how does it compare to GraphQL?" According to recent industry data, over 61% of organizations are now using GraphQL, while REST continues to dominate enterprise environments. Understanding both approaches is essential for making informed architectural decisions.

What is a RESTful API?

A RESTful API (Representational State Transfer) is an architectural style that leverages HTTP protocols to create scalable web services. REST and RESTful services follow six key principles: statelessness, client-server architecture, cacheability, layered system, uniform interface, and code on demand (optional). Unlike the traditional SOAP protocol vs REST debate, where SOAP v REST discussions centered on protocol complexity, RESTful APIs embrace simplicity and web-native patterns.

The fundamental concept behind RESTful API architecture involves treating every piece of data as a resource, accessible through standard HTTP methods (GET, POST, PUT, DELETE). This approach has made REST API RESTful implementations the backbone of countless web applications, from simple CRUD operations to complex enterprise systems.

What is GraphQL?

GraphQL represents a paradigm shift from traditional REST approaches. Developed by Facebook in 2012 and open-sourced in 2015, GraphQL is a query language and runtime for APIs that enables clients to request exactly the data they need. Unlike REST's resource-based approach, GraphQL operates through a single endpoint that can handle complex data fetching scenarios.

The core innovation of GraphQL lies in its declarative data fetching model. When you need to perform a GraphQL query to get number of customers along with their recent orders and contact information, a single request can retrieve all related data. This contrasts sharply with REST, where multiple API calls would be necessary.

GraphQL mutation capabilities further extend its functionality, allowing clients to modify data using the same expressive query language. This unified approach to both reading and writing data represents a significant departure from REST's verb-based HTTP methods.

Historical Context

The evolution from SOAP protocol vs REST to modern GraphQL reflects changing application needs. REST APIs have revolutionized how computer systems communicate over the internet, providing a secure, scalable interface that follows specific architectural rules. However, as applications became more sophisticated and mobile-first, the limitations of REST's fixed data structures became apparent.

GraphQL emerged as a response to these challenges, particularly the over-fetching and under-fetching problems inherent in REST architectures. While REST remains excellent for many use cases, GraphQL's client-driven approach addresses specific pain points in modern application development.

Key Differences: When to Choose GraphQL vs REST API

The choice between GraphQL and REST involves understanding fundamental differences in how each approach handles data fetching, performance optimization, and development workflows.

Data Fetching Approaches

REST uses multiple endpoints for each resource, requiring separate HTTP calls for different data types. A typical REST implementation might require:

GET /api/users/123
GET /api/users/123/orders
GET /api/users/123/profile

This multi-request pattern often leads to over-fetching (receiving unnecessary data) or under-fetching (requiring additional requests). In contrast, GraphQL allows clients to specify exactly what data they need in a single request:

query {
  user(id: 123) {
    name
    email
    orders {
      id
      total
      items {
        name
        price
      }
    }
  }
}

Performance Considerations

Performance characteristics vary significantly between approaches. RESTful APIs excel in scenarios where caching is crucial, as HTTP caching mechanisms are well-established and widely supported. The stateless nature of REST makes it highly scalable for simple operations.

GraphQL shines in bandwidth-constrained environments, particularly mobile applications. By fetching only required data, GraphQL can reduce payload sizes by 30-50% compared to equivalent REST implementations. However, this efficiency comes with increased server-side complexity, as resolvers must efficiently handle arbitrary query combinations.

Development Experience

REST's simplicity makes it accessible to developers at all skill levels. The HTTP-based approach aligns naturally with web development patterns, and debugging tools are mature and widely available. RESTful API documentation follows established conventions, making integration straightforward.

GraphQL offers powerful introspection capabilities and schema-first development, but requires a steeper learning curve. The strongly-typed schema provides excellent developer experience through auto-completion and compile-time validation, but teams must invest time in understanding GraphQL-specific concepts like resolvers, fragments, and query optimization.

Scalability Factors

REST is well-suited for microservices architectures, where each service exposes functionality through well-defined APIs. The stateless nature of RESTful services makes horizontal scaling straightforward, and load balancing strategies are well-established.

GraphQL presents unique scalability challenges in distributed systems. Query complexity can vary dramatically, making resource planning difficult. Advanced GraphQL implementations require sophisticated caching strategies and query analysis to prevent performance degradation.

Technical Implementation: REST vs GraphQL in Practice

Understanding the practical implementation details of both approaches helps developers make informed decisions about which technology best fits their specific requirements.

REST API Implementation Patterns

RESTful API implementation follows well-established patterns centered around HTTP methods and resource-based URLs. A typical REST API for user management might include:

GET    /api/users           # List all users
POST   /api/users           # Create new user
GET    /api/users/123       # Get specific user
PUT    /api/users/123       # Update user
DELETE /api/users/123       # Delete user

This approach leverages HTTP's built-in semantics, making RESTful APIs intuitive for developers familiar with web protocols. Status codes provide clear communication about operation results, and stateless communication ensures scalability.

Versioning in REST typically involves URL-based strategies (/v1/users, /v2/users) or header-based approaches. While this can lead to API proliferation, it provides clear backward compatibility guarantees.

GraphQL Implementation Essentials

GraphQL implementation begins with schema definition, establishing the contract between client and server:

type User {
  id: ID!
  name: String!
  email: String!
  orders: [Order!]!
}

type Order {
  id: ID!
  total: Float!
  createdAt: String!
}

type Query {
  users: [User!]!
  user(id: ID!): User
}

type Mutation {
  createUser(name: String!, email: String!): User!
}

GraphQL mutation operations provide a structured approach to data modification, maintaining the same expressive power as queries. Resolvers handle the actual data fetching logic, allowing for flexible backend integration.

Security Considerations

Both approaches require careful security implementation, but with different focus areas. RESTful APIs benefit from standard HTTP security practices: authentication headers, CORS policies, and input validation at the endpoint level.

GraphQL introduces unique security challenges, particularly around query complexity and depth limiting. Malicious clients could potentially craft expensive queries that strain server resources. Implementing query complexity analysis, depth limiting, and timeout mechanisms becomes crucial for GraphQL security.

Error Handling and Monitoring

REST relies on HTTP status codes for error communication, providing a standardized approach that integrates well with existing monitoring tools. Error responses follow predictable patterns, making debugging straightforward.

GraphQL uses a different error model, where HTTP status is typically 200 even for errors, with actual error information embedded in the response payload. This approach requires specialized monitoring tools and error handling strategies but provides more detailed error context.

API Gateway Management: Optimizing GraphQL and REST APIs

Modern API management requires sophisticated gateway solutions that can handle both REST and GraphQL effectively. API gateways serve as the critical infrastructure layer that enables organizations to manage, secure, and optimize their API ecosystems regardless of the underlying architecture.

Managing RESTful APIs with API Gateway

RESTful APIs integrate naturally with traditional API gateway patterns. Standard gateway features like route configuration, load balancing, and protocol translation work seamlessly with REST's resource-based approach. Caching strategies are particularly effective with RESTful services, as the predictable URL patterns and HTTP semantics enable sophisticated caching policies.

API gateways excel at transforming REST requests and responses, enabling legacy system integration and API evolution without breaking existing clients. Rate limiting and throttling policies can be applied at the resource level, providing granular control over API consumption.

GraphQL API Gateway Integration

GraphQL presents unique challenges and opportunities for API gateway integration. Modern gateways like API7 provide GraphQL-specific features including schema stitching, query complexity analysis, and GraphQL-to-REST transformation capabilities.

Query complexity analysis becomes crucial for protecting backend services from expensive operations. API gateways can implement sophisticated policies that evaluate query depth, field count, and estimated execution time before forwarding requests to GraphQL servers.

Schema federation support allows organizations to compose multiple GraphQL services into a unified API surface, with the gateway handling query planning and execution across distributed services.

Unified API Management Approach

Leading API gateway solutions support multi-protocol environments, enabling organizations to manage both RESTful APIs and GraphQL services through a single management plane. This unified approach provides consistent authentication, authorization, monitoring, and analytics across all API types.

Developer portal integration becomes particularly valuable in mixed environments, as it can generate documentation and provide testing interfaces for both REST endpoints and GraphQL schemas. This consistency improves developer experience and reduces onboarding complexity.

Performance Optimization Techniques

API gateways enable sophisticated performance optimization for both API types. Intelligent caching can be applied to GraphQL queries based on query fingerprinting and field-level cache policies. For RESTful APIs, traditional HTTP caching mechanisms provide excellent performance benefits.

Request and response transformation capabilities allow gateways to optimize data formats, compress payloads, and aggregate multiple backend calls into single client responses. Global load balancing and failover mechanisms ensure high availability for both GraphQL and REST services.

Making the Right Choice: Decision Framework and Future Trends

Selecting between GraphQL and REST requires a structured evaluation of technical requirements, team capabilities, and long-term strategic goals. Rather than viewing this as a binary choice, successful organizations often adopt hybrid approaches that leverage the strengths of both paradigms.

Decision Criteria Matrix

Project requirements should drive the technology choice. RESTful APIs excel in scenarios requiring:

Simple CRUD operations with well-defined resources
Heavy caching requirements
Integration with existing HTTP-based infrastructure
Team familiarity with web standards
Microservices architectures with clear service boundaries

GraphQL provides advantages when projects involve:

Complex data relationships and nested queries
Mobile applications with bandwidth constraints
Rapidly evolving client requirements
Multiple client types with different data needs
Real-time features requiring subscription support

Use Case Scenarios

Enterprise applications often benefit from REST's maturity and simplicity. E-commerce platforms, content management systems, and traditional web applications typically align well with RESTful service patterns. The predictable structure and extensive tooling ecosystem make REST an excellent choice for teams building standard business applications.

GraphQL shines in scenarios requiring flexible data access patterns. Social media platforms, analytics dashboards, and mobile applications often see significant benefits from GraphQL's precise data fetching capabilities. When you need to execute a GraphQL query to get the number of customers along with their transaction history and preferences, the single-request efficiency becomes invaluable.

Future Outlook and Trends

The API landscape continues evolving, with both REST and GraphQL finding distinct niches. REST maintains strong adoption in enterprise environments, while GraphQL usage grows in frontend-driven applications and mobile development.

Emerging trends include hybrid approaches where REST APIs serve as data sources for GraphQL gateways, providing the best of both worlds. API gateway evolution increasingly focuses on protocol translation and unified management capabilities.

Industry adoption data shows continued growth for both approaches, suggesting that the future involves coexistence rather than replacement. Organizations are increasingly adopting API-first strategies that can accommodate multiple paradigms based on specific use case requirements.

Conclusion and Recommendations

The GraphQL vs REST debate oversimplifies what should be a nuanced technical decision. Both approaches offer distinct advantages, and the optimal choice depends on specific project requirements, team expertise, and organizational constraints.

RESTful APIs remain the gold standard for simple, cacheable, and well-understood interaction patterns. Their alignment with HTTP semantics, mature tooling ecosystem, and widespread developer familiarity make them an excellent default choice for many applications.

GraphQL provides compelling advantages for applications requiring flexible data access, precise resource utilization, and rapid iteration. The investment in learning GraphQL concepts pays dividends in scenarios where its strengths align with project needs.

The most successful API strategies often involve thoughtful integration of both approaches, leveraged through sophisticated API gateway solutions that can manage, secure, and optimize diverse API ecosystems. As API management continues evolving, the ability to support multiple paradigms becomes increasingly valuable for maintaining architectural flexibility and meeting diverse client requirements.

Rather than asking "which is better," developers should ask "which approach best serves my specific requirements?" The answer will vary based on context, but understanding the strengths and limitations of both GraphQL and REST enables informed decisions that drive successful API implementations.

Manage User Permissions Effortlessly Using API7-MCP

Yilia — Tue, 20 May 2025 10:21:11 +0000

Introduction

As large language model (LLM) applications experience explosive growth, a pivotal challenge emerges: how can these models transcend mere dialogue boxes to interact seamlessly with our daily files, applications, and web services? Addressing this, Anthropic—the developer behind Claude—officially launched and open-sourced the Model Context Protocol (MCP) in late 2024.

MCP offers a standardized method enabling AI models to securely and controllably connect with and operate external data sources and tools, such as accessing files, querying databases, and invoking APIs. This breakthrough dismantles the traditional isolation of models, significantly expanding AI's capabilities—from a conversational assistant to a hands-on helper capable of executing more specific and complex tasks.

How API7-MCP Enhances API7 Enterprise

Keeping pace with this trend, API7.ai introduced API7-MCP. Leveraging MCP's robust capabilities, API7-MCP facilitates effortless and rapid integration into the LLM ecosystem, further simplifying numerous complex and tedious configuration processes within API7 Enterprise.

This article delves into how to utilize API7-MCP to configure user roles and permissions through natural language, showcasing its powerful functionalities via practical use cases.

Overview of Permission Management Features

Query and edit user roles, assessing user permission risks.
Perform CRUD (Create, Read, Update, Delete) operations on roles.
Perform CRUD operations on permissions and query permission configuration rules.

These features assist users in promptly identifying and addressing permission risks, effectively constructing, adjusting, and managing the entire permission system, ensuring the security and rationality of system permissions.

In this article, we demonstrate using the scenario of configuring personnel permissions for a new-launched business system. In real-world applications, the above functionalities can be flexibly combined to meet actual needs.

Use Case: Permission Configuration for New Business System Launch

Background

Assume an enterprise internally launches a business system named "Intelligent Customer Relationship Management System" (abbreviated as "iCRM"). The system administrator needs to add a new role, "iCRM admin" (responsible for the comprehensive management and maintenance of the iCRM system), and assign this role to the user Tom. Let's achieve this effortlessly using API7-MCP.

Prerequisites

Install API7 Enterprise.
Create a user Tom and icrm gateway group within API7 Enterprise.
Configure API7-MCP in the AI client (here we combine VS Code with the Cline plugin as the AI client).

Steps

1.Input the following request in the Cline dialog box:

"Add a new role 'iCRM admin', which can manage all resources under the icrm gateway group. After creating the role, write and bind a permission policy to it, and assign this role to user Tom."

2.Cline requests to obtain Tom's user ID. Click "Approve" to authorize it.

3.Cline requests to create a permission policy that allows full access to the icrm gateway group. Click "Approve" to authorize it.

4.Cline requests to create the role iCRM admin and attach the newly created permission policy to it. Click "Approve" to authorize it.

5.After successfully creating the role, Cline requests to assign the iCRM admin role to user Tom. Click "Approve" to authorize it.

6.Task completed. The "iCRM admin" role and corresponding permission policy have been successfully created and assigned to user Tom.

Verify

Confirm Role Creation

The custom role "iCRM admin" has been created, described as "Role with permissions to manage all resources under icrm gateway group."

This role has been attached to the permission policy icrm_full_access.

Confirm Permission Policy Creation

Reviewing the permission policy, it allows access to all resources under the icrm gateway group.

Confirm User Role Update

User Tom has been updated from having no role to being assigned the iCRM admin role.

Conclusion

API7-MCP introduces flexibility and security to API management through natural language-based permission configuration, effectively eliminating the complexities of traditional permission management. By leveraging the MCP protocol, users can achieve efficient API management with API7 Enterprise at a lower cost.

The scenario-based example of the iCRM system demonstrates that API7-MCP can adapt to most permission management scenarios. It focuses on building permission architectures while also emphasizing dynamic adjustments to permission policies. Through natural language interactions, it integrates seamlessly into business scenarios, achieving a fusion of AI and business processes. This approach not only reduces the technical costs of enterprise permission management but also builds a scalable API security ecosystem through the standardized MCP protocol.

From stdio to HTTP SSE: Host Your MCP Server with APISIX API Gateway

Yilia — Mon, 21 Apr 2025 10:24:20 +0000

Introduction

In contemporary API infrastructure, HTTP protocols and streaming communications (like SSE, WebSocket) have become mainstream for building real-time, interactive applications. Over the past few months, the Model Context Protocol (MCP) has gained popularity. However, most MCP Servers are implemented via stdio for local environments and cannot be invoked by external services and developers.

To bridge these services with modern API architectures, Apache APISIX has introduced the mcp-bridge plugin. It seamlessly converts stdio-based MCP services into HTTP SSE streaming interfaces and manages them through an API gateway for routing and traffic management.

Model Context Protocol (MCP) Overview

MCP is an open protocol that standardizes how AI applications provide context information to large language models (LLMs). It allows developers to switch between different LLM providers while ensuring data security and facilitating integration with local or remote data sources. Supporting a client-server architecture, MCP servers expose specific functionalities that are accessible to clients via these servers.

What Is the `mcp-bridge` Plugin?

The Apache APISIX mcp-bridge plugin launches a subprocess to manage the MCP Server, takes over its stdio channel, transforms client HTTP SSE requests into MCP protocol calls, and pushes responses back to the client via SSE.

Key features:

📡 Wraps MCP RPC calls into SSE message streams
🔄 Manages subprocess stdio lifecycle with queued RPC scheduling
🗂️ Lightweight MCP session management (including session ID, ping keep-alive, and queuing)
🧰 Supports session sharing across multiple workers for stability in APISIX multi-worker environments

How It Works and Architecture Diagram

Below is a sequence diagram illustrating the working mechanism of the mcp-bridge plugin, helping you to understand the data flow from stdio to SSE:

✅ Highlights:

APISIX manages SSE long-lived connections
The mcp-bridge plugin handles subprocesses, stdio, and scheduling queues
Clients receive real-time subprocess outputs, forming streaming SSE responses

Application Scenarios and Benefits

✅ Typical Application Scenarios

🛠️ Integrating existing MCP/stdio services with web platforms
🖥️ Cross-language and cross-platform subprocess service management

✅ Benefits

🌐 Modernization: Instantly transform stdio services into HTTP SSE APIs
🕹️ Managed: Unified management of subprocess launch and IO lifecycle
📈 Scalability: Session sharing in multi-worker environments for large-scale deployment support
🔄 Traffic Control Integration: Seamless API management system integration with APISIX traffic control, authentication, and rate-limiting plugins

Authentication and Rate Limiting with Apache APISIX Plugins

Apache APISIX provides robust authentication plugins (like OAuth 2.0, JWT, and OIDC) and rate-limiting plugins (such as rate limiting and circuit breakers). These enhance the mcp-bridge plugin, ensuring secure authentication and traffic control for connected MCP services.

Authentication Plugins

Support for OAuth 2.0, JWT, and OIDC plugins to protect APIs and MCP services.
Automatic client identity verification during API gateway requests to prevent unauthorized access.

Rate-Limiting Plugins

Rate Limiting: Restricts each client's request rate to prevent system overload.
Circuit Breaker: Automatically switches or returns errors to avoid system crashes during high traffic or failures.

Adding Authentication and Rate Limiting to MCP Servers

By integrating authentication and rate-limiting plugins with the mcp-bridge plugin, you can enhance API security and ensure system stability in high-concurrency environments.

Roadmap

The current version is a prototype. Future enhancements include:

Currently, MCP sessions are not shared across multiple APISIX instances. For multi-node APISIX clusters, proper session persistence configuration on the front-end load balancer is essential to ensure requests from the same client always go to the same APISIX instance.
The current MCP SSE connection is loop-driven. While the loop doesn't consume many resources (stdio read/write will be synchronous non-blocking calls), it's not efficient. We plan to connect to a message queue for an event-driven, scalable cluster approach.
The MCP session management module is just a prototype. We intend to abstract an MCP proxy server module to support launching MCP servers within APISIX for advanced scenarios. This proxy server module will be event-driven rather than loop-driven.

Summary

The Apache APISIX mcp-bridge plugin significantly simplifies the integration of Model Context Protocol (MCP) services with the HTTP API world. It offers a modern streaming interface management approach for traditional services.

APISIX-MCP: Embracing Intelligent API Management with AI + MCP

Yilia — Wed, 02 Apr 2025 03:03:57 +0000

This article introduces the MCP protocol and its application in APISIX-MCP. APISIX-MCP simplifies API management through natural language interaction, supporting the creation, updating, and deletion of resources.

Preface

With the explosive growth of large-scale AI model applications, many traditional systems are eager to integrate AI capabilities quickly. However, the current landscape of AI tools lacks unified standards, resulting in severe fragmentation. Different models vary in capability and integration methods, creating significant challenges for traditional applications during adoption.

Against this backdrop, in late 2024, Anthropic—the company behind the renowned Claude model—introduced the Model Context Protocol (MCP). MCP positions itself as the "USB-C interface" for AI applications. Just as USB-C standardizes connections for peripherals and accessories, MCP provides a standardized approach for AI models to connect with diverse data sources and tools.

Numerous services and applications have already adopted MCP. For example:

GitHub-MCP enables natural language code submissions and PR creation.
Figma MCP allows AI to generate UI designs directly.
With Browser-tools-MCP, tools like Cursor can debug code by interacting with DOM elements and console logs.

The official MCP repository includes implementations for Google Drive, Slack, Git, and various databases. As an open standard, MCP has gained widespread recognition in the AI community, attracting third-party developers who contribute hundreds of new MCP services daily. Anthropic, as the founder, actively drives MCP’s evolution by refining the protocol and educating developers.

About APISIX-MCP

The rise of MCP offers traditional applications a new technical pathway. Leveraging MCP’s standardized integration capabilities, we developed APISIX-MCP, which bridges large language models with Apache APISIX’s Admin API through natural language interaction. The current implementation supports the following operations:

General Operations

get_resource: Retrieve resources by type (routes, services, upstreams, etc.).
delete_resource: Delete resources by ID.

API Resource Management

create_route/update_route/delete_route: Manage routes.
create_service/update_service/delete_service: Manage services.
create_upstream/update_upstream/delete_upstream: Manage upstreams.
create_ssl/update_ssl/delete_ssl: Manage SSL certificates.
create_or_update_proto: Manage Protobuf definitions.
create_or_update_stream_route: Manage stream routes.

Plugin Operations

get_all_plugin_names: List all available plugins.
get_plugin_info/get_plugins_by_type/get_plugin_schema: Fetch plugin configurations.
create_plugin_config/update_plugin_config: Manage plugin configurations.
create_global_rule/update_global_rule: Manage global plugin rules.
get_plugin_metadata/create_or_update_plugin_metadata/delete_plugin_metadata: Manage plugin metadata.

Security Configuration

get_secret_by_id/create_secret/update_secret: Manage secrets.
create_or_update_consumer/delete_consumer: Manage consumers.
get_credential/create_or_update_credential/delete_credential: Manage consumer credentials.
create_consumer_group/delete_consumer_group: Manage consumer groups.

How to Use APISIX-MCP

APISIX-MCP is now open-sourced and available on npm and GitHub. It can be configured via any MCP-compatible AI client, such as Claude Desktop, Cursor, or the Cline plugin for VSCode.

Below is a step-by-step guide using Cursor:

Open Cursor, click the settings icon, and navigate to the settings page.

Click "Add new global MCP server" to edit the mcp.json configuration file:

   {
     "mcpServers": {
       "apisix-mcp": {
         "command": "npx",
         "args": ["-y", "apisix-mcp"],
         "env": {
           "APISIX_SERVER_HOST": "your-apisix-server-host",
           "APISIX_ADMIN_API_PORT": "your-apisix-admin-api-port",
           "APISIX_ADMIN_API_PREFIX": "your-apisix-admin-api-prefix",
           "APISIX_ADMIN_KEY": "your-apisix-api-key"
         }
       }
     }
   }

In the mcpServers field of the configuration file, add a service apisix-mcp, which can be changed. Then configure the commands for running the MCP service.

command: npx (Node.js package executor).
args: -y (auto-install dependencies) and apisix-mcp (package name).
env: Customize APISIX connection settings (defaults below):

In the env field, you can specify the APISIX service access address, Admin API port, prefix, and authentication key. These environment variables have default values, so if you start APISIX without any custom configuration, you can omit the env field entirely. The default values for each variable are as follows:

| Variable | Description | Default Value |

|---------------------------|--------------------------------------|-----------------------------|

| APISIX_SERVER_HOST | APISIX server host | http://127.0.0.1 |

| APISIX_ADMIN_API_PORT | Admin API port | 9180 |

| APISIX_ADMIN_API_PREFIX | Admin API prefix | /apisix/admin |

| APISIX_ADMIN_KEY | Admin API authentication key | edd1c9f034335f136f87ad84b625c8f1 |

Upon successful configuration, the MCP Servers list will show a green indicator for apisix-mcp, along with available tools.

Note: If setup fails, refer to the APISIX-MCP GitHub documentation for manual builds.

In the chat panel, select Agent mode and choose a model (e.g., Claude Sonnet 3.5/3.7 or GPT-4o).

Next, we can enter relevant operational commands to verify if the MCP service is functioning correctly. Following the workflow in APISIX's Getting Started documentation, we input the following into the dialog box and send the message:

"Help me create a route with path /api for accessing https://httpbin.org upstream, with CORS and rate-limiting plugins. Print the route details after configuration."

Next, in Cursor, you will see a process similar to the MCP tool invocation demonstrated in the video below. Due to the inherent randomness of large AI model responses, the exact operations performed may vary from the example shown.

Here, the auto-execution mode (YOLO Mode) is enabled, allowing Cursor to automatically invoke all tools in the MCP server. From the video, we can observe the AI performing the following operations based on our requirements:

Analyzing the plugins we need to configure, then calling get_plugins_list to retrieve all plugin names
Invoking get_plugin_schema to examine detailed configuration information for different plugins
Calling create_route to create the route
Using update_route to add the previously queried plugin configurations to the route
Executing get_route to verify whether the route was successfully configured and if the configuration is correct

The resulting route configuration includes:

Route ID: httpbin
Path: /api/*
Methods: GET, POST, PUT, DELETE, PATCH, HEAD, OPTIONS

CORS Plugin:

allow_origins: *
allow_methods: *
allow_headers: *
expose_headers: X-Custom-Header
max_age: 3600
allow_credential: false

limit-count Plugin:

count: 100
time_window: 60
key: remote_addr
rejected_code: 429
policy: local

Upstream:

type: roundrobin (load balancing strategy using round-robin)  
upstream node: httpbin.org:443 (backend service address)

Advantages of AI-Driven Operations

In the above process, we accomplished the creation of a route configured with CORS and rate-limiting through just one round of natural language interaction with AI. Compared to manual route configuration, leveraging AI offers several distinct advantages:

Reduced Cognitive Load: Eliminates manual documentation lookup and parameter memorization.
Automated Workflows: AI decomposes tasks (e.g., plugin setup → route creation) without human intervention.
Closed-Loop Validation: Auto-verification ensures correctness.
Iterative Optimization: Continuous dialogue refines configurations.

This interaction model transforms complex configuration processes into natural conversational experiences while maintaining accuracy and verifiability. These capabilities are achieved through the MCP protocol's semantic parsing of requirements, intelligent tool invocation, and final execution via Admin API.

It's important to note that APISIX-MCP isn't designed to completely replace manual configuration, but rather to optimize efficiency for high-frequency operations. Its value shines particularly in configuration debugging and rapid validation scenarios, creating effective complementarity with traditional management approaches. As the MCP ecosystem continues to evolve, we can anticipate deeper integration of such tools in API management, promising more sophisticated capabilities.

Conclusion

MCP enables intelligent operations for complex API systems. APISIX-MCP lowers the barrier to Apache APISIX adoption, with future plans for AI-traffic-specific plugins. The fusion of AI and API management promises smarter, more efficient infrastructure governance.

What Is an AI Gateway: Differences from API Gateway

Yilia — Fri, 28 Mar 2025 03:26:32 +0000

"The future isn't AI gateways—it's API gateways that speak AI."_ This blog explores AI gateways, their differences from API gateways, and why evolved solutions like Apache APISIX AI Gateway are shaping the future.

What Is an AI Gateway? Why Did It Arise in the AI Era?

The AI era has ushered in unprecedented complexity in deploying and managing artificial intelligence (AI) models. Organizations now juggle multiple models—from computer vision to large language models (LLMs)—across diverse environments (cloud, edge, hybrid). Traditional API gateways, designed for general-purpose data traffic, often fall short in addressing the unique challenges posed by AI workloads. This is where AI gateways emerge as critical middleware, acting as a unified control plane for routing, securing, and optimizing AI workloads.

The Rise of AI Gateways

The proliferation of generative AI and LLMs (Large Language Models) has introduced unique challenges:

Token Consumption: LLMs process requests in tokens, requiring granular tracking for cost and performance optimization.
Stream-Type Requests: AI agents often generate real-time, streaming responses (e.g., ChatGPT's incremental output), demanding low-latency handling.
Tool Integration: AI systems increasingly rely on external data sources and APIs (e.g., retrieving live weather data or CRM records).

According to a 2023 Gartner report, over 75% of enterprises now use AI models in production, driving demand for specialized infrastructure. Traditional API gateways, designed for RESTful APIs and static request-response cycles, struggle with these AI-specific demands. Enter the AI gateway—a purpose-built solution to manage AI-native traffic.

AI Agents vs. Traditional Devices: Why Stream-Type Requests Demand Specialized Handling

AI agents (e.g., chatbots, coding assistants) generate fundamentally different traffic patterns than traditional clients:

Metric	Traditional API Requests	AI Agent Requests
Request Type	Synchronous (HTTP GET/POST)	Asynchronous, streaming (SSE)
Latency	Milliseconds	Seconds-minutes (for chunks)
Billing	Per API call	Per token or compute time
Failure Modes	Timeouts, HTTP errors	Partial completions, hallucinations

The Stream-Type Challenge

When an AI agent requests a poem generated by GPT-4, the response is streamed incrementally. Traditional API gateways, built for atomic requests, struggle with:

Partial Responses: Aggregating chunks into a coherent audit log.
Token Accounting: Accurately counting tokens across streaming chunks.
Real-Time Observability: Monitoring latency per token or detecting drift in response quality.

Many purpose-built AI gateways lack distributed tracing, forcing engineers to cobble together metrics. In contrast, API gateways like Apache APISIX provide built-in integrations with Prometheus and Grafana, enabling token-level dashboards.

Two Types of AI Gateways: Purpose-Built vs. API Gateway Evolutions

Today's AI gateways fall into two categories:

Specific Purpose-Built AI Gateways

These are built from the ground up to address AI use cases. Startups like PromptLayer and LangChain offer solutions focused on:

Token-Based Rate Limiting: Enforcing usage quotas based on tokens instead of API calls.
Prompt Engineering Tools: Allowing developers to test and optimize prompts.
AI-Specific Analytics: Tracking metrics like response hallucination rates or token costs.

Example: OpenAI's API uses token-based pricing ($0.06 per 1K tokens for GPT-4), requiring gateways to meter usage precisely. A dedicated AI gateway might integrate token counters directly into its throttling logic.

However, these gateways often lack the observability and scalability of mature API management platforms. For instance, measuring token consumption across distributed microservices can lead to inaccuracies if the gateway lacks distributed tracing capabilities.

Evolved AI Gateways from API Gateways

Established API gateways like Kong, Apache APISIX, and AWS API Gateway are adapting to AI workloads by adding:

Streaming Support: Handling Server-Sent Events (SSE) and WebSockets for real-time AI responses.
Token-Aware Plugins: Extending rate-limiting plugins to track tokens.
LLM Orchestration: Managing multiple AI models (e.g., routing requests to cost-effective models like Mistral-7B for simple tasks).

Mature API gateways leverage decades of experience in security (OAuth, JWT), scalability (load balancing), and monetization—features often missing in AI-first solutions.

Why Evolved AI Gateways Are Winning Long-Term

While purpose-built AI gateways excel in niche scenarios, evolved API gateways are becoming the default choice for three reasons:

Cost Efficiency: Maintaining separate gateways for AI and non-AI traffic doubles operational overhead. Converged systems reduce costs by 30–50% (Gartner, 2023).
Flexibility: Enterprises can't predict which AI models will dominate. Platforms like Apache APISIX allow seamless integration of new LLMs without rearchitecting.
Future-Proofing: As AI becomes embedded in all apps (e.g., AI-powered search in e-commerce), gateways must handle hybrid workloads.

Model Context Protocol (MCP): Bridging AI Assistants and External Tools

To connect AI agents with external data and APIs, the Model Context Protocol (MCP) has emerged as a standardized framework. MCP defines how AI models request and consume external resources, such as:

Data Sources: SQL databases, vector stores (e.g., Pinecone).
APIs: CRM systems, payment gateways.
Tools: Code interpreters, and image generators.

How MCP Works

Context Injection: An AI assistant sends a request with a context header specifying required tools (MCP-Context: weather_api, crm).
Gateway Routing: The AI gateway validates permissions, injects API keys, and routes the request to relevant services.
Response Synthesis: The gateway aggregates API responses (e.g., weather data + CRM contacts) and feeds them back to the AI model.

Example: A user asks, "Email our top client in NYC about today's weather." The AI gateway uses MCP to:

Fetch the top client from Salesforce.
Retrieve NYC weather from OpenWeatherMap.
Pass this context to GPT-4 to draft the email.

Benefits of MCP

Security: Centralized policy enforcement (e.g., masking PII in CRM responses).
Cost Control: Caching frequent data requests (e.g., product catalogs).
Interoperability: Standardizing AI-to-API communication across vendors.

Future of AI Gateways: Convergence with API Monetization

As AI adoption matures, two trends will shape AI gateways:

Trend 1: The Decline of Standalone AI Gateways

Niche AI gateways will struggle to compete with evolved API gateways that offer:

Unified Governance: One platform for REST, GraphQL, and AI APIs.
Monetization Models: Token-based billing, subscription tiers.
Enterprise Features: Role-based access control (RBAC), audit logging.

Under such a trend, AI traffic will flow through traditional API gateways enhanced with AI capabilities.

Trend 2: API Gateways as AI Orchestrators

Future API gateways will act as AI orchestrators, handling:

Model Routing: Directing requests to optimal models based on cost, latency, or accuracy.
Hybrid Workflows: Blending AI and non-AI services (e.g., validating a GPT-4 response against a database).
Token Analytics: Real-time dashboards showing token spend by team or project.

The Bottom Line

In the future, the line between "AI gateway" and "API gateway" will blur. But the unchangeable fact is APIs are the basics of API gateways and AI gateways. Companies that adopt AI-ready API gateways today will gain a strategic edge in scalability, cost control, and innovation.

Conclusion: Embracing AI-API Convergence

AI gateways are not a replacement but an evolution of API gateways. While purpose-built solutions address immediate LLM challenges, their limitations in observability and scalability make them transitional. Established API gateways—enhanced with streaming support, token-aware plugins, and MCP—are poised to dominate.

Solutions like Apache APISIX AI Gateway exemplify this shift, blending AI-native features with battle-tested API management. As AI permeates every app, enterprises must choose platforms that scale beyond siloed use cases. The winners? Adaptable, extensible tools that speak both API and AI.

DEV Community: Yilia

TLS Key Exchange for API Gateway Origins: Cut Handshake Retries Safely

Key Takeaways

Understand the Two TLS Boundaries

Why HelloRetryRequest Changes Latency

Post-Quantum Readiness Is a Compatibility Program

Connection Reuse Is the First Performance Control

Build an Origin TLS Dashboard

A Safe Rollout Checklist

Conclusion

HTTP Connection Reuse: The Hidden API Latency Multiplier

Key Takeaways

Why an Unread Body Can Discard a Healthy Connection

Map the Two Connection Pools

Diagnose Churn Before Tuning Keep-Alive

Configure the Gateway Side Deliberately

Design API Responses for Predictable Consumption

A Safe Rollout Checklist

Conclusion

Bot Traffic Moves: Build API Defenses That Follow Behavior

Key Takeaways

What the Yandex Event Actually Showed

Build Controls That Survive an IP Change

Rate-Limit by the Resource You Need to Protect

Understand Local and Shared Counter Boundaries

Use Behavior for Detection, Identity for Enforcement

Separate Gateway Policy from the Other Defense Layers

Test the Migration Counterexample

Conclusion

Release Apache APISIX Ingress Controller 2.0

Highlights of Apache APISIX Ingress Controller 2.0

Support Gateway API

Introduce Gateway API Extensions

Support APISIX Standalone API-Driven Mode

Support Multi-Data-Plane Deployment Mode

Admin API Mode

Standalone Mode

Conclusion

Building an AI Agent Traffic Management Platform: APISIX AI Gateway in Practice

Introduction: The Turning Point from Dispersed Traffic to Intelligent Governance

Challenges: The Complexity of Multi-Model, Multi-Tenant, Hybrid Cloud

1. Stability Assurance

2. Multi-tenant Isolation

3. Intelligent Scheduling

System Design: Core Architecture of the AI Gateway

Key Selection Criteria for AI Gateways

Fine-Grained AI Traffic Governance and Multi-tenant Isolation

Scenario 1: Automatic Fallback for Hybrid Models

Scenario 2: Token-Based Rate Limiting

Value Delivered by APISIX AI Gateway

1. Simplified Large Model Access, Lowering Usage Barriers

2. Achieved Centralized Resource Management with Service Stability

3. Unified Control with Traffic Security Assurance

AI Gateway Evolution Direction and Outlook

Load Balancing AI/ML API with Apache APISIX

Overview

Quickstart Installation

Secure the Admin API

1. Set an Admin Key

2. Whitelist Management IPs (allow_admin)

3. Restart APISIX

Define Basic Routes for v1 and v2

1. Route for v1

2. Route for v2

Implement Traffic Splitting (50/50)

Verify Split Endpoint Functionality

Load Test & Distribution Validation

Best Practices & Next Steps

References

Announcing APISIX Integration with AI/ML API

Introduction

Proxy to OpenAI via AI/ML API

Prerequisites

Configure the Route

Test the Integration

Verify Response

Core Use Cases

Conclusion

More Resources

GraphQL vs REST API: Which is Better for Your Project in 2025?

Why `HelloRetryRequest` Changes Latency

What Is the `mcp-bridge` Plugin?