Gateway Routing#
The AIBrix gateway is built on Envoy Gateway and acts as the single entry point for all LLM inference requests. It handles dynamic model discovery, request routing, rate limiting, and response streaming. For a deep dive into the internal design, see the AIBrix Router architecture guide.
Dynamic Routing#
When a model or LoRA adapter is deployed, the respective controller automatically creates an HTTPRoute object. The gateway dynamically discovers these routes to forward incoming requests without any manual configuration.
First, retrieve the external IP and port of the Envoy proxy:
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
envoy-aibrix-system-aibrix-eg-903790dc LoadBalancer 10.96.239.246 101.18.0.4 80:32079/TCP 10d
envoy-gateway ClusterIP 10.96.166.226 <none> 18000/TCP,18001/TCP,18002/TCP,19001/TCP 10d
In most Kubernetes setups, LoadBalancer is supported by default. Capture the external IP:
LB_IP=$(kubectl get svc/envoy-aibrix-system-aibrix-eg-903790dc -n envoy-gateway-system -o=jsonpath='{.status.loadBalancer.ingress[0].ip}')
ENDPOINT="${LB_IP}:80"
Verify that the HTTPRoute status is Accepted before sending traffic:
$ kubectl get httproute -A
NAMESPACE NAME HOSTNAMES AGE
aibrix-system aibrix-reserved-router 17m # reserved router
aibrix-system deepseek-r1-distill-llama-8b-router 14m # created for each model deployment
....
$ kubectl describe httproute deepseek-r1-distill-llama-8b-router -n aibrix-system
Name: deepseek-r1-distill-llama-8b-router
Namespace: aibrix-system
Labels: <none>
Annotations: <none>
API Version: gateway.networking.k8s.io/v1
Kind: HTTPRoute
Metadata:
Creation Timestamp: 2025-02-16T17:56:03Z
Generation: 1
Resource Version: 2641
UID: 2f3f9620-bf7c-487a-967e-2436c3809178
Spec:
Parent Refs:
Group: gateway.networking.k8s.io
Kind: Gateway
Name: aibrix-eg
Namespace: aibrix-system
Rules:
Backend Refs:
Group:
Kind: Service
Name: deepseek-r1-distill-llama-8b
Namespace: default
Port: 8000
Weight: 1
Matches:
Headers:
Name: model
Type: Exact
Value: deepseek-r1-distill-llama-8b
Path:
Type: PathPrefix
Value: /
Timeouts:
Request: 120s
Status:
Parents:
Conditions:
Last Transition Time: 2025-02-16T17:56:03Z
Message: Route is accepted
Observed Generation: 1
Reason: Accepted
Status: True
Type: Accepted
Last Transition Time: 2025-02-16T17:56:03Z
Message: Resolved all the Object references for the Route
Observed Generation: 1
Reason: ResolvedRefs
Status: True
Type: ResolvedRefs
Controller Name: gateway.envoyproxy.io/gatewayclass-controller
Parent Ref:
Group: gateway.networking.k8s.io
Kind: Gateway
Name: aibrix-eg
Namespace: aibrix-system
Events: <none>
As of v0.5.0, each model’s HTTPRoute is created with a default set of path prefixes (e.g. /v1/chat/completions, /v1/completions). To expose additional custom paths, add the model.aibrix.ai/model-router-custom-paths annotation to your Deployment, ModelAdapter, or RayClusterFleet manifest:
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
model.aibrix.ai/name: deepseek-r1-distill-llama-8b
model.aibrix.ai/port: "8000"
annotations:
model.aibrix.ai/model-router-custom-paths: /version,/score # comma-separated; spaces and empty entries are ignored
name: deepseek-r1-distill-llama-8b
namespace: default
The model name in the request body must match the model.aibrix.ai/name label in your deployment:
curl -v http://${ENDPOINT}/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "deepseek-r1-distill-llama-8b",
"messages": [{"role": "user", "content": "Say this is a test!"}],
"temperature": 0.7
}'
Attention
AIBrix exposes a public endpoint to the internet. Enable authentication to secure it.
For vLLM backends, pass the --api-key argument or set the VLLM_API_KEY environment variable to require an API key in the Authorization header.
See vLLM OpenAI-Compatible Server for details.
After enabling authentication, include the Authorization header in every request:
curl -v http://${ENDPOINT}/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer any_key" \
-d '{
"model": "deepseek-r1-distill-llama-8b",
"messages": [{"role": "user", "content": "Say this is a test!"}],
"temperature": 0.7
}'
Routing Strategies#
The routing strategy controls how the gateway selects the target pod for each request. It can be set per-request via the routing-strategy header, or globally via the ROUTING_ALGORITHM environment variable.
Some strategies rely on metrics queried from the Prometheus HTTP API (PromQL). See Prometheus API Access for configuration.
General load balancing#
random: routes to a randomly selected pod. Suitable as a baseline or when all pods are equivalent.least-request: routes to the pod with the fewest in-flight requests.least-busy-time: routes to the pod with the least cumulative busy processing time.least-latency: routes to the pod with the lowest average processing latency.least-kv-cache: routes to the pod with the smallest KV cache occupancy (least VRAM used).least-gpu-cache: routes to the pod with the lowest GPU cache utilization.least-utilization: routes to the pod with the lowest overall utilization score.throughput: routes to the pod that has processed the fewest total weighted tokens, favoring underloaded pods.power-of-two: applies power-of-two-choices — randomly samples two pods and selects the better one.
KV-cache aware#
prefix-cache: routes to a pod that already holds a KV cache matching the request’s prompt prefix, with load balancing and multi-turn conversation support.prefix-cache-preble: routes considering both prefix cache hits and pod load, based on Preble: Efficient Distributed Prompt Scheduling for LLM Serving.
Fairness#
vtc-basic: routes using a hybrid score that balances per-user token fairness and pod utilization. A simplified variant of the Virtual Token Counter (VTC) algorithm. See VTC-artifact for background.
SLO-aware#
slo: routes with awareness of per-request service-level objectives.slo-pack-load: SLO-aware routing that consolidates load onto fewer pods.slo-least-load: SLO-aware routing that spreads load to the least loaded pod.slo-least-load-pulling: variant ofslo-least-loadthat pulls metrics directly instead of using cached snapshots.
Specialized#
pd: prefill-decode disaggregation routing. Splits processing between dedicated prefill pods and decode pods for optimized end-to-end latency.curl -v http://${ENDPOINT}/v1/chat/completions \ -H "routing-strategy: pd" \ -H "Content-Type: application/json" \ -d '{ "model": "your-model-name", "messages": [{"role": "user", "content": "Say this is a test!"}], "temperature": 0.7 }'
session-affinity: sticky session routing. Encodes the target pod’s address (IP:Port) as a base64 value in thex-session-idresponse header. Subsequent requests that include this header are routed to the same pod. If that pod is no longer available, the gateway transparently fails over to a new pod and issues a fresh session ID.How it works:
On the first request (no
x-session-id), the gateway picks a ready pod and returns anx-session-idresponse header.The client stores and resends this header on follow-up requests (e.g., in multi-turn conversations).
The gateway decodes the session ID to recover the original pod address and routes to it.
If the pod is unavailable (scaled down or evicted), it falls over to a new pod and issues a new session ID.
Note
x-session-idencodes only network location. It is not a security token and must not be used for authentication or authorization.curl -v http://${ENDPOINT}/v1/chat/completions \ -H "routing-strategy: session-affinity" \ -H "Content-Type: application/json" \ -d '{ "model": "your-model-name", "messages": [{"role": "user", "content": "Say this is a test!"}], "temperature": 0.7 }'
To override the strategy for a single request, pass the routing-strategy header with any of the values above:
curl -v http://${ENDPOINT}/v1/chat/completions \
-H "routing-strategy: least-request" \
-H "Content-Type: application/json" \
-d '{
"model": "your-model-name",
"messages": [{"role": "user", "content": "Say this is a test!"}],
"temperature": 0.7
}'
Rate Limiting#
The gateway supports per-user rate limiting on requests per minute (RPM) and tokens per minute (TPM). Pass the user header with a unique identifier to enable rate-limit enforcement for that client.
For details on managing users, see the User Management documentation.
curl -v http://${ENDPOINT}/v1/chat/completions \
-H "user: your-user-id" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer any_key" \
-d '{
"model": "your-model-name",
"messages": [{"role": "user", "content": "Say this is a test!"}],
"temperature": 0.7
}'
Note
If rate limiting is not needed, the user header can be omitted.
External Filter#
The external-filter header restricts the candidate pod pool using Kubernetes label selector syntax before the routing strategy makes its final selection. It is evaluated before routing and only takes effect when a routing-strategy is also set.
Supported selector forms:
key=valuekey in (a, b)key!=valuecomma-separated list of selectors
See the Kubernetes label selector reference for the full syntax.
curl -v http://${ENDPOINT}/v1/completions \
-H "Content-Type: application/json" \
-H "routing-strategy: random" \
-H "external-filter: environment=production,tier=frontend" \
-d '{
"model": "deepseek-r1-distill-llama-8b",
"prompt": "San Francisco is a",
"max_tokens": 128,
"temperature": 0
}'
Note
Filtering happens before the routing strategy and does not change which pod the strategy considers optimal.
external-filteronly takes effect whenrouting-strategyis set.It further narrows down the pods already selected by
model.aibrix.ai/name.If the filter eliminates all pods, the request fails with
no ready pods for routing.external-filteris optional. When omitted, no extra filtering is applied.
Headers Reference#
This section describes the custom headers used in request processing for routing, debugging, and rate limiting.
Target and General Headers#
Header Name |
Description |
|---|---|
|
Unique request ID associated with the client request. Useful for correlating logs. |
|
Indicates whether request headers were processed correctly. Used for debugging header parsing issues. |
|
The destination pod selected by the routing algorithm. Useful for verifying routing decisions. |
|
The routing strategy applied to this request. |
|
Label selector expression used to further filter candidate pods before routing. |
Routing and Error Debugging Headers#
Header Name |
Description |
|---|---|
|
Identifies errors caused by incorrect user input. |
|
Indicates a routing failure, such as being unable to select a target pod. |
|
Signals that the response body could not be parsed, typically due to an internal error. |
|
Generic error header when no specific cause is identified. |
|
Marks a request body parsing failure, such as invalid JSON. |
|
Indicates that no model was specified in the request. |
|
Indicates that the requested model exists but has no active backend pods. |
|
Indicates that an unsupported routing strategy was specified. |
Streaming Headers#
Header Name |
Description |
|---|---|
|
Signals an error during response streaming. |
|
Indicates an incorrect |
|
Indicates whether usage statistics were included in the streaming response. |
Rate Limiting Headers#
Header Name |
Description |
|---|---|
|
Indicates the RPM (requests per minute) counter was updated successfully. |
|
Indicates the TPM (tokens per minute) counter was updated successfully. |
|
Signals that the request exceeded the allowed RPM limit. |
|
Signals that the request exceeded the allowed TPM limit. |
|
Error encountered while incrementing the RPM counter. |
|
Error encountered while incrementing the TPM counter. |
Debugging Guidelines#
Identify error headers — inspect
x-error-routing,x-error-user,x-error-response-unmarshal, andx-error-response-unknownto locate the root cause. For request body issues, checkx-error-request-body-processingandx-error-no-model-in-request.Verify routing and model assignment — confirm
target-podis set. Ifx-error-no-model-in-requestorx-error-no-model-backendsappears, verify the request body includes a valid model name and that the model has active backend pods. Ifx-error-invalid-routing-strategyis present, confirm the strategy name is supported.Diagnose streaming issues — check
x-error-streamingandx-error-stream. If usage statistics are missing from a streaming response, inspectx-error-no-stream-options-include-usage.Investigate rate limiting — if a request was rejected, check
x-error-rpm-exceededorx-error-tpm-exceeded. If counters failed to update, look forx-error-incr-rpmorx-error-incr-tpm. Successful updates are indicated byx-update-rpmandx-update-tpm.
Verify that all gateway objects have status.conditions == Accepted:
kubectl describe gatewayclass -n aibrix-system
kubectl describe gateway -n aibrix-system
kubectl describe envoypatchpolicy -n aibrix-system
# check for all objects
kubectl describe envoyextensionpolicy -n aibrix-system
# check for all objects
kubectl describe httproute -n aibrix-system
Collect logs from the Envoy proxy and the gateway plugin for deeper investigation:
kubectl get pods -n envoy-gateway-system
NAME READY STATUS RESTARTS AGE
envoy-aibrix-system-aibrix-eg-903790dc-84ccfcbc6b-hw2lq 2/2 Running 0 13m
envoy-gateway-7c7659ffc9-rvm5s 1/1 Running 0 16m
kubectl logs envoy-aibrix-system-aibrix-eg-903790dc-84ccfcbc6b-hw2lq -n envoy-gateway-system
kubectl get pods -n aibrix-system
NAME READY STATUS RESTARTS AGE
aibrix-controller-manager-fb4495448-j9k6g 1/1 Running 0 22m
aibrix-gateway-plugins-6bd9fcd5b9-2bwpr 1/1 Running 0 22m
aibrix-gpu-optimizer-df9db96c8-2fctd 1/1 Running 0 22m
aibrix-kuberay-operator-5bf4985d86-7g4tz 1/1 Running 0 22m
aibrix-metadata-service-9d4cd7f77-mq7tr 1/1 Running 0 22m
aibrix-redis-master-7d6b77c794-bcqxc 1/1 Running 0 22m
kubectl logs aibrix-gateway-plugins-6bd9fcd5b9-2bwpr -n aibrix-system
Observability & Telemetry#
The gateway plugins feature built-in support for distributed tracing using OpenTelemetry (OTel). This allows you to trace end-to-end request lifecycles across the external processing pipeline.
Tracing is strictly opt-in by default to conserve system resources. To enable the telemetry components, you need to complete the following two steps:
Step 1: Enable OpenTelemetry in EnvoyProxy#
Enable openTelemetry.enable in your helm chart or add the telemetry configuration directly to your EnvoyProxy resource.
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
name: aibrix-custom-proxy-config
spec:
telemetry:
tracing:
samplingRate: 100
provider:
type: OpenTelemetry
openTelemetry:
# -- The hostname or IP address of the OpenTelemetry Collector/OTLP receiver.
# Do NOT include the protocol scheme (e.g., http://) or port here.
host: "otel-collector.monitoring.svc.cluster.local"
# -- The port of the OTLP gRPC receiver (Envoy strictly uses gRPC for this).
port: 4317
Step 2: Configure the OTLP Exporter Endpoint#
Next, you must explicitly configure an OTLP exporter endpoint in your environment. You can enable it by setting either the global endpoint or the traces-specific endpoint.
For example, if you are running a cluster local OTel Collector, inject the following environment variables:
env:
# Option 1: Global OTLP endpoint (Recommended)
# NOTE: If protocol is HTTP, the SDK automatically appends '/v1/traces' to this URL.
# If protocol is gRPC, it uses the base URL as-is.
- name: OTEL_EXPORTER_OTLP_ENDPOINT
value: "http://otel-collector.monitoring.svc.cluster.local:4317"
# --- OR ---
# Option 2: Traces-specific endpoint (Uncomment to use instead of Option 1)
# The SDK uses this URL EXACTLY as-is without appending any paths.
# (Example below uses port 4318 for HTTP/Protobuf exports)
# - name: OTEL_EXPORTER_OTLP_TRACES_ENDPOINT
# value: "http://otel-collector.monitoring.svc.cluster.local:4318/v1/traces"
# Specifies the transport protocol. Valid options: grpc, http, or http/protobuf
- name: OTEL_EXPORTER_OTLP_PROTOCOL
value: "grpc"
# Keeps TLS active but skips server certificate validation (useful for self-signed certs)
- name: OTEL_EXPORTER_OTLP_INSECURE_SKIP_VERIFY
value: "true"
# Set Headers if your OTel Collector or APM backend (e.g., Datadog, Honeycomb) requires authentication
- name: OTEL_EXPORTER_OTLP_HEADERS
value: "Authorization=Bearer {{ token }}"
Advanced Configuration
For a comprehensive list of supported telemetry configurations—including sampling rates, TLS options and custom headers, please refer to the full configuration guide in: pkg/plugins/gateway/ENV_VARS.md
Config Profiles#
A config profile lets you embed routing settings directly in a model’s pod annotation, so every request to that model picks up the right configuration automatically. You can define multiple named profiles in one annotation and switch between them per-request with a single header.
Setting up a profile
Add the model.aibrix.ai/config annotation to the pod template of your Deployment, StormService, or RayClusterFleet:
annotations:
model.aibrix.ai/config: |
{
"defaultProfile": "default",
"profiles": {
"default": {
"routingStrategy": "least-latency",
"requestsPerSecond": 100
},
"batch": {
"routingStrategy": "throughput"
}
}
}
Selecting a profile at request time
Add the config-profile header. If the header is absent, the defaultProfile is used:
# Use the "batch" profile for this request
curl http://${ENDPOINT}/v1/chat/completions \
-H "config-profile: batch" \
-H "Content-Type: application/json" \
-d '{"model": "my-model", "messages": [{"role": "user", "content": "Summarize..."}]}'
Profile fields
Field |
Description |
|---|---|
|
The routing algorithm for this profile (e.g. |
|
Model-level RPS cap for this profile. Requests that exceed the limit are rejected with HTTP 429. Omit or set to |
|
Algorithm-specific settings as a nested JSON object. Currently used by the |
Routing strategy priority (highest to lowest):
routing-strategyrequest header — always wins, even if a profile is active.routingStrategyfrom the resolved config profile.ROUTING_ALGORITHMenvironment variable on the gateway plugin.
Backward compatibility: if a pod has no model.aibrix.ai/config annotation, the gateway falls back to the model.aibrix.ai/routing-strategy pod label and the ROUTING_ALGORITHM env. No migration is required for existing deployments.
Prometheus API Access#
Some routing strategies rely on metrics queried from the Prometheus HTTP API (PromQL). Configure the endpoint and optional Basic Auth credentials with the following environment variables.
Environment Variable |
Default |
Description |
|---|---|---|
|
(empty) |
Prometheus HTTP API base URL (e.g. |
|
(empty) |
Kubernetes Secret name containing Basic Auth credentials. When set, takes precedence over the plaintext env vars below. |
|
|
Namespace of the Secret specified by |
|
|
Key in |
|
|
Key in |
|
(empty) |
Basic Auth username. Used only when |
|
(empty) |
Basic Auth password. Used only when |
Example (plaintext env vars)#
export PROMETHEUS_ENDPOINT="http://prometheus-operated.prometheus.svc:9090"
export PROMETHEUS_BASIC_AUTH_USERNAME="prom_user"
export PROMETHEUS_BASIC_AUTH_PASSWORD="prom_pass"
Example (Kubernetes Secret)#
apiVersion: v1
kind: Secret
metadata:
name: prometheus-basic-auth
namespace: aibrix-system
type: Opaque
stringData:
username: prom_user
password: prom_pass
export PROMETHEUS_ENDPOINT="http://prometheus-operated.prometheus.svc:9090"
export PROMETHEUS_BASIC_AUTH_SECRET_NAME="prometheus-basic-auth"
export PROMETHEUS_BASIC_AUTH_SECRET_NAMESPACE="aibrix-system"
Message Size Configuration#
Two independent size limits apply to requests flowing through the gateway. Both default to 4 MiB and must be kept in sync when you raise or lower the limit.
Gateway Plugin gRPC Buffer (AIBRIX_GRPC_MAX_MESSAGE_SIZE_BYTES)#
Envoy communicates with the AIBrix gateway plugin over an external-processing (ext_proc) gRPC connection. This env var controls the maximum size of a single gRPC message on that connection. Requests or responses larger than this value are rejected before they reach routing logic.
Configure it under gatewayPlugin.envs in values.yaml:
gatewayPlugin:
envs:
AIBRIX_GRPC_MAX_MESSAGE_SIZE_BYTES: "4194304" # 4 MiB
Envoy Connection Buffer (clientTrafficPolicy.connection.bufferLimit)#
Envoy itself buffers each incoming HTTP connection up to bufferLimit bytes before it begins processing. If a request body exceeds this limit Envoy returns a 413 before the ext_proc filter is even invoked.
Configure it under envoyGateway.clientTrafficPolicy in values.yaml:
envoyGateway:
clientTrafficPolicy:
connection:
bufferLimit: 4194304 # 4 MiB
Keeping the two values in sync#
Because Envoy drops the connection before the gateway plugin sees it when bufferLimit is the binding limit, both values should always be set to the same size. When you need to support larger payloads (e.g. large KV-cache state-sync messages), update both fields together:
gatewayPlugin:
envs:
AIBRIX_GRPC_MAX_MESSAGE_SIZE_BYTES: "16777216" # 16 MiB
envoyGateway:
clientTrafficPolicy:
connection:
bufferLimit: 16777216 # 16 MiB