ModelClaim High-Density GPU Runtime Pools

Warning

ModelClaim is an experimental feature. It currently supports one engine replica per claim, uses a fixed GPU topology per warm pool, and requires a dedicated kvcached runtime image. Validate the feature with your engine, model, GPU, and workload before using it for production traffic.

ModelClaim lets several independently managed model engines share a warm GPU runtime Pod. Instead of creating one Kubernetes Deployment per model, an operator first creates a pool of topology-homogeneous GPU Pods. A user then creates a ModelClaim for each model that should run in the pool.

The ModelClaim controller selects a compatible Pod, asks the AIBrix runtime agent to start a separate engine process, and publishes the engine’s port to the AIBrix gateway only after the engine is ready. The default experimental image contains vLLM; SGLang requires a custom compatible runtime image. The kvcached framework provides elastic KV-cache memory across colocated engines. An optional pool policy can redistribute KV capacity according to observed requests and put idle vLLM engines into sleep mode.

Architecture

        flowchart LR
  MC["ModelClaim"] --> C["ModelClaim controller"]
  C -->|"select Pod and activate"| R["AIBrix runtime agent"]
  subgraph P["Warm GPU Pod"]
    R --> E1["Engine A / port A / IPC A"]
    R --> E2["Engine B / port B / IPC B"]
    E1 <--> K["kvcached physical pages"]
    E2 <--> K
  end
  C -->|"model, port, state annotation"| G["AIBrix gateway"]
  U["OpenAI client"] --> G --> E1
  G -. "sleeping: wake and retryable 503" .-> R
    

The main responsibilities are:

• ModelClaim declares the served model, artifact, pool selector, engine, and engine arguments. It does not request GPUs directly.

• Warm pool Deployment owns the GPU resources and fixes the Pod-visible GPU topology.

• ModelClaim controller performs placement, lifecycle reconciliation, route gating, and the optional pool policy.

• AIBrix runtime agent starts and supervises one process per claim and exposes actual engine, memory, and request state through a runtime snapshot.

• AIBrix gateway routes a served model to its per-engine port. It can trigger a wake for a sleeping model but does not hold the original request.

Prerequisites

You need:

• an existing Kubernetes cluster with NVIDIA GPUs and the NVIDIA device plugin;

• the latest AIBrix nightly control plane from the main branch;

• a kvcached-compatible image for the selected engine;

• enough host memory for vLLM sleep level 1 and enough /dev/shm capacity for kvcached metadata and IPC;

• a node-local or otherwise persistent model-weight cache;

• kubectl access to create CRDs, Deployments, Services, and ModelClaims.

Install the nightly control plane

ModelClaim is included in the current main branch, including its CRD, controller reconciliation, and gateway wake support. Follow the Nightly Version section of the AIBrix installation guide. The standard config/crd and config/default manifests install the ModelClaim API and use the public nightly controller and gateway images. No custom control-plane build is required.

Verify the installation before creating a runtime pool:

kubectl get crd modelclaims.model.aibrix.ai
kubectl -n aibrix-system rollout status \
  deployment/aibrix-controller-manager --timeout=10m
kubectl -n aibrix-system rollout status \
  deployment/aibrix-gateway-plugins --timeout=10m

Nightly tags are mutable. For a repeatable experiment, record the AIBrix commit and resolved controller and gateway image digests together with the runtime image digest.

Build the experimental runtime image

Only the kvcached runtime agent image needs a feature-specific build. Its dedicated target is intentionally not part of docker-build-all, docker-push-all, or the public nightly image workflow.

Use the same current main checkout used by the nightly installation. Build the image and push it to a registry accessible from the cluster:

git clone https://github.com/vllm-project/aibrix.git
cd aibrix

export AIBRIX_CONTAINER_REGISTRY_NAMESPACE=ghcr.io/your-organization/aibrix
export IMAGE_TAG="$(git rev-parse HEAD)"

IS_MAIN_BRANCH=false make docker-build-kvcached-runtime
IS_MAIN_BRANCH=false make docker-push-kvcached-runtime

KVCACHED_RUNTIME_BASE_IMAGE defaults to ghcr.io/ovg-project/kvcached-vllm:latest. Pin it to a tested digest or tag for a reproducible deployment:

KVCACHED_RUNTIME_BASE_IMAGE=ghcr.io/ovg-project/kvcached-vllm@sha256:<digest> \
IMAGE_TAG=modelclaim-test IS_MAIN_BRANCH=false \
  make docker-build-kvcached-runtime

The default base image contains vLLM. The runtime launcher also has an SGLang path, but using engine: sglang requires a custom base image that contains a compatible SGLang and kvcached integration. Automatic idle sleep currently applies only to vLLM.

Update the warm-pool manifest to use ${AIBRIX_CONTAINER_REGISTRY_NAMESPACE}/kvcached-runtime:${IMAGE_TAG}. Keep the runtime source revision aligned with the nightly control plane revision used for the test.

Create a warm runtime pool

The repository sample creates one single-GPU runtime Pod and a metrics Service:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: warm-runtime-pool-b300
  namespace: default
  labels:
    pool.aibrix.ai/name: b300-pool-a
    pool.aibrix.ai/enabled: "true"
spec:
  replicas: 1
  selector:
    matchLabels:
      app: warm-runtime-pool-b300
  template:
    metadata:
      labels:
        app: warm-runtime-pool-b300
        pool.aibrix.ai/name: b300-pool-a
        pool.aibrix.ai/enabled: "true"
    spec:
      containers:
        - name: aibrix-runtime
          # This image must be built on a kvcached-enabled vLLM base image.
          # The runtime sidecar starts the per-ModelClaim engine processes.
          image: aibrix/kvcached-runtime:dev
          imagePullPolicy: IfNotPresent
          ports:
            - name: runtime
              containerPort: 8080
          readinessProbe:
            httpGet:
              path: /healthz
              port: 8080
            initialDelaySeconds: 3
            periodSeconds: 5
          env:
            - name: AIBRIX_MODEL_RUNTIME_MOCK
              value: "0"
            - name: ENABLE_KVCACHED
              # The image includes kvcached and kvctl, but the runtime agent
              # must not be autopatched. The launcher enables kvcached with a
              # unique IPC name for each child engine process.
              value: "false"
            - name: KVCACHED_AUTOPATCH
              value: "0"
            - name: AIBRIX_WEIGHT_CACHE_DIR
              value: /root/.cache/huggingface
          resources:
            limits:
              nvidia.com/gpu: "1"
          volumeMounts:
            - name: dshm
              mountPath: /dev/shm
            - name: hf-cache
              mountPath: /root/.cache/huggingface
            - name: engine-registry
              mountPath: /var/run/aibrix
      volumes:
        - name: dshm
          emptyDir:
            medium: Memory
            sizeLimit: 16Gi
        - name: hf-cache
          hostPath:
            path: /var/cache/aibrix/huggingface
            type: DirectoryOrCreate
        - name: engine-registry
          emptyDir: {}
---
# Exposes the runtime sidecar /metrics endpoint for the ModelClaim Grafana
# dashboard. Label aibrix.ai/metrics=modelclaim-runtime matches
# observability/monitor/service_monitor_modelclaim_runtime.yaml.
apiVersion: v1
kind: Service
metadata:
  name: warm-runtime-pool-b300-metrics
  namespace: default
  labels:
    app: warm-runtime-pool-b300
    pool.aibrix.ai/name: b300-pool-a
    pool.aibrix.ai/enabled: "true"
    aibrix.ai/metrics: modelclaim-runtime
spec:
  selector:
    app: warm-runtime-pool-b300
  ports:
    - name: runtime
      port: 8080
      targetPort: runtime

Before applying it, review these fields:

pool.aibrix.ai/name

The logical pool selected by ModelClaims. Put the label on both the Deployment and Pod template.

pool.aibrix.ai/enabled: "true"

Required on candidate Pods. Removing or changing it keeps the Pod out of ModelClaim placement.

image

Replace aibrix/kvcached-runtime:dev when using a remote registry or a pinned image.

nvidia.com/gpu

Defines the fixed topology for this pool. Every Pod in one pool should expose the same number of GPUs.

/dev/shm

The sample uses a 16 GiB memory-backed emptyDir. Size it for the chosen kvcached and engine configuration.

AIBRIX_WEIGHT_CACHE_DIR

Points to the model-weight cache inspected by runtime snapshots. The sample mounts a node-local hostPath so recreated Pods on that node can reuse downloads.

/var/run/aibrix

Stores the engine registry. The emptyDir survives an agent process restart within the Pod, but not a Pod replacement.

The runtime image still contains kvcached, kvctl, and the shared-memory control interfaces when ENABLE_KVCACHED=false. This variable controls whether the current Python process is patched; it does not remove kvcached from the image. Keep ENABLE_KVCACHED=false and KVCACHED_AUTOPATCH=0 on the runtime agent. The launcher overrides both values and assigns a unique KVCACHED_IPC_NAME for each child engine. Autopatching the agent itself can load the engine stack into the supervisor, create an unrelated CUDA context, and blur the isolation boundary between independently managed engines.

Apply the pool and wait for the runtime agent:

kubectl apply -f samples/modelclaim/warm-runtime-pool.yaml
kubectl rollout status deployment/warm-runtime-pool-b300 --timeout=10m
kubectl get pods -l pool.aibrix.ai/name=b300-pool-a -o wide

Create ModelClaims

The sample attaches two small Qwen models to the same pool:

apiVersion: model.aibrix.ai/v1alpha1
kind: ModelClaim
metadata:
  name: qwen3-0-6b
  namespace: default
spec:
  modelName: qwen3-0.6b
  podSelector:
    matchLabels:
      pool.aibrix.ai/name: b300-pool-a
  artifactURL: huggingface://Qwen/Qwen3-0.6B
  engine: vllm
  replicas: 1
  engineConfig:
    args:
      --max-model-len: "2048"
---
apiVersion: model.aibrix.ai/v1alpha1
kind: ModelClaim
metadata:
  name: qwen25-0-5b
  namespace: default
spec:
  modelName: qwen2.5-0.5b
  podSelector:
    matchLabels:
      pool.aibrix.ai/name: b300-pool-a
  artifactURL: huggingface://Qwen/Qwen2.5-0.5B-Instruct
  engine: vllm
  replicas: 1
  engineConfig:
    args:
      --max-model-len: "2048"

Apply the claims and watch their lifecycle:

kubectl apply -f samples/modelclaim/modelclaims.yaml
kubectl get modelclaims -w

kubectl wait --for=jsonpath='{.status.phase}'=Active \
  modelclaim/qwen3-0-6b --timeout=15m
kubectl wait --for=jsonpath='{.status.phase}'=Active \
  modelclaim/qwen25-0-5b --timeout=15m

The supported spec fields are:

Field	Required	Description
modelName	No	Served model identifier used in OpenAI-compatible requests. Defaults to the ModelClaim object name.
podSelector	Yes	Selects eligible warm-pool Pods. Candidates must also have the enabled pool label.
artifactURL	Yes	Model artifact location, such as huggingface://, s3://, or gcs://.
engine	No	vllm or sglang. Defaults to vllm. SGLang requires a compatible custom runtime image.
replicas	No	Defaults to 1. The current API only accepts 1.
engineConfig.args	No	Engine CLI flags mapped to string values. Use an empty string for a boolean flag.

For example:

engineConfig:
  args:
    --max-model-len: "4096"
    --enable-prefix-caching: ""

Do not set --gpu-memory-utilization. kvcached owns elastic KV-cache allocation, and the ModelClaim path rejects that flag. Data parallelism is not supported; --data-parallel-size must remain 1.

Configure TP and PP pools

ModelClaim uses a deliberately simple fixed-topology rule for vLLM:

tensor-parallel-size * pipeline-parallel-size
  == GPUs visible to the warm runtime Pod

For a TP=2 model, create a separate pool whose Pods each request two GPUs, then use:

engineConfig:
  args:
    --tensor-parallel-size: "2"
    --pipeline-parallel-size: "1"

Do not use one four-GPU pool to mix TP=1, TP=2, and TP=4 claims. Create one topology-homogeneous pool per shape. Automatic request-driven KV policy and idle sleep are currently limited to single-GPU runtime Pods, even though the fixed TP/PP activation path is supported.

Inspect actual runtime state

ModelClaim status summarizes the lifecycle:

Status	Meaning
Pending / Scheduling	The claim is new or the controller is selecting a compatible Pod.
Loading / Activating	The runtime is downloading or starting the engine. It remains non-routable with port 0.
Active	The runtime reports the engine alive and ready; the gateway has a real per-engine port.
Sleeping	The engine remains resident but is intentionally non-routable. A request can trigger a wake.
Failed	Activation or local restart recovery reached a terminal failure.

Inspect claim status and the routing annotation:

kubectl get modelclaim qwen3-0-6b -o yaml

POD=$(kubectl get pod -l pool.aibrix.ai/name=b300-pool-a \
  -o jsonpath='{.items[0].metadata.name}')
kubectl get pod "$POD" -o json \
  | jq '.metadata.annotations
    | with_entries(select(.key | startswith("modelclaim.aibrix.ai/")))'

An annotation has this form:

{"model":"qwen3-0.6b","port":20000,"state":"active"}

port: 0 means the model is known but not currently routable. It is used while the engine is activating, restarting, sleeping, or failed.

For detailed engine and memory state, port-forward the runtime API:

kubectl port-forward "pod/$POD" 8080:8080

curl -fsS http://localhost:8080/v1/runtime/snapshot | jq .

The snapshot includes per-engine port, IPC name, phase, liveness, readiness, restart count, last error, KV usage and capacity, best-effort HBM peak, request activity, and cached-artifact markers.

Send inference requests

Find the Envoy Service by its ownership labels and start a port-forward:

ENVOY_SERVICE=$(kubectl -n envoy-gateway-system get service \
  --selector=gateway.envoyproxy.io/owning-gateway-namespace=aibrix-system,gateway.envoyproxy.io/owning-gateway-name=aibrix-eg \
  -o jsonpath='{.items[0].metadata.name}')
test -n "$ENVOY_SERVICE"
kubectl -n envoy-gateway-system port-forward \
  "service/$ENVOY_SERVICE" 8888:80

Send an OpenAI-compatible request using spec.modelName:

curl -fsS http://localhost:8888/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "qwen3-0.6b",
    "messages": [{"role": "user", "content": "Say hello."}],
    "max_tokens": 32
  }' | jq .

Enable automatic KV and sleep policy

Policy is optional and is configured as one strict JSON annotation on the warm pool Deployment. It does not add fields to ModelClaim:

kubectl annotate deployment warm-runtime-pool-b300 \
  'pool.aibrix.ai/policy={"reclaim":{"mode":"kv-first","capacityBytes":4294967296,"guaranteedFloorPercent":20},"lifecycle":{"sleepAfterSeconds":60}}' \
  --overwrite

The fields mean:

reclaim.capacityBytes

Total KV capacity that the policy may distribute on each eligible single-GPU runtime. This is not total GPU HBM. Choose it only after accounting for model weights, engine overhead, and headroom.

reclaim.guaranteedFloorPercent

Per-model minimum as a percentage of the configured capacity. The policy also protects currently used and preallocated KV pages. If protected usage already exceeds the total capacity, no new plan is applied.

lifecycle.sleepAfterSeconds

How long a vLLM engine must have complete, initialized, and idle request observations before sleep level 1 is applied.

The controller distributes remaining KV capacity among active models using bounded inflight requests and completion deltas. A configured limit is a kvcached capacity ceiling, not an immediate physical HBM allocation and not an OOM guarantee.

The JSON parser rejects unknown fields. An invalid policy is disabled and reported with an InvalidPoolPolicy Event:

kubectl describe deployment warm-runtime-pool-b300
kubectl get events --field-selector reason=InvalidPoolPolicy

Sleeping request behavior

The gateway does not hold or replay a request while a model wakes. When a binding is sleeping, the gateway starts one deduplicated asynchronous wake and returns:

HTTP/1.1 503 Service Unavailable
Retry-After: 10

The client or an outer gateway must retry. While the engine is waking, later requests can continue to receive 503. The controller restores the real port only after the runtime reports the engine active and ready.

An activating model also returns 503 with Retry-After. A terminally failed model returns 503 without Retry-After.

Runtime reliability

Each ModelClaim engine has an independent process group and supervisor. An engine crash is removed from routing and restarted with exponential backoff; other engines in the Pod are not restarted. After five local restarts, the engine remains Failed with its error visible in the runtime snapshot.

The kvcached runtime image uses tini and a small restart loop around the AIBrix agent. If only the agent process crashes, child engines stay alive. The new agent reads /var/run/aibrix/engines.json, verifies the PID start time and health endpoint, and re-adopts valid engines without changing their port or IPC name.

This recovery does not cross a Pod restart. If the Pod disappears, the controller removes the stale instance and can activate the claim on another compatible Pod. There is no live migration or transparent preservation of in-flight requests.

Observability

The sample metrics Service is selected by observability/monitor/service_monitor_modelclaim_runtime.yaml. Import observability/grafana/AIBrix_ModelClaim_Runtime_Dashboard.json for the current dashboard.

Controller metrics include:

• aibrix_modelclaim_desired_replicas;

• aibrix_modelclaim_ready_replicas;

• aibrix_modelclaim_activating;

• aibrix_modelclaim_activation_total{result};

• aibrix_modelclaim_pool_policy_valid.

Runtime metrics include:

• aibrix:modelclaim_models_resident;

• aibrix:modelclaim_kv_used_bytes{model};

• aibrix:modelclaim_kv_total_bytes{model};

• aibrix:modelclaim_hbm_peak_bytes{model}.

HBM attribution is best effort and is used for observation and placement ranking. It is not a hard admission or reservation signal.

Troubleshooting

Claim remains Scheduling with zero candidates

Confirm that the Pod is Running, has a Pod IP, matches podSelector, and has pool.aibrix.ai/enabled: "true". For vLLM, confirm that TP times PP exactly matches the Pod-visible GPU count.

Claim remains Activating

Inspect the runtime snapshot and engine logs. Weight download, CUDA graph initialization, or engine compilation may take time. The controller intentionally keeps the route at port 0 until /health succeeds.

Activation rejects --gpu-memory-utilization

Remove the flag. The kvcached framework replaces the engine’s fixed KV-memory fraction in this deployment path.

Policy does not change KV limits

Automatic policy requires exactly one visible accelerator, valid request metrics with matching model labels, and at least one observed active model. It does not shrink when observations are incomplete or protected KV usage exceeds the configured capacity.

Model never enters automatic sleep

Automatic idle sleep currently applies only to vLLM. Check that request activity is initialized and observable and that no request is running or waiting.

Requests return 503

Inspect the routing annotation and ModelClaim status. A sleeping or activating model is retryable; a failed model requires operator attention.

Agent restarted but engines also disappeared

Check that the container uses build/container/Dockerfile.kvcached-runtime and its tini supervisor, and that the engine registry is mounted at /var/run/aibrix. A full Pod or container restart does not preserve engine processes.

Cleanup

Delete claims before the warm pool so the finalizer can stop their engines and remove routing annotations:

kubectl delete -f samples/modelclaim/modelclaims.yaml --wait=true
kubectl delete -f samples/modelclaim/warm-runtime-pool.yaml --wait=true

Validation guides

Use the ModelClaim manual GPU validation runbook for the full minikube and real-GPU acceptance procedure. A Chinese version is also available. To validate kvcached and vLLM sleep independently of the AIBrix control plane, use the mechanism guide.

Upstream references:

• kvcached;

• vLLM Sleep Mode;

• Prism: Cost-Efficient Multi-LLM Serving via GPU Memory Ballooning.