Lora Dynamic Loading

Lora Dynamic Loading#

The LoRA Model Adapter support is crucial for improving model density and reducing inference costs in large language models (LLMs). By enabling dynamic loading and unloading of LoRA adapters, it allows multiple models to be deployed more efficiently on a shared infrastructure.

This reduces resource consumption by reusing base models across different tasks while swapping in lightweight LoRA adapters for specific model tuning. The approach optimizes GPU memory usage, decreases cold start times, and enhances scalability, making it ideal for high-density deployment and cost-effective inference in production environments.

High Level Design#

Model Adapter Controller#

We develop a ModelAdapter controller to manage the lora model adapter lifecycle. User can submit a lora Custom Resource and controller will register to the matching pods, configure the service discovery and expose the adapter status.

lora-controller-workflow

ModelAdapter Lifecycle and Phase Transitions#

The ModelAdapter goes through several distinct phases during its lifecycle. Understanding these phases helps users monitor and troubleshoot their LoRA deployments:

Phase Transition Flow:

Pending → Scheduled → Running
   ↓         ↓          ↓
Waiting   Pod        Adapter loaded,
for pods  selected   ready for inference

Loading fails on every candidate pod → Failed (retried until one load succeeds)

Phase Details:

  1. Pending: The controller has started reconciliation but no adapter instance is loaded yet. The Ready condition says why: reason NoReadyPods means no ready pod matches the podSelector (with replicas: 1 it also covers pods that became ready too recently to be scheduled); reason ModelAdapterUnavailable means candidate pods exist but the adapter is not loaded on any of them yet.

  2. Scheduled: Only with replicas: 1. The controller has selected a pod that matches the podSelector and is loading the adapter on it. Pods are validated for readiness and stability before selection. Loading includes:

    • Downloading the adapter from the specified artifactURL

    • Registering the adapter with the vLLM engine

    • Retrying with exponential backoff if loading fails

    Without replicas (load on all matching pods) the adapter moves from Pending straight to Running.

  3. Running: The adapter is loaded on at least one pod and the Kubernetes Service and EndpointSlice for service discovery exist. The LoRA model is accessible through the gateway using the adapter name. readyReplicas out of desiredReplicas in the status shows how many candidate pods have the adapter loaded.

  4. Failed: Loading failed on every candidate pod. The Ready condition carries reason ModelAdapterLoadingError and the per-pod errors in its message. The controller keeps retrying and the adapter returns to Running as soon as one load succeeds.

ResourceCreated is not a step of the normal flow. It shows up as the phase only when creating the Service or EndpointSlice fails; the condition of the same name carries the error details, and the controller keeps retrying. Bound is a condition, not a phase: it is True (together with the Scheduled condition) while the adapter is loaded, and turns False with reason ModelAdapterLoadingError when a loading pass fails, while the phase stays Pending or Failed.

Error Handling and Reliability Features:

  • Retry: A failed load is retried on that pod. The first attempt is immediate. Later attempts wait 10s, 20s, 40s, 80s and 160s, then 5 minutes. The stored retry count stops growing at 6. The controller keeps trying at that cap.

  • Other pods: In the same reconcile, a pod that failed or is still in backoff is skipped and the next candidate is tried. Pods recorded in the scheduled-pods annotation are tried first. The controller does not wait for a retry cap before moving on.

  • Pod health: A pod must be Ready, and must have been Ready for at least 5 seconds, before it is selected.

  • Retriable errors: connection refused, connection reset, timeout, no route to host, network is unreachable, temporary failure, service unavailable and bad gateway. Other errors, such as authentication or an invalid artifact, fail that pod immediately.

Monitoring Phase Transitions:

kubectl get modeladapter shows the phase, the replica counters and the reason of the Ready condition at a glance. While the base model pod is still starting:

$ kubectl get modeladapter
NAME             PHASE     DESIRED   READY   CANDIDATES   REASON        MODEL PATH                                                     AGE
qwen-code-lora   Pending                                  NoReadyPods   huggingface://ai-blond/Qwen-Qwen2.5-Coder-1.5B-Instruct-lora   5s

Once the adapter is loaded:

$ kubectl get modeladapter
NAME             PHASE     DESIRED   READY   CANDIDATES   REASON                  MODEL PATH                                                     AGE
qwen-code-lora   Running   1         1       1            ModelAdapterAvailable   huggingface://ai-blond/Qwen-Qwen2.5-Coder-1.5B-Instruct-lora   2m

Blank counter cells mean zero. -o wide adds the base model, the pods hosting the adapter and the full Ready condition message:

$ kubectl get modeladapter -o wide
NAME             PHASE     DESIRED   READY   CANDIDATES   REASON                  BASE MODEL                 MODEL PATH                                                     AGE   INSTANCES                                       MESSAGE
qwen-code-lora   Running   1         1       1            ModelAdapterAvailable   qwen-coder-1-5b-instruct   huggingface://ai-blond/Qwen-Qwen2.5-Coder-1.5B-Instruct-lora   2m    ["qwen-coder-1-5b-instruct-5587f4c57d-kml6s"]   ModelAdapter default/qwen-code-lora is ready

For the full condition history with timestamps and reasons for each state change, use:

kubectl describe modeladapter <adapter-name>

Model Adapter Service Discovery#

We aim to reuse the Kubernetes Service as the abstraction layer for each lora model. Traditionally, a single pod belongs to one service. For LoRA, one pod hosts several adapters, which does not fit that model. To support lora cases in kubernetes native way, we customize the lora endpoints and allow a single pod with different LoRAs belong to multiple services.

vLLM Engine Changes#

High density Lora support can not be done solely in the control plane side, we also write an RFC about improving the Visibility of LoRA metadata, Dynamic Loading and Unloading, Remote Registry Support, Observability to enhance LoRA management for production grade serving. Please check [RFC]: Enhancing LoRA Management for Production Environments in vLLM for more details. Majority of the changes are done in vLLM, we just need to use the latest vLLM image and that would be ready for production.

Examples#

Here’s one example of how to create a lora adapter.

Prerequisites#

  1. You have a base model deployed in the same namespace.

  2. vLLM engine needs to enable VLLM_ALLOW_RUNTIME_LORA_UPDATING feature flag.

  3. You have a lora model hosted on Huggingface or S3 compatible storage.

  4. (Optional) You have a Huggingface token to enable faster model downloads from Hugging Face.

    # Create a secret with the Hugging Face token
kubectl create secret generic hf-token --from-literal=token=hf_xxxxxxxxxxx

Create base model#

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    model.aibrix.ai/name: qwen-coder-1-5b-instruct # Note: The label value `model.aibrix.ai/name` here must match with the service name.
    model.aibrix.ai/port: "8000"
    adapter.model.aibrix.ai/enabled: "true" # Required for LoRA adapter support
  name: qwen-coder-1-5b-instruct
  namespace: default
spec:
  # Increase replicas for better adapter reliability and load distribution
  # Allows multiple LoRA adapters to be distributed across pods
  replicas: 3
  selector:
    matchLabels:
      model.aibrix.ai/name: qwen-coder-1-5b-instruct
  template:
    metadata:
      labels:
        model.aibrix.ai/name: qwen-coder-1-5b-instruct
        adapter.model.aibrix.ai/enabled: "true"
    spec:
      containers:
        - command:
            - vllm
            - serve
            - --host
            - "0.0.0.0"
            - --port
            - "8000"
            - --uvicorn-log-level
            - warning
            - --model
            - Qwen/Qwen2.5-Coder-1.5B-Instruct
            - --served-model-name
            # Note: The `--served-model-name` argument value must also match the Service name and the Deployment label `model.aibrix.ai/name`
            - qwen-coder-1-5b-instruct
            - --enable-lora
            # Allow at most four distinct LoRA adapters in one vLLM batch.
            - --max-loras
            - "4"
            # Keep up to four adapters in the CPU LoRA cache as well.
            - --max-cpu-loras
            - "4"
          image: vllm/vllm-openai:v0.24.0
          imagePullPolicy: Always
          name: vllm-openai
          env:
            # REQUIRED: Enable runtime LoRA updating for ModelAdapter support
            - name: VLLM_ALLOW_RUNTIME_LORA_UPDATING
              value: "True"
            # Optional: Increase timeout for better reliability during adapter loading
            - name: VLLM_LORA_MODULES_LOADING_TIMEOUT
              value: "300"
            # Optional: Add HF_TOKEN to enable faster model downloads from Hugging Face
            - name: HF_TOKEN
              valueFrom:
                secretKeyRef:
                  name: hf-token
                  key: token
                  optional: true
          ports:
            - containerPort: 8000
              protocol: TCP
          volumeMounts:
            - name: adapter-storage
              mountPath: /tmp/aibrix/adapters
          resources:
            limits:
              nvidia.com/gpu: "1"
            requests:
              nvidia.com/gpu: "1"
          # Health checks for better pod reliability
          livenessProbe:
            httpGet:
              path: /health
              port: 8000
            initialDelaySeconds: 60
            periodSeconds: 30
            timeoutSeconds: 10
            failureThreshold: 3
          readinessProbe:
            httpGet:
              path: /health  
              port: 8000
            initialDelaySeconds: 30
            periodSeconds: 10
            timeoutSeconds: 5
            successThreshold: 1
            failureThreshold: 3
          # Startup checks for model downloading and spinning up engine
          startupProbe:
            httpGet:
              path: /health
              port: 8000
              scheme: HTTP
            initialDelaySeconds: 120  # Waits 120 seconds before doing the first check
            failureThreshold: 120 # Model downloads from Hugging Face is time consuming.
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 1
        - name: aibrix-runtime
          image: aibrix/runtime:v0.8.0-rc.1
          command:
            - aibrix_runtime
            - --port
            - "8080"
          env:
            - name: INFERENCE_ENGINE
              value: vllm
            - name: INFERENCE_ENGINE_ENDPOINT
              value: http://localhost:8000
          ports:
            - containerPort: 8080
              protocol: TCP
          volumeMounts:
            - name: adapter-storage
              mountPath: /tmp/aibrix/adapters
          livenessProbe:
            httpGet:
              path: /healthz
              port: 8080
            initialDelaySeconds: 3
            periodSeconds: 2
          readinessProbe:
            httpGet:
              path: /ready
              port: 8080
            initialDelaySeconds: 5
            periodSeconds: 10
      volumes:
        - name: adapter-storage
          emptyDir: {}

---

apiVersion: v1
kind: Service
metadata:
  labels:
    model.aibrix.ai/name: qwen-coder-1-5b-instruct
    prometheus-discovery: "true"
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: "8080"
  name: qwen-coder-1-5b-instruct # Note: The Service name must match the label value `model.aibrix.ai/name` in the Deployment
  namespace: default
spec:
  ports:
    - name: serve
      port: 8000
      protocol: TCP
      targetPort: 8000
    - name: http
      port: 8080
      protocol: TCP
      targetPort: 8080
  selector:
    model.aibrix.ai/name: qwen-coder-1-5b-instruct
  type: ClusterIP

# Expose endpoint
LB_IP=$(kubectl get svc/envoy-aibrix-system-aibrix-eg-903790dc -n envoy-gateway-system -o=jsonpath='{.status.loadBalancer.ingress[0].ip}')
ENDPOINT="${LB_IP}:80"

# send request to base model
curl -v http://${ENDPOINT}/v1/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "qwen-coder-1-5b-instruct",
        "prompt": "San Francisco is a",
        "max_tokens": 128,
        "temperature": 0
    }'

Create lora model adapter#

apiVersion: model.aibrix.ai/v1alpha1
kind: ModelAdapter
metadata:
  name: qwen-code-lora
  namespace: default
  labels:
    model.aibrix.ai/name: "qwen-code-lora"
    model.aibrix.ai/port: "8000"
spec:
  # Base model that this LoRA adapter extends
  baseModel: qwen-coder-1-5b-instruct
  
  # Pod selector to find pods that can host this adapter
  # Pods must have both labels: model.aibrix.ai/name and adapter.model.aibrix.ai/enabled=true
  podSelector:
    matchLabels:
      model.aibrix.ai/name: qwen-coder-1-5b-instruct
      adapter.model.aibrix.ai/enabled: "true"
  
  # URL for the LoRA adapter artifact
  # Supported formats: huggingface://, s3://, or absolute local path
  artifactURL: huggingface://ai-blond/Qwen-Qwen2.5-Coder-1.5B-Instruct-lora
  
  # Replicas controls adapter distribution across pods:
  # - nil (omitted): Load adapter on ALL matching pods (recommended)
  # - 1: Load adapter on a single pod selected by the scheduler
  # Only nil or 1 are supported. Other values will be rejected.
  # replicas: 1
  
  # The controller does not read schedulerName. Pod selection uses
  # --model-adapter-scheduler-policy on the controller manager
  # (default leastAdapters). The CRD still defaults this field to "default".
  schedulerName: default

  # Lifecycle phases: Pending → Scheduled (replicas: 1 only) → Running, or Failed
  # - Pending: No adapter instance loaded yet; the Ready condition reason says why
  # - Scheduled: A pod was selected for the adapter (single-pod mode)
  # - Running: Adapter loaded on at least one pod, ready for inference requests
  # - Failed: Loading failed on every candidate pod; the controller keeps retrying

If you run kubectl describe modeladapter qwen-code-lora, you will see the status of the lora adapter progressing through different phases:

Phase 1: Pending

$ kubectl describe modeladapter qwen-code-lora
Status:
  Conditions:
    Last Transition Time:  2025-02-16T19:14:50Z
    Message:               Starting reconciliation
    Reason:                ModelAdapterPending
    Status:                Unknown
    Type:                  Initialized
  Phase:  Pending

Phase 2: Scheduled (only with replicas: 1)

$ kubectl describe modeladapter qwen-code-lora
Status:
  Conditions:
    Last Transition Time:  2025-02-16T19:14:50Z
    Message:               Starting reconciliation
    Reason:                ModelAdapterPending
    Status:                Unknown
    Type:                  Initialized
    Last Transition Time:  2025-02-16T19:14:52Z
    Message:               ModelAdapter default/qwen-code-lora has selected 1 pods for scheduling: [qwen-coder-1-5b-instruct-5587f4c57d-kml6s]
    Reason:                Scheduled
    Status:                True
    Type:                  Scheduled
  Phase:  Scheduled

Phase 3: Running

$ kubectl describe modeladapter qwen-code-lora
Status:
  Candidates:  1
  Conditions:
    Last Transition Time:  2025-02-16T19:14:50Z
    Message:               Starting reconciliation
    Reason:                ModelAdapterPending
    Status:                Unknown
    Type:                  Initialized
    Last Transition Time:  2025-02-16T19:14:52Z
    Message:               ModelAdapter default/qwen-code-lora has selected 1 pods for scheduling: [qwen-coder-1-5b-instruct-5587f4c57d-kml6s]
    Reason:                Scheduled
    Status:                True
    Type:                  Scheduled
    Last Transition Time:  2025-02-16T19:14:55Z
    Message:               ModelAdapter default/qwen-code-lora is ready
    Reason:                ModelAdapterAvailable
    Status:                True
    Type:                  Ready
  Desired Replicas:  1
  Instances:
    qwen-coder-1-5b-instruct-5587f4c57d-kml6s
  Phase:           Running
  Ready Replicas:  1

Send request using lora model name to the gateway.

# send request to base model
curl -v http://${ENDPOINT}/v1/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "qwen-code-lora",
        "prompt": "San Francisco is a",
        "max_tokens": 128,
        "temperature": 0
    }'

Here’s the resources created associated with the lora custom resource.

lora-service-discovery-resources

  1. A new Kubernetes service will be created with the exact same name as ModelAdapter name.

2. The podSelector is used to filter the matching pods. In this case, it will match pods with label model.aibrix.ai/name=qwen-coder-1-5b-instruct. Make sure your base model have this label. This ensures that the LoRA adapter is correctly associated with the right pods.

Attention

Note: this is only working with vLLM engine. If you use other engine, feel free to open an issue.

Troubleshooting Phase Transitions#

If your ModelAdapter gets stuck in a particular phase, here are common issues and solutions:

Stuck in Pending Phase:

  • Check if pods matching podSelector exist and are running

  • Verify that pods have the correct labels (model.aibrix.ai/name and adapter.model.aibrix.ai/enabled=true)

  • Ensure pods are in Ready state

Stuck in Scheduled Phase:

  • This was a known issue in earlier versions that has been fixed

  • Check controller logs for any loading errors: kubectl logs -n aibrix-system deployment/aibrix-controller-manager

  • Verify vLLM pods have VLLM_ALLOW_RUNTIME_LORA_UPDATING enabled

Stuck in Failed Phase (loading errors):

  • Check if the artifactURL is accessible and valid

  • For Hugging Face models, ensure the model path exists

  • Check authentication if using private repositories

  • Review controller logs for specific loading errors

  • A retriable error is retried on that pod with the backoff above. The same reconcile also tries the next candidate pod.

Failed to Reach Running Phase:

  • Check if Kubernetes Service creation succeeded: kubectl get svc <adapter-name>

  • Verify EndpointSlice creation: kubectl get endpointslices

  • Check pod readiness and health status

Useful Commands for Debugging:

# Check ModelAdapter status
kubectl describe modeladapter <adapter-name>

# Check controller logs
kubectl logs -n aibrix-system deployment/aibrix-controller-manager

# Check pod logs
kubectl logs <pod-name>

# Check services and endpoints
kubectl get svc,endpointslices -l model.aibrix.ai/name=<adapter-name>

Reliability Features#

AIBrix ModelAdapter includes several advanced reliability features to ensure robust LoRA adapter deployments in production environments:

Automatic Retry Mechanism#

The controller retries a failed load on the same pod:

  • The first attempt runs immediately. There is no wait before it.

  • After a failure the waits are 10s, 20s, 40s, 80s and 160s, then 5 minutes. The base interval is 5 seconds and the wait doubles, capped at 5 minutes.

  • The retry count stored on the adapter stops growing at 6 (MaxLoadingRetries). Loading continues at the capped wait.

  • Retriable errors are the connection and gateway errors listed above. Other errors fail that pod immediately.

# Retry behavior is automatic - no configuration needed
# Controller tracks retry state via annotations:
# - adapter.model.aibrix.ai/retry-count.<pod-name>
# - adapter.model.aibrix.ai/last-retry-time.<pod-name>

Pod Switching and Failover#

When a load fails, or the pod is still waiting out its backoff, the same reconcile tries the next candidate. Pods named in adapter.model.aibrix.ai/scheduled-pods are tried first. The search stops once desiredReplicas pods have loaded the adapter.

Pod choice for replicas: 1 uses one scheduler for the whole controller, set by --model-adapter-scheduler-policy (default leastAdapters). The other policies are random, binPack, leastLatency and leastThroughput. spec.schedulerName is not read. The CRD defaults that field to default, and the samples still set it.

Pod Health Validation#

The controller implements comprehensive pod health checks:

  • Readiness validation: Pods must be in Ready state before selection

  • Stability requirements: Pods must be stable (ready for 5+ seconds) to avoid flapping

  • Continuous monitoring: Unhealthy pods are automatically removed from instances list

  • Startup tolerance: Handles pod startup delays gracefully

Connection Error Handling#

Robust error handling for various network and startup conditions:

Retriable Errors (will trigger retry):
- "connection refused"
- "connection reset"
- "timeout"
- "no route to host"
- "network is unreachable"
- "temporary failure"
- "service unavailable"
- "bad gateway"

Non-Retriable Errors (will fail immediately):
- Authentication errors
- Invalid model format
- Configuration errors

Advanced Configuration Examples#

High-Availability Setup with Multi-Replica:

Here’s a production-ready configuration demonstrating load balancing across multiple pods:

apiVersion: model.aibrix.ai/v1alpha1
kind: ModelAdapter
metadata:
  name: sample-lora-multi-replica
  namespace: default
  labels:
    model.aibrix.ai/name: "sample-lora-multi-replica"
    model.aibrix.ai/port: "8000"
spec:
  # REQUIRED: Base model that this LoRA adapter extends
  baseModel: qwen-coder-1-5b-instruct
  
  # Do not specify the replicas for the adapter
  # The adapter will be loaded on all pods
  # Provides high availability and load distribution
  # replicas: 1
  
  # Pod selector to identify which pods can host this adapter
  # Requires pods to have both labels for proper selection
  podSelector:
    matchLabels:
      model.aibrix.ai/name: qwen-coder-1-5b-instruct
      adapter.model.aibrix.ai/enabled: "true"
  
  # URL for the LoRA adapter artifact
  # Using a real Hugging Face model for this example
  artifactURL: "huggingface://ai-blond/Qwen-Qwen2.5-Coder-1.5B-Instruct-lora"
  
  # The controller does not read schedulerName. With replicas omitted,
  # the adapter is loaded on every matching pod, so no scheduler runs.
  schedulerName: least-adapters

Testing Commands:

# 1. Apply the multi-replica adapter:
kubectl apply -f samples/adapter/adapter-multi-replica.yaml

# 2. Monitor phase transitions:
watch kubectl describe modeladapter sample-lora-multi-replica

# 3. Check controller logs for retry behavior:
kubectl logs -n aibrix-system deployment/aibrix-controller-manager -f

# 4. Test pod failure scenario (delete a pod to trigger switching):
kubectl delete pod <pod-name>
# Watch how controller handles pod loss and reschedules

# 5. Verify final state:
kubectl get modeladapter sample-lora-multi-replica -o yaml

Monitoring and Observability#

Phase Monitoring:

The reliability features are transparent through the standard ModelAdapter status:

# Monitor real-time phase transitions
watch kubectl describe modeladapter <adapter-name>

# Check for retry annotations (during loading failures)
kubectl get modeladapter <adapter-name> -o yaml | grep -A 5 annotations

Controller Logs:

The controller records retries in its log, not as Kubernetes Events. Useful messages are Selected pods for adapter scheduling, Retriable error loading adapter, Non-retriable error loading adapter on pod, Retry count at cap, retrying on max backoff and Successfully loaded adapter on pod.

kubectl logs -n aibrix-system deployment/aibrix-controller-manager -f | grep -E "loading adapter|Selected pods|Retry count"

Conditions:

The Ready and Scheduled conditions are the record of progress. kubectl describe modeladapter <adapter-name> shows them. The controller does not emit Events for retries.

Best Practices for Production#

  1. Deploy Multiple Base Model Replicas:

    # Ensure multiple pods are available for failover
spec:
  replicas: 3  # At least 3 pods for reliability

  2. Use Multiple Adapter Replicas:

    # Distribute adapter across all pods
spec:
  # Load adapter on all pods by omitting replicas field
  # replicas: 1

  3. Configure Appropriate Health Checks:

    # In base model deployment
livenessProbe:
  httpGet:
    path: /health
    port: 8000
  initialDelaySeconds: 60
  periodSeconds: 30
  failureThreshold: 3

  4. Monitor Adapter Health:

    # Regular health monitoring
kubectl get modeladapters -o wide
kubectl describe modeladapter <name>

# Set up alerts on phase transitions
kubectl get events --field-selector involvedObject.kind=ModelAdapter

More configurations#

Multi-LoRA Inference#

vLLM can serve requests for different LoRA adapters in a single batch. SGLang supports a similar feature. The base model in samples/adapter/base.yaml uses the following arguments to allow up to four distinct adapters per batch:

- --enable-lora
- --max-loras
- "4"
- --max-cpu-loras
- "4"

The vLLM max_loras setting limits the number of distinct adapters in one batch, not the total number of requests or ModelAdapter resources. max_cpu_loras controls the CPU LoRA cache and must be greater than or equal to max_loras.

Create one ModelAdapter resource for each LoRA. The adapters must all be compatible with the base model. For a readily reproducible example, the following manifest assigns four names to the same public LoRA artifact. Consequently, their generated outputs may be similar. Replace the artifactURL values with different compatible adapters when testing your own models. Each resource omits spec.replicas, so AIBrix loads every adapter on all matching base-model pods.

# Create one ModelAdapter per LoRA. All adapters target the same base model
# deployment and can be served together because base.yaml sets --max-loras 4.
apiVersion: model.aibrix.ai/v1alpha1
kind: ModelAdapter
metadata:
  name: qwen-code-lora-a
  namespace: default
  labels:
    model.aibrix.ai/name: qwen-code-lora-a
    model.aibrix.ai/port: "8000"
spec:
  baseModel: qwen-coder-1-5b-instruct
  podSelector:
    matchLabels:
      model.aibrix.ai/name: qwen-coder-1-5b-instruct
      adapter.model.aibrix.ai/enabled: "true"
  artifactURL: huggingface://ai-blond/Qwen-Qwen2.5-Coder-1.5B-Instruct-lora
---
# These adapters intentionally reuse the same public artifact so that the
# documentation example works without finding four separate compatible LoRAs.
# Replace each URL with a different compatible adapter for your own workload.
apiVersion: model.aibrix.ai/v1alpha1
kind: ModelAdapter
metadata:
  name: qwen-code-lora-b
  namespace: default
  labels:
    model.aibrix.ai/name: qwen-code-lora-b
    model.aibrix.ai/port: "8000"
spec:
  baseModel: qwen-coder-1-5b-instruct
  podSelector:
    matchLabels:
      model.aibrix.ai/name: qwen-coder-1-5b-instruct
      adapter.model.aibrix.ai/enabled: "true"
  artifactURL: huggingface://ai-blond/Qwen-Qwen2.5-Coder-1.5B-Instruct-lora
---
apiVersion: model.aibrix.ai/v1alpha1
kind: ModelAdapter
metadata:
  name: qwen-code-lora-c
  namespace: default
  labels:
    model.aibrix.ai/name: qwen-code-lora-c
    model.aibrix.ai/port: "8000"
spec:
  baseModel: qwen-coder-1-5b-instruct
  podSelector:
    matchLabels:
      model.aibrix.ai/name: qwen-coder-1-5b-instruct
      adapter.model.aibrix.ai/enabled: "true"
  artifactURL: huggingface://ai-blond/Qwen-Qwen2.5-Coder-1.5B-Instruct-lora
---
apiVersion: model.aibrix.ai/v1alpha1
kind: ModelAdapter
metadata:
  name: qwen-code-lora-d
  namespace: default
  labels:
    model.aibrix.ai/name: qwen-code-lora-d
    model.aibrix.ai/port: "8000"
spec:
  baseModel: qwen-coder-1-5b-instruct
  podSelector:
    matchLabels:
      model.aibrix.ai/name: qwen-coder-1-5b-instruct
      adapter.model.aibrix.ai/enabled: "true"
  artifactURL: huggingface://ai-blond/Qwen-Qwen2.5-Coder-1.5B-Instruct-lora

From the AIBrix repository root, create or update the base model. If you completed the single-LoRA example above, delete its adapter first so that all four cache slots are available. Then create the four adapters and wait until they are ready:

kubectl apply -f samples/adapter/base.yaml
kubectl rollout status deployment/qwen-coder-1-5b-instruct --timeout=10m

kubectl delete modeladapter qwen-code-lora --ignore-not-found
kubectl apply -f samples/adapter/adapter-multi-lora.yaml

kubectl wait --for=jsonpath='{.status.phase}'=Running \
    modeladapter/qwen-code-lora-a \
    modeladapter/qwen-code-lora-b \
    modeladapter/qwen-code-lora-c \
    modeladapter/qwen-code-lora-d \
    --timeout=10m

In one terminal, forward a local port to the AIBrix gateway and leave the command running:

ENVOY_SERVICE=$(kubectl get service -n envoy-gateway-system \
    --selector=gateway.envoyproxy.io/owning-gateway-namespace=aibrix-system,gateway.envoyproxy.io/owning-gateway-name=aibrix-eg \
    -o jsonpath='{.items[0].metadata.name}')

kubectl port-forward -n envoy-gateway-system \
    "service/${ENVOY_SERVICE}" 8080:80

In a second terminal, send one request to each adapter concurrently. The response from each adapter is saved and then printed after all four requests finish:

ENDPOINT=localhost:8080
RESULT_DIR=$(mktemp -d)

for MODEL in qwen-code-lora-a qwen-code-lora-b \
             qwen-code-lora-c qwen-code-lora-d; do
    curl --fail-with-body --silent --show-error \
        "http://${ENDPOINT}/v1/completions" \
        -H "Content-Type: application/json" \
        -d "{
            \"model\": \"${MODEL}\",
            \"prompt\": \"Implement merge sort in Python.\",
            \"max_tokens\": 256,
            \"temperature\": 0
        }" > "${RESULT_DIR}/${MODEL}.json" &
done

wait

for MODEL in qwen-code-lora-a qwen-code-lora-b \
             qwen-code-lora-c qwen-code-lora-d; do
    echo "=== ${MODEL} ==="
    cat "${RESULT_DIR}/${MODEL}.json"
    echo
done

Model Registry#

Currently, we support Huggingface model registry, S3 compatible storage and local file system.

  1. If your model is hosted on Huggingface, you can use the artifactURL with huggingface:// prefix to specify the model url. vLLM will download the model from Huggingface and load it into the pod in runtime.

  2. If you put your model in S3 compatible storage, you have to use AIBrix AI Runtime at the same time. You can use the artifactURL with s3:// prefix to specify the model url. AIBrix AI Runtime will download the model from S3 on the pod and load it with local model path in vLLM.

  3. If you use shared storage like NFS, you can use the artifactURL with a / absolute path to specify the model url (/models/yard1/llama-2-7b-sql-lora-test as an example). You must mount that path into the pod.

Model api-key Authentication#

User may pass in the argument --api-key or environment variable VLLM_API_KEY to enable the server to check for API key in the header.

vllm serve --api-key sk-kFJ12nKsFakefVmGpj3QzX65s4RbN2xJqWzPYCjYu7wT3BFake

We already have an example and you can kubectl apply -f samples/adapter/base-api-key.yaml.

In that case, lora model adapter can not query the vLLM server correctly, showing {"error":"Unauthorized"} error. You need to update additionalConfig field to pass in the API key.

apiVersion: model.aibrix.ai/v1alpha1
kind: ModelAdapter
metadata:
  name: qwen-code-lora-with-key
  namespace: default
  labels:
    model.aibrix.ai/name: "qwen-code-lora-with-key"
    model.aibrix.ai/port: "8000"
spec:
  baseModel: qwen-coder-1-5b-instruct
  podSelector:
    matchLabels:
      model.aibrix.ai/name: qwen-coder-1-5b-instruct
      adapter.model.aibrix.ai/enabled: "true"
  artifactURL: huggingface://ai-blond/Qwen-Qwen2.5-Coder-1.5B-Instruct-lora
  additionalConfig:
    api-key: sk-kFJ12nKsFakefVmGpj3QzX65s4RbN2xJqWzPYCjYu7wT3BFake
  schedulerName: default

You need to send the request with --header 'Authorization: Bearer your-api-key'

# send request to base model
curl -v http://${ENDPOINT}/v1/completions \
    -H "Content-Type: application/json" \
    -H "Authorization: Bearer sk-kFJ12nKsFakefVmGpj3QzX65s4RbN2xJqWzPYCjYu7wT3BFake" \
    -d '{
        "model": "qwen-code-lora-with-key",
        "prompt": "San Francisco is a",
        "max_tokens": 128,
        "temperature": 0
    }'

Runtime Support Sidecar#

A standard install starts the controller manager with --enable-runtime-sidecar. With that flag, the ModelAdapter controller calls the runtime on port 8080 when the pod has an aibrix-runtime container, and calls the engine on port 8000 otherwise. The runtime then registers the adapter with the engine. See AI Engine Runtime.

The flag’s default in the binary is off. The install manifest turns it on. To load adapters by calling vLLM directly, remove --enable-runtime-sidecar from the controller manager:

kubectl edit deployment aibrix-controller-manager -n aibrix-system

 args:
 - --leader-elect
 - --health-probe-bind-address=:8081
 - --metrics-bind-address=:8080
 - --enable-runtime-sidecar  # remove this line

Private Artifact Storage Support#

AIBrix supports loading LoRA adapters from private cloud storage backends including AWS S3, Google Cloud Storage (GCS), and Volcano Engine TOS. This feature requires the AIBrix runtime sidecar to handle artifact delegation - downloading artifacts from private storage and forwarding them to the inference engine.

Supported Storage Backends:

  • AWS S3: s3://bucket/path/to/adapter

  • Google Cloud Storage: gs://bucket/path/to/adapter

  • Volcano Engine TOS: tos://bucket/path/to/adapter

  • Private HuggingFace: huggingface://org/private-model (with token)

  • HTTP(S) with Authentication: Any URL with custom headers

How It Works:

Without Runtime (Direct Mode):
Controller → Inference Engine (port 8000)
✓ Public HTTP(S), public HuggingFace
✗ Private S3, GCS (not supported)

With Runtime (Artifact Delegation):
Controller → Runtime (port 8080) → Downloads Artifact → Engine
✓ S3, GCS, private HuggingFace, all storage types

The runtime sidecar:

  1. Receives the artifact URL and credentials from the controller

  2. Downloads the artifact from private storage to local filesystem

  3. Forwards the local path to the inference engine

  4. Manages artifact lifecycle (cleanup on unload)

Prerequisites:

  1. AIBrix runtime sidecar deployed alongside your inference engine

  2. Controller flag --enable-runtime-sidecar=true enabled

  3. Kubernetes secrets configured with storage credentials

  4. Shared volume between runtime and engine containers

Note

When the AIBrix sidecar injection webhook is enabled, AIBrix injects a shared volume named adapter-storage mounted at /tmp/aibrix/adapters into both the runtime and engine containers. The runtime downloads LoRA adapter artifacts into this directory by default. See DefaultAdapterVolumeName and DefaultAdapterMountPath in pkg/webhook/sidecar_injection.go.

Setup: AWS S3 Storage#

Step 1: Create S3 Credentials Secret

apiVersion: v1
kind: Secret
metadata:
  name: s3-credentials
  namespace: default
type: Opaque
stringData:
  aws_access_key_id: "AKIAIOSFODNN7EXAMPLE"
  aws_secret_access_key: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
  aws_region: "us-west-2"

Step 2: Deploy Pod with Runtime Sidecar

apiVersion: v1
kind: Pod
metadata:
  name: vllm-llama3
  labels:
    model.aibrix.ai/name: llama-3
    adapter.model.aibrix.ai/enabled: "true"
spec:
  containers:
  # Inference engine
  - name: vllm
    image: vllm/vllm-openai:latest
    ports:
    - containerPort: 8000
    env:
    - name: VLLM_ALLOW_RUNTIME_LORA_UPDATING
      value: "True"
    volumeMounts:
    - name: adapter-storage
      mountPath: /tmp/aibrix/adapters

  # AIBrix runtime sidecar
  - name: aibrix-runtime
    image: aibrix/runtime:latest
    ports:
    - containerPort: 8080
    env:
    - name: INFERENCE_ENGINE
      value: "vllm"
    - name: INFERENCE_ENGINE_ENDPOINT
      value: "http://localhost:8000"
    volumeMounts:
    - name: adapter-storage
      mountPath: /tmp/aibrix/adapters
    - name: credentials
      mountPath: /var/run/secrets/aibrix
      readOnly: true

  volumes:
  - name: adapter-storage
    emptyDir: {}
  - name: credentials
    secret:
      secretName: s3-credentials

Step 3: Create ModelAdapter with S3 URL

apiVersion: model.aibrix.ai/v1alpha1
kind: ModelAdapter
metadata:
  name: sql-lora-s3
spec:
  baseModel: llama-3
  podSelector:
    matchLabels:
      model.aibrix.ai/name: llama-3
  # S3 artifact URL
  artifactURL: s3://my-company-bucket/lora-adapters/sql-adapter/
  # Reference to S3 credentials
  credentialsSecretRef:
    name: s3-credentials
  replicas: 1

Step 4: Verify Artifact Delegation

# Check ModelAdapter status
kubectl describe modeladapter sql-lora-s3

# Should show: Phase: Running
# Controller logs will show:
# "Using runtime path for artifact delegation"

# Runtime logs will show download progress:
kubectl logs <pod-name> -c aibrix-runtime
# "Downloading artifact from s3://... to /tmp/aibrix/adapters/sql-lora-s3"
# "Successfully downloaded artifact for sql-lora-s3"
# "Forwarding load request to engine with local path"

Setup: Google Cloud Storage (GCS)#

Step 1: Create GCS Credentials Secret

apiVersion: v1
kind: Secret
metadata:
  name: gcs-credentials
type: Opaque
stringData:
  gcp_service_account_json: |
    {
      "type": "service_account",
      "project_id": "my-project",
      "private_key_id": "key-id",
      "private_key": "-----BEGIN PRIVATE KEY-----\n...\n-----END PRIVATE KEY-----\n",
      "client_email": "service-account@my-project.iam.gserviceaccount.com",
      "client_id": "123456789",
      "auth_uri": "https://accounts.google.com/o/oauth2/auth",
      "token_uri": "https://oauth2.googleapis.com/token",
      "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs"
    }

Step 2: Create ModelAdapter with GCS URL

apiVersion: model.aibrix.ai/v1alpha1
kind: ModelAdapter
metadata:
  name: code-lora-gcs
spec:
  baseModel: llama-3
  podSelector:
    matchLabels:
      model.aibrix.ai/name: llama-3
  # GCS artifact URL
  artifactURL: gs://my-gcs-bucket/lora-adapters/code-adapter/
  credentialsSecretRef:
    name: gcs-credentials
  replicas: 1

Setup: Private HuggingFace Models#

Step 1: Create HuggingFace Token Secret

apiVersion: v1
kind: Secret
metadata:
  name: huggingface-credentials
type: Opaque
stringData:
  huggingface_token: "hf_xxxxxxxxxxxxxxxxxxxxxxxxxxxxx"

Step 2: Create ModelAdapter with Private HuggingFace URL

apiVersion: model.aibrix.ai/v1alpha1
kind: ModelAdapter
metadata:
  name: private-hf-lora
spec:
  baseModel: llama-3
  podSelector:
    matchLabels:
      model.aibrix.ai/name: llama-3
  # Private HuggingFace model
  artifactURL: huggingface://my-org/private-lora-adapter
  credentialsSecretRef:
    name: huggingface-credentials
  replicas: 1

Troubleshooting Artifact Delegation#

Runtime Not Receiving Requests:

# Check controller is configured for runtime
kubectl get deployment aibrix-controller-manager -n aibrix-system -o yaml | grep enable-runtime-sidecar
# Should show: - --enable-runtime-sidecar=true

# Check runtime pod is running
kubectl get pods -l app=vllm
kubectl logs <pod-name> -c aibrix-runtime

Download Failures:

# Check runtime logs for specific errors
kubectl logs <pod-name> -c aibrix-runtime

# Common issues:
# - "Access denied": Check credentials in secret
# - "Not found": Verify artifact URL is correct
# - "Invalid credentials": Verify secret format matches expected keys

Credential Issues:

# Verify secret exists and has correct keys
kubectl get secret s3-credentials -o yaml

# For S3, required keys:
# - aws_access_key_id
# - aws_secret_access_key
# - aws_region (optional)

# For GCS, required keys:
# - gcp_service_account_json

Engine Connection Errors:

# Verify runtime can reach engine
kubectl exec <pod-name> -c aibrix-runtime -- curl http://localhost:8000/health

# Check shared volume is mounted correctly
kubectl describe pod <pod-name>
# Both containers should have /tmp/aibrix/adapters mounted

Supported URL Schemes Summary:

URL Scheme

Example

Requires Runtime

Requires Credentials

s3://

s3://bucket/path/

Yes

Yes

gs://

gs://bucket/path/

Yes

Yes

huggingface:// (private)

huggingface://org/model

Yes

Yes (token)

huggingface:// (public)

huggingface://meta-llama/model

No

No

http:// / https://

https://example.com/adapter

No

Optional

Best Practices:

  1. Use Separate Secrets per Storage Type: Create dedicated secrets for S3, GCS credentials rather than mixing them

  2. Test Credentials First: Verify credentials work by manually downloading from storage before creating ModelAdapter

  3. Monitor Runtime Logs: Watch runtime logs during first deployment to catch credential or connectivity issues early

  4. Shared Storage Volume: Ensure the adapter-storage volume is properly shared between runtime and engine containers

  5. Artifact Size: Large artifacts may take time to download; adjust timeout values if needed

Runtime Metrics#

The runtime’s /metrics endpoint, the aibrix: metric names, and METRICS_ENABLE_TRANSFORMATION / METRICS_RAW_PASSTHROUGH_MODE are documented in AI Engine Runtime. The runtime process only starts with INFERENCE_ENGINE=vllm. Rule sets for SGLang and TRT-LLM exist in the code and are not selectable today.