Multi-Node Inference

Multi-Node Inference#

Distributed inference splits and processes an LLM across multiple nodes or devices. This approach is needed for large models that exceed the memory capacity of a single machine.

AIBrix provides two orchestration paths for multi-node inference:

  1. Ray-based Orchestration (RayClusterFleet / RayClusterReplicaSet): Uses KubeRay for intra-application worker placement and coordination, with Kubernetes managing replica scaling and rollouts.

  2. Native PodSet Orchestration (StormService): Kubernetes-native multi-role and multi-node grouping via podGroupSize without requiring KubeRay.

Choosing an Orchestration Abstraction#

Operators can pick the abstraction matching their deployment topology and infrastructure setup:

Abstraction

Infrastructure Requirement

Best Suited For

RayClusterFleet

KubeRay operator installed

Standard multi-node vLLM deployments where Ray handles process placement and worker coordination.

StormService

Native Kubernetes (no KubeRay required)

Prefill-Decode (PD) disaggregated setups, custom multi-role architectures, or direct engine-native distributed backends (like SGLang or vLLM with MPI/NCCL and RDMA networking).

KubeRay Orchestration (RayClusterFleet)#

In distributed computing, managing multi-node inference requires coordination at two layers: fine-grained task execution inside the cluster, and standard operational management from Kubernetes.

Ray handles intra-application task scheduling and worker communication well, but relies on external systems for cluster lifecycle operations. Kubernetes excels at container scheduling, autoscaling, and rolling updates.

AIBrix combines both: Ray handles internal distributed computation, while Kubernetes manages replica lifecycle and environment setup.

Two key APIs manage Ray clusters: RayClusterReplicaSet and RayClusterFleet. These mirror Kubernetes ReplicaSet and Deployment patterns. In most cases, RayClusterFleet is the primary resource to configure.

mix-grain-orchestration

  • Ray Framework Focus: Ray handles intra-application orchestration. Each application instance corresponds to a single Ray cluster.

  • Kubernetes Layer: Kubernetes operates at the outer layer, handling Ray cluster creation, autoscaling, and rolling updates.

  • Service Encapsulation: Services map to Ray clusters representing application instances rather than single pods.

Attention

We already submitted our ideas to the KubeRay community.

How it works#

        graph TD
    Fleet["RayClusterFleet<br/>rollout, revision history, pause"] -->|owns| RS["RayClusterReplicaSet<br/>keeps N Ray clusters alive"]
    RS -->|creates| RC1["RayCluster (KubeRay)"]
    RS -->|creates| RC2["RayCluster (KubeRay)"]
    subgraph RC1
        H1["head pod<br/>inference engine + GPU"]
        W1["worker pod(s)<br/>GPU"]
    end
    subgraph RC2
        H2["head pod"]
        W2["worker pod(s)"]
    end
    GW["Gateway"] -.routes only to head pods.-> H1
    GW -.-> H2

The layers, from the outside in:

  • RayClusterFleet carries the rollout semantics of a Deployment: a rolling update or recreate strategy, revision history, paused, minReadySeconds and a progress deadline. Every change to spec.template produces a new RayClusterReplicaSet and the fleet shifts replicas between old and new sets according to strategy.

  • RayClusterReplicaSet keeps a fixed number of KubeRay RayCluster objects running and replaces any that disappear.

  • RayCluster is KubeRay’s resource. Its spec comes from spec.template.spec of the fleet, with only the fleet-name label added to the head and worker pod templates, so anything KubeRay supports (rayVersion, headGroupSpec, workerGroupSpecs, rayStartParams) is available.

  • The engine runs on the head pod with Ray as its distributed executor (--distributed-executor-backend ray for vLLM). Worker pods only run ray start and contribute their GPUs to the Ray cluster.

Readiness. A Ray cluster counts as ready only when KubeRay reports both the RayClusterProvisioned and HeadPodReady conditions as True and every desired worker is ready. Those conditions are produced by KubeRay’s RayClusterStatusConditions feature gate, which the AIBrix installation instructions enable. Without the gate the fleet can never report ready replicas.

Routing. The gateway discovers model pods by the model.aibrix.ai/name label, but it ignores pods labelled ray.io/node-type: worker. Requests are therefore routed to head pods only. The fleet controller stamps every pod with orchestration.aibrix.ai/raycluster-fleet-name so that metrics and routing state can be mapped back to the fleet that owns the pod.

Prerequisites#

  • The KubeRay operator. It is optional for the rest of AIBrix and only needed for RayClusterFleet and RayClusterReplicaSet. Install it with the Helm command in Installation; that command pins a patched operator image and turns on the RayClusterStatusConditions feature gate that readiness depends on.

  • GPU nodes for the head pod and each worker pod.

  • An engine image that contains Ray. Official vLLM images from v0.6.6 onward work out of the box; for older versions see Container Image Requirements below.

Configuration reference#

Both resources live in the orchestration.aibrix.ai/v1alpha1 API group.

RayClusterFleet spec

Field

Type

Description

replicas

int32

Number of Ray clusters to run. Defaults to 1.

selector

LabelSelector

Must match the labels in template.metadata.labels. Required.

template

RayClusterTemplateSpec

metadata and spec for each Ray cluster. spec is a KubeRay RayClusterSpec and is passed through, with the fleet-name label added to the pod templates.

strategy

DeploymentStrategy

Recreate or RollingUpdate with maxSurge and maxUnavailable, same semantics as a Deployment.

minReadySeconds

int32

How long a Ray cluster must stay ready before it counts as available.

revisionHistoryLimit

int32

Number of old RayClusterReplicaSet objects to keep for rollback.

paused

bool

Stop the controller from acting on template changes.

progressDeadlineSeconds

int32

Seconds after which a stalled rollout is reported as failed in status.conditions.

RayClusterFleet status reports replicas, updatedReplicas, readyReplicas, availableReplicas, unavailableReplicas, observedGeneration, conditions and scalingTargetSelector. The fleet exposes the Kubernetes scale subresource, so kubectl scale rayclusterfleet <name> --replicas=N works, and a PodAutoscaler can use kind: RayClusterFleet as its scaleTargetRef.

RayClusterReplicaSet spec is the subset a ReplicaSet would have: replicas, selector, template and minReadySeconds. You normally never create one directly.

Labels and annotations that matter

Key

Purpose

model.aibrix.ai/name (label)

Set it on the head and worker pod templates. The gateway discovers the model’s pods through this label. (The PodAutoscaler uses the fleet’s scale selector instead.)

ray.io/overwrite-container-cmd: "true" (annotation on the Ray cluster template)

Tells KubeRay to respect the container command and args you wrote instead of generating its own ray start command. KubeRay still injects the generated command into the env var KUBERAY_GEN_RAY_START_CMD so you can run it yourself, which is what the sample does. The generated variable does not include ulimit, so set that in your own command.

ray.io/node-type (label, set by KubeRay)

head or worker. The gateway skips worker pods when routing.

orchestration.aibrix.ai/raycluster-fleet-name (label, set by the fleet controller)

Maps a pod back to its fleet. Do not set it yourself.

Parallelism sizing. With the Ray executor, the engine’s tensor-parallel size must equal the number of GPUs in the whole Ray cluster (head plus workers). The sample below runs --tensor-parallel-size 2 on a head pod with one GPU and one worker pod with one GPU.

RayClusterFleet Example#

Below is a RayClusterFleet example deploying a two-node distributed inference cluster:

apiVersion: orchestration.aibrix.ai/v1alpha1
kind: RayClusterFleet
metadata:
  name: qwen-coder-7b-instruct
  labels:
    app.kubernetes.io/name: aibrix
    app.kubernetes.io/managed-by: kustomize
spec:
  replicas: 1
  selector:
    matchLabels:
      model.aibrix.ai/name: qwen-coder-7b-instruct
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
  template:
    metadata:
      labels:
        model.aibrix.ai/name: qwen-coder-7b-instruct
      annotations:
        ray.io/overwrite-container-cmd: "true"
    spec:
      rayVersion: "2.10.0"
      headGroupSpec:
        rayStartParams:
          dashboard-host: "0.0.0.0"
        template:
          metadata:
            labels:
              model.aibrix.ai/name: qwen-coder-7b-instruct
          spec:
            containers:
              - name: ray-head
                image: vllm/vllm-openai:v0.7.1
                command: ["/bin/bash", "-c"]
                args:
                  - >
                    ulimit -n 65536 &&
                    apt update && apt install -y wget net-tools && pip3 install ray[default] pyarrow pandas &&
                    echo "[INFO] Starting Ray head node..." &&
                    eval "$KUBERAY_GEN_RAY_START_CMD" &

                    echo "[INFO] Waiting for Ray dashboard to be ready..." &&
                    until curl --max-time 5 --fail http://127.0.0.1:8265 > /dev/null 2>&1; do
                      echo "[WAITING] $(date -u +'%Y-%m-%dT%H:%M:%SZ') - Ray dashboard not ready yet...";
                      sleep 2;
                    done &&
                    echo "[SUCCESS] Ray dashboard is available!" &&

                    vllm serve Qwen/Qwen2.5-Coder-7B-Instruct \
                      --served-model-name qwen-coder-7b-instruct \
                      --tensor-parallel-size 2 \
                      --distributed-executor-backend ray \
                      --host 0.0.0.0 \
                      --port 8000 \
                      --dtype half
                ports:
                  - containerPort: 6379
                    name: gcs-server
                  - containerPort: 8265
                    name: dashboard
                  - containerPort: 10001
                    name: client
                  - containerPort: 8000
                    name: service
                resources:
                  limits:
                    cpu: "4"
                    nvidia.com/gpu: 1
                  requests:
                    cpu: "4"
                    nvidia.com/gpu: 1
              - name: aibrix-runtime
                image: aibrix/runtime:v0.3.0
                command:
                  - aibrix_runtime
                  - --port
                  - "8080"
                env:
                  - name: INFERENCE_ENGINE
                    value: vllm
                  - name: INFERENCE_ENGINE_ENDPOINT
                    value: http://localhost:8000
                  - name: PYTORCH_CUDA_ALLOC_CONF
                    value: "expandable_segments:True"
                ports:
                  - containerPort: 8080
                    protocol: TCP
                livenessProbe:
                  httpGet:
                    path: /healthz
                    port: 8080
                  initialDelaySeconds: 3
                  periodSeconds: 2
                readinessProbe:
                  httpGet:
                    path: /ready
                    port: 8080
                  initialDelaySeconds: 5
                  periodSeconds: 10
                resources:
                  limits:
                    cpu: "1"
                  requests:
                    cpu: "1"
      workerGroupSpecs:
        - groupName: small-group
          replicas: 1
          minReplicas: 1
          maxReplicas: 5
          rayStartParams: {}
          template:
            metadata:
              labels:
                model.aibrix.ai/name: qwen-coder-7b-instruct
            spec:
              containers:
                - name: ray-worker
                  image: vllm/vllm-openai:v0.7.1
                  env:
                    - name: MY_POD_IP
                      valueFrom:
                        fieldRef:
                          fieldPath: status.podIP
                  command: [ "/bin/bash", "-c" ]
                  args:
                    - >
                      ulimit -n 65536 &&
                      eval "$KUBERAY_GEN_RAY_START_CMD --node-ip-address=$MY_POD_IP" &&
                      tail -f /dev/null
                  lifecycle:
                    preStop:
                      exec:
                        command: [ "/bin/sh", "-c", "ray stop" ]
                  resources:
                    limits:
                      cpu: "4"
                      nvidia.com/gpu: 1
                    requests:
                      cpu: "4"
                      nvidia.com/gpu: 1

---

apiVersion: v1
kind: Service
metadata:
  name: qwen-coder-7b-instruct
  labels:
    model.aibrix.ai/name: qwen-coder-7b-instruct
    prometheus-discovery: "true"
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: "8080"
spec:
  selector:
    model.aibrix.ai/name: qwen-coder-7b-instruct
  ports:
    - name: serve
      port: 8000
      protocol: TCP
      targetPort: 8000
    - name: http
      port: 8080
      protocol: TCP
      targetPort: 8080

---

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: qwen-coder-7b-instruct-router
  namespace: aibrix-system
spec:
  parentRefs:
    - group: gateway.networking.k8s.io
      kind: Gateway
      name: aibrix-eg
      namespace: aibrix-system
  rules:
    - backendRefs:
        - group: ""
          kind: Service
          name: qwen-coder-7b-instruct
          namespace: default
          port: 8000  # or 8000 if you're not using the runtime sidecar
          weight: 1
      matches:
        - headers:
            - name: model
              type: Exact
              value: qwen-coder-7b-instruct
          path:
            type: PathPrefix
            value: /v1/completions
        - headers:
            - name: model
              type: Exact
              value: qwen-coder-7b-instruct
          path:
            type: PathPrefix
            value: /v1/chat/completions
      timeouts:
        request: 120s

What the sample is doing, section by section:

  • The head container raises the file-descriptor limit, installs the Ray dashboard dependencies, runs the KubeRay-generated ray start command in the background, waits until the Ray dashboard on port 8265 answers, and only then launches vllm serve with --distributed-executor-backend ray. Waiting for the dashboard matters: vLLM connects to the Ray cluster at startup and fails if the head is not up yet.

  • The worker container runs the generated ray start command with its own pod IP and then blocks with tail -f /dev/null. A preStop hook calls ray stop so the node leaves the cluster cleanly.

  • The AI Runtime sidecar on the head pod exposes standardized metrics on port 8080 and provides the liveness and readiness probes for the pod. See AI Engine Runtime.

  • The Service selects pods by model.aibrix.ai/name and carries the prometheus-discovery: "true" label so metrics are scraped.

  • The HTTPRoute attaches the model to the AIBrix gateway by matching the model header. This is the same route shape used for single-pod deployments; see Deploying Gateway.

Verify the deployment#

# Fleet, its replica set, and the KubeRay clusters it created
kubectl get rayclusterfleet
kubectl get rayclusterreplicaset
kubectl get raycluster

# Head and worker pods
kubectl get pods -l ray.io/node-type=head
kubectl get pods -l ray.io/node-type=worker

The fleet CRD defines no extra printer columns, so compare the counts directly:

kubectl get rayclusterfleet qwen-coder-7b-instruct \
  -o jsonpath='{.status.readyReplicas}/{.spec.replicas}{"\n"}'

The fleet is healthy when both numbers match. Then send a request through the gateway exactly as you would for a single-pod model:

kubectl -n envoy-gateway-system port-forward service/envoy-aibrix-system-aibrix-eg-903790dc 8888:80 &

curl http://localhost:8888/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "model: qwen-coder-7b-instruct" \
  -d '{"model": "qwen-coder-7b-instruct", "messages": [{"role": "user", "content": "hello"}]}'

Troubleshooting#

The fleet never reports ready replicas. Run kubectl describe raycluster <name> and look at Status.Conditions. AIBrix requires RayClusterProvisioned and HeadPodReady to be True. If those conditions are absent entirely, the KubeRay operator was installed without the RayClusterStatusConditions feature gate; reinstall it with the command from the installation guide.

Head pod restarts, or vLLM exits with a Ray connection error. The engine started before the Ray head was up. Keep the dashboard wait loop from the sample in front of vllm serve. Also confirm rayVersion in the template matches the Ray version inside the image; a mismatch prevents workers from joining.

Worker pods stay Pending. Each worker requests a GPU. Check node capacity with kubectl describe node and confirm the nvidia.com/gpu request in workerGroupSpecs can be satisfied.

The gateway returns an error for the model although the pods are Running. The gateway only routes to head pods that carry model.aibrix.ai/name and are Ready. Check that the label is on the head pod template (not only on the fleet) and that the readiness probe on the runtime sidecar (port 8080, /ready) is passing.

Native PodSet Orchestration (StormService)#

For deployments that do not run KubeRay, or for disaggregated architectures requiring explicit role separation (such as separate Prefill and Decode roles), AIBrix provides native multi-node grouping via StormService.

Using podGroupSize within a role template, StormService allocates multiple synchronized pods for each replica instance and injects deterministic distributed environment variables (such as $POD_GROUP_INDEX and $PODSET_NAME). This enables engine-native Tensor Parallelism (TP) across multiple nodes.

Key capabilities:

  • No external dependencies: Runs directly on Kubernetes without installing KubeRay.

  • Multi-Role and Disaggregation support: Allows defining separate roles (such as routing, prefill, and decode) with distinct resource profiles and pod group sizes within a single service definition.

  • Deterministic rank and discovery: Pods within a group discover peers via predictable headless service DNS entries (such as ${PODSET_NAME}-0.${STORM_SERVICE_NAME}).

StormService Multi-Node TP Sample#

Below is a complete multi-node Tensor Parallelism example with Prefill/Decode disaggregation (2-node prefill and 2-node decode with podGroupSize: 2 and --nnodes 2 --tp-size 2):

apiVersion: orchestration.aibrix.ai/v1alpha1
kind: StormService
metadata:
  name: tp-1p1d
spec:
  replicas: 1
  updateStrategy:
    type: InPlaceUpdate
  stateful: true
  selector:
    matchLabels:
      app: tp-1p1d
  template:
    metadata:
      labels:
        app: tp-1p1d
    spec:
      roles:
        - name: routing
          replicas: 1
          stateful: true
          template:
            spec:
              containers:
                - name: mini-lb
                  image: aibrix/sglang-router:v0.1.6
                  command: [ "sh", "-c" ]
                  args:
                    - |
                      python3 -m sglang_router.launch_router \
                        --pd-disaggregation \
                        --policy random \
                        --service-discovery \
                        --service-discovery-port 30000 \
                        --prefill-selector storm-service-name=$STORM_SERVICE_NAME role-name=prefill stormservice.orchestration.aibrix.ai/pod-group-index=0 \
                        --decode-selector storm-service-name=$STORM_SERVICE_NAME role-name=decode stormservice.orchestration.aibrix.ai/pod-group-index=0 \
                        --service-discovery-namespace default
        - name: prefill
          replicas: 1
          podGroupSize: 2
          stateful: true
          template:
            metadata:
              annotations:
                k8s.volcengine.com/pod-networks: |
                  [
                    {
                      "cniConf":{
                          "name":"rdma"
                      }
                    }
                  ]
              labels:
                model.aibrix.ai/name: qwen3-8B
                model.aibrix.ai/port: "30000"
                model.aibrix.ai/engine: sglang
            spec:
#              nodeSelector:
#                kubernetes.io/hostname: 192.168.0.6
              containers:
                - name: prefill
                  image: aibrix/sglang:v0.4.9.post3-cu126-nixl-v0.4.1
                  command: ["sh", "-c"]
                  args:
                    - |
                      python3 -m sglang.launch_server \
                        --model-path /models/Qwen3-8B \
                        --served-model-name qwen3-8B \
                        --host 0.0.0.0 \
                        --port 30000 \
                        --disaggregation-mode prefill \
                        --disaggregation-transfer-backend=nixl \
                        --trust-remote-code \
                        --dist-init-addr "${PODSET_NAME}-0.${STORM_SERVICE_NAME}.default.svc.cluster.local:5000" \
                        --nnodes 2 \
                        --node-rank $POD_GROUP_INDEX \
                        --tp-size 2 \
                        --mem-fraction-static 0.8 \
                        --log-level debug
                  env:
                    - name: GLOO_SOCKET_IFNAME
                      value: eth0
                    - name: NCCL_SOCKET_IFNAME
                      value: eth0
                    - name: NCCL_IB_DISABLE
                      value: "0"
                    - name: NCCL_IB_GID_INDEX
                      value: "7"
                    - name: NCCL_DEBUG
                      value: "INFO"
                    - name: UCX_TLS
                      value: ^gga
                  volumeMounts:
                    - name: model-vol
                      mountPath: /models
                    - mountPath: /dev/shm
                      name: shared-mem
                  resources:
                    limits:
                      nvidia.com/gpu: 1
                      vke.volcengine.com/rdma: "1"
                  securityContext:
                    capabilities:
                      add:
                        - IPC_LOCK
              volumes:
                - name: model-vol
                  hostPath:
                    path: /root/models
                    type: Directory
                - emptyDir:
                    medium: Memory
                  name: shared-mem
        - name: decode
          replicas: 1
          podGroupSize: 2
          stateful: true
          template:
            metadata:
              annotations:
                k8s.volcengine.com/pod-networks: |
                  [
                    {
                      "cniConf":{
                          "name":"rdma"
                      }
                    }
                  ]
              labels:
                model.aibrix.ai/name: qwen3-8B
                model.aibrix.ai/port: "30000"
                model.aibrix.ai/engine: sglang
            spec:
#              nodeSelector:
#                kubernetes.io/hostname: 192.168.0.6
              containers:
                - name: decode
                  image: aibrix/sglang:v0.4.9.post3-cu126-nixl-v0.4.1
                  command: ["sh", "-c"]
                  args:
                    - |
                      python3 -m sglang.launch_server \
                        --model-path /models/Qwen3-8B \
                        --served-model-name qwen3-8B \
                        --host 0.0.0.0 \
                        --port 30000 \
                        --disaggregation-mode decode \
                        --disaggregation-transfer-backend=nixl \
                        --trust-remote-code \
                        --dist-init-addr "${PODSET_NAME}-0.${STORM_SERVICE_NAME}.default.svc.cluster.local:5000" \
                        --nnodes 2 \
                        --node-rank $POD_GROUP_INDEX \
                        --tp-size 2 \
                        --mem-fraction-static 0.8 \
                        --log-level debug
                  env:
                    - name: GLOO_SOCKET_IFNAME
                      value: eth0
                    - name: NCCL_SOCKET_IFNAME
                      value: eth0
                    - name: NCCL_IB_DISABLE
                      value: "0"
                    - name: NCCL_IB_GID_INDEX
                      value: "7"
                    - name: NCCL_DEBUG
                      value: "INFO"
                    - name: UCX_TLS
                      value: ^gga
                  volumeMounts:
                    - name: model-vol
                      mountPath: /models
                    - mountPath: /dev/shm
                      name: shared-mem
                  resources:
                    limits:
                      nvidia.com/gpu: 1
                      vke.volcengine.com/rdma: "1"
                  securityContext:
                    capabilities:
                      add:
                        - IPC_LOCK
              volumes:
                - name: model-vol
                  hostPath:
                    path: /root/models
                    type: Directory
                - emptyDir:
                    medium: Memory
                  name: shared-mem

Container Image Requirements#

Attention

Starting from v0.6.6, essential packages to run distributed inference with the official vLLM container image distribution are included out of the box. If you use earlier versions, follow the guidance below to build a compatible image.

If you are using an earlier vLLM version, you have two options:

  • Use our built image aibrix/vllm-openai:v0.6.1.post2-distributed.

  • Build your own image following these steps:

FROM vllm/vllm-openai:v0.6.1.post2
RUN apt update && apt install -y wget
RUN pip3 install ray[default]
ENTRYPOINT [""]

docker build -t aibrix/vllm-openai:v0.6.1.post2-distributed .

See also

Prefill-Decode Disaggregation (PD)

The complete guide to prefill/decode disaggregation with StormService.