Version

Scale the model

On this page

A ModelService can front more than one ModelDeployment. Here you add a second deployment, pinned to a different region, and point the same service at both. The endpoint you already curled stays the same. Behind it, traffic now load-balances across two regions.

graph LR
    subgraph fleet ["Fleet"]
        IC1["cluster-a\nsmall GPU"]
        IC2["cluster-b\nlarger GPU"]
    end

    subgraph ml ["ML team"]
        MD1["ModelDeployment\nqwen-demo"]
        MD2["ModelDeployment\nqwen-west\ntargets cluster-b"]
        MS["ModelService qwen\n/ml-team/qwen/v1/..."]
    end

    IC1 --> MD1
    IC2 --> MD2
    MD1 --> MS
    MD2 --> MS

Add a second deployment

The new deployment targets a larger-GPU cluster you added in the last step. On EKS, GKE, and AKS it pins to a second region with a clusterSelector. On Nebius, which runs one region per project, the capacity selector alone routes it to the H100 tier:

# A second ModelDeployment, pinned to a different region with clusterSelector.
# It carries its own deployment name, so the ModelService can front it alongside
# qwen-demo and serve both from one endpoint.
apiVersion: modelplane.ai/v1alpha1
kind: ModelDeployment
metadata:
  name: qwen-west
  namespace: ml-team
spec:
  replicas: 1
  # clusterSelector filters which InferenceClusters this deployment can land on.
  # This pins the replica to the us-west cluster you added in "Scale the fleet".
  template:
    spec:
      clusterSelector:
        matchLabels:
          modelplane.ai/region: us-west
      engines:
      - name: qwen
        members:
        - role: Standalone
          nodeSelector:
            devices:
            - name: gpu
              count: 1
              selectors:
          # L40S (46068Mi) qualifies; L4 (23034Mi) does not.
              - cel: |
                  device.capacity["gpu.nvidia.com"].memory.compareTo(quantity("40Gi")) >= 0
          template:
            spec:
              containers:
              - name: engine
                image: vllm/vllm-openai:v0.23.0
                args:
                - --model=Qwen/Qwen2.5-0.5B-Instruct
                - --dtype=half

# A second ModelDeployment, pinned to a different region with clusterSelector.
# It carries its own deployment name, so the ModelService can front it alongside
# qwen-demo and serve both from one endpoint.
apiVersion: modelplane.ai/v1alpha1
kind: ModelDeployment
metadata:
  name: qwen-west
  namespace: ml-team
spec:
  replicas: 1
  # clusterSelector filters which InferenceClusters this deployment can land on.
  # This pins the replica to the us-west cluster you added in "Scale the fleet".
  template:
    spec:
      clusterSelector:
        matchLabels:
          modelplane.ai/region: us-west
      engines:
      - name: qwen
        members:
        - role: Standalone
          nodeSelector:
            devices:
            - name: gpu
              count: 1
              selectors:
          # A100 40GB (40960Mi) qualifies; L4 (23034Mi) does not.
          # Threshold at 35Gi not 40Gi to avoid the boundary on A100's exact reported VRAM.
              - cel: |
                  device.capacity["gpu.nvidia.com"].memory.compareTo(quantity("35Gi")) >= 0
          template:
            spec:
              containers:
              - name: engine
                image: vllm/vllm-openai:v0.23.0
                args:
                - --model=Qwen/Qwen2.5-0.5B-Instruct
                - --dtype=half

# A second ModelDeployment, pinned to a different region with clusterSelector.
# It carries its own deployment name, so the ModelService can front it alongside
# qwen-demo and serve both from one endpoint.
apiVersion: modelplane.ai/v1alpha1
kind: ModelDeployment
metadata:
  name: qwen-west
  namespace: ml-team
spec:
  replicas: 1
  # clusterSelector filters which InferenceClusters this deployment can land on.
  # This pins the replica to the eastus cluster you added in "Scale the fleet".
  template:
    spec:
      clusterSelector:
        matchLabels:
          modelplane.ai/region: eastus
      engines:
      - name: qwen
        members:
        - role: Standalone
          nodeSelector:
            devices:
            - name: gpu
              count: 1
              selectors:
          # A100 (81920Mi) qualifies; A10 (24564Mi) does not.
              - cel: |
                  device.capacity["gpu.nvidia.com"].memory.compareTo(quantity("35Gi")) >= 0
          template:
            spec:
              containers:
              - name: engine
                image: vllm/vllm-openai:v0.23.0
                args:
                - --model=Qwen/Qwen2.5-0.5B-Instruct
                - --dtype=half

# A second ModelDeployment fronted alongside qwen-demo from one endpoint.
# Nebius runs one region per project, so instead of pinning to another region
# this deployment selects the larger GPU (H100) you added in "Scale the fleet".
# The VRAM selector alone routes it to the H100 cluster - no clusterSelector.
apiVersion: modelplane.ai/v1alpha1
kind: ModelDeployment
metadata:
  name: qwen-west
  namespace: ml-team
spec:
  replicas: 1
  template:
    spec:
      engines:
      - name: qwen
        members:
        - role: Standalone
          nodeSelector:
            devices:
            - name: gpu
              count: 1
              selectors:
          # H100 (81559Mi) qualifies; L40S (46068Mi) does not.
              - cel: |
                  device.capacity["gpu.nvidia.com"].memory.compareTo(quantity("60Gi")) >= 0
          template:
            spec:
              containers:
              - name: engine
                image: vllm/vllm-openai:v0.23.0
                args:
                - --model=Qwen/Qwen2.5-0.5B-Instruct
                - --dtype=half

Wait until its replica is Ready, then check placement. You now have one replica per region:

kubectl get modelreplica -n ml-team

NAME              CLUSTER       SYNCED   READY   COMPOSITION                   AGE
qwen-demo-7323a   eks-us-east   True     True    modelreplicas.modelplane.ai   42m
qwen-west-92535   eks-us-west   True     True    modelreplicas.modelplane.ai   8m

Front both with one service

Update the ModelService to select both deployments. Each entry in spec.endpoints adds its matching replicas to the same endpoint:

# The same ModelService as before, now selecting two deployments. Each entry in
# spec.endpoints adds every ModelEndpoint matching its selector to the same
# OpenAI-compatible endpoint, so /ml-team/qwen/ load-balances across qwen-demo
# and qwen-west, wherever their replicas run.
apiVersion: modelplane.ai/v1alpha1
kind: ModelService
metadata:
  name: qwen
  namespace: ml-team
spec:
  endpoints:
  - selector:
      matchLabels:
        modelplane.ai/deployment: qwen-demo
  - selector:
      matchLabels:
        modelplane.ai/deployment: qwen-west

The endpoint URL doesn’t change. Clients that had this URL before still have it; they don’t know the fleet changed. The gateway load-balances across both regions, and losing one region keeps the other serving. Send the same request as before:

ADDRESS=$(kubectl get ms qwen -n ml-team -o jsonpath='{.status.address}')

kubectl run -i --rm curl-test \
  --image=curlimages/curl \
  --restart=Never \
  --env="ADDRESS=$ADDRESS" \
  -- sh -c 'curl -v "$ADDRESS/v1/chat/completions" \
  -H "Content-Type: application/json" \
  -d "{\"model\":\"Qwen/Qwen2.5-0.5B-Instruct\",\"messages\":[{\"role\":\"user\",\"content\":\"What is Kubernetes in one sentence?\"}],\"max_tokens\":100}"'

That’s the tour

You stood up a control plane, built a multi-region GPU fleet, deployed a model across it, and ended with one stable endpoint serving requests. The platform team published hardware. The ML team described what the model needs. Modelplane placed them and served behind a single endpoint.

Clean up tears everything down when you’re done.

For more on the resources you used:

Modelplane is in active development and we’re building in the open. If you’re running your own inference fleet and want to shape where this goes, we’d love to hear from you. Star the repository, join us in Slack, or read the manifesto.