Scaling out with Dynamo

NVIDIA Dynamo is an open-source framework for distributed inference. Use it with SGLang or vLLM to serve Beam across multiple replicas and GPU nodes. Its KV-aware router balances worker load with prompt-cache reuse, and it supports running prefill and decode on separate workers. See NVIDIA’s Beam deployment recipe for deployment instructions, or Serving in production with Kubernetes to use the reflection-serving Helm chart.

Pull a maintained image

NVIDIA’s NGC images bundle Dynamo with your selected engine. Use the same image for the frontend and workers.

SGLang
vLLM

Pull the image

docker pull nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.6.0-beam-1.0-501b-dev.1

Pull the image

docker pull nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.6.0-beam-1.0-501b-dev.1

Build from public source

Prerequisites:

Linux x86-64 build machine
Docker with BuildKit enabled
Git
Python 3 with venv support

The commands build each repository’s default branch. Use the same Docker daemon for both steps.

SGLang
vLLM

Build the engine image

SGLang’s upstream Dockerfile builds the runtime image and includes a packaged version of Dynamo. BRANCH_TYPE=local builds the checked-out SGLang source.

Build the engine image

git clone https://github.com/sgl-project/sglang.git
cd sglang
docker build --platform linux/amd64 \
  -f docker/Dockerfile --target runtime \
  --build-arg BRANCH_TYPE=local \
  --build-arg INSTALL_FLASHINFER_JIT_CACHE=1 \
  -t beam-sglang:local .
cd ..

Add Dynamo

Build Dynamo from source to replace the packaged version in the engine image. The RUNTIME_IMAGE arguments below select beam-sglang:local as the runtime base. The rendered Dockerfile builds Dynamo’s wheels and installs them into that base, producing beam-sglang-dynamo:local for the frontend and workers.

Add Dynamo

git clone https://github.com/ai-dynamo/dynamo.git
cd dynamo
python3 -m venv .build-venv
.build-venv/bin/pip install jinja2 pyyaml
.build-venv/bin/python container/render.py \
  --framework sglang --target runtime \
  --platform linux/amd64 --cuda-version 13.0 \
  --output-short-filename
docker build --platform linux/amd64 \
  -f container/rendered.Dockerfile \
  --build-arg RUNTIME_IMAGE=beam-sglang \
  --build-arg RUNTIME_IMAGE_TAG=local \
  -t beam-sglang-dynamo:local .
cd ..

Build the engine image

vLLM’s upstream Dockerfile builds vLLM and its API server. INSTALL_KV_CONNECTORS=true adds NIXL dependencies.

Build the engine image

git clone https://github.com/vllm-project/vllm.git
cd vllm
docker build --platform linux/amd64 \
  -f docker/Dockerfile --target vllm-openai \
  --build-arg INSTALL_KV_CONNECTORS=true \
  -t beam-vllm:local .
cd ..

Add Dynamo

The engine image above does not include Dynamo. The RUNTIME_IMAGE arguments below select beam-vllm:local as the runtime base. The rendered Dockerfile builds Dynamo’s wheels and installs them into that base, producing beam-vllm-dynamo:local for the frontend and workers.

Add Dynamo

git clone https://github.com/ai-dynamo/dynamo.git
cd dynamo
python3 -m venv .build-venv
.build-venv/bin/pip install jinja2 pyyaml
.build-venv/bin/python container/render.py \
  --framework vllm --target runtime \
  --platform linux/amd64 --cuda-version 13.0 \
  --output-short-filename
docker build --platform linux/amd64 \
  -f container/rendered.Dockerfile \
  --build-arg RUNTIME_IMAGE=beam-vllm \
  --build-arg RUNTIME_IMAGE_TAG=local \
  -t beam-vllm-dynamo:local .
cd ..

Get started

Self-hosting

Third-party providers

Tools

Pull a maintained image

Build from public source

​Pull a maintained image

​Build from public source

Pull a maintained image

Build from public source