Serve with vLLM

vLLM recipe for Beam

The full reference for serving Beam with vLLM.

This page walks through one setup, for development and evaluation: vLLM in Docker on one server with four B300 GPUs, using the NVFP4 weights and the DFlash2 drafter for speculative decoding. Before you start, check the requirements and download the weights. The commands below use the MODEL_DIR that page sets.

Get the image

Select a vLLM image release with Beam and DFlash support, and pin it by digest:

Pull and pin the image

export IMAGE='vllm/vllm-openai:reflection-beam'
docker pull "$IMAGE"
export IMAGE="$(docker image inspect "$IMAGE" \
  --format '{{index .RepoDigests 0}}')"

Start the server

Run on your GPU server:

Start the server

docker run --rm --init --name beam \
  --gpus '"device=0,1,2,3"' --shm-size=32g \
  -p 127.0.0.1:8000:8000 \
  --mount "type=bind,src=$MODEL_DIR,dst=/weights,readonly" \
  -e HF_HUB_OFFLINE=1 -e TRANSFORMERS_OFFLINE=1 \
  --entrypoint vllm "$IMAGE" \
  serve /weights/nvfp4 \
  --host 0.0.0.0 --port 8000 \
  --served-model-name Beam-1.0-501B-A23B \
  --tensor-parallel-size 4 \
  --dtype bfloat16 \
  --max-model-len 262144 --max-num-seqs 8 \
  --gpu-memory-utilization 0.95 \
  --enable-prefix-caching \
  --enable-auto-tool-choice \
  --tool-call-parser beam --reasoning-parser beam \
  --default-chat-template-kwargs '{"allow_missing_thinking": false}' \
  --structured-outputs-config '{"backend":"xgrammar","reasoning_parser":"beam","disable_any_whitespace":true}' \
  --speculative-config '{"method":"dflash","model":"/weights/dflash2","num_speculative_tokens":8}'

The container reads the models offline from /weights and splits Beam across the four GPUs with tensor parallelism. This configuration accepts up to 262,144 tokens of context and eight concurrent requests, parses tool calls and reasoning with the beam parsers, and drafts eight tokens at a time with DFlash2. Structured outputs use the xgrammar backend.allow_missing_thinking set to false makes a request fail if an earlier assistant turn is missing its reasoning, which would otherwise degrade Beam’s output. See Reasoning across turns.

Make a request

Once the server finishes loading, check that it lists Beam-1.0-501B-A23B:

List the served models

curl --fail-with-body http://127.0.0.1:8000/v1/models

Then send a chat request:

Send a chat request

curl --fail-with-body http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "Beam-1.0-501B-A23B",
    "messages": [{"role": "user", "content": "Explain tensor parallelism briefly."}],
    "max_tokens": 2048
  }'

A request that doesn’t set temperature or top_p uses Beam’s recommended 0.7 and 0.99, which vLLM reads from the checkpoint’s generation_config.json.

Stop the server

docker stop --time 60 beam

We’re grateful to Inferact and NVIDIA for their guidance and support in developing and refining our Beam serving recipes for vLLM.

Get started

Self-hosting

Third-party providers

Tools

vLLM recipe for Beam