Serve with SGLang

SGLang recipe for Beam

The full reference for serving Beam with SGLang.

This page walks through one setup, for development and evaluation: SGLang in Docker on one server with four B300 GPUs, using the NVFP4 weights and the DFlash2 drafter for speculative decoding. Before you start, check the requirements and download the weights. The commands below use the MODEL_DIR that page sets.

Get the image

Select an SGLang image release with Beam and DFlash support, and pin it by digest:

Pull and pin the image

export IMAGE='lmsysorg/sglang:beam'
docker pull "$IMAGE"
export IMAGE="$(docker image inspect "$IMAGE" \
  --format '{{index .RepoDigests 0}}')"

Start the server

Run on your GPU server:

Start the server

docker run --rm --init --name beam \
  --gpus '"device=0,1,2,3"' --shm-size=32g \
  -p 127.0.0.1:8000:8000 \
  --mount "type=bind,src=$MODEL_DIR,dst=/weights,readonly" \
  -e HF_HUB_OFFLINE=1 -e TRANSFORMERS_OFFLINE=1 \
  --entrypoint python3 "$IMAGE" \
  -m sglang.launch_server \
  --host 0.0.0.0 --port 8000 \
  --model-path /weights/nvfp4 --served-model-name Beam-1.0-501B-A23B \
  --tp-size 4 --dtype bfloat16 \
  --context-length 262144 --max-running-requests 8 \
  --mem-fraction-static 0.9 --max-total-tokens 262144 \
  --attention-backend trtllm_mha --moe-runner-backend flashinfer_trtllm \
  --kv-cache-dtype fp8_e4m3 \
  --chunked-prefill-size 8192 --max-prefill-tokens 8192 \
  --cuda-graph-max-bs-decode 8 --page-size 64 \
  --swa-full-tokens-ratio 0.25 \
  --tool-call-parser beam --reasoning-parser beam \
  --default-chat-template-kwargs '{"allow_missing_thinking": false}' \
  --speculative-algorithm DFLASH \
  --speculative-draft-model-path /weights/dflash2 \
  --speculative-draft-load-format safetensors \
  --speculative-num-draft-tokens 8 \
  --speculative-draft-attention-backend trtllm_mha \
  --speculative-draft-kv-cache-dtype fp8_e4m3 \
  --speculative-draft-window-size 4096 \
  --json-model-override-args '{"max_position_embeddings":262144}' \
  --weight-loader-disable-mmap --weight-loader-drop-cache-after-load \
  --model-loader-extra-config '{"enable_multithread_load":false}'

The container reads the models offline from /weights and splits Beam across the four GPUs with tensor parallelism. This configuration accepts up to 262,144 tokens of context and eight concurrent requests, parses tool calls and reasoning with the beam parsers, and drafts eight tokens at a time with DFlash2.allow_missing_thinking set to false makes a request fail if an earlier assistant turn is missing its reasoning, which would otherwise degrade Beam’s output. See Reasoning across turns.

Make a request

Once the server finishes loading, check that it lists Beam-1.0-501B-A23B:

List the served models

curl --fail-with-body http://127.0.0.1:8000/v1/models

Then send a chat request:

Send a chat request

curl --fail-with-body http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "Beam-1.0-501B-A23B",
    "messages": [{"role": "user", "content": "Explain tensor parallelism briefly."}],
    "max_tokens": 2048
  }'

A request that doesn’t set temperature or top_p uses Beam’s recommended 0.7 and 0.99, which SGLang reads from the checkpoint’s generation_config.json.

Stop the server

docker stop --time 60 beam

We’re grateful to RadixArk and NVIDIA for their guidance and support in developing and refining our Beam serving recipes for SGLang.

Get started

Self-hosting

Third-party providers

Tools

SGLang recipe for Beam