Serve with TokenSpeed

TokenSpeed recipe for Beam

The full reference for serving Beam with LightSeek’s TokenSpeed.

This page walks through one setup, for development and evaluation: TokenSpeed in Docker on one server with four B300 GPUs, using the NVFP4 weights and the DFlash2 drafter for speculative decoding. Before you start, check the requirements and download the weights. The commands below use the MODEL_DIR that page sets.

Get the image

Select a TokenSpeed image release with Beam and DFlash2 support, including the Beam tool-call and reasoning parsers in its bundled gateway, and pin it by digest:

Pull and pin the image

export IMAGE='lightseekorg/tokenspeed:beam'
docker pull "$IMAGE"
export IMAGE="$(docker image inspect "$IMAGE" \
  --format '{{index .RepoDigests 0}}')"

Start the server

Run on your GPU server:

Start the server

docker run --rm --init --name beam \
  --gpus '"device=0,1,2,3"' --shm-size=64g \
  -p 127.0.0.1:8000:8000 \
  --mount "type=bind,src=$MODEL_DIR,dst=/weights,readonly" \
  -e HF_HUB_OFFLINE=1 -e TRANSFORMERS_OFFLINE=1 \
  --entrypoint tokenspeed "$IMAGE" \
  serve /weights/nvfp4 \
  --host 0.0.0.0 --port 8000 \
  --served-model-name Beam-1.0-501B-A23B \
  --tensor-parallel-size 4 --dtype bfloat16 \
  --quantization nvfp4 --kv-cache-dtype fp8_e4m3 \
  --max-model-len 131072 --max-num-seqs 8 \
  --gpu-memory-utilization 0.9 \
  --chunked-prefill-size 8192 --max-cudagraph-capture-size 8 \
  --attention-backend trtllm_mha --moe-backend flashinfer_trtllm \
  --enable-prefix-caching --disable-kvstore \
  --tool-call-parser beam --reasoning-parser beam \
  --drafter-attention-backend trtllm_mha \
  --speculative-config '{"method":"dflash","model":"/weights/dflash2","num_speculative_tokens":16}'

The container reads the models offline from /weights and splits Beam across the four GPUs with tensor parallelism. This configuration accepts up to 131,072 tokens of context and eight concurrent requests, parses tool calls and reasoning with the beam parsers, and drafts 16 tokens at a time with DFlash2. TokenSpeed’s draft width matches the drafter model’s block_size.

Make a request

Once the server finishes loading, check that it lists Beam-1.0-501B-A23B:

List the served models

curl --fail-with-body http://127.0.0.1:8000/v1/models

Then send a chat request:

Send a chat request

curl --fail-with-body http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "Beam-1.0-501B-A23B",
    "messages": [{"role": "user", "content": "Explain tensor parallelism briefly."}],
    "max_tokens": 2048,
    "temperature": 0.7,
    "top_p": 0.99,
    "chat_template_kwargs": {"allow_missing_thinking": false}
  }'

The request sets Beam’s recommended temperature and top_p. TokenSpeed doesn’t read them from the checkpoint, so send them with every request.It also sets allow_missing_thinking to false, so the request fails if an earlier assistant turn is missing its reasoning, which would otherwise degrade Beam’s output. TokenSpeed has no server-wide default for chat template options, so send this with every request too. See Reasoning across turns.

Stop the server

docker stop --time 60 beam

We’re grateful to LightSeek and NVIDIA for their guidance and support in developing and refining our Beam serving recipes for TokenSpeed.

Get started

Self-hosting

Third-party providers

Tools

TokenSpeed recipe for Beam