Use HF Transformers

This guide runs Beam on your own compute with Hugging Face Transformers. It starts with the high-level pipeline, then uses generate for full control over the prompt and the parsed reply, and covers reasoning effort, tool calling, and parallelism.

Requirements

The checkpoint is reflection/Beam-1.0-501B-A23B. Its BF16 weights take about 1 TB, so the examples split the model across 8 GPUs with more than 1 TB of combined memory. Install Transformers:

Install Transformers

pip install -U transformers torch

Save each example as a script and launch it with one process per GPU:

Launch on 8 GPUs

torchrun --nproc-per-node 8 generate.py

Every process runs the whole script, so each one prints the output.

Quick start with pipeline

pipeline applies the chat template, generates, and parses the reply into thinking and content. DistributedConfig(tp_size=8, ep_size=8) shards the model across the 8 GPUs and places whole experts on each one. See Parallelism.

generate.py

from transformers import DistributedConfig, pipeline

pipe = pipeline(
    "text-generation",
    model="reflection/Beam-1.0-501B-A23B",
    model_kwargs={"distributed_config": DistributedConfig(tp_size=8, ep_size=8)},
)

messages = [{"role": "user", "content": "Explain in one sentence why the sky is usually blue."}]
messages = pipe(messages, max_new_tokens=2048)[0]["generated_text"]
print(messages[-1]["thinking"])
print(messages[-1]["content"])

messages.append({"role": "user", "content": "And why are sunsets red?"})
messages = pipe(messages, max_new_tokens=2048)[0]["generated_text"]
print(messages[-1]["content"])

The pipeline returns the whole conversation, so you can append the next user message and call it again.

Generate with full control

Load the model and tokenizer yourself to pass chat template arguments, inspect the prompt, or change how you generate. Pass the prompt to parse_response as prefix: the chat template writes the start of the reply into the prompt, and the parser needs it.

generate.py

from transformers import AutoModelForCausalLM, AutoTokenizer, DistributedConfig

model_id = "reflection/Beam-1.0-501B-A23B"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id, distributed_config=DistributedConfig(tp_size=8, ep_size=8)
)

messages = [{"role": "user", "content": "Explain in one sentence why the sky is usually blue."}]
inputs = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt", return_dict=True
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=2048)
generated = tokenizer.decode(outputs[0, inputs["input_ids"].shape[1] :])
message = tokenizer.parse_response(generated, prefix=inputs["input_ids"][0])
print(message["thinking"])
print(message["content"])

generate and pipeline sample with Beam’s recommended temperature of 0.7 and top_p of 0.99, which the checkpoint sets as defaults. Pass temperature and top_p to change them, or do_sample=False to decode greedily. The later examples reuse this model and tokenizer. To parse a reply as it streams, tokenizer.get_response_parser(prefix=...) returns a parser that you feed() text chunks to. See response parsing.

Reasoning effort

Beam always reasons before it answers. Pass reasoning_effort to apply_chat_template to choose how much, as an integer or a label: minimal (8), low (32), medium (64), high (128), xhigh (256), or max (512).

inputs = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt", return_dict=True, reasoning_effort="low"
).to(model.device)

Without reasoning_effort, the template uses max. Beam wasn’t trained on efforts below 8, so it doesn’t follow smaller values. With pipeline, pass tokenizer_encode_kwargs={"reasoning_effort": "low"}. If generation hits max_new_tokens while Beam is still reasoning, the parsed message has thinking and no content. The hosted API has its own effort values and defaults. See Reasoning.

Multi-turn conversations

Append each parsed message to the history unchanged. Beam’s output degrades without its earlier reasoning, which the chat template reads from thinking, reasoning_content, or reasoning.

messages.append(message)
messages.append({"role": "user", "content": "And why are sunsets red?"})
inputs = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt", return_dict=True
).to(model.device)

By default, an assistant message without reasoning renders with an empty reasoning block. Pass allow_missing_thinking=False to raise an error instead when add_generation_prompt=True. Set it when you serve Beam, so that clients which drop reasoning get an error instead of lower-quality replies. The Harmony format guide covers reasoning across turns.

Tool calling

Pass the same tools to apply_chat_template and parse_response. The parser uses each tool’s schema to cast arguments to their types. Without tools, every argument is a string.

def get_weather(city: str):
    """
    Get the current weather in a city.

    Args:
        city: The city name.
    """
    return "Sunny, 22 C"


messages = [{"role": "user", "content": "What's the weather like in Paris right now?"}]
inputs = tokenizer.apply_chat_template(
    messages, tools=[get_weather], add_generation_prompt=True, return_tensors="pt", return_dict=True
).to(model.device)
outputs = model.generate(**inputs, max_new_tokens=2048)
generated = tokenizer.decode(outputs[0, inputs["input_ids"].shape[1] :])
message = tokenizer.parse_response(generated, prefix=inputs["input_ids"][0], tools=[get_weather])
print(message["tool_calls"])
# [{'type': 'function', 'function': {'name': 'get_weather', 'arguments': {'city': 'Paris'}}}]

messages.append(message)
for call in message["tool_calls"]:
    result = get_weather(**call["function"]["arguments"])
    messages.append({"role": "tool", "name": call["function"]["name"], "content": result})

Render messages with the same tools and generate again to get the answer. Parsed tool calls have no IDs, and Beam matches results to calls by position, so add the results in the order of the calls.

Chat template options

Argument	Effect
`reasoning_effort`	How much Beam reasons. See Reasoning effort.
`model_identity`	Replaces the default first line of the system message.
`builtin_tools`	Names of tools from `tools` to declare in the system message. Only for `search` and `browse` with the definitions Beam was trained with. The Harmony format guide covers built-in tools.
`allow_missing_thinking`	Defaults to `True`. Set to `False` to raise an error when an earlier assistant message has no reasoning, as recommended for serving.

Put system and developer messages at the start of the conversation. The template drops them anywhere else.

Parallelism

With DistributedConfig(tp_size=8, ep_size=8), each GPU holds 32 of the 256 experts in every MoE layer, and tokens are sent to the GPU that holds their experts. With DistributedConfig(tp_size=8) alone, each GPU holds a slice of every expert instead. See expert parallelism and tensor parallelism. Prefer either over device_map="auto", which places whole layers on each GPU, runs them one GPU at a time, and can run out of memory while loading Beam.

Fine-tuning

The chat template marks the tokens Beam generated. Pass return_assistant_tokens_mask=True with return_dict=True to apply_chat_template, and it returns assistant_masks, covering each assistant message’s reasoning, answer, and tool calls. TRL’s SFTConfig(assistant_only_loss=True) uses this mask to train only on those tokens. We’re grateful to Hugging Face for their guidance and support in developing and refining our Beam recipes for Transformers.

Get started

Self-hosting

Third-party providers

Tools

Requirements

Quick start with pipeline

Generate with full control

Reasoning effort

Multi-turn conversations

Tool calling

Chat template options

Parallelism

Fine-tuning

​Requirements

​Quick start with pipeline

​Generate with full control

​Reasoning effort

​Multi-turn conversations

​Tool calling

​Chat template options

​Parallelism

​Fine-tuning

Requirements

Quick start with pipeline

Generate with full control

Reasoning effort

Multi-turn conversations

Tool calling

Chat template options

Parallelism

Fine-tuning