> ## Documentation Index
> Fetch the complete documentation index at: https://developers.reflection.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Use HF Transformers

> Generate text, control reasoning, and call tools with Beam in Hugging Face Transformers

This guide runs Beam on your own compute with [Hugging Face Transformers](https://huggingface.co/docs/transformers). It starts with the high-level `pipeline`, then uses `generate` for full control over the prompt and the parsed reply, and covers reasoning effort, tool calling, and parallelism.

## Requirements

The checkpoint is `reflection/Beam-1.0-501B-A23B`. Its BF16 weights take about 1 TB, so the examples split the model across 8 GPUs with more than 1 TB of combined memory. Install Transformers:

```bash Install Transformers icon="terminal" theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
pip install -U transformers torch
```

Save each example as a script and launch it with one process per GPU:

```bash Launch on 8 GPUs icon="terminal" theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
torchrun --nproc-per-node 8 generate.py
```

Every process runs the whole script, so each one prints the output.

## Quick start with pipeline

`pipeline` applies the chat template, generates, and parses the reply into `thinking` and `content`. `DistributedConfig(tp_size=8, ep_size=8)` shards the model across the 8 GPUs and places whole experts on each one. See [Parallelism](#parallelism).

```python generate.py icon="python" highlight={6} theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
from transformers import DistributedConfig, pipeline

pipe = pipeline(
    "text-generation",
    model="reflection/Beam-1.0-501B-A23B",
    model_kwargs={"distributed_config": DistributedConfig(tp_size=8, ep_size=8)},
)

messages = [{"role": "user", "content": "Explain in one sentence why the sky is usually blue."}]
messages = pipe(messages, max_new_tokens=2048)[0]["generated_text"]
print(messages[-1]["thinking"])
print(messages[-1]["content"])

messages.append({"role": "user", "content": "And why are sunsets red?"})
messages = pipe(messages, max_new_tokens=2048)[0]["generated_text"]
print(messages[-1]["content"])
```

The pipeline returns the whole conversation, so you can append the next user message and call it again.

## Generate with full control

Load the model and tokenizer yourself to pass chat template arguments, inspect the prompt, or change how you generate. Pass the prompt to `parse_response` as `prefix`: the chat template writes the start of the reply into the prompt, and the parser needs it.

```python generate.py icon="python" highlight={16} theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
from transformers import AutoModelForCausalLM, AutoTokenizer, DistributedConfig

model_id = "reflection/Beam-1.0-501B-A23B"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id, distributed_config=DistributedConfig(tp_size=8, ep_size=8)
)

messages = [{"role": "user", "content": "Explain in one sentence why the sky is usually blue."}]
inputs = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt", return_dict=True
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=2048)
generated = tokenizer.decode(outputs[0, inputs["input_ids"].shape[1] :])
message = tokenizer.parse_response(generated, prefix=inputs["input_ids"][0])
print(message["thinking"])
print(message["content"])
```

`generate` and `pipeline` sample with Beam's recommended `temperature` of `0.7` and `top_p` of `0.99`, which the checkpoint sets as defaults. Pass `temperature` and `top_p` to change them, or `do_sample=False` to decode greedily.

The later examples reuse this `model` and `tokenizer`. To parse a reply as it streams, `tokenizer.get_response_parser(prefix=...)` returns a parser that you `feed()` text chunks to. See [response parsing](https://huggingface.co/docs/transformers/main/en/chat_response_parsing).

## Reasoning effort

Beam always reasons before it answers. Pass `reasoning_effort` to `apply_chat_template` to choose how much, as an integer or a label: `minimal` (8), `low` (32), `medium` (64), `high` (128), `xhigh` (256), or `max` (512).

```python icon="python" highlight={2} theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
inputs = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt", return_dict=True, reasoning_effort="low"
).to(model.device)
```

Without `reasoning_effort`, the template uses `max`. Beam wasn't trained on efforts below 8, so it doesn't follow smaller values. With `pipeline`, pass `tokenizer_encode_kwargs={"reasoning_effort": "low"}`. If generation hits `max_new_tokens` while Beam is still reasoning, the parsed message has `thinking` and no `content`.

The hosted API has its own effort values and defaults. See [Reasoning](/reasoning).

## Multi-turn conversations

Append each parsed message to the history unchanged. Beam's output degrades without its earlier reasoning, which the chat template reads from `thinking`, `reasoning_content`, or `reasoning`.

```python icon="python" highlight={1} theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
messages.append(message)
messages.append({"role": "user", "content": "And why are sunsets red?"})
inputs = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt", return_dict=True
).to(model.device)
```

By default, an assistant message without reasoning renders with an empty reasoning block. Pass `allow_missing_thinking=False` to raise an error instead when `add_generation_prompt=True`. Set it when you serve Beam, so that clients which drop reasoning get an error instead of lower-quality replies. The [Harmony format guide](/harmony-format#reasoning-across-turns) covers reasoning across turns.

## Tool calling

Pass the same `tools` to `apply_chat_template` and `parse_response`. The parser uses each tool's schema to cast arguments to their types. Without `tools`, every argument is a string.

```python icon="python" highlight={13,17} theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
def get_weather(city: str):
    """
    Get the current weather in a city.

    Args:
        city: The city name.
    """
    return "Sunny, 22 C"


messages = [{"role": "user", "content": "What's the weather like in Paris right now?"}]
inputs = tokenizer.apply_chat_template(
    messages, tools=[get_weather], add_generation_prompt=True, return_tensors="pt", return_dict=True
).to(model.device)
outputs = model.generate(**inputs, max_new_tokens=2048)
generated = tokenizer.decode(outputs[0, inputs["input_ids"].shape[1] :])
message = tokenizer.parse_response(generated, prefix=inputs["input_ids"][0], tools=[get_weather])
print(message["tool_calls"])
# [{'type': 'function', 'function': {'name': 'get_weather', 'arguments': {'city': 'Paris'}}}]

messages.append(message)
for call in message["tool_calls"]:
    result = get_weather(**call["function"]["arguments"])
    messages.append({"role": "tool", "name": call["function"]["name"], "content": result})
```

Render `messages` with the same `tools` and generate again to get the answer. Parsed tool calls have no IDs, and Beam matches results to calls by position, so add the results in the order of the calls.

## Chat template options

| Argument | Effect |
| - | - |
| `reasoning_effort` | How much Beam reasons. See [Reasoning effort](#reasoning-effort). |
| `model_identity` | Replaces the default first line of the system message. |
| `builtin_tools` | Names of tools from `tools` to declare in the system message. Only for `search` and `browse` with the definitions Beam was trained with. The [Harmony format guide](/harmony-format#built-in-tools) covers built-in tools. |
| `allow_missing_thinking` | Defaults to `True`. Set to `False` to raise an error when an earlier assistant message has no reasoning, as recommended for serving. |

Put `system` and `developer` messages at the start of the conversation. The template drops them anywhere else.

## Parallelism

With `DistributedConfig(tp_size=8, ep_size=8)`, each GPU holds 32 of the 256 experts in every MoE layer, and tokens are sent to the GPU that holds their experts. With `DistributedConfig(tp_size=8)` alone, each GPU holds a slice of every expert instead. See [expert parallelism](https://huggingface.co/docs/transformers/main/en/expert_parallelism) and [tensor parallelism](https://huggingface.co/docs/transformers/main/en/perf_infer_gpu_multi).

Prefer either over `device_map="auto"`, which places whole layers on each GPU, runs them one GPU at a time, and can run out of memory while loading Beam.

## Fine-tuning

The chat template marks the tokens Beam generated. Pass `return_assistant_tokens_mask=True` with `return_dict=True` to `apply_chat_template`, and it returns `assistant_masks`, covering each assistant message's reasoning, answer, and tool calls. TRL's `SFTConfig(assistant_only_loss=True)` uses this mask to train only on those tokens.

We're grateful to **Hugging Face** for their guidance and support in developing and refining our Beam recipes for Transformers.