> ## Documentation Index
> Fetch the complete documentation index at: https://developers.reflection.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Beam Verifier

> Check a deployment's correctness, and compare its performance to reference numbers

The `beam-verifier` CLI has two parts, each run against a live Beam endpoint.

<CardGroup cols={2}>
  <Card title="Verify" icon="circle-check" href="#verify-your-endpoint">
    Checks functional correctness of your endpoint.

    **Needs:** a `/v1` base URL that serves chat completions.
  </Card>

  <Card title="Evaluate" icon="chart-line" href="#evaluate-your-endpoint">
    Evaluates your endpoint on main benchmarks.

    **Needs:** a `/v1` base URL.
  </Card>
</CardGroup>

Start with [verify](#verify-your-endpoint) to catch any API and conformance problems. Use [evaluate](#evaluate-your-endpoint) to measure benchmark scores and compare against our reference scores.

## Install Beam Verifier

Requires Python 3.12 or later and [uv](https://docs.astral.sh/uv/).

```bash icon="terminal" theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
git clone https://github.com/reflection-oss/beam-verifier.git
cd beam-verifier
uv tool install .
```

To confirm the install worked, run a command that needs no endpoint:

```bash icon="terminal" theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
beam-verifier list-datasets
```

If it prints the dataset table, you're set.

## Verify your endpoint

`beam-verifier verify` checks that your deployment complies with our chat API contract. It runs the functional conformance suite against a live `/v1/chat/completions` endpoint: parameter acceptance, request validation, chat, tool calling, structured output, `tool_choice`, multi-turn, and streaming. [Evaluate](#evaluate-your-endpoint) assumes a green `verify` run first.

### Run the required tests

You need `beam-verifier` [installed](#install-beam-verifier). In the commands below, replace `http://localhost:8000/v1` with your endpoint's base URL and `my-model` with your model ID. If your endpoint requires an API key, pass it with `--api-key`.

```bash icon="terminal" theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
beam-verifier verify --base-url http://localhost:8000/v1 --model-name my-model
```

<Warning>
  `--model-name` must match the ID the endpoint serves under, not the checkpoint name.
</Warning>

A conforming endpoint passes every case. The expected result is 100%.

The required tests check several aspects of the API, split into three suites:

| Suite | File | Checks |
| - | - | - |
| Request contract | `e2e/test_request_contract.py` | minimal request accepted; each parameter value accepted or rejected; preserve\_thinking behavior; malformed requests rejected with HTTP 400 |
| Response contract | `e2e/test_response_contract.py` | structured output conforms to the requested JSON schema |
| Tool protocol | `e2e/test_tool_protocol.py` | tool-call shape; parallel calls in one turn; `tool_choice` required / none / named / auto; tool result round-trip and grounding |

Each suite runs in both transport modes, buffered and streaming, wherever the behavior should not depend on transport.

### Optional tests

These run only when their flag is passed to the command (`--parse-url`, `--format-url`, `--completions-url`); otherwise they are skipped. They hit a parser, formatter and completions endpoint to localize any failure that is not detectable from the chat endpoint.

| Flag | What it checks |
| - | - |
| `--parse-url` | Fixed Harmony output tokens parse into the expected content, reasoning and tool calls |
| `--format-url` | The chat template renders the pinned gold token IDs exactly |
| `--completions-url` | Raw generations keep Harmony framing, channels and tool-call syntax |

```bash icon="terminal" theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
beam-verifier verify \
  --base-url http://localhost:8000/v1 --model-name my-model \
  --parse-url http://localhost:8000/parse \
  --format-url http://localhost:8000/format \
  --completions-url http://localhost:8000/v1/completions
```

#### Parser

Accepts `tokens` (integer token IDs) and `tools` (OpenAI-style tool definitions). Returns a top-level assistant message with `role: "assistant"`, `content` (string or null), and, when present, `reasoning_content` (string or null) and `tool_calls` (array or null). Each tool call must have `type: "function"` and a `function` containing `name` and JSON-encoded string `arguments`, with values matching the supplied tool schema. Returns HTTP 400 for malformed or truncated tool-call inputs.

Example request and response for `POST /parse`:

<CodeGroup>
  ```json Request theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
  {
    "tokens": [151669, 77091, 151672, 34484, 151671, 93945, 1112, 151670,
               151669, 77091, 151672, 11822, 151671, 13048, 1052, 0, 151674],
    "tools": []
  }
  ```

  ```json Response theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
  {
    "role": "assistant",
    "content": "Hi there!",
    "reasoning_content": "Thinking...",
    "tool_calls": []
  }
  ```
</CodeGroup>

#### Formatter

Accepts `messages`, optional `tools`, and the formatting options `add_generation_prompt`, `add_special_tokens`, and `chat_template_kwargs`. Applies the endpoint's chat template and tokenizer and returns `tokens`: a nonempty array of integer IDs matching the fixtures exactly, including special tokens. Returns HTTP 400 for invalid chat inputs.

Example request and response for `POST /format`:

<CodeGroup>
  ```json Request theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
  {
    "messages": [{"role": "user", "content": "Hi"}],
    "add_generation_prompt": false,
    "add_special_tokens": false
  }
  ```

  ```json Response theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
  {
    "tokens": [151669, 8948, 151671, 2610, 525, 50130, 11, 458, 15235, 17847,
               16176, 553, 42876, 382, 2, 7818, 11744, 25, 6358, 11, 1590, 13,
               13434, 1969, 387, 5230, 369, 1449, 1943, 13, 151670, 151669,
               872, 151671, 13048, 151670]
  }
  ```
</CodeGroup>

#### Raw completions

Generates directly from a preformatted Harmony `prompt`, honoring `max_tokens`, `temperature`, `stream: false`, and `n: 1`. Returns `choices` with exactly one entry containing `text` (string) and `finish_reason` (`"stop"` or `"length"`). Honors `skip_special_tokens: false` and `no_stop_trim: true` to preserve all Harmony special tokens, including the final `<|return|>` or `<|call|>`. Truncated completions fail the suite.

### Read the result

By default results are stored under `.runs/<timestamp>-<model>-verify/`:

```bash icon="terminal" theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
beam-verifier view                     # the latest run, in the terminal
beam-verifier view --failed --full     # only rows that scored 0, untruncated
beam-verifier report -o report.html    # self-contained HTML: scores, traces, grader output
```

## Evaluate your endpoint

`beam-verifier eval` scores your deployment on knowledge/reasoning benchmarks (GPQA-Diamond and AIME, through InspectAI). Use these scores to compare against our published results.

### Pick datasets

```bash icon="terminal" theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
beam-verifier list-datasets
```

| Dataset ID | Reference score | Runner | Scope |
| - | - | - | - |
| `gpqa-diamond` | 90.6% | <Badge color="gray" size="sm">InspectAI</Badge> | Graduate-level science |
| `aime2026` | 97.7% | <Badge color="gray" size="sm">InspectAI</Badge> | Competition mathematics |

<Note>
  Reference scores are Beam's BF16 results from [Beam's model card](https://huggingface.co/reflection/Beam-1.0-501B-A23B), measured at temperature 0.7 and top\_p 0.99. The model card also lists FP8 and NVFP4 scores.
</Note>

These are the datasets `eval` runs when you name none. `list-datasets` also shows optional ones without a reference score: GSM8K and three InfiniteBench long-context sets.

### Run an eval

You need a green [`verify`](#verify-your-endpoint) run first.

Start with a smoke run of five examples per dataset, to check connectivity and scoring before committing to a full run:

```bash Smoke run icon="terminal" highlight={3} theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
beam-verifier eval gpqa-diamond aime2026 \
  --base-url http://localhost:8000/v1 --model-name my-model \
  --num-examples 5
```

#### Configure the run

```bash icon="terminal" theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
beam-verifier eval \
  --base-url http://localhost:8000/v1 --model-name my-model \
  --parallel-tasks 8 \
  --reasoning-effort high
```

* `--parallel-tasks` — how many tasks run concurrently against the endpoint. It has the biggest effect on wall-clock time. A task may make more than one request.
* `--reasoning-effort`, `--temperature`, `--max-tokens`, `--top-p` — sent with each request. Each dataset's preset sets temperature, top\_p and reasoning effort, and a value you pass overrides it. An unset `--max-tokens` sends nothing.
* `--model-kwargs KEY=VALUE` — merged into every model request, wherever the request is made from.
* `--harness-kwargs KEY=VALUE` — forwarded verbatim to the eval library's own API (`epochs=4`, `retry_on_error=5` for InspectAI).

`--model-kwargs` and `--harness-kwargs` each accept multiple `KEY=VALUE` pairs after one flag, with YAML values. Repeating the flag also works. Run `beam-verifier eval --help` for all options.

#### Knowledge and reasoning benchmarks

These run through [InspectAI](https://github.com/UKGovernmentBEIS/inspect_ai) and need nothing beyond the endpoint.

```bash icon="terminal" theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
beam-verifier eval gpqa-diamond aime2026 \
  --base-url http://localhost:8000/v1 --model-name my-model \
  --harness-kwargs epochs=4 retry_on_error=5   # Forward InspectAI options through --harness-kwargs
```

InspectAI logs land in `<run>/inspect_logs/<dataset-id>/`.

#### Resume

```bash icon="terminal" theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
beam-verifier eval --resume .runs/<run-directory> \
  --base-url http://localhost:8000/v1 --model-name my-model
```

The dataset list comes from the run and cannot change, and `--base-url` / `--model-name` must match the saved ones. The API key is never written to the artifacts, so supply it again. Saved settings are reused; a setting named again overrides the saved value, and the resumed run reports both the override and anything it could not apply.

#### Read the result and view your traces

These commands generate a detailed report with all the traces and detailed results per task:

```bash icon="terminal" theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
beam-verifier view                              # latest run
beam-verifier view --dataset gpqa-diamond --failed --full
beam-verifier view --row 12                     # one full trace
beam-verifier report -o report.html
```

<Warning>
  * An ungraded task counts as zero in the aggregate score. If no task was graded at all, the run reports no metrics.
</Warning>