Beam Verifier

The beam-verifier CLI has two parts, each run against a live Beam endpoint.

Verify

Checks functional correctness of your endpoint.Needs: a /v1 base URL that serves chat completions.

Evaluate

Evaluates your endpoint on main benchmarks.Needs: a /v1 base URL.

Start with verify to catch any API and conformance problems. Use evaluate to measure benchmark scores and compare against our reference scores.

Install Beam Verifier

Requires Python 3.12 or later and uv.

git clone https://github.com/reflection-oss/beam-verifier.git
cd beam-verifier
uv tool install .

To confirm the install worked, run a command that needs no endpoint:

beam-verifier list-datasets

If it prints the dataset table, you’re set.

Verify your endpoint

beam-verifier verify checks that your deployment complies with our chat API contract. It runs the functional conformance suite against a live /v1/chat/completions endpoint: parameter acceptance, request validation, chat, tool calling, structured output, tool_choice, multi-turn, and streaming. Evaluate assumes a green verify run first.

Run the required tests

You need beam-verifier installed. In the commands below, replace http://localhost:8000/v1 with your endpoint’s base URL and my-model with your model ID. If your endpoint requires an API key, pass it with --api-key.

beam-verifier verify --base-url http://localhost:8000/v1 --model-name my-model

--model-name must match the ID the endpoint serves under, not the checkpoint name.

A conforming endpoint passes every case. The expected result is 100%. The required tests check several aspects of the API, split into three suites:

Suite	File	Checks
Request contract	`e2e/test_request_contract.py`	minimal request accepted; each parameter value accepted or rejected; preserve_thinking behavior; malformed requests rejected with HTTP 400
Response contract	`e2e/test_response_contract.py`	structured output conforms to the requested JSON schema
Tool protocol	`e2e/test_tool_protocol.py`	tool-call shape; parallel calls in one turn; `tool_choice` required / none / named / auto; tool result round-trip and grounding

Each suite runs in both transport modes, buffered and streaming, wherever the behavior should not depend on transport.

Optional tests

These run only when their flag is passed to the command (--parse-url, --format-url, --completions-url); otherwise they are skipped. They hit a parser, formatter and completions endpoint to localize any failure that is not detectable from the chat endpoint.

Flag	What it checks
`--parse-url`	Fixed Harmony output tokens parse into the expected content, reasoning and tool calls
`--format-url`	The chat template renders the pinned gold token IDs exactly
`--completions-url`	Raw generations keep Harmony framing, channels and tool-call syntax

beam-verifier verify \
  --base-url http://localhost:8000/v1 --model-name my-model \
  --parse-url http://localhost:8000/parse \
  --format-url http://localhost:8000/format \
  --completions-url http://localhost:8000/v1/completions

Parser

Accepts tokens (integer token IDs) and tools (OpenAI-style tool definitions). Returns a top-level assistant message with role: "assistant", content (string or null), and, when present, reasoning_content (string or null) and tool_calls (array or null). Each tool call must have type: "function" and a function containing name and JSON-encoded string arguments, with values matching the supplied tool schema. Returns HTTP 400 for malformed or truncated tool-call inputs. Example request and response for POST /parse:

{
  "tokens": [151669, 77091, 151672, 34484, 151671, 93945, 1112, 151670,
             151669, 77091, 151672, 11822, 151671, 13048, 1052, 0, 151674],
  "tools": []
}

{
  "role": "assistant",
  "content": "Hi there!",
  "reasoning_content": "Thinking...",
  "tool_calls": []
}

Formatter

Accepts messages, optional tools, and the formatting options add_generation_prompt, add_special_tokens, and chat_template_kwargs. Applies the endpoint’s chat template and tokenizer and returns tokens: a nonempty array of integer IDs matching the fixtures exactly, including special tokens. Returns HTTP 400 for invalid chat inputs. Example request and response for POST /format:

{
  "messages": [{"role": "user", "content": "Hi"}],
  "add_generation_prompt": false,
  "add_special_tokens": false
}

{
  "tokens": [151669, 8948, 151671, 2610, 525, 50130, 11, 458, 15235, 17847,
             16176, 553, 42876, 382, 2, 7818, 11744, 25, 6358, 11, 1590, 13,
             13434, 1969, 387, 5230, 369, 1449, 1943, 13, 151670, 151669,
             872, 151671, 13048, 151670]
}

Raw completions

Generates directly from a preformatted Harmony prompt, honoring max_tokens, temperature, stream: false, and n: 1. Returns choices with exactly one entry containing text (string) and finish_reason ("stop" or "length"). Honors skip_special_tokens: false and no_stop_trim: true to preserve all Harmony special tokens, including the final <|return|> or <|call|>. Truncated completions fail the suite.

Read the result

By default results are stored under .runs/<timestamp>-<model>-verify/:

beam-verifier view                     # the latest run, in the terminal
beam-verifier view --failed --full     # only rows that scored 0, untruncated
beam-verifier report -o report.html    # self-contained HTML: scores, traces, grader output

Evaluate your endpoint

beam-verifier eval scores your deployment on knowledge/reasoning benchmarks (GPQA-Diamond and AIME, through InspectAI). Use these scores to compare against our published results.

Pick datasets

beam-verifier list-datasets

Dataset ID	Reference score	Runner	Scope
`gpqa-diamond`	90.6%	InspectAI	Graduate-level science
`aime2026`	97.7%	InspectAI	Competition mathematics

Reference scores are Beam’s BF16 results from Beam’s model card, measured at temperature 0.7 and top_p 0.99. The model card also lists FP8 and NVFP4 scores.

These are the datasets eval runs when you name none. list-datasets also shows optional ones without a reference score: GSM8K and three InfiniteBench long-context sets.

Run an eval

You need a green verify run first. Start with a smoke run of five examples per dataset, to check connectivity and scoring before committing to a full run:

Smoke run

beam-verifier eval gpqa-diamond aime2026 \
  --base-url http://localhost:8000/v1 --model-name my-model \
  --num-examples 5

Configure the run

beam-verifier eval \
  --base-url http://localhost:8000/v1 --model-name my-model \
  --parallel-tasks 8 \
  --reasoning-effort high

--parallel-tasks — how many tasks run concurrently against the endpoint. It has the biggest effect on wall-clock time. A task may make more than one request.
--reasoning-effort, --temperature, --max-tokens, --top-p — sent with each request. Each dataset’s preset sets temperature, top_p and reasoning effort, and a value you pass overrides it. An unset --max-tokens sends nothing.
--model-kwargs KEY=VALUE — merged into every model request, wherever the request is made from.
--harness-kwargs KEY=VALUE — forwarded verbatim to the eval library’s own API (epochs=4, retry_on_error=5 for InspectAI).

--model-kwargs and --harness-kwargs each accept multiple KEY=VALUE pairs after one flag, with YAML values. Repeating the flag also works. Run beam-verifier eval --help for all options.

Knowledge and reasoning benchmarks

These run through InspectAI and need nothing beyond the endpoint.

beam-verifier eval gpqa-diamond aime2026 \
  --base-url http://localhost:8000/v1 --model-name my-model \
  --harness-kwargs epochs=4 retry_on_error=5   # Forward InspectAI options through --harness-kwargs

InspectAI logs land in <run>/inspect_logs/<dataset-id>/.

Resume

beam-verifier eval --resume .runs/<run-directory> \
  --base-url http://localhost:8000/v1 --model-name my-model

The dataset list comes from the run and cannot change, and --base-url / --model-name must match the saved ones. The API key is never written to the artifacts, so supply it again. Saved settings are reused; a setting named again overrides the saved value, and the resumed run reports both the override and anything it could not apply.

Read the result and view your traces

These commands generate a detailed report with all the traces and detailed results per task:

beam-verifier view                              # latest run
beam-verifier view --dataset gpqa-diamond --failed --full
beam-verifier view --row 12                     # one full trace
beam-verifier report -o report.html

An ungraded task counts as zero in the aggregate score. If no task was graded at all, the run reports no metrics.

Get started

Self-hosting

Third-party providers

Tools

Verify

Evaluate

Install Beam Verifier

Verify your endpoint

Run the required tests

Optional tests

Parser

Formatter

Raw completions

Read the result

Evaluate your endpoint

Pick datasets

Run an eval

Configure the run

Knowledge and reasoning benchmarks

Resume

Read the result and view your traces

Verify

Evaluate

​Install Beam Verifier

​Verify your endpoint

​Run the required tests

​Optional tests

​Parser

​Formatter

​Raw completions

​Read the result

​Evaluate your endpoint

​Pick datasets

​Run an eval

​Configure the run

​Knowledge and reasoning benchmarks

​Resume

​Read the result and view your traces

Install Beam Verifier

Verify your endpoint

Run the required tests

Optional tests

Parser

Formatter

Raw completions

Read the result

Evaluate your endpoint

Pick datasets

Run an eval

Configure the run

Knowledge and reasoning benchmarks

Resume

Read the result and view your traces