Note on the benchmark. We ran BFCL v4 without its web-search category, which needs a search API key. BFCL’s overall formula counts a missing category as zero, so the overall scores on this page are lower than a full run would give, for the stock models and the fine-tunes alike. The before-and-after deltas are unaffected, because both sides were scored the same way.
What you’ll build
BASE_MODEL changed, take the E2B from 30.98 to 33.30 at 54% fewer output tokens and hold the 12B at 48.32 against 49.98 at 64% fewer, so what you learn on the E4B carries to the other sizes. Every row was measured with thinking on and the model’s own thoughts kept in the chat history; tokens are the model’s output summed over the 4,906 scored entries.
<details>
<summary>The numbers behind the chart</summary>
The 12B row is one seed; three seeds of the same recipe average 47.1, and run-to-run variance at this learning rate is about plus or minus 3 points overall, plus or minus 8 on multi-turn and plus or minus 15 on memory.
</details>
What you’ll learn
- A thought on every turn, trained only on the model’s own spans. Gemma 4’s template renders a thought only on tool-call turns and the last round; the recipe opens it on every assistant turn, and the labels cover the thought, the calls, the reply and the closing token.
- Why the thoughts are this length. On the 12B, verbose thoughts score 47.70 at 4.21M tokens, the recipe’s purpose-style thoughts 47.13 at 3.16M (three-seed mean; the served seed 48.32 at 2.54M), concise thoughts 42.38 at 2.32M. The prompt asks for enough to justify the decision and each argument, and nothing more.
- One epoch at 3e-6. Higher rates over-fit the data’s habits and cost the 12B multi-turn accuracy; a second epoch helps the small models by about a point and hurts the 12B.
- Compare fairly. One handler for the stock model and the fine-tune, thinking on, thoughts kept in history; compare multi-turn and memory across seeds, not single runs.
What you need
- Pinned by
download.shandrequirements.txt: the three Gemma 4 instruction-tuned checkpoints, which are public, and the training stack of our runs on Python 3.12 with CUDA 13.0 wheels fortorch. train.jsonlfrom the tool-calling reasoning data recipe in this cookbook; itsLIMIT=20mode gives a small file for a dry run. The evaluation needs BFCL, whicheval_install.shinstalls, and an SGLang build with Gemma 4 support.
Run it
Train
train.sh renders the conversations with the thinking gate open on every assistant turn and masks everything the model did not generate, fine-tunes for one epoch at a learning rate of 3e-6 under FSDP, and exports the epoch checkpoint behind the generation gate. It refuses a RUN_DIR that already holds checkpoints. The last line of a successful run:
loss is the training loss of the logged step, which jumps from step to step because every step sees different conversations; mean_token_accuracy is the share of trained tokens the model predicts correctly. The run folder holds dataset/ and dataset_val/, checkpoints/checkpoint-117/ with trainer_state.json and train_info.json beside it, export/ with SMOKE_OK, and the filled-in fsdp.yaml.
</details>
Then print the losses and check the export:
Evaluate before and after
Score the stock model first, so the baseline is your own, then the fine-tune, under one handler: thinking on, and the model’s own thoughts kept in the chat history. BFCL v4 scores six sections over 4,906 entries.
score/data_overall.csv, with the web-search category counted as zero on both sides. The diagnostics count multi-turn entries that hit BFCL’s step limit, repeated the same call three times in a row, or ended in an inference error. Tokens and latency sum the server’s usage over every call of an entry, excluding the 111 memory prerequisite entries that BFCL does not score.
</details>
The same eval_serve.sh serves the export for your own use, with Gemma 4’s tool-call and reasoning parsers on.
Common fixes
COMMON_FIXES.md lists the symptoms we hit, their causes and the fix for each: out of memory in the first steps, a chat template the recipe does not know, an export that printsSMOKE_FAIL, a serving image that cannot load the exported config, a transformers that predates Gemma 4, a held-out loss that does not fall, inference errors in the analysis, and a registry entry that breaks the overall number.
Next steps
- Train on your own data. One conversation per line as
{"id": ..., "tools": [...], "messages": [...]}, with OpenAI-styletools, areasoning_contentfield on every assistant message and structuredtool_callswhere the assistant calls a tool. Keep a thought on every assistant turn, including text replies, and hold a few hundred conversations out. - Change the dose with care. Higher learning rates over-fit the data’s habits; a second epoch helps the small models by about a point and hurts the 12B.
- Compare seeds, not single runs, when a difference is within a few points on multi-turn or memory; those sections move by 8 to 15 points between seeds of the same recipe.