Quick reference for agents working on tinker-cookbook. Detailed guidance is in the skills under skills/.
tinker-cookbook is a client library with training and eval code built on the Tinker service (hosted by Thinking Machines Lab) and the Tinker SDK (a separate repo with just the API). You author training/eval loops that run on a CPU machine; Tinker executes the heavy GPU work.
Skills: This repo ships two Claude Code skills in skills/: research (SFT, RL, DPO, distillation, evaluation, model selection, experiment methodology) and debug (performance, correctness, renderer, and error triage). Install via /plugin marketplace add thinking-machines-lab/tinker-cookbook, then use /tinker:research or /tinker:debug.
Agents often struggle with the nested type hierarchy.
Core types:
Datum=model_input(ModelInput) +loss_fn_inputs(dict of TensorData)ModelInput= list of chunks (EncodedTextChunk, ImageChunk)TensorData= wrapper for numpy/torch arrays with shape info
Helper functions (use these instead of manual construction):
datum_from_model_input_weights(model_input, weights, max_length)- SL datum creation (supervised/common.py)conversation_to_datum(messages, renderer, max_length, train_on_what)- Full pipeline (supervised/data.py)renderer.build_supervised_example(messages)- Returns (ModelInput, weights)ModelInput.from_ints(tokens)- Create from token listTensorData.from_numpy(arr)/TensorData.from_torch(tensor)- Wrap arrays
Builder pattern: Config objects are chz dataclasses (SupervisedDatasetBuilder, RLDatasetBuilder, EnvGroupBuilder). They expose .build()/__call__() returning runtime objects.
Key code locations:
- SL:
tinker_cookbook/supervised/train.py - RL:
tinker_cookbook/rl/train.py - DPO:
tinker_cookbook/preference/train_dpo.py - Renderers:
tinker_cookbook/renderers/ - Completers:
tinker_cookbook/completers.py - RL types:
tinker_cookbook/rl/types.py - Rollout strategies:
tinker_cookbook/rl/rollout_strategy.py(FailFast, RetryOnFailure) - Logging:
tinker_cookbook/utils/logtree.py,tinker_cookbook/rl/rollouts.py - Recipes:
tinker_cookbook/recipes/
Training outputs: RL and SL training write human-readable HTML reports and machine-readable JSON files (metrics, rollout transcripts, per-trajectory summaries) to log_path. Point agents at a log_path directory to analyze training runs — metrics.jsonl for scalar metrics, *_rollout_summaries.jsonl for per-trajectory data, and *_logtree.json for full rollout transcripts including model responses.
Subscript suffixes for tensor names: _P (problems), _G (groups), _T (tokens), _D (datums). Example: tokens_P_G_T[p][g][t]
Code style:
- Explicit typing; avoid
Any/type: ignore - Use
safezip,timed,scopehelpers @chz.chzdecorator for config serializationml_log.log_metricsfor metrics;logtreefor transcripts
Env lifecycle: Env objects are single-use (no reset). Create via EnvGroupBuilder.
tinker-cookbook is a public repository. Keep internal context out of commits (even intermediate commits), and write with an external audience in mind.
For an identifier to pass to the SDK, call service_client.get_server_capabilities().supported_models — authoritative, includes :peft: long-context variants, but requires TINKER_API_KEY. For a read-only lookup with pricing and context length (no auth), browse https://tinker-docs.thinkingmachines.ai/tinker/models/.
-
Sequential API calls: The #1 performance mistake. Always use
_asyncvariants and submit calls back-to-back before awaiting. Useasyncio.gatherfor concurrent evaluation — never sequential loops over API calls. Tinker is designed for high request concurrency from a single Python process; when exact limits matter, check the current SDK/client configuration rather than inventing small caps. For very high parallelism, shard work across multiple Python processes and pass pickled sampling clients where appropriate. -
Sampler desync: Create a new sampling client after saving weights. A stale client silently samples from old weights.
-
LoRA LR: Use
hyperparam_utils.get_lr(model_name)- LoRA needs ~10x higher LR than full fine-tuning. -
Renderer mismatch: Use
model_info.get_recommended_renderer_name()— never hardcode renderer names.Never call
tokenizer.encode(prompt)directly on a chat-tuned model (gpt-oss, Llama-3-Instruct, Qwen-Instruct, etc.). Raw encoding skips the chat template, producing OOD prompt tokens. The sampler and trainer then take subtly different code paths on those OOD inputs, and per-token sampler/trainer logprob KL can inflate by 5×+ (max ratios in the tens), silently breaking PPO/CISPO/GRPO importance ratios. Usetokenizer.apply_chat_template(messages, tokenize=True, add_generation_prompt=True)(take["input_ids"]from the returnedBatchEncoding) or a cookbook renderer for the prompt. Rawencodeis correct only for base / continued-pretraining / NLL-eval workflows where there is no conversation. -
Type construction: Use helper functions, not manual dict construction. See
supervised/data.pyandsupervised/common.py. -
Group semantics: RL advantages are centered within each group.
-
DPO: Start with
dpo_beta=0.1, LR~1e-5.
# Unit tests (no API needed, colocated *_test.py files)
pytest tinker_cookbook/
# Smoke tests (requires TINKER_API_KEY + network)
pytest tests/For debugging, shrink workloads via n_batches, batch_size, group_size in dataset builders.