feat: add Verifiers rollout integration

[willccbb]

Summary

Add an opt-in Verifiers V1 rollout adapter selected by one new argument:

--verifiers-config /path/to/verifiers.toml

The adapter loads a V1 EnvConfig, lets Verifiers own grouped episode execution and reward computation, and returns ordinary Miles Sample groups. Miles continues to own model selection, sampling settings, concurrency, filtering, reward overrides, advantages, and optimization.

flowchart LR
    C["Miles CLI + EnvConfig"] --> V["Verifiers V1 environment"]
    V --> R["Renderers TrainClient"]
    R --> S["Miles SGLang /generate"]
    S --> T["Verifiers traces + group rewards"]
    T --> M["Miles Sample groups"]
    M --> O["Miles advantages + optimizer"]

Loading

This deliberately follows Miles' existing rollout-function extension point instead of changing shared sample conversion or advantage code.

Configuration and ownership

• --verifiers-config replaces the prompt dataset with the configured V1 taskset.
• --hf-checkpoint and --sglang-tokenizer-path provide model and renderer identity; no duplicate Verifiers model option is added.
• --n-samples-per-prompt and --n-samples-per-eval-prompt define each fixed rollout group.
• Existing Miles rollout/eval sampling, context limits, concurrency, LoRA, deterministic inference, reward hooks, dynamic filtering, routing, and resume position retain their normal meanings.
• Verifiers runs the environment lifecycle, including runtimes, user simulators, tools, multi-turn episodes, retries, per-rollout rewards, and group rewards.
• Renderers owns message formatting, tool parsing, token attribution, and multi-turn bridging.

The optional dependency group uses verifiers>=0.2.0,<0.2.1, renderers>=0.1.8, and the compatible openai-agents range. The ceiling is intentional: Verifiers 0.2.1 requires OpenAI >=2.9 while the current SGLang 0.5.15 dependency pins OpenAI 2.6.1. Supporting 0.2.1 should follow once that upstream dependency constraint is compatible.

Supported scope

The adapter supports text-only V1 environments using the Chat Completions dialect, including grouped rollouts and group-level reward computation in both training and evaluation. It also preserves structured prompts, exact sampled token IDs/logprobs, Miles reward hooks, dynamic oversampling/filtering, LoRA requests, and consistent-hash session routing.

Unsupported combinations fail explicitly rather than retrying indefinitely:

• partial rollout, because a live Verifiers episode cannot be resumed
• custom Miles chat-template files, because Renderers owns formatting
• Responses/Anthropic dialects, streaming and auxiliary relay routes
• multimodal inputs, OPD metadata, routing replay, and indexer replay
• traces with multiple graph branches, including compaction

The graph-branch restriction is intentional for this first integration. Miles currently flattens multiple training samples without preserving the enclosing Verifiers trace-group boundary, so accepting these traces would silently compute the wrong group-relative advantages. Single-branch multi-turn and tool traces are supported.

Validation

Automated

• 433 passed, 29 skipped across tests/fast/rollout
• 74 passed in the focused adapter, V1 runtime contract, argument, and sample-conversion suites
• all repository pre-commit hooks passed on all 11 changed files
• Python 3.10 import path verified to skip the Python 3.11-only Verifiers module cleanly; the optional integration runs on Python 3.11+
• independent final review caught and covered eval-only grouped-reward validation and deterministic unsupported-request failures

Real two-GPU end-to-end run

Qwen3-0.6B completed a full Miles optimizer update on Verifiers' code-golf-v1 environment:

• 3 task groups x 4 rollouts = 12 generated samples
• 9 completed and 3 length-truncated; 7 passed the program checks
• mean aggregate reward 0.9167, range [0, 2]
• 4,245 sampled response tokens
• all three fastest group-reward sums were exactly 0.5
• optimizer update and both SGLang weight refreshes completed successfully

W&B run

Learning signal

Qwen3-4B-Instruct completed 50 Miles updates on reverse-text-v1 (3,200 trajectories) with no Verifiers errors, non-finite values, or token-alignment failures:

• mean training reward, first 5 updates: 0.2807
• mean training reward, last 5 updates: 0.5891
• held-out reward: 0.01511 before training to 0.55511 after training (+0.5400)
• bootstrap 95% CI for held-out improvement: [0.49878, 0.57921]
• 165 held-out prompts improved, 33 were unchanged, and 2 regressed

W&B run

Documentation

The user guide documents installation through the optional Miles extra, Prime CLI environment installation (./environments, prime env install ...), the single added configuration option, ownership boundaries, and unsupported cases. The implementation follows the rollout-function integration pattern used by prior merged work such as #1487 and #1633.

[gemini-code-assist]

Code Review

This pull request integrates Verifiers environments into Miles, allowing training on Verifiers tasksets and harnesses. It introduces the VerifiersRolloutFn adapter, a custom MilesSGLangTransport for translating requests, CLI argument parsing and validation for --verifiers-config, and corresponding documentation and tests. The review comments identify two key issues: a potential TypeError in evaluation when computing average rewards on dictionary-structured rewards, and an unbounded memory leak in MilesSGLangTransport due to the _seen_sessions set growing without limit.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.