[rl] Pin renderers 0.1.11, render with TorchTitan's tokenizer, forward renderer knobs strictly

[felipemello1]

Summary

• Pin renderers==0.1.11 (first release with bring-your-own-tokenizer; transformers is now an optional extra we do not install) instead of tracking git+...@main.
• Render with TorchTitan's own HuggingFaceTokenizer: RendererConfig.build(tokenizer=...) wraps it in RendererTokenizer (the renderers.OffsetTokenizer protocol) instead of loading transformers.AutoTokenizer. The controller reads pad_id off its own tokenizer instead of renderer._tokenizer.
• RendererConfig is now name + options: the options dict is passed unchanged to the renderer's typed pydantic config, which rejects options that renderer does not have (extra="forbid") instead of TorchTitan silently dropping them ([rl] RendererConfig silently drops the removed preserve_* renderer knobs pytorch/torchtitan#4365). TorchTitan mirrors no renderer fields; renderer names are the library's names.

Why

requirements.txt tracked renderers main. renderers#88 replaced preserve_all_thinking / preserve_thinking_between_tool_calls with thinking_retention; our build() filtered knobs by field.name in config_type.model_fields, so both became inert with no failing test (pytorch#4365). Two in-tree configs were already hitting the same silent drop: RendererConfig(name="gpt_oss", enable_thinking=False) (gpt-oss has no enable_thinking) and RendererConfig(name="muse_glimmer", enable_thinking=True).

renderers 0.1.11 ships the BYO-tokenizer protocol we were waiting on (TODO(renderers#70) in renderer.py and controller.py), so the transformers import and the renderer._tokenizer reach-in can go.

Previous behavior

# renderer.py
tokenizer = AutoTokenizer.from_pretrained(tokenizer_path)          # transformers
args = {f.name: v for f in fields(self) ... if f.name in config_type.model_fields}  # silent drop
return create_renderer(tokenizer, config_type(**args))
# controller.py
pad_id=self.renderer._tokenizer.eos_token_id

New behavior

# controller.py / rollouter.py / generate.py
tokenizer = HuggingFaceTokenizer(tokenizer_path=hf_assets_path)
renderer = config.renderer.build(tokenizer=tokenizer)
pad_id = tokenizer.eos_id

# renderer.py
def build(self, *, tokenizer: HuggingFaceTokenizer) -> Renderer:
    return create_renderer(RendererTokenizer(tokenizer), self.to_renderers_config())

def to_renderers_config(self) -> BaseRendererConfig:
    ...  # refuses "auto"/"default" (need HF apply_chat_template), registers muse_glimmer
    config_cls = type(config_from_name(self.name))
    return config_cls(**self.options)   # pydantic extra="forbid" raises on unsupported options

RendererConfig(name="qwen3", options={"enable_thinking": False})
RendererConfig(name="gpt-oss", options={"reasoning_effort": "low"})
RendererConfig(name="deepseek-v3", options={"enable_thinking": False}).build(tokenizer=tok)
# ValidationError: Extra inputs are not permitted [enable_thinking]

RendererConfig fields: name: str (library renderer name: qwen3, gpt-oss, llama-3, deepseek-v3, or in-tree muse_glimmer) and options: dict[str, Any], passed through to that renderer's config; the renderers docs list each renderer's options. Removed: the mirrored fields (enable_thinking, tool_parser, reasoning_parser, preserve_*), and the _RENDERER_BY_MODEL map (gpt_oss -> gpt-oss is the only name that changes). Recipes move to options={...}. CLI override of renderer options is not supported (tyro cannot add dict keys); the 8 redundant --renderer.enable-thinking False flags in tests/integration_tests.py are deleted (their recipes already set it).

"auto" was never useful here: renderers resolves it by exact match of tokenizer.name_or_path against MODEL_RENDERER_MAP, and hf_assets_path is a local directory, so it always fell back to DefaultRenderer.

Validation

• Unit: tests/test_renderer.py (new, 12 tests): strict forwarding, unsupported-knob and unknown-name errors, auto/default refusal, union round-trip, RendererTokenizer protocol conformance, and byte-identical parity (token ids, is_content, sampled_mask, message_indices) between RendererTokenizer and transformers.AutoTokenizer under Qwen3Renderer. 168 tests pass across the touched RL test files. The same parity also holds on the real Qwen3-0.6B tokenizer with tools, parse_response, bridge_to_next_turn, and stop ids (offline probe).
• Alphabet sort, Qwen3-0.6B, 2 GPUs (trainer TP=1, 1 generator TP=1, 30 steps): validation reward mean 0.181 -> 0.548, max 0.514 -> 1.000.
• DAPO math, Qwen3-0.6B, 2 GPUs, 2k response cap: DAPO_RESULT_PLACEHOLDER
• grep -rn "transformers" torchtitan/experiments/rl --include='*.py' | grep import now only matches the vLLM registry.

Review focus

• RendererTokenizer (wraps the tokenizers.Tokenizer backend as _tokenizer_backend): encode never adds BOS/EOS (TorchTitan's encode would, per add_bos defaults); unk_token_id = None because the tokenizers backend returns None for unknown tokens; __call__ returns offset_mapping so is_content stays precise.
• to_renderers_config() as a separate method: PR [rl] Add optional Verifiers rollout integration pytorch/torchtitan#4356 needs the config-only seam (verifiers builds its own renderer in the env process from the typed config).
• gpt-oss alphabet sort: enable_thinking=False was silently dropped before (gpt-oss has no such option and always has an analysis channel), so it ran at reasoning_effort="medium". It now sets options={"reasoning_effort": "low"}, the closest expression of the recipe's intent; this is a behavior change. Muse Glimmer's enable_thinking=True was also dropped; Muse always renders reasoning by default, so nothing replaces it.
• Unit tests test_alphabet_sort / test_rollout_worker now pass the repo's debug tokenizer (tests/assets/tokenizer) because the worker opens a real tokenizer.

Rebase notes for PR pytorch#4356

Those files only exist in pytorch#4356, so they are edited there when it rebases onto this PR.

• Rename as_renderers_config() -> to_renderers_config() in examples/verifiers/components/rollouter.py and tests/test_verifiers_example.py; RendererConfig(name="qwen3", enable_thinking=True) -> options={"enable_thinking": True} in examples/verifiers/config_registry.py.
• Add renderers[transformers]==0.1.11 to examples/verifiers/requirements.txt: verifiers 0.3.0 calls renderers.base.load_tokenizer, which needs transformers.
• Add a serialized-config parity test (TrainClientConfig(renderer=..., renderer_model_name=...) round-trip vs RendererConfig.build).
• MuseGlimmerRendererConfig is not a member of the renderers.RendererConfig union, so the verifiers path cannot carry it yet.

Risks / open questions

• Multimodal (qwen3-vl) rendering needs renderers[multimodal] and an HF processor; no RL config uses it, unchanged by this PR.

🤖 Generated with Claude Code