[rl] Pin renderers 0.1.11, render with TorchTitan's tokenizer, take the renderer's typed config directly by felipemello1 · Pull Request #4444 · pytorch/torchtitan

felipemello1 · 2026-09-03T15:51:26Z

TLDR:

pin renderers
Use our tokenizer and remove transformers dependency
Simplify code by using config classes directly rather than trying to map model_name -> config_type
Simplify code by removing the need of registering muse glimmer

Summary

TitanRL uses renderers to turn chat messages into token ids and parse completions back. Until now we wrapped it in our own RendererConfig:

# before: recipe
renderer=RendererConfig(name="qwen3", enable_thinking=False)

# before: RendererConfig.build()
tokenizer = AutoTokenizer.from_pretrained(tokenizer_path)                  # transformers
args = {f.name: v for f in fields(self) if f.name in config_type.model_fields}   # silent drop
return create_renderer(tokenizer, config_type(**args))

Problems:

Silent drops ([rl] RendererConfig silently drops the removed preserve_* renderer knobs #4365). build() forwarded knobs by name-matching against the renderer's config and dropped the rest. When renderers replaced preserve_all_thinking with thinking_retention, the knob went inert with no error; two in-tree recipes were already hitting the same thing (enable_thinking on gpt-oss and Muse Glimmer, which have no such option).
A second tokenizer. build() loaded transformers.AutoTokenizer even though TorchTitan already has the tokenizer, and the controller then reached into renderer._tokenizer for pad_id.
Transformers dependency

Solutions:

renderers==0.1.11 accepts a bring-your-own tokenizer and drops the transformers dependency:

Recipes hold the library's typed config directly: We don't have a wrapper anymore. We don't try to redirect 'qwen3to the right config, e.g.renderer=RendererConfig(name="qwen3", enable_thinking=False)`. Instead, we do:
```
from renderers import Qwen3RendererConfig, GptOssRendererConfig
renderer=Qwen3RendererConfig(enable_thinking=False)
renderer=GptOssRendererConfig(reasoning_effort="low")
```

We load our own tokenizer:build_renderer pairs that config with TorchTitan's tokenizer, through a small adapter (RendererTokenizer) that exposes the interface renderers expects.

# controller / rollout worker / generate.py
tokenizer = HuggingFaceTokenizer(tokenizer_path=hf_assets_path)
config.renderer = build_renderer(tokenizer=tokenizer, config=config.renderer)

def build_renderer(*, tokenizer: HuggingFaceTokenizer, config: BaseRendererConfig) -> Renderer:
    if config.name == "auto": raise ValueError(...)      # unmatched local paths fall back unsafely
    if config.name == "default": raise ValueError(...)   # needs HF apply_chat_template
    renderer_tokenizer = RendererTokenizer(tokenizer)
    if isinstance(config, TorchTitanRendererConfig):     # renderer implemented in TorchTitan
        return config.renderer_cls(renderer_tokenizer, config)
    return create_renderer(tokenizer=renderer_tokenizer, config=config)

We skip registration of new renderers*: Muse Glimmer's renderer lives in TorchTitan, so the renderer's registry cannot find it. We skip the need for the renderers registry that maps config -> renderer class. Check TorchTitanRendererConfig.

Validation

Alphabet sort, Qwen3-0.6B, 2 GPUs: validation reward 0.181 -> 0.548.

…he renderer's typed config directly renderers 0.1.11 makes transformers optional and accepts a bring-your-own tokenizer. RL now renders with TorchTitan's HuggingFaceTokenizer (wrapped in RendererTokenizer to satisfy renderers.OffsetTokenizer) instead of loading transformers.AutoTokenizer, and the controller reads pad_id off its own tokenizer instead of renderer._tokenizer. TorchTitan's RendererConfig wrapper is removed. Controller.Config.renderer is the library's own typed pydantic config (Qwen3RendererConfig(enable_thinking=False), GptOssRendererConfig(reasoning_effort="low"), ...), so a wrong option fails when the recipe is constructed instead of being silently dropped (pytorch#4365), and TorchTitan mirrors no renderer fields. build_renderer(tokenizer, config) is the one TorchTitan-side seam. A renderer that ships in TorchTitan (Muse Glimmer) has a TorchTitanRendererConfig that names its renderer class, and build_renderer constructs it directly, so nothing is written into renderers' registry. AutoRendererConfig and DefaultRendererConfig are rejected with the reason: the former depends on an exact model-ID match that local asset paths do not reliably preserve; the latter needs Hugging Face-compatible apply_chat_template semantics, which TorchTitan's template rendering does not provide. Renderer options are set in the recipe (tyro.conf.Suppress, like model_spec); the 8 redundant --renderer.enable-thinking CLI flags in the integration tests are removed. Configurable.to_dict learns pydantic model_dump so the wandb config stays JSON. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

wwwjn · 2026-09-03T16:10:36Z

                    "--async-loop.num-samples-per-prompt 2",
                    "--trainer.training.max_context_length 1024",
                    "--trainer.training.num_tokens_per_microbatch_per_dp_rank 2048",
-                    "--renderer.enable-thinking False",


Can we still pass CLI to disable thinking? Maybe this config can not be set by CLI because the base render class doesn't have this field, so we can not pass it.

I explicitly added enable-thinking=False for these CI tests because CI machine are easy to OOM (with 24GB memory), Can monitor if the CI works fine after removing it?

HosseinKaviani-H · 2026-09-03T16:38:01Z

Thanks for raising this. The silent drop is what actually impacted the work in #4145. The Muse Glimmer recipe sets enable_thinking=True but that renderer has no such field so it was dropped and the recipe read one way and ran another. Typed configs kill that whole class of bug. One caveat from the same experience: they catch a wrong name, not wrong behavior, so our renderer bugs all passed unit tests and only showed up in a real rollout.

Here is Qwen3, which takes the create_renderer path. Is the TorchTitanRendererConfig path covered anywhere? A short rollout on the Muse Glimmer recipe would cover it.

tianyu-l · 2026-09-04T02:11:46Z

+                elif hasattr(val, "model_dump"):  # pydantic, e.g. renderer configs
+                    return _convert(val.model_dump())


This is supposed to work with only native configs; o/w it'd be bloated eventually.

tianyu-l · 2026-09-04T02:14:03Z

    ) -> None:
        """Build runtime dependencies after the worker actor is spawned."""
-        self._renderer = renderer_config.build(tokenizer_path=hf_assets_path)
+        tokenizer = HuggingFaceTokenizer(tokenizer_path=hf_assets_path)


this doesn't sound right -- it should be from tokenizer config build.

tianyu-l · 2026-09-04T02:14:54Z

    TORCHTITAN_CONFIG_FORMAT,
    TORCHTITAN_WORKER_CLS,
 )
+from torchtitan.experiments.rl.renderer import build_renderer


should be config.build

tianyu-l · 2026-09-04T02:18:47Z

+    return create_renderer(tokenizer=renderer_tokenizer, config=config)
+
+
+class RendererTokenizer:


Maybe

Suggested change

class RendererTokenizer:

class RendererTokenizerWrapper:

to not confuse with HuggingFaceTokenier's functionality

tianyu-l · 2026-09-04T02:20:32Z

+    renderer_cls: ClassVar[type[Renderer]]
+
+
+def build_renderer(


Instead of this, we should have a Renderer class and Renderer.Config in torchtitan and build from there. HuggingFaceTokenizer is the example https://github.com/pytorch/torchtitan/blob/main/torchtitan/components/tokenizer.py#L94

What's the benefit? Well the main one is it can be swapped to be other renderer impl.

tianyu-l · 2026-09-04T02:22:26Z


 import dataclasses

+from renderers import Qwen3RendererConfig


I think this is because Prime-rl "happens" to be using similar way of configuring, but the "proper" and robust way to use something in other library is to build our own wrapper class.

See https://github.com/pytorch/torchtitan/pull/4356/changes#r3931881516

felipemello1 requested review from fegin, tianyu-l, wconstab and wwwjn as code owners September 3, 2026 15:51

pytorch-bot Bot added the ciflow/rl label Sep 3, 2026

meta-cla Bot added the CLA Signed This label is managed by the Meta Open Source bot. label Sep 3, 2026

felipemello1 assigned HosseinKaviani-H Sep 3, 2026

felipemello1 force-pushed the 80-renderers-typed branch from 0d1de2a to bfdb125 Compare September 3, 2026 15:56

felipemello1 force-pushed the 80-renderers-typed branch from bfdb125 to 390eb7e Compare September 3, 2026 15:59

wwwjn reviewed Sep 3, 2026

View reviewed changes

felipemello1 force-pushed the 80-renderers-typed branch from 0c92da9 to 390eb7e Compare September 3, 2026 18:06

tianyu-l requested changes Sep 4, 2026

View reviewed changes

tianyu-l mentioned this pull request Sep 4, 2026

[rl] Add optional Verifiers rollout integration #4356

Open

Navigation Menu

Uh oh!

[rl] Pin renderers 0.1.11, render with TorchTitan's tokenizer, take the renderer's typed config directly - #4444

[rl] Pin renderers 0.1.11, render with TorchTitan's tokenizer, take the renderer's typed config directly#4444
felipemello1 wants to merge 1 commit into
pytorch:mainfrom
felipemello1:80-renderers-typed

felipemello1 commented Sep 3, 2026 •

edited

Loading

Uh oh!

wwwjn Sep 3, 2026

Uh oh!

HosseinKaviani-H commented Sep 3, 2026 •

edited

Loading

Uh oh!

tianyu-l Sep 4, 2026

Uh oh!

tianyu-l Sep 4, 2026

Uh oh!

tianyu-l Sep 4, 2026

Uh oh!

tianyu-l Sep 4, 2026

Uh oh!

tianyu-l Sep 4, 2026

Uh oh!

tianyu-l Sep 4, 2026

Uh oh!

tianyu-l Sep 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

		elif hasattr(val, "model_dump"): # pydantic, e.g. renderer configs
		return _convert(val.model_dump())

		return create_renderer(tokenizer=renderer_tokenizer, config=config)


		class RendererTokenizer:

Uh oh!

Conversation

felipemello1 commented Sep 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Validation

Uh oh!

Choose a reason for hiding this comment

Uh oh!

HosseinKaviani-H commented Sep 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

felipemello1 commented Sep 3, 2026 •

edited

Loading

HosseinKaviani-H commented Sep 3, 2026 •

edited

Loading