[rl] Pin renderers 0.1.11, render with TorchTitan's tokenizer, take the renderer's typed config directly

[felipemello1]

TLDR:

• pin renderers
• Use our tokenizer and remove transformers dependency
• Simplify code by using config classes directly rather than trying to map model_name -> config_type
• Simplify code by removing the need of registering muse glimmer

Summary

TitanRL uses renderers to turn chat messages into token ids and parse completions back. Until now we wrapped it in our own RendererConfig:

# before: recipe
renderer=RendererConfig(name="qwen3", enable_thinking=False)

# before: RendererConfig.build()
tokenizer = AutoTokenizer.from_pretrained(tokenizer_path)                  # transformers
args = {f.name: v for f in fields(self) if f.name in config_type.model_fields}   # silent drop
return create_renderer(tokenizer, config_type(**args))

Problems:

1. Silent drops ([rl] RendererConfig silently drops the removed preserve_* renderer knobs pytorch/torchtitan#4365). build() forwarded knobs by name-matching against the renderer's config and dropped the rest. When renderers replaced preserve_all_thinking with thinking_retention, the knob went inert with no error; two in-tree recipes were already hitting the same thing (enable_thinking on gpt-oss and Muse Glimmer, which have no such option).
2. A second tokenizer. build() loaded transformers.AutoTokenizer even though TorchTitan already has the tokenizer, and the controller then reached into renderer._tokenizer for pad_id.
3. Transformers dependency

Solutions:

renderers==0.1.11 accepts a bring-your-own tokenizer and drops the transformers dependency:

1. Recipes hold the library's typed config directly: We don't have a wrapper anymore. We don't try to redirect 'qwen3to the right config, e.g.renderer=RendererConfig(name="qwen3", enable_thinking=False)`. Instead, we do:

   from renderers import Qwen3RendererConfig, GptOssRendererConfig
   renderer=Qwen3RendererConfig(enable_thinking=False)
   renderer=GptOssRendererConfig(reasoning_effort="low")

2. We load our own tokenizer:build_renderer pairs that config with TorchTitan's tokenizer, through a small adapter (RendererTokenizer) that exposes the interface renderers expects.

   # controller / rollout worker / generate.py
   tokenizer = HuggingFaceTokenizer(tokenizer_path=hf_assets_path)
   renderer = build_renderer(tokenizer=tokenizer, config=config.renderer)
   pad_id = tokenizer.eos_id
   
   def build_renderer(*, tokenizer: HuggingFaceTokenizer, config: BaseRendererConfig) -> Renderer:
       if config.name == "auto": raise ValueError(...)      # unmatched local paths fall back unsafely
       if config.name == "default": raise ValueError(...)   # needs HF apply_chat_template
       renderer_tokenizer = RendererTokenizer(tokenizer)
       if isinstance(config, TorchTitanRendererConfig):     # renderer implemented in TorchTitan
           return config.renderer_cls(renderer_tokenizer, config)
       return create_renderer(tokenizer=renderer_tokenizer, config=config)

3. We skip registration of new renderers*: Muse Glimmer's renderer lives in TorchTitan, so the renderer's registry cannot find it. We skip the need for the renderers registry that maps config -> renderer class. Check TorchTitanRendererConfig.

Validation

• Alphabet sort, Qwen3-0.6B, 2 GPUs: validation reward 0.181 -> 0.548.