[rl] Add optional Verifiers rollout integration

[wwwjn]

WIP: need to be cleaned up.

Summary

• Add a reusable Verifiers rollout adapter under torchtitan/experiments/rl/rollout/verifiers.
• Add a verifiers example that reuses the existing DAPO Math datasets, training configuration, and reward logic while delegating episode execution to Verifiers.
• Use the Verifiers null harness with a local subprocess runtime. No Docker or remote sandbox is required, and no tools are exposed to the model.
• Keep Verifiers optional: existing recipes and shared TitanRL imports do not require the verifiers package.
• Leave the original dapo_math implementation unchanged.

The example can be launched with:

python -m torchtitan.experiments.rl.train \
  --module verifiers \
  --config rl_dapo_qwen3_4b_verifiers_8k

100-step comparison

Both runs used Qwen3-4B-Base on 8 GPUs with identical overrides:

• 100 optimizer steps
• seed 42 for trainer and generator
• 2 prompts per step and 4 samples per prompt
• sequence length 4096 and maximum completion length 256
• validation disabled
• zero-standard-deviation reward groups retained
• W&B online logging enabled

100-step metric	Native dapo_math	verifiers	Delta
Metric span	836.8 s	805.7 s	-3.7%
W&B total runtime	923.3 s	897.6 s	-2.8%
Mean step time	6.872 s	6.656 s	-3.1%
Full-step throughput	192.1 tokens/s	205.5 tokens/s	+7.0%
Forward/backward throughput	750.9 tokens/s	777.1 tokens/s	+3.5%
Mean decode latency	4387.5 ms	5052.1 ms	+15.1%
Mean response length	163.1 tokens	168.0 tokens	+3.0%
Mean reward	0	0	same
Mean loss	0	0	same

W&B runs:

• Native DAPO: https://meta.wandb.io/jianiw/titan_rl/runs/htox9jjk
• Verifiers: https://meta.wandb.io/jianiw/titan_rl/runs/lydh8lb2

The rollout distribution are similar:

This single-run comparison does not show an end-to-end throughput penalty from the Verifiers integration. The decode metric was slower, but generated responses were also longer, and model-state synchronization variance dominated the total step time.

[wwwjn]

Dataset is defined through verifiers