An independent, from-scratch implementation of Recurrent Looped Transformer (RLT), with measured results on two GPUs, controlled ablations, and a scannable variant (RLTX) built to test whether RLT's cost can be removed without losing its capability.
Every number here is measured. Raw JSON is in results/. Where a result contradicts
something we expected — or something we ourselves claimed earlier in the work — it is reported as
measured, with the ablation that settles it.
RLTX: Trading Sequential Depth for a Parallel Scan in the Recurrent Looped Transformer (Volkopat) — theory, design, the changes relative to RLT, the full performance study, and an honest accounting of prior art we converged on independently.
PDF (10 pages, typeset) · Markdown · LaTeX source in
paper/rltx.tex
This implements and evaluates the architecture proposed in:
Recurrent Looped Transformer Yifan Zhang —
yifanzhangresearch@gmail.comSeptember 12, 2026
All architectural ideas under test are from that paper: the causal encoder with recurrent decoder, the gated merge of Eq. (2.9)–(2.11), prompt–response consistency (Prop. 3.1), the document-packing rules of App. A.3, and the RL replay contract of §5.3.
This repository is an independent reproduction and extension. It is not affiliated with the author, and any errors in implementation, measurement, or interpretation are ours.
| question | answer |
|---|---|
| Do the paper's invariants hold in an independent implementation? | Yes, to machine precision (≤ 5.0e-16 in float64) |
| Does the recurrence buy real state-tracking capability? | Yes. 100% on S₅ at 4× training length, where Transformers are at chance |
| Does it help language modelling? | No. At 16M params on enwik8 it loses to a Transformer and to its own recurrence-off ablation |
| What does the recurrence cost? | 21–86× a Transformer's step time, and the gap widens on better hardware |
| Can an associative scan remove that cost? | Partly — 27× faster, but most of that is an architecture change, not the scan |
| Does the scannable variant keep the capability? | Yes — 100% on S₅, 8/8 seeds |
The result we would most want a reader to take away is the fourth row: the efficiency gap is not a constant-factor implementation problem — it grows with hardware generation.
A complete implementation of the forward pass, prefill/consume/generate paths, and RL replay
contract, with 165 tests across 11 modules (src/tests/), run in float64 so that "holds
exactly" is distinguishable from "holds approximately". One test module per paper claim.
Measured on both GPUs (results/*/exp1_invariants.json):
| paper claim | measured |
|---|---|
| Prop. 3.1 — invariance to the serving split | max abs logit difference 4.996e-16 across 15 prompt/response split points |
| Prop. B.1 — no future leakage | exactly 0.0; control perturbation registers 0.180 |
| §5.3 — RL replay reproduces behaviour probabilities | importance-ratio error 8.88e-16 at identical parameters |
| App. A.3 — document packing isolation | isolated delta 3.33e-16; deliberately leaked control 0.734 |
Each verification is paired with a control that should fail, so a passing number means the test can detect a violation rather than being vacuous.
§2.4, parallel (Jacobi/DEER) iteration. The paper notes the recurrence is not equivalent to a parallel decoder pass. Measured: after i iterations, exactly i positions are exact — 1, 2, 3 … 12 — converging only at iteration 12 (3.89e-16). The recurrence is genuinely sequential; iteration buys one position per sweep.
Precision. float64 split drift 5.48e-16; float32 2.46e-07; bfloat16 shows zero split drift but 8.61e-03 error against float64 — the invariant looks perfect in bf16 only because quantization hides it. Reporting split-drift alone in low precision is misleading.
Both GPUs, bf16, identical configurations (results/*/bench_*.json). Step time in ms,
ratio = RLT ÷ plain Transformer.
| scenario | config | 5090 RLT | 5090 Tf | ratio | B200 RLT | B200 Tf | ratio |
|---|---|---|---|---|---|---|---|
| tiny | d128 L4 b256 T32 | 506.2 | 21.7 | 23.3× | 312.0 | 14.9 | 20.9× |
| small | d256 L4 b128 T64 | 1023.5 | 24.3 | 42.1× | 617.0 | 15.2 | 40.6× |
| medium | d512 L6 b64 T128 | 3010.7 | 73.1 | 41.2× | 1871.3 | 21.8 | 85.6× |
| large | d1024 L8 b32 T128 | 4057.2 | 166.3 | 24.4× | 2477.7 | 31.5 | 78.8× |
RTX 5090 Laptop, 25.7 GB, 82 SMs, torch 2.7.1 · B200, 191.5 GB, 148 SMs, torch 2.8.0
The gap widens on better hardware. Moving from the 5090 to the B200:
- the Transformer gets 1.46× / 1.60× / 3.35× / 5.29× faster;
- RLT gets 1.62× / 1.66× / 1.61× / 1.64× faster — essentially flat.
RLT's per-token recurrence is launch-bound and sequential, so it cannot convert extra parallel
hardware into throughput; the Transformer can. At large that turns a 24.4× penalty on the 5090
into 78.8× on the B200. Newer accelerators make the trade worse, which makes this an
architectural cost rather than a kernel-tuning gap.
Task: the word problem over S₅ (running composition of permutations), NC¹-complete. Trained at
length 32, evaluated to 128. Labels are never inputs, so nothing can be copied. 3 seeds
(results/rtx5090/exp2_state_tracking_s5.json). Final-position accuracy:
| model | params | L=16 | L=32 | L=64 | L=128 |
|---|---|---|---|---|---|
| RLT | 985K | 1.000 | 1.000 | 1.000 | 1.000 |
| RLT α=0 (recurrence off) | 985K | 0.914 | 0.083 | 0.009 | 0.008 |
| Plain Transformer 8L | 1.52M | 0.970 | 0.322 | 0.008 | 0.007 |
| Plain Transformer 4L | 770K | 0.871 | 0.065 | 0.007 | 0.008 |
Majority-class baseline is 2.15%, so every non-RLT row is at chance beyond the training length. The α=0 ablation is the controlled comparison — identical parameters, block count, and memory, with only the recurrent path zeroed — and it fails. The capability comes from the recurrence specifically, not from parameters or depth.
Note the shape of the Transformer rows: 97% at L=16, 32% at L=32, chance thereafter. They learn the task in-distribution and fail to extrapolate.
Two scales, and the result reverses between them.
Small (764K params, toy corpus) — results/b200/exp6_language_model.json, val bits/char:
| model | params | val bpc |
|---|---|---|
| RLT | 764K | 2.6159 |
| RLT α=0 | 764K | 2.9021 |
| Plain 6L | 1.15M | 3.1129 |
Here RLT beats a Transformer with 1.5× more parameters, and the recurrence helps.
Real (16M params, enwik8) — results/b200/exp9_real_lm_enwik8.json, test bits/char:
| model | params | matched steps | matched wall-clock |
|---|---|---|---|
| RLT | 16.0M | 1.8556 | 1.8556 |
| RLT α=0 | 16.0M | 1.8196 | — |
| Plain 6L | 18.8M | 1.8204 | 1.4998 |
| Plain 12L | 37.5M | 1.7582 | 1.5017 |
RLT loses to its own ablation. Same parameters, same blocks, same memory — turn the recurrence off and bits/char improves (1.8196 vs 1.8556). At matched wall-clock it is not close (1.4998 vs 1.8556), because the Transformer completes far more steps in the same time.
The honest reading: the parameter-efficiency advantage visible at 764K inverts by 16M on real text, and the α=0 control identifies the recurrence itself as the cause. We flag this because we initially drew the opposite conclusion from the small-scale run alone.
RLT's token-to-token map runs through a full nonlinear decoder stack, which makes it non-associative and therefore unparallelisable. RLTX keeps a per-token state but moves the nonlinearity inside the per-token block and carries state with an associative affine map:
x_t = Block(tokens) parallel across t
(A_t, b_t) = Carrier(x_t) parallel across t
h_t = A_t h_{t-1} + b_t exact parallel scan, O(log T) depth
The scan is exact, not an approximation: parallel vs sequential agreement 3.05e-16 in float64.
RLTX changes two things versus RLT at once: it drops the sequential schedule and replaces a per-token decoder stack with a single affine map. Comparing RLTX to RLT cannot separate them, so we added the control — RLTX with a sequential scan, which holds the architecture fixed and restores only the schedule. B200, d_model=256, 4 layers, d_state=32, batch=128, ms/step:
| T | RLT | RLTX-seq | RLTX-scan | Transformer | architecture | schedule | total |
|---|---|---|---|---|---|---|---|
| 32 | 308.1 | 29.8 | 24.9 | 17.3 | 10.33× | 1.20× | 12.37× |
| 128 | 1272.9 | 85.5 | 51.5 | 25.7 | 14.89× | 1.66× | 24.71× |
| 512 | 5318.1 | 530.2 | 196.6 | 78.0 | 10.03× | 2.70× | 27.05× |
The associative scan is worth 2.70×, not 27×. The other ~10× comes from replacing the per-token decoder stack with an affine map — an architecture change unrelated to associativity. Any claim of the form "making the recurrence parallel recovers the speed" should be decomposed this way before it is believed.
Two things survive: the schedule factor grows monotonically with T (1.20 → 2.70) while the architecture factor peaks and declines, so the scan is the component that scales; and the scan is what holds RLTX at 2.52× a Transformer at T=512, where RLTX-seq is 6.80× and rising.
S₅, d_state=8, 8 seeds, final-token accuracy at L=128 (baseline 2.15%):
| carrier | mean | seeds converged |
|---|---|---|
| orthogonal (4 Householder factors) | 1.000 | 8/8 |
| Householder product, learnable β | 1.000 | 8/8 |
| gated orthogonal | 0.865 | 7/8 |
| tropical (max-plus) | 0.009 | 0/8 |
| diagonal signed | 0.009 | 0/8 |
The capability RLT demonstrates is retained by a scannable carrier at ~1/27th the cost — but only for carriers whose transition matrices are orthogonal. Diagonal and semiring carriers are at chance.
Reset automaton, d_state=8, 8 seeds, baseline 5.7%:
| carrier | S₅ | reset automaton |
|---|---|---|
| Householder product, learnable β | 100% (8/8) | 100% (8/8) |
| gated orthogonal | 88% (7/8) | 100% (8/8) |
| pure orthogonal | 100% (8/8) | 0% (0/8) |
| tropical | 0% | 12% |
A pure orthogonal matrix is invertible, so it cannot destroy state — and a reset is exactly the non-invertible primitive. It fails in 0 of 8 seeds. A learnable β lets an eigenvalue reach 0 (a projection, i.e. erasure), which is why that carrier spans both.
The usual rule for Householder-product carriers is quoted as "n_h factors model permutations of up to n_h+1 elements" — a statement about the group. We tested it by varying the generating alphabet while holding the group fixed. d_state=8, 6 seeds, mean final accuracy at L=128:
| alphabet | group | max reflection length r | n_h=1 | n_h=2 | n_h=3 | n_h=4 |
|---|---|---|---|---|---|---|
| adjacent transpositions | S₅ | 1 | 0.673 | 0.836 | 1.000 | 1.000 |
| 3-cycles | A₅ | 2 | 0.014 | 0.750 | 0.811 | 0.999 |
| two 4-cycles | S₅ | 3 | 0.010 | 0.017 | 0.837 | 0.839 |
| 5-cycle + swap | S₅ | 4 | 0.005 | 0.009 | 0.092 | 1.000 |
Performance departs from chance at exactly n_h = r in every row. The decisive contrast is rows 1 and 4: the same 120-element S₅, presented two ways. By adjacent transpositions (r=1) a single reflection per step suffices; by a 5-cycle (r=4) it is at chance until n_h=4.
The requirement is set by the alphabet's per-step reflection length, not by the group being tracked. We report this as "onset at n_h = r" rather than a threshold theorem — at the boundary, 4/6 and 5/6 seeds converge rather than 6/6.
Practical consequence: methods that linearise a pretrained Transformer into a DeltaNet-style target fix the conversion at a single Householder factor (n_h=1), the top-left cell of this grid. Adding a second factor initialised to β=0 is identity at initialisation and strictly more expressive.
Three bugs, found late, that silently corrupted measurements:
v / (v.norm() + 1e-6)never produces a unit vector, soI − 2vvᵀwas never exactly a reflection. Measured in float64:|QᵀQ − I| = 4.09e-06. Replacing with.clamp_min()gives 1.11e-15 and singular values of exactly 1.0000000000.- A global
self.apply(init)zeroed every carrier's deliberate bias, so a carrier meant to start near-orthogonal (β≈2) started at β=1 — eigenvalue 0, a projection that annihilates the state. That carrier sat at chance for the whole project until fixed; afterwards it reaches 100% on both tasks. - A degenerate task. An early max-subarray formulation had a single distinct label at L=128,
so a constant predictor scored 100%. All numbers from that version were discarded, and every
task now reports a measured majority-class baseline rather than uniform
1/n.
A methodological note worth propagating: S₅ training is bimodal — a seed either converges to ~1.0 or collapses to chance. A 3-seed mean can therefore read 1.00, 0.67 or 0.33 with no underlying difference in capability. Every table here reports the fraction of seeds that converged alongside the mean. Means alone are not a measure of capability on this task.
- Scale. State-tracking work is ≤1M parameters and ≤128 tokens; LM work is ≤37.5M. Nothing here speaks to behaviour at production scale.
- Task realism. S₅ and the reset automaton are synthetic, chosen because fixed-depth models provably struggle with them. We did not find a natural-data task where the capability demonstrably pays.
- Expressivity trade. RLTX is bounded to the finite-state class by construction; RLT, being non-associative, is not so bounded. The 27× is bought with expressivity, and that trade is not characterised here.
- Ablation scope. The α=0 ablation isolates the recurrence but shares the encoder/decoder split with RLT; it is not a test of that split.
- Seeds. 8 seeds on the headline carrier comparisons, 6 on the alphabet grid, 3 elsewhere.
The LM comparison is 3 runs of an identical configuration with no seed control (the script has
no
--seedflag), so its spread reflects nondeterministic initialisation, not controlled seeding.
Everything needed to reproduce the tables is in src/. No dependency beyond PyTorch.
src/
├── rlt/ the RLT implementation
│ ├── model.py forward (§A.3), prefill (§A.1), consume/generate (§A.2)
│ ├── layers.py shared layer core, encoder/decoder blocks, gated merge (2.9–2.11)
│ ├── caches.py SWA cache, encoder cache, memory bank, state, prefix snapshots
│ ├── modules.py RMSNorm, RoPE at explicit absolute positions, SwiGLU, attention
│ ├── rl.py rollout and the §5.3 replay contract, importance ratios, losses
│ ├── train.py lm/label losses, document packing (§A.3), train and eval loops
│ ├── baselines.py plain Transformer used for every comparison
│ └── tasks/synthetic.py S₅, parity, mod-sum, recall, copy, max-subarray, reset automaton
├── rltx/ the scannable variant
│ ├── scan.py associative affine scan, plus tropical and min-max scans
│ ├── carriers.py diagonal, signed-diagonal, Householder, gated-orthogonal, dense,
│ │ tropical, min-max — the objects the §5 tables compare
│ ├── model.py RLTX
│ ├── tasks.py group word problems with a controlled generating alphabet (§5.4)
│ ├── ringing.py change-ringing place-notation expansion (unused; see Limitations)
│ └── experiments/ the §5 experiment scripts
├── experiments/ the §1–§4 experiment scripts (exp1–exp9)
└── tests/ 165 tests, one module per paper claim
cd src
pip install torch pytest
# verification (§1) — runs in float64, no GPU required
pytest tests/ -q
python experiments/exp1_invariants.py
# hardware comparison (§2)
python experiments/exp3_efficiency.py
# state tracking (§3) — RLT vs α=0 vs Transformers on S₅
python experiments/exp2_state_tracking.py --task s5 --seeds 3
# language modelling (§4)
python experiments/exp6_language_model.py
python experiments/exp9_real_lm.py # enwik8; downloads the corpus
# the scannable variant (§5)
python rltx/experiments/exp_speed.py --lens 32,64,128,256,512 --batch 128 \
--d-model 256 --layers 4 --d-state 32 --carrier orthox4
python rltx/experiments/exp_solvability.py --task s5 --d-state 8 --seeds 8 \
--carriers gatedorthox4,orthox4,householderx4,tropical,diagonal_signed
python rltx/experiments/exp_reflength.py --d-state 8 --seeds 6Each experiment writes JSON and prints its own verdict, including an explicit
INCONCLUSIVE when no arm beat its measured baseline — so a failed training run cannot be
mistaken for a scientific result.
Measured with Python 3.13.5 / PyTorch 2.7.1+cu128 (RTX 5090) and Python 3.11.13 / PyTorch 2.8.0+cu128 (B200).
Two categories, kept separate so they are never confused:
Primary artifacts — written directly by the run, unmodified. These back §1, §2, §3 and §4:
results/rtx5090/ bench_RTX5090, exp1_invariants, exp2_state_tracking_s5,
exp4_feedback_stability, exp5_rl
results/b200/ bench_B200, exp1_invariants, exp6_language_model,
exp8_examples, exp9_real_lm_enwik8
results/rltx/ rltx_speed, rltx_wreath, rltx-ortho, rltx-beta,
trop_s5, margin_orthox4, rltx_lm2, spectrum_s5
Transcribed values — results/rltx/transcribed_cluster_runs.json. These runs executed on the
B200 and wrote JSON to a shared volume that was cleaned before the files were retrieved; the
numbers are transcribed from the runs' captured stdout. They back §5.1–5.4. Each entry carries
the exact command that produced it and is reproducible from the code in the parent repository, but
it is second-hand and not byte-verified. Readers who need primary artifacts for those tables
should re-run the listed commands.
The results/rltx/*.json primary files are from an earlier configuration (d_state=32, 2–3
seeds) that we later established is bimodal; §5 uses the d_state=8, 6–8 seed reruns instead.
They are included for completeness, not as support for the §5 tables.