Repository navigation
docs(reproduce): add paper reproduction scripts and guide - #3
Merged
Merged
Conversation
added 8 commits
April 16, 2026 23:09
Scans autonomous-learning-library/{examples,all/scripts,all/r3,all/presets}
and emits a per-category Markdown (or --json) table with paths, default
frame counts, and env/agent hints. Pure stdlib so it runs on the Jetson
Orin venv without extra deps.
This is the first piece of the new reproduce/ folder that documents how
to reproduce the RTSS 2023 R^3 paper on the modernized PyTorch 2.8 +
Gymnasium stack.
run_smoke_classic.sh exercises the modernized ALL stack on CartPole-v1 with DQN for a tiny number of frames (default 5000). Used as a fallback sanity gate when the Atari ROMs are not yet installed on the host. Writes everything under /experiment/zexin/R3/reproduce/runs/ to respect the Jetson Orin disk-layout rule (NVMe only).
run_smoke.sh is the canonical smoke gate for the R^3 reproduction:
1. Runs all R^3 algorithmic unit tests (deadline, batch_size_control,
runtime_coordinator, replay_buffer_size).
2. Preflights ale_py + Pong ROM availability with actionable hints if
missing.
3. Trains DQN on PongNoFrameskip-v4 for FRAMES (default 20000) frames
via the installed all-atari CLI.
4. Parses tensorboard logs and prints final returns/mean to stdout.
Exit codes (0 PASS, 1 unit-test fail, 2 ALE missing, 3 train crash,
4 log-parse fail) make it suitable as an automated CI gate. Output goes
to /experiment/zexin/R3/reproduce/runs/smoke_<timestamp>/ on the NVMe.
run_short.sh trains DQN on PongNoFrameskip-v4 for 250000 frames and is
the bridge between the smoke gate and a full paper run. Designed to be
detached from the user's shell:
- Recommended invocation is via tmux (see header docstring).
- --background flag self-daemonizes via setsid+nohup as a fallback when
tmux is not available.
- Maintains a short_LATEST symlink under reproduce/runs/ so the user
can always 'tail -F short_LATEST/short.log'.
- Emits progress.json with all parsed returns/mean snapshots at the end.
Documented inspection commands include 'tmux attach -t r3-short' and
'tensorboard --logdir reproduce/runs/short_LATEST/runs --bind_all'.
run_full_paper.sh is the catalog + driver for every paper experiment that is reproducible on a single Jetson Orin (Atari only). Each catalog entry maps a name to (env, agent, frames, seeds, paper figure, hours estimate, notes). Modes: --list inventory + estimates (no training) --dry-run print exact all-atari invocations, do not execute --only NAME run a single named experiment from the catalog (default) run every in-scope catalog entry serially The DonkeyCar / MUSHR-DL case study from the paper is intentionally listed as out-of-scope and never executed: it requires a separate x86 host running the Unity DonkeySim, so single-Orin reproduction is impossible. The header and reproduce/README.md both link to the manual setup path for users who really need those numbers. Output is written to /experiment/zexin/R3/reproduce/runs/full_<ts>/ with per-experiment subdirs and an INDEX.md summary.
reproduce/README.md is the single entry point for reproducing the RTSS
2023 R^3 paper on the modernized PyTorch 2.8 + Gymnasium + JetPack 6.2
stack. Sections:
1. System requirements (JetPack 6.2 / Orin AGX recommended).
2. Output convention — every script writes under
/experiment/zexin/R3/reproduce/runs/ on the NVMe, never the eMMC
root.
3. The four scripts (smoke, classic smoke, short, full) with usage,
tmux/tensorboard recipes, and exit-code semantics.
4. Skipped reproductions — explicitly documents why DonkeyCar / MUSHR-DL
are out of scope on a single Orin (require x86 + Unity DonkeySim).
5. Current status of R^3 runtime modules vs. agent presets, so a
reader knows exactly what the smoke / short / full runs prove today.
6. Troubleshooting table covering the most common failure modes
(missing ROMs, OOM, disk fills up, command not found, SSH drop).
ALL's all-atari CLI (autonomous-learning-library/all/scripts/train_atari.py
+ all/environments/atari.py) automatically appends 'NoFrameskip-v4' to
the env name passed on the command line. Passing the full
'PongNoFrameskip-v4' therefore produced 'PongNoFrameskip-v4NoFrameskip'
which gymnasium does not know about.
Use the short names ('Pong', 'Breakout', 'Seaquest') in:
- reproduce/run_smoke.sh
- reproduce/run_short.sh
- reproduce/run_full_paper.sh catalog
- reproduce/README.md catalog table + ENV override example
Discovered while running the smoke test on Jetson Orin AGX.
Starting in ale-py 0.11, the package no longer auto-registers Atari envs
into gymnasium on import. Calling gymnasium.make('PongNoFrameskip-v4')
therefore raises NameNotFound even when ale_py is installed and Pong's
ROM is bundled, which broke the reproduce/run_smoke.sh Atari path on
Jetson Orin (JetPack 6.2 venv ships ale-py 0.11.2).
Add a guarded gymnasium.register_envs(ale_py) at the top of
all/environments/atari.py. The try/except keeps the import optional so
users who do not want Atari support do not need ale-py installed.
Tested on Orin AGX with the smoke run.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a new top-level
reproduce/folder that is the single entry point forreproducing the RTSS 2023 R³ paper on the modernized PyTorch 2.8 + Gymnasium +
JetPack 6.2 stack. Nothing in the existing
autonomous-learning-library/treeis touched — this is purely additive documentation + driver scripts.
What's in
reproduce/README.mdlist_experiments.pyrun_smoke.shrun_smoke_classic.shrun_short.sh--background-friendlyrun_full_paper.sh--list/--dry-run/--only NAMEAll scripts write under
/experiment/zexin/R3/reproduce/runs/so they respectthe Jetson Orin disk-layout rule (NVMe only, never the eMMC root).
Out-of-scope reproductions
The DonkeyCar / MUSHR-DL case study from the paper is intentionally not
runnable from these scripts: it requires a separate x86 PC running the
Unity-based
donkey_simbinary.run_full_paper.shlists those experimentsunder "Out-of-scope" and
reproduce/README.mddocuments the manual setup ifyou specifically need to reproduce them.
Validation
python reproduce/list_experiments.pyproduces a sensible inventory locally.bash reproduce/run_full_paper.sh --listprints the catalog with figuremappings and wall-clock estimates.
run_smoke.shwas launched on Jetson Orin (orin1) — see the agent runreport for the actual PASS/FAIL outcome and the
run_short.shdaemonizedtmux session details.
Atomic commits
Each file landed as its own commit so the diff is easy to review piece by
piece:
feat(reproduce): add list_experiments.py inventory toolfeat(reproduce): add CartPole classic-control smoke testfeat(reproduce): add Atari Pong smoke test (primary 5-10 min gate)feat(reproduce): add 30-60 min Atari short paper-validation scriptfeat(reproduce): add full-paper Atari catalog driverdocs(reproduce): add full README index for paper reproduction guideNotes for the reviewer
autonomous-learning-library/all/r3/) is unit-testedbut not yet wired as a default hook into the agent presets on the modernized
branch.
reproduce/README.md§5 makes this status explicit so the smoke /short / full runs are not over-claimed.