Repository navigation
fix(train): tch-backend rpath build.rs + unreachable test thresholds + --val-dir / --eval-only (rebased residual) - #1014
Merged
ruvnet merged 1 commit intoNov 11, 2026
Conversation
Owner
|
✅ Reviewed — valuable, and it exposes a CI blind spot. The deeper lesson here is the one worth acting on: the |
…t thresholds, --val-dir and --eval-only (ruvnet#1014) Rebased residual of ruvnet#1014 onto main. The tch 0.24 compile fixes, clamp_min, floor-division decode and E0507 test fix from the original PR already landed on main (ruvnet#1008 / ADR-155), so only what main still lacks is carried here: - build.rs: with `tch-backend` + LIBTORCH_USE_PYTORCH=1, embed an rpath to the PyTorch lib dir. On main the lib unit-test binary dies at load with "libtorch_cpu.so: cannot open shared object file". Strict no-op unless the feature AND the env var are set (default / --no-default-features / wheel builds unchanged). Honours PYTHON_SYS_EXECUTABLE like torch-sys. - losses.rs: two gaussian-peak tests asserted values that are mathematically unreachable for a half-pixel keypoint: exp(-0.5/8) = 0.9394 (< 0.95) and exp(-0.5/4.5) = 0.8948 (< 0.9). Thresholds 0.93 / 0.89 with the math. - trainer.rs: evaluate() and new evaluate_with_dump() share ONE loop (evaluate_inner with an optional JSONL sink), so --eval-only measures with exactly the metric training validation uses. - train.rs: --val-dir <DIR> uses a held-out MM-Fi dir as validation, taking precedence over the automatic subject-disjoint split; any subject overlap with --data-dir is refused as a leak (ADR-155 Tier-1.2). The automatic split and the synthetic smoke-test fallback are unchanged. --eval-only --checkpoint <FILE> --val-dir <DIR> [--dump-preds <JSONL>] loads a checkpoint and runs the validation loop standalone. Local evidence (PyTorch 2.11, LIBTORCH_USE_PYTORCH=1, --test-threads=1): main 353 pass / 2 fail (+ loader error without LD_LIBRARY_PATH); this branch 355/355 with no LD_LIBRARY_PATH. Co-Authored-By: claude-flow <ruv@ruv.net>
ruvnet
force-pushed
the
fix/trainer-tch024-and-real-validation
branch
from
November 11, 2026 13:13
57dc6f7 to
2c03e84
Compare
| #[arg(long, value_name = "FILE", requires = "eval_only")] | ||
| dump_preds: Option<PathBuf>, | ||
|
|
||
| /// Override the checkpoint output directory from the config. |
| log_config_summary(&config); | ||
|
|
||
| // ------------------------------------------------------------------ | ||
| // Eval-only mode: load checkpoint, run the validation loop, exit |
There was a problem hiding this comment.
Suggested change
| // Eval-only mode: load checkpoint, run the validation loop, exit | |
| // Eval-only mode: load checkpoint, run the validation loop, exit```suggestion | |
| // Eval-only mode: load checkpoint, run the validation loop, exit**[factory reset]()** |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rebased onto main: reduced scope
Rebased onto main (
dea1cc23, after #2524). Most of the original PR already landed on main through #1008 / ADR-155:TryFromconversionsnumelcast,EdgeRefimport,clamp_min(1.0)losses.rsE0507 test fixThose hunks are dropped. This branch is one commit carrying only what main still lacks. It was re-ported onto main's current code, not replayed from the old diff. Authorship is kept (@stuinfla).
What remains, and why it is still needed
build.rsrpath (tch-backend +LIBTORCH_USE_PYTORCH=1).cargo test -p wifi-densepose-train --features tch-backend --libbuilds, but the test binary dies at load withlibtorch_cpu.so: cannot open shared object file.CARGO_FEATURE_TCH_BACKEND, so it is a strict no-op for default,--no-default-featuresand wheel builds. It honoursPYTHON_SYS_EXECUTABLEthe way torch-sys does.Two gaussian-peak tests asserted unreachable values (still failing on main). A half-pixel keypoint gives a maximum of:
The thresholds are now 0.93 / 0.89. The original PR's comment gave 0.8007 and a 0.79 threshold. The measured value is 0.8948, so the comment and the tighter threshold are corrected.
--val-dir <DIR>: a held-out MM-Fi validation set.--data-diris refused as a leak (ADR-155 §Tier-1.2).[SMOKE-TEST]synthetic fallback are unchanged. The original else-branch, which routed synthetic validation throughrun_training, is not carried, because it would regress ADR-155.--eval-only --checkpoint <FILE> --val-dir <DIR> [--dump-preds <JSONL>].Trainer::evaluateand the newevaluate_with_dumpshare one loop (evaluate_innerwith an optional sink), so eval-only uses exactly the metric that training validation uses.Evidence (local; CI does not build
tch-backend)Run with PyTorch 2.11,
LIBTORCH_USE_PYTORCH=1and--test-threads=1.dea1cc23LD_LIBRARY_PATHis setLD_LIBRARY_PATHcargo test -p wifi-densepose-train --no-default-featuresrustfmt --checkis clean on the touched files.--data-dir train --val-dir valtrains 3 epochs and logs "REAL held-out validation set … subjects [2]".--eval-onlyreloadsbest_*.safetensors, printsEVAL RESULT, and--dump-predswrites 22 JSONL lines (one per window).--val-dirpointing at the training dir is refused, naming the shared subject.--eval-onlywithout--checkpoint.No version bump and no publish. The changes are tch-feature-gated or limited to the
trainbinary, and the published default build is unchanged.Refs #1010 (closed).
🤖 Generated with claude-flow