Repository navigation
fix(lerobot): namespace cache by resolved revision - #328
kstonekuan merged 4 commits into
Conversation
|
👋 Hi @mikemikimike — thank you so much for your first contribution to HFlow! A maintainer will review your pull request as soon as possible. In the meantime:
💡 Tip: one open pull request per contributor at a time. Issues with an assignee are taken; everything else is fair game. We are excited to have you here and appreciate your help making the project better! 🙌 |
kstonekuan
left a comment
There was a problem hiding this comment.
Thanks, and welcome. The fix is the right one-liner and the test earns its place: routing tag-a and branch-a to the same sha and asserting the marker file survives proves reuse and separation in the same assertion, rather than just checking two directory names differ.
One thing before merge, and it's a consequence of your change rather than something you inherited.
Until this PR the cache path was a constant. Now it ends in a string that came off the network, and nothing checks its shape. _hf_repo_info only requires a non-empty str:
resolved_revision = repository_information.get("sha")
if not isinstance(resolved_revision, str) or not resolved_revision.strip():So DoD 5 ("the cache remains entirely below output_dir / "_lerobot_cache"") is stated but not enforced:
sha='8f2c1ab9...' -> /tmp/probe-328-out/_lerobot_cache/8f2c1ab9... below: True
sha='../../../../tmp/probe-328-escaped' -> /tmp/probe-328-escaped below: False
sha='/tmp/probe-328-absolute' -> /tmp/probe-328-absolute below: False
To be clear about the size of this: it needs Hugging Face to return a hostile sha, so it is not a live path traversal and I don't want the commit message calling it one. It matters because the DoD names it, and because your change is what turned a constant path segment into a remote-controlled one.
Refuse a sha that isn't a commit hash where it's resolved, next to the check above. A commit sha is hex, so something like this is enough:
if not re.fullmatch(r"[0-9a-f]{7,64}", resolved_revision):
raise ValueError(f"Hugging Face returned a malformed commit sha for {repo_id}@{revision}")Then a case in your existing parametrized style asserting ../../evil is refused. Your test stubs _hf_repo_info, so this one wants to go at the real function.
Gate is otherwise clean on your head: 1252 passed / 6 skipped.
kstonekuan
left a comment
There was a problem hiding this comment.
LGTM, merging. DoD 5 is enforced now rather than assumed:
ACCEPTED '8f2c1ab9d4e5f60718293a4b5c6d7e8f90a1b2c3' -> below _lerobot_cache: True
ACCEPTED '8f2c1ab' -> below _lerobot_cache: True
refused '../../../../tmp/probe-328-escaped'
refused '/tmp/probe-328-absolute'
refused '..'
refused 'C:/windows'
refused '8f2c1ab9/../../../etc'
Putting it in _hf_repo_info rather than at the join is the right place: the sha is wrong there, not just unsafe as a path, and every future caller of the resolved revision gets it. The match="malformed commit sha" in your test matters too, since a bare pytest.raises(ValueError) there would also pass on the pre-existing empty-sha refusal.
I merged current main in and pushed as 62c3156, because main moved a long way under this and #326 landed in both files you touch. That surfaced a real interaction, which is why I mention it rather than just fixing it quietly:
FAILED tests/test_lerobot_converter.py::test_hf_repo_info_valid_json_still_resolves
#326's fixture uses {"sha": "abc123"}, six hex characters, one short of your {7,64} minimum. Your constraint is right and the fixture was the unrealistic part, so I changed it to abc1234. Nothing about either change is wrong; it's just a collision only visible on the merged result, which is the argument for the bound you picked being a real one.
Full gate on the merged result: clean, 1340 passed / 6 skipped.
One small thing, no action needed. Your namespacing test uses sha-a and sha-b, which the validator you just added would reject. It passes because that test stubs _hf_repo_info wholesale, so it never sees them. Fine as is, but if you touch it again, hex fixtures would keep the two tests telling the same story.
main Hebbian-Robotics#328 namespaces the materialized archive under _lerobot_cache/<resolved sha>; export resolved the flat path and would fail on a fresh materialization. Resolve the single namespaced subdirectory after the import and fail loudly on an unexpected layout. The test fake mirrors the namespaced layout. Signed-off-by: Ayamullah Khan <ayamullahkhan04@gmail.com>
main Hebbian-Robotics#328 namespaces the materialized archive under _lerobot_cache/<resolved sha>; export resolved the flat path and would fail on a fresh materialization. Resolve the single namespaced subdirectory after the import and fail loudly on an unexpected layout. The test fake mirrors the namespaced layout. Signed-off-by: Ayamullah Khan <ayamullahkhan04@gmail.com>
…t Dataset v3 (fixes #190) (#273) * feat(lerobot): export an HFlow-curated selection as a loadable v3 dataset (fixes #190) * fix(lerobot): repoint export at the public importer API Review follow-up on #273: - Tests run for real now: dropped pytest.importorskip("examples.lerobot.prepare") (module-scope skip, the whole file collapsed into one silent skip). The suite uses the sys.path namespace-import approach the repo already uses in examples/egosuite_evaluation/tests; 8 tests execute, no skips. - export.py no longer touches examples/lerobot/prepare private helpers. The source archive is materialized through the public hflow.import_lerobot_dataset entry point (hflow import lerobot) and consumed from the _lerobot_cache artifacts it writes; chunk fetches became existence checks against that archive. --camera-keys is now required so the importer validates the requested cameras (the old "default: all" branch could not resolve keys on the public surface). - Rebased onto current main (covers the first-class importer move). - README export examples updated for --camera-keys. Validation: pytest tests/test_lerobot_export.py -> 8 passed (0 skipped); ruff check/format clean; ty check clean; full suite 1256 passed, 6 skipped, 2 pre-existing failures (catalog_ui flake re-passed; end_to_end ffmpeg filter_script:v unsupported by the WSL ffmpeg build, reproduced on clean main without this change). Signed-off-by: Ayamullah Khan <ayamullahkhan04@gmail.com> * test(lerobot): pin provenance digests and v3 structural validation Second review round found _sha256 and _validate_v3 could both be neutered with no test noticing. Add a digest test asserting each provenance digest equals the sha256 of the file it names and that distinct files get distinct digests, plus one rejection test per structural check (incoherent info.json, window != length, missing video). RED-verified: with both checks neutered the four new tests fail while the eight existing ones still pass. Signed-off-by: Ayamullah Khan <ayamullahkhan04@gmail.com> * fix(lerobot): consume the revision-namespaced importer cache main #328 namespaces the materialized archive under _lerobot_cache/<resolved sha>; export resolved the flat path and would fail on a fresh materialization. Resolve the single namespaced subdirectory after the import and fail loudly on an unexpected layout. The test fake mirrors the namespaced layout. Signed-off-by: Ayamullah Khan <ayamullahkhan04@gmail.com> * fix(lerobot): carry source (chunk, file) identity through the curated export Round-3 export-logic findings from review, plus the #353 cache coupling. The exporter wrote every camera to chunk-000/file-000.mp4 and skipped the copy when that path existed, so a selection spanning more than one source video chunk published the wrong bytes while the episode rows still carried their own window offsets. The destination now keeps the source (chunk, file) identity: distinct source files land in distinct output files, each episode row references the file its frames actually come from, and the copy is deduplicated by identity instead of by first-write. The importer now separates video cache files by index (#353), so the cache lookup reads the {camera}-chunk{chunk}-file{file}.mp4 name and the window reader keeps a 0 chunk/file index instead of dropping it as falsy. Episode data refs are plain integers again and one data parquet is written per selected episode, so the exported dataset reads back with its own reader and the data_path template formats over every row. Sources without the v3 index column are refused loudly instead of silently writing a global frame_index over the per-episode one. The dead official-API validation pass (lerobot never installed anywhere) is dropped; the structural checks now own validation, including that every data and video reference formats through the dataset templates and resolves to an existing file. total_videos and the provenance digests are per-file. Signed-off-by: Ayamullah Khan <ayamullahkhan04@gmail.com> * test(lerobot): multi-chunk exports, read-back, and template-format refusals Every round-3 finding is pinned by at least one test that fails on the old code and passes now. The fixture writes the #353 cache names (chunk and file index) and gains a second video chunk with distinguishable bytes. test_export_multi_chunk_videos asserts both source files land byte-exact in distinct destinations, the second chunk is no longer absent, episode rows reference the file their frames come from, total_videos counts distinct files, and the provenance digests differ per file. test_export_output_readback feeds the exported episode parquet through the exporter own reader and substitutes the data_path template, the exact loadability contract the review described. test_export_validation_failure_leaves_no_destination pins the atomic publish guarantee: a validator refusal propagates and leaves a pre-existing destination untouched and no new destination behind. test_validate_v3_rejects_non_integer_data_refs stages the review exact chunk-000 string and asserts the refusal. The noncontiguous test now asserts the plain data file indexes and that the per-episode data file exists with a per-episode frame_index restart. Signed-off-by: Ayamullah Khan <ayamullahkhan04@gmail.com> --------- Signed-off-by: Ayamullah Khan <ayamullahkhan04@gmail.com>
_stop_duckdb_ui_server called stop_ui_server() whenever the server start hook ran, but the function only exists once the ui extension is loaded. When it is not (e.g. the lifecycle test monkeypatches the start hook), DuckDB auto-installs the extension from the network during shutdown -- seconds on a slow connection, and pointless: there is no server to stop if the extension was never loaded. That network fetch inside a teardown another thread waits on with a fixed timeout is almost certainly the flaky test_catalog_ui timing failure reported on Hebbian-Robotics#326, Hebbian-Robotics#327 and Hebbian-Robotics#328. Guard the call with a duckdb_extensions() loaded check: production always LOADs ui before the server starts, so the stop call is preserved, and a connection that never loaded the extension now closes directly. Pin both shutdown branches with tests that record the statements issued on the connection (Hebbian-Robotics#336 DoD): the stubbed-starter path must issue no stop_ui_server() call, and a session whose ui extension is loaded must still stop its server. The recording proxy does not delegate the stop call to DuckDB -- on a connection without the loaded extension that would itself trigger the network auto-install this fix removes, and asserting DuckDB's install behavior is not the point. Closes Hebbian-Robotics#336
_stop_duckdb_ui_server called stop_ui_server() whenever the server start hook ran, but the function only exists once the ui extension is loaded. When it is not (e.g. the lifecycle test monkeypatches the start hook), DuckDB auto-installs the extension from the network during shutdown -- seconds on a slow connection, and pointless: there is no server to stop if the extension was never loaded. That network fetch inside a teardown another thread waits on with a fixed timeout is almost certainly the flaky test_catalog_ui timing failure reported on #326, #327 and #328. Guard the call with a duckdb_extensions() loaded check: production always LOADs ui before the server starts, so the stop call is preserved, and a connection that never loaded the extension now closes directly. Pin both shutdown branches with tests that record the statements issued on the connection (#336 DoD): the stubbed-starter path must issue no stop_ui_server() call, and a session whose ui extension is loaded must still stop its server. The recording proxy does not delegate the stop call to DuckDB -- on a connection without the loaded extension that would itself trigger the network auto-install this fix removes, and asserting DuckDB's install behavior is not the point. Closes #336
Summary
Closes #294
Validation