Repository navigation
fix(lerobot): disable B-frames so the converter output decodes in full - #251
Conversation
Hebbian-Robotics#250) The converter transcodes each episode window with libx264's default medium preset, which enables B-frames. hflow's write_access_units_to_mp4 remuxes the resulting raw Annex B stream to MP4 with -c copy, and that remux drops the reorder-buffer tail at decode: the muxed file carries 303 samples but only 301 decode, so camera_frame_stats undercounts decoded_frame_count on every healthy episode (both svla cameras measured 301/303 while the source slices decode 303/303 with zero errors). Encode with bframes=0: the identical remux then decodes 303/303. Verified end to end on svla_so101_pickplace @ f641879 ep0 - App.process now records decoded_frame_count 303/303 on both cameras (was 301/303). Regression test: converter output remuxed via write_access_units_to_mp4 must decode every source frame (fails on the old encode, passes now). ruff/format/ty clean; full converter test file 6/6. Signed-off-by: Ayamullah Khan <ayamullahkhan04@gmail.com>
pytestmark applies to the whole module, so on a machine without system ffmpeg the five metadata tests skipped too, and none of those touch ffmpeg. Without ffmpeg on PATH it is now 5 passed, 1 skipped instead of 6 skipped.
kstonekuan
left a comment
There was a problem hiding this comment.
Merging. The diagnosis on #250 is the best piece of debugging this repo has had, and the fix is one line because you did the work to earn it.
Validated on Linux: ruff check, ruff format --check, ty check, 1187 passed, 6 skipped.
I reverted bframes=0 back to aud=1 and the new test failed, so it catches the thing it was written for rather than merely exercising the path.
The layer-by-layer attribution is what makes this reviewable. Source decodes 303/303, converter output decodes 303/303 as raw Annex B, the remux container holds 303 samples and the decoder emits 301, and re-encoding with bframes=0 restores 303 through the identical remux. Four measurements, each isolating one layer, and the conclusion follows from them rather than from a plausible story. Trying -muxdelay 0, genpts, max_interleave_delta 0, and a two-step remux before concluding no flag recovers it is the part most people skip.
One thing worth adding to the record: the converter was emitting non-conforming output. docs/FORMAT.md:152 item 4 requires no B-frames, so a canonical stream should never have had them. Nothing caught it because the doctor does not classify picture types, which #236 documented three hours before you hit this. So the chain is: a documented unchecked constraint, a converter quietly violating it, and a silent two-frame loss downstream. That is the first real consequence of that gap, and it is a good argument for closing it.
One fixup pushed (09adc9d): the pytestmark skip was module-level, so on a machine without system ffmpeg all six tests in the file skipped, including the five metadata tests that never touch ffmpeg. Scoped it to the one test that shells out. Without ffmpeg on PATH it is now 5 passed, 1 skipped instead of 6 skipped.
Your two longer-term notes on #250 are both right and both want separate issues. I would take the measurement one first: decoded_frame_count being below message_count on a healthy episode is exactly the blind spot, and a companion to frame_deficit_pct would have surfaced this without anyone running ffmpeg by hand. #250 stays open and assigned to you for whichever you want to carry.
What
One-line converter change:
_transcode_mp4_to_h264now encodes withbframes=0(was libx264 medium default, which enables B-frames).Why
Fixes #250. The raw Annex B -> MP4
-c copyremux inwrite_access_units_to_mp4drops the B-frame reorder-buffer tail at decode: the muxed file carries all 303 samples but only 301 decode, socamera_frame_statsundercountsdecoded_frame_counton every healthy episode converted from a B-frame stream.frame_deficit_pctcannot see it (no messages are missing).Evidence chain (svla_so101_pickplace @ f641879, ep0, plain ffmpeg):
write_access_units_to_mp4command: 303 samples, 301 decoded (last two frames; reproduced under -muxdelay 0, genpts, max_interleave_delta 0, and a two-step remux)bframes=0decodes 303/303 through the identical remuxValidation
write_access_units_to_mp4must decode every source frame (fails on the old encode, passes now).pytest tests/test_lerobot_converter.py-> 6/6 passed.hflow.App(...).process(...), then the catalog measurements parquet readsdecoded_frame_count303/303 on both/observation.images.upand/observation.images.side(was 301/303).ruff check+ruff format --check+ty checkon both changed files: clean.Fixes #250