Skip to content

fix(lerobot): disable B-frames so the converter output decodes in full - #251

Merged
kstonekuan merged 2 commits into
Hebbian-Robotics:mainfrom
ayam04:fix/lerobot-decode-tail-loss
Aug 29, 2026
Merged

kstonekuan merged 2 commits into
Hebbian-Robotics:mainfrom
ayam04:fix/lerobot-decode-tail-loss

Conversation

@ayam04

@ayam04 ayam04 commented Aug 29, 2026

Copy link
Copy Markdown
Contributor

What

One-line converter change: _transcode_mp4_to_h264 now encodes with bframes=0 (was libx264 medium default, which enables B-frames).

Why

Fixes #250. The raw Annex B -> MP4 -c copy remux in write_access_units_to_mp4 drops the B-frame reorder-buffer tail at decode: the muxed file carries all 303 samples but only 301 decode, so camera_frame_stats undercounts decoded_frame_count on every healthy episode converted from a B-frame stream. frame_deficit_pct cannot see it (no messages are missing).

Evidence chain (svla_so101_pickplace @ f641879, ep0, plain ffmpeg):

  • source slices decode 303/303, full chunks 11939/11939, zero errors
  • converter emits 303 access units; raw Annex B decodes 303/303
  • remuxed via the exact write_access_units_to_mp4 command: 303 samples, 301 decoded (last two frames; reproduced under -muxdelay 0, genpts, max_interleave_delta 0, and a two-step remux)
  • control: same encode with bframes=0 decodes 303/303 through the identical remux

Validation

  • Regression test added: converter output remuxed through write_access_units_to_mp4 must decode every source frame (fails on the old encode, passes now). pytest tests/test_lerobot_converter.py -> 6/6 passed.
  • Full pipeline on the pinned corpus with the fixed converter: hflow.App(...).process(...), then the catalog measurements parquet reads decoded_frame_count 303/303 on both /observation.images.up and /observation.images.side (was 301/303).
  • ruff check + ruff format --check + ty check on both changed files: clean.

Fixes #250

ayam04 and others added 2 commits August 29, 2026 05:55
Hebbian-Robotics#250)

The converter transcodes each episode window with libx264's default medium
preset, which enables B-frames. hflow's write_access_units_to_mp4 remuxes
the resulting raw Annex B stream to MP4 with -c copy, and that remux drops
the reorder-buffer tail at decode: the muxed file carries 303 samples but
only 301 decode, so camera_frame_stats undercounts decoded_frame_count on
every healthy episode (both svla cameras measured 301/303 while the source
slices decode 303/303 with zero errors).

Encode with bframes=0: the identical remux then decodes 303/303. Verified
end to end on svla_so101_pickplace @ f641879 ep0 - App.process now records
decoded_frame_count 303/303 on both cameras (was 301/303).

Regression test: converter output remuxed via write_access_units_to_mp4
must decode every source frame (fails on the old encode, passes now).
ruff/format/ty clean; full converter test file 6/6.

Signed-off-by: Ayamullah Khan <ayamullahkhan04@gmail.com>
pytestmark applies to the whole module, so on a machine without system
ffmpeg the five metadata tests skipped too, and none of those touch ffmpeg.
Without ffmpeg on PATH it is now 5 passed, 1 skipped instead of 6 skipped.

@kstonekuan kstonekuan left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Merging. The diagnosis on #250 is the best piece of debugging this repo has had, and the fix is one line because you did the work to earn it.

Validated on Linux: ruff check, ruff format --check, ty check, 1187 passed, 6 skipped.

I reverted bframes=0 back to aud=1 and the new test failed, so it catches the thing it was written for rather than merely exercising the path.

The layer-by-layer attribution is what makes this reviewable. Source decodes 303/303, converter output decodes 303/303 as raw Annex B, the remux container holds 303 samples and the decoder emits 301, and re-encoding with bframes=0 restores 303 through the identical remux. Four measurements, each isolating one layer, and the conclusion follows from them rather than from a plausible story. Trying -muxdelay 0, genpts, max_interleave_delta 0, and a two-step remux before concluding no flag recovers it is the part most people skip.

One thing worth adding to the record: the converter was emitting non-conforming output. docs/FORMAT.md:152 item 4 requires no B-frames, so a canonical stream should never have had them. Nothing caught it because the doctor does not classify picture types, which #236 documented three hours before you hit this. So the chain is: a documented unchecked constraint, a converter quietly violating it, and a silent two-frame loss downstream. That is the first real consequence of that gap, and it is a good argument for closing it.

One fixup pushed (09adc9d): the pytestmark skip was module-level, so on a machine without system ffmpeg all six tests in the file skipped, including the five metadata tests that never touch ffmpeg. Scoped it to the one test that shells out. Without ffmpeg on PATH it is now 5 passed, 1 skipped instead of 6 skipped.

Your two longer-term notes on #250 are both right and both want separate issues. I would take the measurement one first: decoded_frame_count being below message_count on a healthy episode is exactly the blind spot, and a companion to frame_deficit_pct would have surfaced this without anyone running ffmpeg by hand. #250 stays open and assigned to you for whichever you want to carry.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

camera_frame_stats undercounts by two on B-frame streams: the raw-to-MP4 remux drops the reorder-buffer tail

2 participants