Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/how-to/write-a-converter.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ Non-camera channels have no HFlow-specific payload contract. Their topic, schema

Camera handling is selected by exact schema name. A custom schema name that happens to contain images is treated as an ordinary pass-through channel, so it will not appear in `Episode.cameras` and camera checks will not inspect it. The accepted camera inputs are:

- `sensor_msgs/msg/CompressedImage` or `foxglove.CompressedImage`: the message encoding and schema must be decodable by the installed ROS 2 or protobuf MCAP decoder. Every message on one channel must use the same image format, and that format string must contain `jpeg`, `jpg`, or `png`. The payloads must contain valid images of that format. For a channel with at least two messages, the median difference between consecutive log times must be positive so HFlow can infer a frame rate. HFlow accepts a one-message channel using a 1 fps fallback and drops an empty camera channel with a warning. See [`_decode_compressed_images`](../../src/hflow/transform.py), [`_input_codec_for_image_format`](../../src/hflow/transform.py), and [`estimate_fps_from_log_times`](../../src/hflow/video.py).
- `sensor_msgs/msg/CompressedImage` or `foxglove.CompressedImage`: the message encoding and schema must be decodable by the installed ROS 2 or protobuf MCAP decoder. Every message on one channel must use the same image format, and that format string must contain `jpeg`, `jpg`, or `png`. The payloads must contain valid images of that format. For a channel with at least two messages, the median difference between consecutive log times must be positive so HFlow can infer a frame rate. HFlow accepts a one-message channel using a 1 fps fallback and preserves an empty declared camera channel as an empty canonical video channel. See [`_decode_compressed_images`](../../src/hflow/transform.py), [`_input_codec_for_image_format`](../../src/hflow/transform.py), and [`estimate_fps_from_log_times`](../../src/hflow/video.py).
- `foxglove.CompressedVideo` or `foxglove_msgs/msg/CompressedVideo`: the message encoding and schema must be decodable. Each message must declare `format="h264"` and contain one Annex B access unit. Every keyframe must include SPS and PPS, and the first message on a channel must be a keyframe. HFlow inserts a missing access-unit delimiter without re-encoding when the message can also be serialized by its protobuf or ROS 2 CDR encoder; it rejects other stream repairs. The current validator does not inspect whether an otherwise accepted stream uses B-frames, so a converter passing through H.264 must still disable them to satisfy the canonical output convention. See [`_resolve_decoder`, `_resolve_encoder`, and `_validate_passthrough_video_payload`](../../src/hflow/transform.py).

Raw `sensor_msgs/msg/Image` and `foxglove.RawImage` channels are not supported. Record compressed JPEG or PNG images instead. Any derived topic configured by the pipeline must also be absent from the source because HFlow refuses to create a second channel with that topic; this is normally a pipeline-author concern rather than converter logic. These refusals are in [`write_canonical_episode`](../../src/hflow/transform.py).
Expand Down
6 changes: 3 additions & 3 deletions src/hflow/transform.py
Original file line number Diff line number Diff line change
Expand Up @@ -756,7 +756,9 @@ def group_for(info: TopicInfo) -> str:
payloads = camera_payloads[channel_id]
topic = infos[channel_id].topic
if not payloads:
logger.warning("camera topic %r has no messages; dropping it", topic)
# Preserve the declared channel below even when there are no
# messages. Channel declarations are source information and
# participate in the canonical content identity (#528).
continue
images, frame_ids, input_codec = _decode_compressed_images(
topic, infos[channel_id], payloads
Expand Down Expand Up @@ -895,8 +897,6 @@ def output_topic(message: _OutgoingMessage) -> str:
for source_channel_id in sorted(infos, key=lambda cid: (infos[cid].topic, cid)):
info = infos[source_channel_id]
if source_channel_id in camera_payloads:
Comment on lines 897 to 899

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Empty camera breaks media step

When a source declares an empty camera, this change adds it to Episode.cameras. The built-in contact-sheet step then calls Episode.frames() for it. Episode.video() raises because there are no timestamps to estimate a frame rate, so the media step fails. Keep the declaration, but skip cameras with no frames when making contact sheets.

if not camera_payloads[source_channel_id]:
continue
if video_schema_id is None:
video_schema_id = writer.register_schema(
name=CANONICAL_VIDEO_SCHEMA_NAME,
Expand Down
57 changes: 57 additions & 0 deletions tests/test_transform.py
Original file line number Diff line number Diff line change
Expand Up @@ -133,6 +133,63 @@ def test_camera_channels_became_compressed_video(
assert joint_channel.message_encoding == "cdr"


def _write_state_source(path: Path, *, include_empty_camera: bool) -> Path:
from mcap.writer import Writer as StockWriter

with path.open("wb") as stream:
writer = StockWriter(stream)
writer.start(profile="", library="test")
if include_empty_camera:
camera_schema_id = writer.register_schema(
name="sensor_msgs/msg/CompressedImage",
encoding="ros2msg",
data=b"",
)
writer.register_channel(
topic="/camera/empty",
message_encoding="cdr",
schema_id=camera_schema_id,
)
state_schema_id = writer.register_schema(name="state", encoding="jsonschema", data=b"{}")
state_channel_id = writer.register_channel(
topic="/state",
message_encoding="json",
schema_id=state_schema_id,
)
writer.add_message(
state_channel_id,
log_time=1_000_000_000,
publish_time=1_000_000_000,
data=b'{"value": 1}',
)
writer.finish()
return path


def test_empty_camera_declaration_survives_transform_and_changes_identity(tmp_path: Path) -> None:
with_camera = _write_state_source(tmp_path / "with-camera.mcap", include_empty_camera=True)
without_camera = _write_state_source(tmp_path / "without-camera.mcap", include_empty_camera=False)
canonical_with = tmp_path / "with-camera.canonical.mcap"
canonical_without = tmp_path / "without-camera.canonical.mcap"

write_canonical_episode(with_camera, canonical_with)
write_canonical_episode(without_camera, canonical_without)

with canonical_with.open("rb") as stream:
summary = make_reader(stream).get_summary()
assert summary is not None
by_topic = {
channel.topic: (channel, summary.schemas[channel.schema_id])
for channel in summary.channels.values()
}

camera_channel, camera_schema = by_topic["/camera/empty"]
assert camera_schema.name == CANONICAL_VIDEO_SCHEMA_NAME
assert camera_schema.encoding == "protobuf"
assert camera_channel.message_encoding == "protobuf"
assert canonical_with.read_bytes() != canonical_without.read_bytes()


def test_state_messages_pass_through_byte_for_byte(
source_episode: Path, canonical_episode: Path
) -> None:
Expand Down
Loading