Skip to content

nest: request keyframes + advertise sprop-parameter-sets (fix slow/failed consumer opens) - #2368

Open
ajplotkin wants to merge 2 commits into
AlexxIT:masterfrom
ajplotkin:nest-keyframe-sprop
Open

nest: request keyframes + advertise sprop-parameter-sets (fix slow/failed consumer opens)#2368
ajplotkin wants to merge 2 commits into
AlexxIT:masterfrom
ajplotkin:nest-keyframe-sprop

Conversation

@ajplotkin

Copy link
Copy Markdown

Two small, self-contained improvements to the Nest (nest:) WebRTC source, both gated on FormatName == "nest/webrtc" so nothing else is affected. Together they make Nest cameras usable as a warm RTSP/HLS/consumer source — right now a consumer that joins a Nest stream can wait a long time to start, or fail to determine the video size at all.

1. Request periodic keyframes for the Nest source

OnTrack in pkg/webrtc/conn.go already sends an RTCP PictureLossIndication every 2s to keep the source emitting keyframes — but only for ModePassiveProducer (WHIP/browser push). Nest is an active-pull source (ModeActiveProducer), so it was excluded, and an idle Nest camera lets its keyframe interval drift long. Any consumer that joins mid-GOP (RTSP live view, an on-demand frame.jpeg, MSE) then waits up to a full keyframe interval to start.

This enables the existing keyframe-request loop for the Nest source, gated on FormatName (not the mode) so other ModeActiveProducer WebRTC sources — e.g. battery cameras where a forced 2s IDR would be power/bandwidth hostile — are untouched. PLI is media-plane RTCP, so there's no impact on the Nest/SDM API command quota.

Measured on a Nest doorbell: keyframe interval ~3.3s → ~1.8s; on-demand frame.jpeg grabs drop from several seconds to ~1s.

Also added defer t.Stop() on the ticker (the existing pattern relies on GC of an unreferenced ticker — fine on Go ≥1.23, but explicit is safer).

2. Advertise sprop-parameter-sets in the RTSP SDP

The Nest source's H264 codec FmtpLine comes from Google's WebRTC SDP answer, which carries profile-level-id but no sprop-parameter-sets (WebRTC sends SPS/PPS in-band). The RTSP server copies FmtpLine verbatim into DESCRIBE, so an RTSP consumer gets no SPS/PPS in the SDP and can't determine the video dimensions until an in-band keyframe arrives — which forces a large -probesize for ffmpeg, or fails outright at a small one.

This captures SPS (NAL 7) / PPS (NAL 8) from the incoming H264 RTP in the OnTrack receive loop — handling STAP-A, which is how libwebrtc typically bundles SPS+PPS+IDR — and appends sprop-parameter-sets to the codec FmtpLine once. Every consumer that clones the codec (RTSP) or reads GetParameterSet (MSE/MP4) then benefits. This mirrors what pkg/dvrip already does for H265.

Verified in a live DESCRIBE after the change:

a=fmtp:97 ...;profile-level-id=64001f;sprop-parameter-sets=Z00AIJ...,aO48gA==

Notes / honesty

  • Both are gated on FormatName == "nest/webrtc"; no behavior change for any other source.
  • Codec.Match ignores FmtpLine, so appending sprop can't affect media matching/renegotiation.
  • The FmtpLine append happens on the receive goroutine and is written once; with the keyframe request above, SPS/PPS are captured before any consumer's DESCRIBE. A -race build could flag a consumer connecting during the sub-2s warm-up window; the existing H265 append in pkg/dvrip is unguarded in the same way. Happy to wrap it in a small mutex or capture-into-locals-then-assign if preferred.
  • These don't make stream-copy opens instant: ffmpeg still aligns copy output to the next keyframe, so the ~2s keyframe interval (now bounded by patch 1) remains the floor. The win is that the interval is bounded and the SDP is complete, so consumers start predictably instead of hanging.

Context: I'm using this to serve Nest cameras to Apple HomeKit via Homebridge over local RTSP, so a single warm go2rtc stream is shared instead of each viewer opening its own concurrent Google WebRTC stream (Nest enforces a concurrent-stream limit). Both patches were needed to make that path reliable.

ajplotkin and others added 2 commits July 18, 2026 10:59
pkg/webrtc/conn.go sends an RTCP PictureLossIndication every 2s to keep the
source emitting keyframes, but only for ModePassiveProducer (WHIP/browser push).
Nest is an active-pull WebRTC source (ModeActiveProducer), so it was excluded --
its keyframe interval drifts long when idle, and any consumer that joins mid-GOP
(RTSP live view, snapshot grab) waits up to a full keyframe interval to start.

Enable the keyframe request for the Nest source, gated on FormatName
"nest/webrtc" rather than the mode so other ModeActiveProducer WebRTC sources
(ring/tuya battery cams etc.) aren't forced into 2s IDRs, which would be battery-
and bandwidth-hostile.

PLI is media-plane RTCP -- no SDM API quota impact. Also add defer ticker.Stop()
(the upstream pattern relies on GC of an unreferenced ticker, fine on go>=1.23
but explicit is safer).

Measured on a Nest doorbell: keyframe interval ~3.3s -> ~1.8s; on-demand
frame.jpeg grabs drop to ~1s.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…IBE)

The Nest WebRTC source's H264 codec FmtpLine comes from Google's SDP answer,
which carries profile-level-id but no sprop-parameter-sets (WebRTC sends SPS/PPS
in-band). go2rtc's RTSP server copies FmtpLine verbatim into the DESCRIBE SDP, so
an RTSP consumer (e.g. ffmpeg for HomeKit live view) gets no SPS/PPS in the SDP
and can't learn video dimensions until an in-band keyframe arrives -- forcing a
large -probesize and slow starts (or failing outright at low probesize).

Capture SPS (NAL 7) / PPS (NAL 8) from the incoming H264 RTP in the OnTrack
receive loop (handling STAP-A bundling, which is how libwebrtc packs SPS+PPS+IDR)
and append sprop-parameter-sets to the codec FmtpLine once. Every consumer that
clones the codec (RTSP) or reads GetParameterSet (MSE/MP4) then benefits.

Gated on FormatName "nest/webrtc" like the keyframe-request patch. Upstream does
the equivalent for H265 in pkg/dvrip. With preload + the 2s keyframe request,
SPS/PPS are captured well before any DESCRIBE.

This lets RTSP consumers use a small -probesize safely (dimensions come from the
SDP). Note it does not by itself make stream-copy opens instant: ffmpeg still
aligns copy output to the next keyframe, so the ~2s keyframe interval (bounded by
the PLI patch) remains the floor.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@ajplotkin
ajplotkin force-pushed the nest-keyframe-sprop branch from e428c99 to 133c89b Compare July 18, 2026 09:13
ajplotkin added a commit to ajplotkin/go2rtc that referenced this pull request Jul 18, 2026
… math, udp4/sync caveats, credit adopted PRs

- conn.go: inline note that the sprop FmtpLine append is intentionally lock-free
  (captured before any DESCRIBE in practice; mirrors upstream H265; Codec.Match
  ignores FmtpLine). Comment-only, no behavior change.
- NEST-HOMEKIT-SNAPSHOTS.md: M7 (plugin-version fingerprint 1.1.23 + explicit
  install order plugin->PR#212->patches), L1 (udp4 drops TCP ICE tradeoff),
  L3 (per-camera vs per-project quota scoping for many cameras), L9 (sync only
  on camera-set change + a go2rtc restart drops warm streams), execSync->fs
  refresh matching the deployed code, and credit for adopted PRs AlexxIT#2351/AlexxIT#2327/
  AlexxIT#2193/AlexxIT#2194/AlexxIT#2368.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant