isapi: support Hikvision AAC two-way talk - #2354
Open
JoeyGE0 wants to merge 4 commits into
Open
Conversation
Hikvision AAC /audioData uses [u32be len][ADTS]. Accept AAC channels, pass sessionId on open, and transcode WebRTC PCMU/PCMA to AAC via ffmpeg. Co-authored-by: Cursor <cursoragent@cursor.com>
Hardcoded `ffmpeg` often missing from go2rtc PATH in Frigate; probe common /usr/lib/ffmpeg/*/bin/ffmpeg locations. Co-authored-by: Cursor <cursoragent@cursor.com>
Match WebRTC Opus before PCMU/PCMA when the camera is AAC, wrap Opus in Ogg for ffmpeg, and use nobuffer/low_delay/flush. Keep G.711 passthrough unchanged and add [isapi] debug logs. Co-authored-by: Cursor <cursoragent@cursor.com>
WebRTC often hands track.Codec=ANY; path selection must use the negotiated Opus/PCMU/AAC codec so AAC cams don't fall through to raw G.711 write. Co-authored-by: Cursor <cursoragent@cursor.com>
|
Thanks for this, working a treat for me :D |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Extend Hikvision
isapi://two-way talk beyond G.711 µ/A-law so cameras withaudioCompressionType=AACwork with Frigate / WebRTC talk.On probed firmware (e.g. DS-2CD1383G2-LIUF/SL V5.8.41 and similar LiveGuard / G3 speaker cams), AAC talk works; G.711 is sometimes listed in capabilities but
PUT .../openreturns 403 — AAC is required on those units.Wire format for AAC on
/ISAPI/System/TwoWayAudio/channels/<id>/audioDatais not documented in the public ISAPI PDF as ADTS. Working cameras emit/expect:G.711 path is unchanged: when the camera reports G.711ulaw/alaw, payloads are still written raw (no ffmpeg), same as today.
Motivation
Related: #1911
Many modern Hikvision speaker cameras default (or only allow) AAC for two-way audio. Stock
pkg/isapionly accepted G.711, so Frigate Live talk simply did not work on those cams even though the speaker works in Hik-Connect.Separately, forcing or relying on G.711 is a worse experience where AAC is available: G.711 is narrowband / telephone quality, and on some firmware G.711
openis advertised but fails (403). This change exists so go2rtc can talk to cameras that are actually on AAC, with optional Opus→AAC for clearer WebRTC mic audio, while leaving the existing G.711 path alone for cams that still use it.Validated with live Frigate + WebRTC mic and a local lab tool against real hardware.
Behaviour (AAC cameras)
AAC; sample rate fromaudioSamplingRate(XML often uses kHz, e.g.16→ 16000 Hz).close→ short settle →open→ parse optionalsessionId→PUT .../audioData?sessionId=....ffmpeg(Ogg wrap via pion oggwriter) → AAC-LC ADTS →u32belength prefix → socketffmpeg→ AAC-LC ADTS → length prefixnobuffer/low_delay/flush_packets); also probe common Frigate paths (/usr/lib/ffmpeg/7.0/bin/ffmpeg, etc.) because Frigate’s go2rtc process often does not have bareffmpegonPATH.track.Codec(WebRTC often reportsANY).Compatibility notes (measured)
ffmpeg:#audio=opusis for listen, not talk)microphoneVolume(name swap); this PR does not set volumeisapi://+ RTSP#backchannel=0Test plan
ffmpegon PATH still finds/usr/lib/ffmpeg/.../ffmpegANYOut of scope
Notes for reviewers
Frigate testing was done via a JoeyGE0 go2rtc release overlaid on Frigate Full Access Dev; the patch itself is only
pkg/isapi(+ tinyinternal/isapilogger wire-up). Happy to trim logs or path heuristics if you prefer stricter upstream style.