Skip to content

isapi: support Hikvision AAC two-way talk - #2354

Open
JoeyGE0 wants to merge 4 commits into
AlexxIT:masterfrom
JoeyGE0:isapi-aac-talk
Open

isapi: support Hikvision AAC two-way talk#2354
JoeyGE0 wants to merge 4 commits into
AlexxIT:masterfrom
JoeyGE0:isapi-aac-talk

Conversation

@JoeyGE0

@JoeyGE0 JoeyGE0 commented Jul 14, 2026

Copy link
Copy Markdown

Summary

  • Extend Hikvision isapi:// two-way talk beyond G.711 µ/A-law so cameras with audioCompressionType=AAC work with Frigate / WebRTC talk.

  • On probed firmware (e.g. DS-2CD1383G2-LIUF/SL V5.8.41 and similar LiveGuard / G3 speaker cams), AAC talk works; G.711 is sometimes listed in capabilities but PUT .../open returns 403 — AAC is required on those units.

  • Wire format for AAC on /ISAPI/System/TwoWayAudio/channels/<id>/audioData is not documented in the public ISAPI PDF as ADTS. Working cameras emit/expect:

    [uint32 big-endian frame length][ADTS AAC frame] …
    
  • G.711 path is unchanged: when the camera reports G.711ulaw/alaw, payloads are still written raw (no ffmpeg), same as today.

Motivation

Related: #1911

Many modern Hikvision speaker cameras default (or only allow) AAC for two-way audio. Stock pkg/isapi only accepted G.711, so Frigate Live talk simply did not work on those cams even though the speaker works in Hik-Connect.

Separately, forcing or relying on G.711 is a worse experience where AAC is available: G.711 is narrowband / telephone quality, and on some firmware G.711 open is advertised but fails (403). This change exists so go2rtc can talk to cameras that are actually on AAC, with optional Opus→AAC for clearer WebRTC mic audio, while leaving the existing G.711 path alone for cams that still use it.

Validated with live Frigate + WebRTC mic and a local lab tool against real hardware.

Behaviour (AAC cameras)

  1. Dial — accept AAC; sample rate from audioSamplingRate (XML often uses kHz, e.g. 16 → 16000 Hz).
  2. Openclose → short settle → open → parse optional sessionIdPUT .../audioData?sessionId=....
  3. Advertise for backchannel match (order matters):
    • Opus (preferred for WebRTC mic quality)
    • AAC (if a producer already sends AAC)
    • PCMU / PCMA (fallback; same match as classic G.711 talk)
  4. Send:
    • Opus → ffmpeg (Ogg wrap via pion oggwriter) → AAC-LC ADTS → u32be length prefix → socket
    • PCMU/PCMA → ffmpeg → AAC-LC ADTS → length prefix
    • AAC track → ADTS (RTP demux / encode) → length prefix
  5. ffmpeg — prefer low latency (nobuffer / low_delay / flush_packets); also probe common Frigate paths (/usr/lib/ffmpeg/7.0/bin/ffmpeg, etc.) because Frigate’s go2rtc process often does not have bare ffmpeg on PATH.
  6. Path selection — use the matched producer codec, not track.Codec (WebRTC often reports ANY).

Compatibility notes (measured)

Item Observation
Sample rate Probed cam locks AAC talk to 16 kHz
Bitrate 16 / 32 / 64 kbps all transport OK; encode uses 64k
RTSP listen codec Independent of TwoWayAudio (Frigate ffmpeg:#audio=opus is for listen, not talk)
Volume XML On some cams loudspeaker volume node is named microphoneVolume (name swap); this PR does not set volume
Frigate No Frigate core changes required — streams need isapi:// + RTSP #backchannel=0

Test plan

  • AAC cam + Frigate WebRTC talk → speaker (several cams; Opus→AAC)
  • G.711 cam path still raw (code path unchanged; recommend maintainers re-check on a G.711-only unit)
  • Frigate without ffmpeg on PATH still finds /usr/lib/ffmpeg/.../ffmpeg
  • Matched codec used when track reports ANY
  • Maintainer: G.711ulaw/alaw regression on a non-AAC Hik unit
  • Maintainer: session close / no stuck talk after consumer stop

Out of scope

  • Native AAC encode without ffmpeg
  • Changing Frigate UI / AEC behaviour
  • Receiving camera mic over ISAPI GET (talk sendonly only, same as before)

Notes for reviewers

Frigate testing was done via a JoeyGE0 go2rtc release overlaid on Frigate Full Access Dev; the patch itself is only pkg/isapi (+ tiny internal/isapi logger wire-up). Happy to trim logs or path heuristics if you prefer stricter upstream style.

Josiah Clark and others added 4 commits July 15, 2026 11:15
Hikvision AAC /audioData uses [u32be len][ADTS]. Accept AAC channels,
pass sessionId on open, and transcode WebRTC PCMU/PCMA to AAC via ffmpeg.

Co-authored-by: Cursor <cursoragent@cursor.com>
Hardcoded `ffmpeg` often missing from go2rtc PATH in Frigate;
probe common /usr/lib/ffmpeg/*/bin/ffmpeg locations.

Co-authored-by: Cursor <cursoragent@cursor.com>
Match WebRTC Opus before PCMU/PCMA when the camera is AAC, wrap Opus in Ogg for ffmpeg, and use nobuffer/low_delay/flush. Keep G.711 passthrough unchanged and add [isapi] debug logs.

Co-authored-by: Cursor <cursoragent@cursor.com>
WebRTC often hands track.Codec=ANY; path selection must use the
negotiated Opus/PCMU/AAC codec so AAC cams don't fall through to raw G.711 write.

Co-authored-by: Cursor <cursoragent@cursor.com>
@JoeyGE0 JoeyGE0 changed the title isapi: support Hikvision AAC two-way talk (length-prefixed ADTS) isapi: support Hikvision AAC two-way talk Jul 15, 2026
@conorlap

Copy link
Copy Markdown

Thanks for this, working a treat for me :D

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants