Add UniFi Protect Opus talkback source by mpapierski · Pull Request #2340 · AlexxIT/go2rtc

mpapierski · 2026-07-09T09:50:43Z

This PR adds native UniFi Protect talkback support to go2rtc via a new unifi-talkback: source.

The important behavior is that talkback stays passive until a client actually sends microphone audio. Viewing a stream with normal video+audio does not open a UniFi Protect talkback session or activate the camera speaker. When microphone RTP starts flowing through go2rtc’s existing backchannel path, go2rtc creates a Protect talkback session, starts FFmpeg, transcodes the mic audio to the format Protect requested, and streams it to the returned session URL. Only opus codec with 22khz is known to be advertised for the talkback channel (tested with G4 Doorbell Pro and Doorbell Lite; if anyone has different UniFi doorbells, please let me know your experience).

To set it up you need API Key for your local unifi account, and camera IDs.

export PROTECT='https://192.168.1.1'
export API_KEY="<YOUR UNIFI API KEY>"
curl -sk \
  -H "X-API-KEY: $API_KEY" \
  "$PROTECT/proxy/protect/integration/v1/cameras" \
  > /tmp/unifi-cameras.json
jq -r '.[] | [.id, .name] | @tsv' /tmp/unifi-cameras.json

You need to copy the id from the first column to use given camera.

Here's my config I'm using for two-way audio for both my G4 Doorbell Pro and Doorbell Lite.

streams:
  gate_medium:
    - 'rtsps://192.168.1.1:7441/<rtsp url from unifi protect>?enableSrtp#backchannel=0'
    - 'unifi-talkback:https://192.168.1.1?camera_id=<gate_id>&api_key=<api_key>'
  gate_kiosk:
    - 'ffmpeg:rtsps://192.168.1.1:7441/<rtsp url from unifi protect>?enableSrtp#video=h264#audio=opus'
    - 'unifi-talkback:https://192.168.1.1?camera_id=<gate_id>&api_key=<api_key>'
  door_medium:
    - 'rtsps://192.168.1.1:7441/<rtsp url from unifi protect>?enableSrtp#backchannel=0'
    - 'unifi-talkback:https://192.168.1.1?camera_id=<door_id>&api_key=<api_key>'
  door_kiosk:
    - 'ffmpeg:rtsps://192.168.1.1:7441/<rtsp url from unifi protect>?enableSrtp#video=h264#audio=opus'
    - 'unifi-talkback:https://192.168.1.1?camera_id=<door_id>&api_key=<api_key>'

I'm using the passthrough gate_medium stream to get the lower-resolution source stream directly, without any re-encoding. In my setup, Protect uses HEVC, which isn't suitable for direct WebRTC playback on the kiosk device.

To solve this, I added a separate door_kiosk stream. It ingests the same upstream source but transcodes it into a format that's compatible with the Android tablet over WebRTC (H.264 video + Opus audio).

NOTE: You may not need "medium" stream if your "kiosk" device is powerful enough. I just found out that my Doorbell Lite has 2K resolution and Galaxy Tab A11+ needs several seconds to negotiate live view. I found a medium to be the fastest stream in my case.

Also, the assumption is that the UDMP can comfortably serve two clients from the lower-resolution stream. On the go2rtc VM, the only transcoding performed is:

Video: HEVC → H.264 for the kiosk stream.
Audio: Opus 48 kHz → Opus 22 kHz for talkback, matching what UniFi advertises.

On the home assistant side I'm using WebRTC Camera (aka go2rtc) addon with his card config:

type: custom:webrtc-camera
mode: webrtc,mse,hls,mjpeg
ui: true
streams:
  - url: rtsp://192.168.50.206:8554/door_kiosk
    name: Live
    mode: webrtc
    media: video
  - url: rtsp://192.168.50.206:8554/door_kiosk
    name: Talk
    mode: webrtc
    media: video,audio,microphone

Now, when I click on the card it will switch from Live to Talk and back to Live. In "Talk" mode my voice from mic of my android "kiosk" tablet is heard on the doorbell's speaker. So two-way audio works really well now.

Disclaimer: this is my first contribution, and the code is AI-assisted, although it works well in my home setup.

I’d really appreciate any feedback, help, or guidance needed to make this mergeable. I know many people are looking forward to having two-way audio from Home Assistant on their UniFi doorbells, so I’m happy to adjust the implementation as needed.

Closes #1760 and #279 (and other unifi doorbell two-way audio related issues)

Add a native unifi-talkback source that exposes UniFi Protect speaker support through go2rtc's existing audio backchannel model. The source performs a passive Protect camera metadata lookup and advertises an audio sendonly media only when featureFlags.hasSpeaker is true. Talkback activation is deferred until microphone RTP actually arrives. This keeps normal video/audio viewing and passive probing from opening a Protect talkback session or activating the camera speaker. On first RTP, the source creates a public Protect talkback session, validates that the returned codec is opus, and starts an FFmpeg RTP bridge to the session URL. The browser/go2rtc input side accepts common backchannel codecs OPUS/48000/2, PCMU/8000, and PCMA/8000. The Protect output side remains Opus-only and is transcoded with FFmpeg to the sampling rate and mono layout returned by the Protect API. The FFmpeg bridge uses a local RTP input described by SDP on stdin, then outputs libopus RTP to the Protect session URL. The local UDP writer is unconnected so early RTP packets sent before FFmpeg has bound its input port do not fail the whole talkback session. FFmpeg stderr is routed into the UniFi logger for easier real-device debugging. Add lifecycle logging around Protect API requests and responses, talkback session creation, FFmpeg start/stop, and RTP write failures. Trace logs include Protect response bodies, with session URLs and API keys registered through the existing credentials redaction path. Also register configured unifi-talkback API keys during module init so /api/streams output is redacted before the source is first dialed. Make credential replacement prefer longer overlapping secrets so a registered session URL cannot be partially redacted by a shorter URL prefix and leak a query token suffix. Document the new source, add module/schema/navigation entries, and add focused tests for URL parsing, secret redaction, passive speaker advertisement, no session creation during probe, first-RTP session startup, FFmpeg command/SDP generation, Opus-only Protect codec handling, G.711 input SDP generation, and cleanup behavior. Refs AlexxIT#1760

t3hk0d3 · 2026-07-09T11:08:06Z

+		"-i", "pipe:0",
+		"-map", "0:a:0",
+		"-vn",
+		"-c:a", "libopus",


i would make it param that falls back to libopus.

Technically Unifi Protect API provides what audio codec should be used.

Addressed. Session's codec is passed here, and opus is mapped to libopus. Keep in mind I haven't found any evidence through docs or other means that there may be other audio codecs advertised

AFAIK Unifi Cameras also support AAC and Vorbis.

https://github.com/uilibs/uiprotect/blob/main/src/uiprotect/stream.py#L70

AAC is used thru ADTS endpoint, while OPUS is used thru RTP

I added vorbis, and aac to the mapping with the appropriate container format in 3074051

Thread the codec returned by the Protect talkback-session API into FFmpeg command generation instead of hardcoding libopus at the call site. The mapping currently accepts only opus and translates it to libopus, preserving the Opus-only behavior while making the codec choice explicit. Reject unsupported codecs from the same mapping path with a clear "unifi: unsupported talkback codec: <codec>" error so talkback fails before starting FFmpeg or opening local RTP resources.

t3hk0d3

LGTM!

mpapierski · 2026-07-28T08:38:30Z

Any update on the possibility of merging this? I've been running it for a couple of weeks now at home and haven't found any issues. @AlexxIT

mpapierski mentioned this pull request Jul 9, 2026

Unifi 2-way audio #279

Open

t3hk0d3 reviewed Jul 9, 2026

View reviewed changes

mpapierski added 2 commits July 9, 2026 15:24

Support UniFi talkback AAC and Vorbis codecs

3074051

t3hk0d3 approved these changes Jul 9, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add UniFi Protect Opus talkback source - #2340

Add UniFi Protect Opus talkback source#2340
mpapierski wants to merge 3 commits into
AlexxIT:masterfrom
mpapierski:unifi-talkback

mpapierski commented Jul 9, 2026 •

edited

Loading

Uh oh!

t3hk0d3 Jul 9, 2026

Uh oh!

mpapierski Jul 9, 2026

Uh oh!

t3hk0d3 Jul 9, 2026 •

edited

Loading

Uh oh!

mpapierski Jul 9, 2026

Uh oh!

t3hk0d3 left a comment

Uh oh!

mpapierski commented Jul 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

mpapierski commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

t3hk0d3 Jul 9, 2026

Choose a reason for hiding this comment

Uh oh!

mpapierski Jul 9, 2026

Choose a reason for hiding this comment

Uh oh!

t3hk0d3 Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mpapierski Jul 9, 2026

Choose a reason for hiding this comment

Uh oh!

t3hk0d3 left a comment

Choose a reason for hiding this comment

Uh oh!

mpapierski commented Jul 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

mpapierski commented Jul 9, 2026 •

edited

Loading

t3hk0d3 Jul 9, 2026 •

edited

Loading