Support for alpha/transparency per apple interoperability proposal

[Joe-Palmer]

In May 2019 Apple released and implemented a proposal for alpha channel support (so called transparent videos/movies) encoded in HEVC. Currently, it seems only Apple tools (avconvert and Compressor) support this format. ffmpeg with the current libx265 fails while encoding as it does not support a pixel format with alpha support (ffmpeg relevant ticket https://trac.ffmpeg.org/ticket/7965 ).

Is it planned or has it been evaluated providing support for videos with alpha? Or is it something that can already be accomplished with command line flags?

https://bitbucket.org/multicoreware/x265_git/issues/attachments/577/multicoreware/x265_git/1604508977.83/577/HEVC-Video-with-Alpha-Interoperability-Profile.pdf

[farindk]

Thanks for the link to the spec document and sorry for ignoring it for so long.
I am open to adding alpha support to libde265, but the main question for me is: would there be anyone using it? I mean the software, not alpha video as such.
libde265 is currently mainly used in libheif for still images and there, alpha is encoded in a different way and already supported.
ffmpeg has its own decoder. So who would switch to libde265 just because of alpha support?

Also note that large parts of the spec document are about the mp4 container. libde265 is a lower level library that does not read mp4. So we would need a software reading mp4 and using libde265 for h265 decoding.

[Joe-Palmer]

No problem, glad you picked it up now! We actually use Gstreamer at the moment and I see there is a plugin available so that is how we would switch to using libde265. We receive live streams from iOS devices and have developed the ability to encode depth data (from the TrueDepth or LiDAR cameras) into the alpha channel but currently can't decode this data on the servers.

Would it be helpful to provide a sample file with this depth data encoded into the alpha channel?

[farindk]

A sample file (apart from the puppet video I got from the ffmpeg ticket) would be nice of course. If it is too large, let me know and I'll tell you a place where you can upload it.

I had a look into Annex-F of the standard, which describes the decoding of the auxiliary streams. This is no small change, Annex F is 110 pages. Since I have to do this in my "free time", I'm not sure when that would be possible.

BTW: Annex F names two types of auxiliary streams: alpha and depth. If you're actually storing depth data, it might be better to put it into the depth channel (if Apple supports that). There might also be metadata associated how to interpret the depth data (HEIF has such a header, see my libheif, which supports it).

[Joe-Palmer]

OK, interesting. I'm not sure if Apple supports the depth channel (or I don't think it did when we did the original work), we will investigate further. I actually have a budget to get this work done (and a quote to do the work) so if you are open to being sponsored to develop this as open source, I would be very open to this. Is this option worth exploring?

[Apidcloud]

We would also use libde265 with ffmpeg if it supported the alpha channel. To my (very limited) knowledge there's no decoder working with ffmpeg that supports it

[farindk]

A first working version of the decoder with alpha and depth layer support can be found in the alpha-depth branch. The API is not stable yet.

You can check which auxiliary layers the stream contains with dec265 --info and then decode that layer with dec265 --layer #. There is also a new decoder API (`de265_audecoder') for this. This is work in progress, comments welcome.