Segfault in `libde265` in `gst-discoverer-1.0`

[pjungkamp]

Problem

Running gst-discoverer-1.0 Dune_trunc.h265 causes a segfault in libde265.so.
Dune_trunc.h265 contains the first 10 seconds of the 4K h265 bitstream for the movie Dune.
Running dec265 didn't have any problems with this file.

Dune_trunc.zip

Debugging

I decided to post an issue on libde265 because I think that the issue boils down to a double free in libde265.
If gstreamer itself is the more likely culprit I'll repost this issue there.

gdb backtrace

Thread 4 "h265parse0:sink" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff5a8e6c0 (LWP 1370002)]
0x00007ffff5230aa4 in image_unit::~image_unit (this=0x7fffec779580, __in_chrg=<optimized out>) at decctx.cc:194
194         delete tasks[i];                                                                                                                                                        
(gdb) bt
#0  0x00007ffff5230aa4 in image_unit::~image_unit (this=0x7fffec779580, __in_chrg=<optimized out>) at decctx.cc:194
#1  0x00007ffff5233c3e in decoder_context::decode_some (this=this@entry=0x7fffec034aa0, did_work=did_work@entry=0x7ffff5a8d460) at decctx.cc:796
#2  0x00007ffff5235ab0 in decoder_context::read_slice_NAL (this=this@entry=0x7fffec034aa0, reader=..., nal=nal@entry=0x7fffec016670, nal_hdr=...) at decctx.cc:698
#3  0x00007ffff5235c91 in decoder_context::decode_NAL (this=this@entry=0x7fffec034aa0, nal=0x7fffec016670) at decctx.cc:1240
#4  0x00007ffff5235f8c in decoder_context::decode (this=0x7fffec034aa0, more=0x7ffff5a8d56c) at decctx.cc:1328
#5  0x00007ffff5c2bf43 in gst_libde265_dec_handle_frame () from /nix/store/m986c3np2pll6s48kvbw59bi9rmb80qm-gst-plugins-bad-1.22.5/lib/gstreamer-1.0/libgstde265.so
#6  0x00007ffff7989ce2 in gst_video_decoder_decode_frame () from /nix/store/4k1pg5wr83dk829qzfispkskdwr2x038-gst-plugins-base-1.22.5/lib/libgstvideo-1.0.so.0
#7  0x00007ffff798a16a in gst_video_decoder_chain_forward () from /nix/store/4k1pg5wr83dk829qzfispkskdwr2x038-gst-plugins-base-1.22.5/lib/libgstvideo-1.0.so.0
#8  0x00007ffff798cd5a in gst_video_decoder_chain () from /nix/store/4k1pg5wr83dk829qzfispkskdwr2x038-gst-plugins-base-1.22.5/lib/libgstvideo-1.0.so.0
#9  0x00007ffff7e33ee9 in gst_pad_chain_data_unchecked () from /nix/store/z964cncjvwsiic8bphinlkb5z99rr25f-gstreamer-1.22.5/lib/libgstreamer-1.0.so.0
#10 0x00007ffff7e363f1 in gst_pad_push_data () from /nix/store/z964cncjvwsiic8bphinlkb5z99rr25f-gstreamer-1.22.5/lib/libgstreamer-1.0.so.0
#11 0x00007ffff7e3d76b in gst_pad_push () from /nix/store/z964cncjvwsiic8bphinlkb5z99rr25f-gstreamer-1.22.5/lib/libgstreamer-1.0.so.0
#12 0x00007ffff78c7d40 in gst_base_transform_chain () from /nix/store/z964cncjvwsiic8bphinlkb5z99rr25f-gstreamer-1.22.5/lib/libgstbase-1.0.so.0
#13 0x00007ffff7e33ee9 in gst_pad_chain_data_unchecked () from /nix/store/z964cncjvwsiic8bphinlkb5z99rr25f-gstreamer-1.22.5/lib/libgstreamer-1.0.so.0
#14 0x00007ffff7e363f1 in gst_pad_push_data () from /nix/store/z964cncjvwsiic8bphinlkb5z99rr25f-gstreamer-1.22.5/lib/libgstreamer-1.0.so.0
#15 0x00007ffff7e3d76b in gst_pad_push () from /nix/store/z964cncjvwsiic8bphinlkb5z99rr25f-gstreamer-1.22.5/lib/libgstreamer-1.0.so.0
#16 0x00007ffff78a85f8 in gst_base_parse_push_frame () from /nix/store/z964cncjvwsiic8bphinlkb5z99rr25f-gstreamer-1.22.5/lib/libgstbase-1.0.so.0
#17 0x00007ffff5bcbe6c in gst_h265_parse_handle_frame () from /nix/store/m986c3np2pll6s48kvbw59bi9rmb80qm-gst-plugins-bad-1.22.5/lib/gstreamer-1.0/libgstvideoparsersbad.so
#18 0x00007ffff78a33b9 in gst_base_parse_handle_buffer () from /nix/store/z964cncjvwsiic8bphinlkb5z99rr25f-gstreamer-1.22.5/lib/libgstbase-1.0.so.0
#19 0x00007ffff78a3bb7 in gst_base_parse_scan_frame () from /nix/store/z964cncjvwsiic8bphinlkb5z99rr25f-gstreamer-1.22.5/lib/libgstbase-1.0.so.0
#20 0x00007ffff78a704e in gst_base_parse_loop () from /nix/store/z964cncjvwsiic8bphinlkb5z99rr25f-gstreamer-1.22.5/lib/libgstbase-1.0.so.0
#21 0x00007ffff7e6d911 in gst_task_func () from /nix/store/z964cncjvwsiic8bphinlkb5z99rr25f-gstreamer-1.22.5/lib/libgstreamer-1.0.so.0
#22 0x00007ffff7c8911a in g_thread_pool_thread_proxy (data=<optimized out>) at ../glib/gthreadpool.c:350
#23 0x00007ffff7c887cd in g_thread_proxy (data=0x7ffff0000d40) at ../glib/gthread.c:831
#24 0x00007ffff7a9bdd4 in start_thread (arg=<optimized out>) at pthread_create.c:444
#25 0x00007ffff7b1d9b0 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Here is the actual frame where a delete seems to cause the segfault.

(gdb) frame
#0  0x00007ffff5230aa4 in image_unit::~image_unit (this=0x7fffec779580, __in_chrg=<optimized out>) at decctx.cc:194
194	    delete tasks[i];
(gdb) print this->tasks[i]
$5 = (thread_task *) 0x7fffef3a4520

My first guess would be a double free in the image_unit data structure, though I haven't dug into libde265 code yet.

System

$ gst-discoverer-1.0 --gst-version
GStreamer Core Library version 1.22.5

$ dec265 --help
 dec265  v1.0.12

$ uname -a
Linux yoga9 6.5.6 #1-NixOS SMP PREEMPT_DYNAMIC Fri Oct  6 11:16:31 UTC 2023 x86_64 GNU/Linux

$ cat /etc/os-release
BUG_REPORT_URL="https://github.com/NixOS/nixpkgs/issues"
BUILD_ID="23.11.20231009.f99e5f0"
DOCUMENTATION_URL="https://nixos.org/learn.html"
HOME_URL="https://nixos.org/"
ID=nixos
LOGO="nix-snowflake"
NAME=NixOS
PRETTY_NAME="NixOS 23.11 (Tapir)"
SUPPORT_URL="https://nixos.org/community.html"
VERSION="23.11 (Tapir)"
VERSION_CODENAME=tapir
VERSION_ID="23.11"

[farindk]

Thanks. Since dec265 apparently works, my uninformed guess is that this is an issue in the libde265dec gstreamer plugin. I'll have a closer look later.

[farindk]

I tried with a slightly different gstreamer version, but could not reproduce it.

$ gst-discoverer-1.0 --gst-version
GStreamer Core Library version 1.20.3

your command line gives:

$ gst-discoverer-1.0 ../../issues/issue-425/Dune_trunc.h265 
Analyzing file:///home/.../Dune_trunc.h265
Done discovering file:///home/.../Dune_trunc.h265

Properties:
  Duration: 0:00:00.000000000
  Seekable: yes
  Live: no
  video #0: H.265 (Main 10 Profile)
    Stream ID: 715a97afea0372969f3dcd1dc7068e9c56d6283986affb477604b9405a379b2b
    Width: 3840
    Height: 2160
    Depth: 30
    Frame rate: 24000/1001
    Pixel aspect ratio: 1/1
    Interlaced: false
    Bitrate: 0
    Max bitrate: 0

Ubuntu 22.04, libde265 v1.0.12

[pjungkamp]

I'll check some older NixOS commits and report back soon.

If it's reproducible on older NixOS versions I'll check some other distros in containers in case it's a weird Nix-related build bug.

[pjungkamp]

As your Ubuntu seems to work I tried a simple debian container, but both bookworm and buster segfault.

I started off from the official dockerhub debian:buster and debian:bookworm containers. E.g.

docker run --rm -it debian:bookworm

Then I installed gstreamer using:

apt update && apt install gstreamer1.0-{tools,plugins-base,plugins-base-apps,plugins-good,plugins-bad}

Here's the buster segfault:

root@308ac31fa206:/# gst-discoverer-1.0 /mnt/Dune_trunc.h265 
Analyzing file:///mnt/Dune_trunc.h265
Segmentation fault (core dumped)
root@308ac31fa206:/# gst-discoverer-1.0 --gst-version        
GStreamer Core Library version 1.14.4

Here's the bookworm segfault:

root@aaa749e91f5f:/# gst-discoverer-1.0 /mnt/Dune_trunc.h265 
Analyzing file:///mnt/Dune_trunc.h265
Segmentation fault (core dumped)
root@aaa749e91f5f:/# gst-discoverer-1.0 --gst-version        
GStreamer Core Library version 1.22.0

The coredumps in my host's log suggest it's the same kind of crash (at least it's in libde265):

Nov 08 22:32:08 kernel: traps: h265parse0:sink[791363] general protection fault ip:7fd53873e6a2 sp:7fd525ff2e60 error:0 in libde265.so.0.1.4[7fd538711000+51000]
Nov 08 22:32:08 kernel: traps: h265parse0:sink[791358] general protection fault ip:7fd53873e6a2 sp:7fd5287f7e60 error:0 in libde265.so.0.1.4[7fd538711000+51000]
Nov 08 22:32:08 systemd[1]: Started Process Core Dump (PID 791377/UID 0).
Nov 08 22:32:08 systemd-coredump[791378]: [🡕] Process 791336 (gst-discoverer-) of user 0 dumped core.
                                                
                                                Stack trace of thread 7728:
                                                #0  0x00007fd53873e6a2 n/a (/usr/lib/x86_64-linux-gnu/libde265.so.0.1.4 + 0x346a2)
                                                #1  0x0001000000000002 n/a (n/a + 0x0)
                                                ELF object binary architecture: AMD x86-64

All these tests have been made on my x86_64-linux laptop (Intel Core i5 1240P, integrated graphics).
I also reproduced the crash on a server running NixOS (Intel Xeon E5-2680 v3, headless).

It seems to be happening irrespective of what package I use.
I'll check with my Arch Linux machine next, maybe something about the NixOS kernel is weird.

@farindk Should I move to a different bug tracker? Can you give me tips on where to start debugging?

[pjungkamp]

I can't reproduce on Arch Linux... Weird...

$ gst-discoverer-1.0 Dune_trunc.h265 
Analyzing file:///home/pjungkamp/Dune_trunc.h265
Done discovering file:///home/pjungkamp/Dune_trunc.h265

Properties:
  Duration: 0:00:00.000000000
  Seekable: yes
  Live: no
  video #0: H.265 (Main 10 Profile)
    Stream ID: 3eab9ccfcebb0e0fbd197b50b018c8c210f8c18cb0cfda46bd80a57a4e97c24c
    Width: 3840
    Height: 2160
    Depth: 30
    Frame rate: 24000/1001
    Pixel aspect ratio: 1/1
    Interlaced: false
    Bitrate: 0
    Max bitrate: 0

So an issue on the packaging side might be warranted. But I'm concerned about that this also happened in containers.

[AngryPhantom]

Guys, I'm not sure if it's related but I've run hdrcopy by mistake today (instead of another command in bash) and just found out what it is (a part of libde265). This command just gives me a 'Segmentation fault' message. I dunno if it's ok. Just decided to report. I'm running Arch Linux and libde265 is 1.0.12.

[farindk]

Guys, I'm not sure if it's related but I've run hdrcopy by mistake today (instead of another command in bash) and just found out what it is (a part of libde265). This command just gives me a 'Segmentation fault' message. I dunno if it's ok. Just decided to report. I'm running Arch Linux and libde265 is 1.0.12.

Not related. hdrcopy is also not meant to be installed by the distribution. It's for my own testing during development only.

[AngryPhantom]

@farindk Thank you. So it's not actully used and segfault is normal?

[farindk]

@farindk Thank you. So it's not actully used and segfault is normal?

It's only development tool. It segfaults when the parameters are not correct. There is no input validation of any kind. It should not be installed as it is pretty useless for a normal user. I've opened #428 to make sure it is removed from any distributions.

[AngryPhantom]

@farindk Got it. Thank you.