Repository navigation
Conversation
Previously, a dissolve between two opaque 4K60 H.264 clips at 100 Mbit/s ran at 8.3 fps on a 16-core desktop, when played on one thread as the Kdenlive monitor does. Most of this time went to color conversions that changed nothing. Kdenlive sets fix_background_alpha on every dissolve, and with this property, the luma transition requested both frames as rgba and converted the result back to yuv422. It did this only to make the padding that filter_resize adds transparent, so for opaque video, the four conversions had no effect. The round trip also slowed down the work after the transition, because it left the result with an alpha mask that was fully opaque. Kdenlive composites its tracks with a qtblend transition, which saw this mask and so composited the frame in rgba too. The transition now requests yuv422 and marks both frames with fix_background_alpha. The padding stays transparent, because when filter_resize pads a marked frame that has no alpha mask, it first adds an opaque one. The alpha of the result also stays correct: when one frame is translucent, the transition gives the other frame an opaque mask if it has none, as the round trip did, so that alpha_over can still compute it. With this change, the same dissolve runs at 73 fps. Its luma is also closer to the source, because the rgba round trip clipped colors that rgb cannot represent. In the first frame of the dissolve, luma is now identical to the decoded source, and before, the two differed by a PSNR of 38 dB. Apart from this, the output does not change. These cases give the same result as before: - A 16:9 clip in a 4:3 profile gets transparent padding. - With alpha_over, a dissolve into a clip with transparency shows the track below through it, with the same colors in every frame, also with a luma map and with invert. - With fix_background_alpha off, the output is bit-identical.
Previously, the luma transition split the dissolve of translucent frames into slices, but it mixed opaque frames line by line on the calling thread. For 4K frames, this loop took a large part of the time of each frame. Opaque frames now use slices too, as translucent frames do. The output does not change, and on a 16-core desktop, a dissolve between two 4K60 H.264 clips at 100 Mbit/s goes from 73 to 77 fps.
Previously, the conversion of progressive 8-bit 4:2:0 video to yuyv422 was slow, and it put the chroma half a line too low. The producer does this conversion on every frame of most video, and libswscale caused both problems: - MLT interpolates chroma vertically (SWS_BILINEAR), so libswscale ran its general scaler for this conversion. The producer also created a new scaler for each column slice of each frame. - libswscale copied chroma line n to output line 2n+1, and averaged two chroma lines for the even lines, whatever src_v_chr_pos said. The producer now does this conversion itself. In 4:2:0 video such as H.264 and MPEG-2, each chroma line lies between two luma lines, so each output line mixes the nearest chroma line (3/4) with the next nearest one (1/4). The conversion runs in line slices and uses SSE2 where available. The new conversion does not convert between YUV matrices, but libswscale does, for example for SD video in an HD profile. So the producer uses the new conversion only when the source and the destination use the same matrix. For this check, the choice of the libswscale matrix moves into mlt_get_sws_coefficients(). Other source formats also still use libswscale. Because the chroma is now in the correct position, the output changes where the chroma changes from one line to the next. At the sharp color edges of a test pattern, it changes by up to 113 levels. Luma does not change, and SD clips (tagged BT.601 or untagged) in an HD profile still use libswscale, so their output is bit-identical to before. On a 16-core desktop, a dissolve between two 4K60 H.264 clips at 100 Mbit/s goes from 77 to 104 fps.
Previously, when playback reached a clip whose file was not open yet, it stopped while MLT probed the file three times. Kdenlive opens files only at this point, so this happened at the start of clips. For a seekable file with audio and video, producer_open() called avformat_find_stream_info() on three contexts: 1. the first context; 2. the context that get_basic_info() reopens after its seek test; 3. a third context for audio. Each probe decodes video frames on the calling thread. For a 4K60 H.264 clip at 100 Mbit/s, one probe takes about 100 ms on a 16-core desktop. For common formats, the probe of the audio context is not necessary. The audio context reads the same file as the video context, and MP4/MOV and Matroska/WebM files keep all their stream information in the header. So for these formats, the audio context now gets a copy of what the probe of the video context found. Other formats are still probed, and so is the seek-test context. The copy gives the same result as a probe: the decoded pictures and audio do not change, also after seeks, for MP4, MOV, Matroska and WebM files with AAC, Opus, Vorbis, FLAC, MP3, PCM and AC-3 audio. This is also true for files with several audio streams or with the audio stream first, for fragmented MP4, and for files with an audio offset or an edit list. With this change, a dissolve into a 4K60 H.264 clip at 100 Mbit/s whose file is not open yet goes from 40 to 47 fps.
Previously, when playback reached a clip whose file was not open yet, it also stopped for the probe of the seek-test context. To test if a file is seekable, get_basic_info() seeks in the probed context and keeps it aside. Then it opens the file a second time and probes the new context with avformat_find_stream_info(), which decodes video frames on the calling thread. The audio context already gets its stream information without a probe, from find_stream_info_from(). The reopened context now does the same, so MP4/MOV and Matroska/WebM files skip this probe too. The reopened context has the same stream information as after a probe, and the decoded pictures and audio do not change, also after seeks. This was checked with MP4, MOV, Matroska (H.264 and HEVC), WebM, AVI and album art files. One difference remains, but MLT does not use it: with H.264 and HEVC in Matroska, the packets of the reopened context can have other dts values, because only a probe sets has_b_frames. Their pts values, which MLT uses, are the same. With this change, the start of such a clip takes less time. On a 16-core desktop, a dissolve into a 4K60 H.264 clip at 100 Mbit/s whose file is not open yet goes from 48 to 56 fps. In a Kdenlive project with such clips, the first frame of a dissolve now takes about 260 ms, and it took 290 to 305 ms before.
Previously, on a 16-core desktop, each frame of a 4K dissolve spent about 3 ms to copy the two decoded frames into the image cache of the avformat producer. The cause was that mlt_frame_clone_image() copied each 16 MB yuv422 image with one memcpy(), and one thread cannot use all of the memory bandwidth. mlt_frame_clone() and mlt_frame_clone_image() now copy large images and alpha masks in slices. The limit is 4 MB, so 4K images use slices, and smaller buffers, such as 1080p yuv422 images, are still copied with one memcpy(). A slice job that starts more slice jobs can deadlock the slice pool. This cannot happen here, because no slice job in MLT clones a frame. To check this, melt ran with several render threads (real_time from -1 to -8) and different slice counts, and it finished normally. The output does not change. With this change, the two copies take about 1.7 ms, and a dissolve between two 4K60 H.264 clips at 100 Mbit/s goes from 104 to 113 fps.
Previously, during forward playback, the producer made a deep copy of each decoded frame in last_good_frame. It needs the last good frame to conceal decoding errors: when a frame fails to decode, the producer shows the last good frame in its place. But the image cache already held a copy of the same frame, so for a 4K clip, each frame cost a second 16 MB copy that was useful only after an error. When the image cache is on, the producer now keeps only the position of the last good frame, and gets the frame from the cache when decoding fails. The cache can lose that frame in two cases, so the producer copies the frame before they happen: - when frames before it go into the cache, which happens during backward playback or after a seek back; - when the cache is reset. Without the image cache, the producer still copies every frame. The producer conceals errors with the same frames as before. This was checked with clips with corrupted parts, played forward, played backward, after a seek back, after a cache reset, and with a small image cache or none. With this change, forward playback is faster. On a 16-core desktop, a dissolve between two 4K60 H.264 clips at 100 Mbit/s goes from 61 to 69 fps, and in a Kdenlive project with such clips, from 73 to 86 fps.
e3c8337 to
25a34c4
Compare
iniw
changed the title
ddennedy
left a comment
ddennedy
left a comment
There was a problem hiding this comment.
"avformat: convert yuv420p to yuyv422 without libswscale" is completely unacceptable. It is not acceptable to pull custom pixel format conversion into the producer. "SD video in an HD profile" probably refers to different color spaces, and proper colorspace conversion is required instead of simply reorganizing sample values. Keep the separation of concerns and leverage the knowledge about chroma co-siting, colorspace conversion, acceleration etc. must continue to be outsourced to the libswscale that is more capable than anything I want to maintain in MLT as a primary code path.
| // transparent. This gives the same result as padding an rgba image, without | ||
| // converting the image to rgba and back. | ||
| if (format == mlt_image_yuv422 && !alpha | ||
| && mlt_properties_get_int(properties, "fix_background_alpha")) { |
There was a problem hiding this comment.
This creates an undocumented coupling between transtion_luma and filter_resize. And "fix_background_alpha" doesn't mean anything in this filter. Can you think of a different way to accomplish this? I wonder if we could just infer that the alpha should be created if resize_alpha != 255. In fact, if we did this, it would remove the need for the fix_background_alpha property on transition_luma.
There was a problem hiding this comment.
I agree, but the default for "resize_alpha" is 0 since it does not exist by default. Only certain things set it to 255. So, the logic should be to create the alpha if it does not exist and the "resize_alpha" property exists on the frame. And use the property value instead of hard-code 255 in the memset.
The "fix_background_alpha" property exists on transition luma to version the behavior. Are you saying that 3.5 years is long enough, and the fidelity of old projects no longer relevant?
There was a problem hiding this comment.
I think the "fix" property has gone on lone enough. If there are old projects that currently do not set fix_background_alpha=1, then they are getting inconsistent results depending on whether an upstream service is requesting RGB or YUV. In my opinion, that is a bug that should be fixed.
Current Behavior:
- If an upstream service is requesting RGB, then they are getting transparent alpha every time.
- If an upstream service is requesting YUV, then they are getting opaque or transparent depending on fix_background_alpha.
Proposed Behavior:
- The transition should operate on transparent background regardless of the format requested
That could be accomplished by always setting "resize_alpha=0" in the transition and then conditionally creating the alpha in filter_resize if the "resize_alpha" property exists and not 255.
| ctx.src_alpha += slice_start * ctx.src_width; | ||
|
|
||
| for (int i = 0; i < slice_height; i++) { | ||
| composite_line_yuv(ctx.dst_image, |
There was a problem hiding this comment.
Is the only difference between this function and dissolve_slice_yuv422 that it calls composite_line_yuv() instead of composite_line_yuv_float()? If so, what is the difference? Could everything use the same dissolve slice function and simplify this whole file?
There was a problem hiding this comment.
As Dan said, we can not put conversion code in this file. But if SWS is not returning correct chroma location, we should figure that out. Is it not calling mlt_get_sws_flags() as the scaler does. Also, if creating multiple sws contexts is expensive, maybe we could find a way to preserve them - like adding the conversion to the filtergraph that is sometimes created. I'm not sure - I haven't thought about it that much.
This branch has not been deployed
3 participants
Backstory
I was using Kdenlive to edit some relatively heavy clips (4K, 60 fps, 100 Mbit/s H.264 screen recordings) and noticed that during dissolve transitions the preview became unusably laggy: playback dropped to about 8 fps for the length of each dissolve, while the rest of the timeline played in real time. Preview Render made the dissolves smooth, but only until the next change.
A profile of the dissolve showed that decoding was not the problem. Most of the time went to pixel format conversions on a single thread:
transition_lumaconverted both 4K frames to rgba and back to yuv422 because offix_background_alpha, which Kdenlive sets on every dissolve, and theqtblendtransition that Kdenlive uses to composite its tracks then converted the result to rgba again.Summary
This PR is a series of 7 patches that incrementally improve the performance of MLT in that scenario, with the final goal of real-time dissolve transitions. They should also help outside this case, because the conversions that they remove or make faster happen for most video, at any resolution. The work was guided by profiling MLT (with
perf) in a small program that plays a timeline the way the Kdenlive monitor does: one thread gets each frame as yuv422 together with its audio.Each patch explains its change in its commit message, so please review this PR commit by commit.
Results
On my machine (AMD Ryzen 9 9950X, 16 cores and 32 threads, FFmpeg 8.1.2, GCC 16.2), with the test below: a 1-second dissolve between two generated 4K60 H.264 clips at 100 Mbit/s, built on master and after each patch. "First play" is the first time playback reaches the dissolve, when the file of the second clip is not open yet. "Replay" plays it again with the files open. Each value is the middle of 3 runs.
The first play is lower than the replay because of one long frame: when playback reaches the second clip, MLT opens its file, probes it and decodes from the last keyframe up to the first frame that it shows. In my real project, that frame takes about 255 ms (it took 460 ms on master), and the dissolve plays at 85 fps on the first play and 99 fps on replay.
Correctness
The requirement for each patch was that the pictures and the audio do not change. Patches 1 and 3 change the picture on purpose, because the old output was less accurate, and their commit messages explain the change and how large it is. Each commit message also lists the checks that were made for that patch.
To check this, I built master and each commit, played the same projects with each build, and dumped the image (yuv422) and the audio (s16) of every frame. Then I compared each commit with the commit before it, byte for byte. Where a patch changes the picture on purpose, I compared both outputs with the frames that ffmpeg decodes from the source file, to make sure that the new output is closer to the source and not only different.
The test projects were laid out like Kdenlive projects (tracks composited with
qtblend, dissolves and luma wipes withfix_background_alpha), and the test clips were made to reach the code that each patch changes.Reproducing
The test needs
ffmpeg, a C compiler,pkg-configand the build dependencies of MLT. Put the four files below in one directory, then:measure.shbuilds the checkout intobuild/and installs it intoinst/inside it, so run it once on master and once with this branch checked out. Output on my machine:To compare outputs, add a dump prefix as the last argument of
bench. It writes the images of the measured frames to<prefix>.img0and<prefix>.img1(yuyv422, one file for each play) and all of the audio to<prefix>.pcm0and<prefix>.pcm1(s16, 48 kHz, stereo). For example, to compare the pictures of two builds:ffmpeg -f rawvideo -pix_fmt yuyv422 -s 3840x2160 -i master.img1 \ -f rawvideo -pix_fmt yuyv422 -s 3840x2160 -i branch.img1 -lavfi psnr -f null -gen-clips.sh: makes the test clipsdissolve.mlt: a timeline laid out like a Kdenlive projectbench.c: plays the timeline like the Kdenlive monitormeasure.sh: builds MLT and runs the testAI Disclosure
Claude Opus 5.5 did nearly all of this work on its own in Claude Code, after I described the problem to it: it profiled MLT, wrote the patches and the test above, compared each patch with master, and helped me write this description. I guided it and checked that the results hold in Kdenlive, where the dissolves now play in real time.
I had never read MLT's code before, so I cannot judge whether these changes fit the design of the project as well as they fix my problem. I am sending them because I think they are useful to other users too. If you prefer not to accept changes written like this, I understand, and I will keep the patches in my local build, where they work for me.