fix(simulation): prevent unbounded memory growth in GstCameraSystem#27873
Open
porhJ wants to merge 1 commit into
Open
fix(simulation): prevent unbounded memory growth in GstCameraSystem#27873porhJ wants to merge 1 commit into
porhJ wants to merge 1 commit into
Conversation
github-actions
Bot
added
kind:bug
Contributor
|
This PR looks like too much of LLM usage. Reduce the over explained comments please. Also why ur baseline is Ubuntu 22? |
Author
|
Sorry, I will reduce the comments. |
Each CameraStream pushed camera frames into an appsrc with no queue bound (block=false, enough-data unhandled) and never read the pipeline bus, so when the pipeline stalled the frames accumulated silently at frame_size x fps (~110 MB/s per camera at 1280x960@30) until the host froze. The stall is guaranteed on modern NVIDIA drivers because the nvh264enc path sets legacy NVENC properties they reject, killing caps negotiation before data ever flows, while set_state() only being checked for FAILURE (pipelines with sinks return ASYNC) hid the error. Bound the appsrc queue (max-buffers=3, leaky-type=downstream) so a stalled pipeline drops frames instead of hoarding them, surface bus errors/warnings through gzerr/gzwarn, verify the async transition to PLAYING, retry once with software x264 when the NVIDIA encoder fails, drop the driver-incompatible encoder properties, and guard the frame memcpy against undersized frames. Also make ~CameraStream() call stop() unconditionally: after a pipeline failure (_running == false) the streaming thread could still be joinable, and destroying a joinable std::thread terminates the process. Signed-off-by: PasinJ <porh8pasin@gmail.com>
porhJ
force-pushed
the
porhJ/fix-gstcamera-unbounded-appsrc
branch
from
bbd2835 to
240227d
Compare
Author
|
Thank you very much for the suggestion. |
dakejahl
approved these changes
Jul 14, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problems
Running models with camera sensors (e.g. gz_x500_mono_cam_down) will freeze the host (both with gui and headless). On 24 GB ram laptop around 3 minutes until everything freeze. Reduce camera sensors resolution would only delay the freezing.
Causes
The nvh264enc path (default, useCuda=true) sets legacy NVENC properties (preset=2, rc-mode, qp-const, zerolatency) that modern drivers reject. The failure is silent because the plugin never reads the pipeline bus and treats the ASYNC state-change return as success. This makes the downstream never consumes any frames. And appsrc doesn't have upper-bound (
block=false) so it just keeps accepting one frame per camera tick until it takes all memory and eventually freezes the host.Solution
(1) Bound the appsrc queue (max-buffers=3, leaky-type=downstream) and surface errors via gzerr/gzwarn, verifying the async transition to PLAYING.
(2) On NVIDIA encoder failure, retry once with software x264enc (runPipeline(useCuda) refactor).
(3) Drop the driver-incompatible NVENC properties, keeping only bitrate.
(4) Join/detach the streaming thread before destruction so a joinable std::thread can't terminate() the process.
Test
Measuring on Ubuntu 22.04 / GStreamer 1.20.3 / gz-sim 8.13 Harmonic / RTX 5050 (driver 595.71.05)
Reproduction of the original bug
Any camera world on a host where nvh264enc exists but cannot initialize.