fix: sanitize Linux /tmp asset paths on runtime param updates; rate-limit FileNotFoundError (#916)#918
Open
livepeer-tessa wants to merge 1 commit intomainfrom
Open
fix: sanitize Linux /tmp asset paths on runtime param updates; rate-limit FileNotFoundError (#916)#918livepeer-tessa wants to merge 1 commit intomainfrom
livepeer-tessa wants to merge 1 commit intomainfrom
Conversation
…FileNotFoundError Issue #916: ltx2 i2v_image path from client's /tmp crashes every chunk Root cause (two parts): 1. _sanitize_initial_params was only called at pipeline LOAD time, not during runtime parameter updates received over WebSocket. When the frontend sends i2v_image mid-session (e.g. user picks a Reference Image while already streaming), the path from the client machine (/tmp/.daydream-scope/assets/foo.png) is forwarded raw to the fal.ai worker where that /tmp path doesn't exist. 2. Even with the right path, a race condition between CDN download and the first chunk causes FileNotFoundError on every chunk — 2500+ per session in the observed window — flooding Grafana logs. Fixes: - Add _sanitize_asset_path and _sanitize_initial_params static methods to PipelineManager (mirrors the fix from PR #827 for Windows paths, extended to also catch Linux /tmp paths from foreign machines). - Call _sanitize_initial_params in PipelineManager._load_pipeline_implementation for plugin pipelines (load-time fix). - Call _sanitize_initial_params in FrameProcessor.update_parameters for all runtime parameter updates (WebSocket mid-session updates). - In PipelineProcessor.process_chunk, catch FileNotFoundError separately and rate-limit repeated log entries for the same missing path to one per 30s, preventing log floods while still making the error visible. Tests: 9 new tests in TestSanitizeAssetPath covering Windows paths, Linux /tmp foreign paths, relative paths, None values, and list params. Signed-off-by: Tessa (livepeer-tessa) <tessa@livepeer.org>
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Contributor
🚀 fal.ai Preview Deployment
Livepeer Runner
Testing Livepeer Mode |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Issue #916: The ltx2 pipeline was generating ~2500 FileNotFoundError per session when a user sets an
i2v_imagereference image.Root Cause (2 parts)
1. Path sanitization only happened at load time, not during runtime updates
_sanitize_initial_paramswas called in_load_pipeline_implementationbut never during WebSocket parameter updates. When the frontend sendsi2v_imagemid-session (user picks a Reference Image while already streaming), the path — e.g./tmp/.daydream-scope/assets/foo.pngfrom the client's machine — gets forwarded raw to the fal.ai worker, where that/tmppath doesn't exist.PR #827 added sanitization for Windows paths (e.g.
C:\Users\...) but didn't address:2. FileNotFoundError floods logs on every chunk
Even with the correct path, a race condition between CDN download and the first inference chunk causes
FileNotFoundErroron every processed frame — easily 2500+ per session in the observed window — flooding Grafana.Session Timeline (from issue)
Fix
1. Add
_sanitize_asset_path/_sanitize_initial_paramstoPipelineManagerSame logic as PR #827 (Windows path detection) extended to also rewrite Unix absolute paths that fall outside the worker's configured assets directory — catching the Linux-
/tmp-from-different-machine case.2. Apply sanitization at runtime in
FrameProcessor.update_parametersThis covers mid-session WebSocket parameter updates (e.g. user picking a Reference Image while already streaming).
3. Apply sanitization at load time in
_load_pipeline_implementationCovers plugin pipelines (like ltx2) at load time.
4. Rate-limit repeated
FileNotFoundErrorinPipelineProcessorCatch
FileNotFoundErrorseparately and suppress repeated log entries for the same missing path, emitting at most 1 error per path per 30 seconds. This prevents log floods for any transient race condition during CDN download.Tests
9 new tests in
TestSanitizeAssetPath:C:\Users\...)C:/Users/...)/tmppaths from a foreign machine (new — issue [fal.ai] ltx2: i2v_image FileNotFoundError on Linux /tmp path — 2500 chunk errors per session (race condition) #916)Nonevaluesi2v_imageWindows pathi2v_imageLinux/tmppath (new)imageslistRelated