Repository navigation
NAT-549: Handle standardized-payload resume safely - #172
eugene-mux merged 67 commits into
Conversation
23c52fc to
1b8e194
Compare
1b8e194 to
18e2d78
Compare
There was a problem hiding this comment.
I think some of this new transport logic should be simplified. Going line-by-line doesn't make much sense, this is more overall design feedback:
- Transport Logic can assume that overlapping PUTs are accepted (the same byte ranges can be uploaded multiple times without errors)
- Generated Payloads and "Original Payloads" can share the same chunk response validation logic
- We don't need to worry about blocking job resumes while the Transformer's resources are cleared. The Transformer releases MediaCodecs synchronously, and repeatedly re-starting an upload while it standardizes is not likely to be a production case
- The logic that handles generated-file resume near the top of
createUploadJobis kind of hard to follow. It might help to break it into helper functions. This might also help ensure that no work or cases are duplicated.originalSelectedis persisted, so you can skip all of the standardization-related logic if the original file was already selected- you only need to
query()the upload endpoint if you are re-generating a standardized file - If standardization didn't finish during the last upload attempt (engine died or something), you can safely delete the standardized file and standardize again. Nothing will have uploaded.. This might already be assumed in code, but it's hard to tell.
Also can we use "Standardized" instead of "Generated?" That would be much clearer and match up with the terminology used elsewhere in the codebase
|
Also, consider thinking of the upload process in terms of its overall state transitions: If the original file is selected, selection/standardization can't happen again. Once standardization finishes it can't be restarted. Once transport starts you can't standardize again (and have to fail the upload if the file was deleted, whether it's standardized or original). Once a chunk is started, it can be re-sent. Once a chunk is finished successfully, it can't be re-sent. Following that process a little more literally in |
|
Thanks Emily, really helpful review! Simplified the flow around the state transitions you described, shared transport and retries, removed the release waits, and switched to “Standardized” naming. Pushed the changes. |
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit f06f88f. Configure here.
daytime-em
left a comment
There was a problem hiding this comment.
Thanks for taking the feedback! LGTM 👍
93548a8
into
feature/nat-534-hevc-standard-ingest

Description
Resume standardized uploads using the retained validated payload and confirmed progress. Original and standardized files share chunk transport, including retries of overlapping byte ranges.
Changes
Validation
Note
High Risk
Changes core upload persistence, resume blocking, and HTTP acknowledgement semantics for Direct Upload destinations—incorrect behavior could splice wrong bytes or leave destinations unusable.
Overview
This PR reworks resume and restart for uploads that use locally standardized (converted) payloads, and tightens how chunk PUTs are acknowledged and retried.
Resume & persistence: Standardized outputs are tracked with durable state (
PayloadIdentity, preparation phase, owned file path,network_started). Progress and the selected file are kept across pause, process death, andforceRestart(now documented as replacing the attempt while keeping payload and acknowledged bytes—not resetting to byte zero). If a completed standardized file is missing, tampered with, or the destination may already have partial bytes, resume fails withUploadResumeBlockedExceptionand can permanently block that Direct Upload URL instead of re-exporting or guessing offsets.Transport: Chunk upload logic is split:
UploadProtocolvalidates responses (308Range, success codes), handles boundedRetry-After, and uses a no-redirect HTTP client; per-chunk retries move toUploadJobFactorywith overlapping-range retries.ChunkWorkeronly performs a single PUT and reports progress via one-shot request bodies.Preparation / encoding: Renames generated → standardized across the pipeline; original selection is persisted atomically before transport. SDR export encodes in landscape encoder dimensions while validation compares display size using rotation metadata (portrait encoding flag removed).
Manager behavior: Cached job restore no longer drops entries when the source file is missing at restore time (handled elsewhere in the new persistence model).
Reviewed by Cursor Bugbot for commit a62c556. Bugbot is set up for automated code reviews on this repo. Configure here.