Repository navigation
Conversation
Persist operation ownership before SQL locking and checkpoint each batch before mutations and continuation delivery. Redeliveries can resume reads or saved handoffs while refusing to replay potentially partial mutations. Gate new operations by organization, cap event batches at a configurable 50 events, and restrict Postgres and Snuba replacements to owned hashes.
github-actions
Bot
added
the
Scope: Backend
Treat legacy reservation release as best-effort so a Redis cleanup failure cannot abort processing after SQL locks commit. Cover continued processing and preservation of the original exception. Set the recoverable unmerge batch cap to 250 events to halve the previous 500-event batch while limiting additional per-batch overhead.
Default new unmerge chains to 250 events per batch. Remove the newly introduced option and runtime cap so existing chains retain the batch size carried in their task arguments.
Track the current batch and stage in one operation checkpoint. Use SET NX for fingerprint ownership, resume reads and saved handoffs, and reject replay after mutations start. Remove partial ownership, separate child checkpoints, explicit expiry arguments, and source-specific recovery handling. Reuse Redis release helpers and keep the conditional checkpoint write in a named Lua script.
![sentry-warden[bot]](https://avatars.githubusercontent.com/in/2755838?s=60&v=4){kind=small}
Checkpoint SQL unlocks separately from Redis owner removal so cancellation can resume safely after a worker interruption. Retire rejected requests and release their ownership when fingerprints have moved or disappeared before validation.
Keep empty successive batches resumable until their lock cleanup completes. Preserve the mutation guard before initial truncation and nonempty batches, and reuse staged lock release for completion.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
An unmerge can set its SQL fingerprint locks and then lose a worker while loading events. On redelivery, those locks are excluded, so the task loses track of its own fingerprints and can leave them stuck. Handing off the next batch also has a window where delivery may succeed without the current worker receiving confirmation.
This records fingerprint ownership in Redis and keeps one checkpoint per operation with its current batch and stage. A retry can resume reading events or resend a saved continuation. SQL locks are reused only when Redis confirms ownership; conflicting ownership or missing recovery state causes an error. Event payloads are loaded before resetting source aggregates, and batches interrupted after mutations begin require inspection rather than automatic replay.
Separately, this lowers the default batch size from 500 to 250 events for new unmerge chains. This is independent of the recovery mechanism, but loading fewer payloads at once and doing less work per task may also mitigate worker memory pressure and processing timeouts. Existing chains retain the batch size in their task arguments.
Deploy the API and all issue workers before enabling
organizations:unmerge-recovery. Legacy lock acquisition also respects Redis ownership; reservation cleanup failures are logged without aborting processing.This is a temporary mitigation. Existing orphaned locks and coordination between concurrent merge/unmerge operations remain outside its scope.