notifier: take the write lock when updating LastEval in sendEvaluatorRequests - #867
Open
ZayanKhan-12 wants to merge 1 commit into
Open
notifier: take the write lock when updating LastEval in sendEvaluatorRequests#867ZayanKhan-12 wants to merge 1 commit into
ZayanKhan-12 wants to merge 1 commit into
Conversation
…Requests sendEvaluatorRequests held only the cluster's read lock while writing groupInfo.LastEval. A read lock does not exclude other readers of the same group state, and two instances of sendEvaluatorRequests can overlap briefly after a ZooKeeper session bounce (manageEvalLoop spawns a new loop once it reacquires the lock, while the previous loop can still be mid-iteration before observing doEvaluations == false), so the write can race. Writes to shared state need the write lock. The lock is only held for the in-memory iteration - the evaluator request send happens in a spawned goroutine - so this does not hold the lock across any channel sends or network calls. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
sendEvaluatorRequestsholds only the cluster's read lock (consumerGroup.Lock.RLock()) while writinggroupInfo.LastEval. A read lock permits concurrent access, so this write can race:sendEvaluatorRequestscan briefly overlap after a ZooKeeper session bounce —manageEvalLoopspawns a new loop as soon as it reacquires the ZK lock, while the previous loop can still be mid-iteration before it observesdoEvaluations == false. Both then read and writeLastEvalunder read locks concurrently.Fix
Take the write lock for the group iteration. The lock only covers the in-memory loop — the evaluator request send already happens in a spawned goroutine — so no channel sends or network calls happen under the lock, and contention is unchanged in practice.
Testing
go test ./core/internal/notifier/andgo test -race ./...pass (the pre-existingcoordinator_race_test.gobehavior under-racebuild tags is unchanged).🤖 Generated with Claude Code