[v0.1.x-branch] Backport #966: waved: fix mailbox-compat test tempdir cleanup flake under -race - #975
Merged
Conversation
TestStartMailboxIngressConcurrentIncompatible (and its sibling) start the durable serverconn egress actor with StartEgress but tore it down with a deferred, fire-and-forget runtime.Stop(). Stop() only cancels the actor context and returns immediately, so the egress goroutine could keep issuing queries against the shared SQLite handle while the t.Cleanup chain closed the DB and removed the temp dir. An in-flight connection re-materializes the WAL/-shm sidecar files, so the tempdir RemoveAll raced and intermittently failed with "directory not empty" under -race. Switch both tests to StopAndWait so the egress goroutine has fully exited before cleanup runs. Reproduced reliably at -count=300 -cpu=4 before the change; passes cleanly across 1000+ iterations after. (cherry picked from commit c112d2e)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Backport of #966
Problem
The
unit-racejob intermittently fails with:(seen e.g. in the run for #895, but the test and code are on
mainand theflake is not reorg-related). Despite running under
-race, this is not adata-race report — Go's
t.TempDir()callst.Errorfwhen itsRemoveAllfails, which fails the test.
Root cause
TestStartMailboxIngressConcurrentIncompatibleandTestStartMailboxIngressAlreadyIncompatiblestart the durable serverconnegress actor with
StartEgress()but tore it down with a deferred,fire-and-forget
runtime.Stop().DurableActor.Stop()only cancels theactor context and returns immediately (
a.stopOnce.Do(func(){ a.cancel() }));only
StopAndWait/Waitblock ona.done.So the teardown ordering was:
defer Stop()— signals cancel, returns instantlyt.Cleanup:DB.Close()t.Cleanup:os.RemoveAll(tempdir)The egress goroutine kept issuing queries against the shared SQLite handle
during steps 2–3. An in-flight connection re-materializes the WAL/
-shmsidecar files, so
RemoveAll's finalrmdirraces and fails withdirectory not empty.Fix
Switch both tests to
StopAndWait(context.Background())in the deferredteardown so the egress goroutine has fully drained before the DB is closed
and the temp dir removed. Test-only change.
Verification
-count=300 -cpu=4 -racefailed dozens of times.go vet,make fmt-changed,make lint-changed-local(0 issues) all pass.Note (separate, pre-existing)
While stressing this I observed a distinct, much rarer data race originating
at
baselib/actor/durable_actor.go:508(the egress worker goroutine) — ~1 inseveral thousand iterations, identical on
mainand the reorg branches anduntouched by this change. It is orthogonal to the reported cleanup flake and
worth a separate investigation; I could not reliably reproduce it to capture a
full trace.