Fix flaky TestDisposeStaleCryptoKeys caused by mock clock ticker handling#7128
Open
amartinezfayo wants to merge 2 commits into
Open
Fix flaky TestDisposeStaleCryptoKeys caused by mock clock ticker handling#7128amartinezfayo wants to merge 2 commits into
amartinezfayo wants to merge 2 commits into
Conversation
…ling Signed-off-by: Agustín Martínez Fayó <amartinezfayo@gmail.com>
There was a problem hiding this comment.
Pull request overview
This PR stabilizes TestDisposeStaleCryptoKeys in the gcp_kms key manager plugin tests by making mock-clock/ticker synchronization deterministic, avoiding races and intermediate-time observations that could previously lead to hangs/timeouts.
Changes:
- Make test CryptoKeys already stale by setting
spire-last-updateolder thanmaxStaleDuration. - Drain the dispose task “initialized” notification before advancing the mock clock, fixing the ticker baseline race.
- Advance the mock clock exactly one ticker period at a time and remove the now-unneeded
maxDurationhelper.
Signed-off-by: Agustín Martínez Fayó <amartinezfayo@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TestDisposeStaleCryptoKeysdrivesdisposeCryptoKeysTaskthrough the mock clock, but its correctness depended on goroutine scheduling in two ways, so it would occasionally hang until the test timed out.disposeCryptoKeysTaskcreates its ticker and only then emits an initialization notification, yet the test advanced the clock before draining that notification. When the task goroutine won the race and created its ticker at the starting time, the ticker's baseline was earlier than the test assumed. The first dispose run then executed while the CryptoKeys were not yet stale, and because the task blocks on the unbuffered notification channel after each run, that non-stale run's notification stalled the single task goroutine before it could reach the run that schedules destruction. The test then waited forever for a destroy notification that never arrived.The test also advanced the clock by many ticker periods at once.
github.com/andres-erbsen/clockadvances its notion of "now" incrementally as it fires each intermediate tick and yields between them, so a multi-period advance fires the ticker while "now" is still at an intermediate value. The dispose run could observe a not-yet-stale clock even though the advance targeted a stale time.This PR makes the handshake deterministic. The test drains the initialization notification before advancing the clock, so the ticker's baseline is fixed. The CryptoKeys start with a last-update timestamp older than
maxStaleDurationso they are already stale, and the clock is advanced exactly one ticker period at a time, which fires the ticker once with the clock already at the advanced time. Stale detection no longer depends on which intermediate clock value the running task happens to observe.This is a follow-up to #7081, which addressed the
Sleep-based retry loops in othergcp_kmstests but not the ticker handshake in this one.An example of a failure is here: https://github.com/spiffe/spire/actions/runs/28537752222/job/84603474025