ci: fail a PR if any test gets more than 1.5x slower than master - #4028
Merged
Conversation
u9g
force-pushed
the
test-duration-regression
branch
from
August 30, 2026 15:25
e6f8adf to
921003b
Compare
u9g
changed the title
The spec reporter now also writes a JSON of per-test durations (DURATIONS=...). CI restores the last master run's durations for the version group from actions/cache, and test/compareDurations.js fails the job if a test that passed in both runs took more than 1.5x its baseline (ignoring jumps under 5s, which are server/network jitter). Every push to master saves a fresh baseline, so the numbers track the tests as they change.
u9g
force-pushed
the
test-duration-regression
branch
from
August 30, 2026 15:27
921003b to
555505c
Compare
Member
|
I like it but I think it might fail. Let's try it out and adapt if anything doesn't work |
This was referenced Aug 31, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Recent PRs (#3977, #3984, #3999, #3990, ...) cut a lot of fixed sleeps out of the external tests. Nothing stops them creeping back in, and a fixed wall-clock budget doesn't work here since runtime is dominated by server startup and varies per version. So: compare per-test durations against master's last run and fail on a 1.5x regression.
How
test/common/durationsReporter.js: the spec reporter, plus a{ "<test title>": ms }file of passing tests written to$DURATIONS.mocha_testnow uses it; log output is unchanged.test/common/compareDurations.js <baselineDir> <currentDir>: for each test present in both runs, fail if it took more than 1.5x its baseline and at least 5s longer (so sub-second tests can't fail on jitter). Prints abaseline -> nowtable per version. Tests without a baseline are skipped.ci.yml: restore the last master baseline for the version group fromactions/cache(durations-<job-index>-*), run the comparison after the tests, and on pushes to master save the run as the new baseline. Rebaselining is automatic, no committed files.Only tests that passed count, and with
--retries 3mocha records the passing attempt's duration, so a flaky retry doesn't register as a slowdown.Verification
Locally against the internal tests: 632 durations recorded with the spec output intact; the compare script exits 1 when a test goes 1s -> 20s, 0 when identical, and skips cleanly with no baseline. The first CI run on master seeds the cache; PRs before that just print "no baseline yet".