ci: comment on PRs with tests that got slower instead of failing them - #4048
Merged
Conversation
Test durations are too noisy for a hard gate: fishing and nether vary by 2-3x between identical runs, and master itself is failing the 1.5x check. Keep the comparison, but post the slower tests as a PR comment instead. Fork PRs run CI with a read-only token, so the matrix jobs upload the slower tests as an artifact and a workflow_run job posts (or updates) the comment. The PR number travels in the artifact because workflow_run.pull_requests is empty for forks.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #4028. Test durations are too noisy for a hard gate: fishing and nether vary by 2-3x between identical runs (see #4045's CI, and master's own run at faa2e33 failed the same check). Keep the comparison, but post the tests that got more than 1.5x slower as a PR comment instead of failing the job.
How it works:
compareDurations.jswrites the slower tests to a file and never exits non-zero.slower-<job-index>artifact.duration-comment.ymlruns onworkflow_runof CI, downloads the artifacts and creates or updates a single marked comment on the PR. When a later push has no slower tests, the same comment is updated to say so.Fork PRs get a read-only token, so the comment cannot be posted from the CI run itself;
workflow_run.pull_requestsis also empty for forks, which is why the PR number travels in the artifact.Since the new workflow triggers on
workflow_run, it only starts running once this is merged to master. The script was smoke-tested locally with fake duration files.