[release/2.9] Fix flaky reentrant backward test by passing explicit grad for non-scalar output (#179704)#3418
Open
darren-amd wants to merge 1 commit into
Open
Conversation
…alar output (pytorch#179704) Fixes pytorch#179703 More detailed description of the problem is in the issue pytorch#179703 ## What this PR changes In `test/test_autograd.py`, `_test_reentrant_parent_error_on_cpu` [(THIS LINE)](https://github.com/pytorch/pytorch/blob/3b39608061aac9da750ad3954197400b191286c3/test/test_autograd.py#L12574) defines a reentrant autograd function whose backward currently does: ```python reentrant_root.backward() ``` This PR changes it to: ```python reentrant_root.backward(grad) ``` ## Why this solution `reentrant_root` is non-scalar (`[3, 3]`). For non-scalar outputs, `.backward()` requires an explicit gradient seed. Calling `.backward()` without that seed can raise: `RuntimeError: grad can be implicitly created only for scalar outputs` ## Effect on test behavior Before this change, the test had two competing error sources: 1. Intended error: `"Simulate error"` from `SimulateBackwardError` 2. Accidental error: non-scalar `.backward()` without explicit grad Because execution order is nondeterministic, CI could intermittently fail when (2) happened first. After this change, the accidental error source is removed. The test now consistently validates only the intended behavior. Pull Request resolved: pytorch#179704 Approved by: https://github.com/jeffdaily (cherry picked from commit 24cdfbd)
|
Jenkins build for a3a3339c97329634a42aaa5e61c2b028bd861ab1 commit finished as FAILURE |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cherry-pick of upstream pytorch#179704 (commit 24cdfbd) into release/2.9.
Why
test/test_autograd.py::TestAutogradDeviceTypeCUDA::test_reentrant_parent_error_on_cpu_cudais flaky on ROCm Linux CI.ReentrantFunc.backward()callsreentrant_root.backward()on a non-scalar[3, 3]tensor with no explicit gradient, which is always invalid. This races two errors through the autograd engine: the intended"Simulate error"and an accidentalRuntimeError: grad can be implicitly created only for scalar outputs. On ROCm the accidental error wins, soassertRaisesRegex("Simulate error")fails.The one-line fix (
reentrant_root.backward()->reentrant_root.backward(grad)) removes the accidental error source and makes the test deterministic. Already present on upstreammainandrelease/2.10; this backports it torelease/2.9.Change
test/test_autograd.py - 1 line, inside
_test_reentrant_parent_error_on_cpu.Fixes ROCm/TheRock#6170