Skip to content

[release/2.12] [ROCm] Backport LayerNorm gamma/beta backward fix for wave32 (RDNA)#3425

Open
amd-maradosa wants to merge 1 commit into
release/2.12from
backport/layernorm-wave32-183864
Open

[release/2.12] [ROCm] Backport LayerNorm gamma/beta backward fix for wave32 (RDNA)#3425
amd-maradosa wants to merge 1 commit into
release/2.12from
backport/layernorm-wave32-183864

[ROCm] Fix LayerNorm backward kernel for AMD Strix Halo GPUs (#183864)

06aff09
Select commit
Loading
Failed to load commit list.
ROCm Repo Management API / Tests / Tests / Test Distributed / Run pytorch_distributed_2 failed Jul 13, 2026 in 0s

failed: 4, skipped: 28, passed: 130

failed: 4, skipped: 28, passed: 130

Details

TestDistBackendWithSpawn.test_ddp_apply_optim_in_backward

RuntimeError: Process 1 exited with error code 10 and exception:
Traceback (most recent call last):
  File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 971, in run_test
    getattr(self, test_name)()
  File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 818, in wrapper
    fn()
  File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3444, in wrapper
    method(*args, **kwargs)
  File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 235, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 4893, in test_ddp_apply_optim_in_backward
    self._test_ddp_apply_optim_in_backward(
  File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 4875, in _test_ddp_apply_optim_in_backward
    self.assertEqual(
  File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 4441, in assertEqual
    raise error_metas.pop()[0].to_error(  # type: ignore[index]
AssertionError: Tensor-likes are not close!

Mismatched elements: 6 / 3072 (0.2%)
Greatest absolute difference: 2.288818359375e-05 at index (0, 512) (up to 1e-05 allowed)
Greatest relative difference: 2.253430284326896e-05 at index (1, 448) (up to 1.3e-06 allowed)
Params not equal at iteration 3

To execute this test, run the following from the base repo dir:
    python test/distributed/test_distributed_spawn.py TestDistBackendWithSpawn.test_ddp_apply_optim_in_backward

This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
Stack trace 
Traceback (most recent call last):
  File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 816, in wrapper
    self._join_processes(fn)
  File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1086, in _join_processes
    self._check_return_codes(fn, elapsed_time)
  File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1126, in _check_return_codes
    raise RuntimeError(error)
RuntimeError: Process 1 exited with error code 10 and exception:
Traceback (most recent call last):
  File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 971, in run_test
    getattr(self, test_name)()
  File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 818, in wrapper
    fn()
  File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3444, in wrapper
    method(*args, **kwargs)
  File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 235, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 4893, in test_ddp_apply_optim_in_backward
    self._test_ddp_apply_optim_in_backward(
  File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 4875, in _test_ddp_apply_optim_in_backward
    self.assertEqual(
  File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 4441, in assertEqual
    raise error_metas.pop()[0].to_error(  # type: ignore[index]
AssertionError: Tensor-likes are not close!

Mismatched elements: 6 / 3072 (0.2%)
Greatest absolute difference: 2.288818359375e-05 at index (0, 512) (up to 1e-05 allowed)
Greatest relative difference: 2.253430284326896e-05 at index (1, 448) (up to 1.3e-06 allowed)
Params not equal at iteration 3

To execute this test, run the following from the base repo dir:
    python test/distributed/test_distributed_spawn.py TestDistBackendWithSpawn.test_ddp_apply_optim_in_backward

This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
Standard error 
/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/backends/cudnn/__init__.py:175: UserWarning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (Triggered internally at /var/lib/jenkins/pytorch/torch/csrc/cuda/Module.cpp:1990.)
  torch._C._cuda_set_cudnn_benchmark_limit(_benchmark_limit)
Standard out 
Process 1 terminated with exit code 10, terminating remaining processes.

TestDistBackendWithSpawn.test_ddp_apply_optim_in_backward_grad_as_bucket_view_false

RuntimeError: Process 0 exited with error code 10 and exception:
Traceback (most recent call last):
  File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 971, in run_test
    getattr(self, test_name)()
  File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 818, in wrapper
    fn()
  File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3444, in wrapper
    method(*args, **kwargs)
  File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 235, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 4902, in test_ddp_apply_optim_in_backward_grad_as_bucket_view_false
    self._test_ddp_apply_optim_in_backward(
  File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 4875, in _test_ddp_apply_optim_in_backward
    self.assertEqual(
  File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 4441, in assertEqual
    raise error_metas.pop()[0].to_error(  # type: ignore[index]
AssertionError: Tensor-likes are not close!

Mismatched elements: 6 / 3072 (0.2%)
Greatest absolute difference: 3.62396240234375e-05 at index (1, 1017) (up to 1e-05 allowed)
Greatest relative difference: 3.636888322944287e-06 at index (2, 664) (up to 1.3e-06 allowed)
Params not equal at iteration 3

To execute this test, run the following from the base repo dir:
    python test/distributed/test_distributed_spawn.py TestDistBackendWithSpawn.test_ddp_apply_optim_in_backward_grad_as_bucket_view_false

This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
Stack trace 
Traceback (most recent call last):
  File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 816, in wrapper
    self._join_processes(fn)
  File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1086, in _join_processes
    self._check_return_codes(fn, elapsed_time)
  File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1126, in _check_return_codes
    raise RuntimeError(error)
RuntimeError: Process 0 exited with error code 10 and exception:
Traceback (most recent call last):
  File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 971, in run_test
    getattr(self, test_name)()
  File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 818, in wrapper
    fn()
  File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3444, in wrapper
    method(*args, **kwargs)
  File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 235, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 4902, in test_ddp_apply_optim_in_backward_grad_as_bucket_view_false
    self._test_ddp_apply_optim_in_backward(
  File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 4875, in _test_ddp_apply_optim_in_backward
    self.assertEqual(
  File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 4441, in assertEqual
    raise error_metas.pop()[0].to_error(  # type: ignore[index]
AssertionError: Tensor-likes are not close!

Mismatched elements: 6 / 3072 (0.2%)
Greatest absolute difference: 3.62396240234375e-05 at index (1, 1017) (up to 1e-05 allowed)
Greatest relative difference: 3.636888322944287e-06 at index (2, 664) (up to 1.3e-06 allowed)
Params not equal at iteration 3

To execute this test, run the following from the base repo dir:
    python test/distributed/test_distributed_spawn.py TestDistBackendWithSpawn.test_ddp_apply_optim_in_backward_grad_as_bucket_view_false

This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
Standard error 
/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/backends/cudnn/__init__.py:175: UserWarning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (Triggered internally at /var/lib/jenkins/pytorch/torch/csrc/cuda/Module.cpp:1990.)
  torch._C._cuda_set_cudnn_benchmark_limit(_benchmark_limit)
Standard out 
Process 0 terminated with exit code 10, terminating remaining processes.

TestDistBackendWithSpawn.test_ddp_apply_optim_in_backward_grad_as_bucket_view_false

RuntimeError: Process 3 exited with error code 10 and exception:
Traceback (most recent call last):
  File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 971, in run_test
    getattr(self, test_name)()
  File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 818, in wrapper
    fn()
  File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3444, in wrapper
    method(*args, **kwargs)
  File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 235, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 4902, in test_ddp_apply_optim_in_backward_grad_as_bucket_view_false
    self._test_ddp_apply_optim_in_backward(
  File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 4875, in _test_ddp_apply_optim_in_backward
    self.assertEqual(
  File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 4441, in assertEqual
    raise error_metas.pop()[0].to_error(  # type: ignore[index]
AssertionError: Tensor-likes are not close!

Mismatched elements: 3 / 3072 (0.1%)
Greatest absolute difference: 2.765655517578125e-05 at index (0, 570) (up to 1e-05 allowed)
Greatest relative difference: 2.346949941056664e-06 at index (0, 570) (up to 1.3e-06 allowed)
Params not equal at iteration 3

To execute this test, run the following from the base repo dir:
    python test/distributed/test_distributed_spawn.py TestDistBackendWithSpawn.test_ddp_apply_optim_in_backward_grad_as_bucket_view_false

This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
Stack trace 
Traceback (most recent call last):
  File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 816, in wrapper
    self._join_processes(fn)
  File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1086, in _join_processes
    self._check_return_codes(fn, elapsed_time)
  File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1126, in _check_return_codes
    raise RuntimeError(error)
RuntimeError: Process 3 exited with error code 10 and exception:
Traceback (most recent call last):
  File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 971, in run_test
    getattr(self, test_name)()
  File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 818, in wrapper
    fn()
  File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3444, in wrapper
    method(*args, **kwargs)
  File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 235, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 4902, in test_ddp_apply_optim_in_backward_grad_as_bucket_view_false
    self._test_ddp_apply_optim_in_backward(
  File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 4875, in _test_ddp_apply_optim_in_backward
    self.assertEqual(
  File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 4441, in assertEqual
    raise error_metas.pop()[0].to_error(  # type: ignore[index]
AssertionError: Tensor-likes are not close!

Mismatched elements: 3 / 3072 (0.1%)
Greatest absolute difference: 2.765655517578125e-05 at index (0, 570) (up to 1e-05 allowed)
Greatest relative difference: 2.346949941056664e-06 at index (0, 570) (up to 1.3e-06 allowed)
Params not equal at iteration 3

To execute this test, run the following from the base repo dir:
    python test/distributed/test_distributed_spawn.py TestDistBackendWithSpawn.test_ddp_apply_optim_in_backward_grad_as_bucket_view_false

This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
Standard error 
/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/backends/cudnn/__init__.py:175: UserWarning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (Triggered internally at /var/lib/jenkins/pytorch/torch/csrc/cuda/Module.cpp:1990.)
  torch._C._cuda_set_cudnn_benchmark_limit(_benchmark_limit)
Standard out 
Process 3 terminated with exit code 10, terminating remaining processes.

TestDistBackendWithSpawn.test_ddp_apply_optim_in_backward

RuntimeError: Process 3 exited with error code 10 and exception:
Traceback (most recent call last):
  File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 971, in run_test
    getattr(self, test_name)()
  File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 818, in wrapper
    fn()
  File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3444, in wrapper
    method(*args, **kwargs)
  File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 235, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 4893, in test_ddp_apply_optim_in_backward
    self._test_ddp_apply_optim_in_backward(
  File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 4875, in _test_ddp_apply_optim_in_backward
    self.assertEqual(
  File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 4441, in assertEqual
    raise error_metas.pop()[0].to_error(  # type: ignore[index]
AssertionError: Tensor-likes are not close!

Mismatched elements: 6 / 3072 (0.2%)
Greatest absolute difference: 2.1457672119140625e-05 at index (0, 999) (up to 1e-05 allowed)
Greatest relative difference: 1.4788884072913788e-05 at index (0, 448) (up to 1.3e-06 allowed)
Params not equal at iteration 3

To execute this test, run the following from the base repo dir:
    python test/distributed/test_distributed_spawn.py TestDistBackendWithSpawn.test_ddp_apply_optim_in_backward

This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
Stack trace 
Traceback (most recent call last):
  File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 816, in wrapper
    self._join_processes(fn)
  File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1086, in _join_processes
    self._check_return_codes(fn, elapsed_time)
  File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1126, in _check_return_codes
    raise RuntimeError(error)
RuntimeError: Process 3 exited with error code 10 and exception:
Traceback (most recent call last):
  File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 971, in run_test
    getattr(self, test_name)()
  File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 818, in wrapper
    fn()
  File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3444, in wrapper
    method(*args, **kwargs)
  File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 235, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 4893, in test_ddp_apply_optim_in_backward
    self._test_ddp_apply_optim_in_backward(
  File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 4875, in _test_ddp_apply_optim_in_backward
    self.assertEqual(
  File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 4441, in assertEqual
    raise error_metas.pop()[0].to_error(  # type: ignore[index]
AssertionError: Tensor-likes are not close!

Mismatched elements: 6 / 3072 (0.2%)
Greatest absolute difference: 2.1457672119140625e-05 at index (0, 999) (up to 1e-05 allowed)
Greatest relative difference: 1.4788884072913788e-05 at index (0, 448) (up to 1.3e-06 allowed)
Params not equal at iteration 3

To execute this test, run the following from the base repo dir:
    python test/distributed/test_distributed_spawn.py TestDistBackendWithSpawn.test_ddp_apply_optim_in_backward

This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
Standard error 
/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/backends/cudnn/__init__.py:175: UserWarning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (Triggered internally at /var/lib/jenkins/pytorch/torch/csrc/cuda/Module.cpp:1990.)
  torch._C._cuda_set_cudnn_benchmark_limit(_benchmark_limit)
Standard out 
Process 3 terminated with exit code 10, terminating remaining processes.
View more details on ROCm Repo Management API