[release/2.10] [ROCm] derive the hipcub compatible cccl version from 'HIPCUB_CCCL_VERSION'#3439
Closed
jithunnair-amd wants to merge 2 commits into
Closed
ROCm Repo Management API / Tests / Tests / Test Distributed / Run pytorch_distributed_2
failed
Jul 17, 2026 in 0s
failed: 2, skipped: 23, passed: 94
failed: 2, skipped: 23, passed: 94
Details
TestDistBackendWithSpawn.test_ddp_apply_optim_in_backward_grad_as_bucket_view_false
RuntimeError: Process 0 exited with error code 10 and exception:
Traceback (most recent call last):
File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
getattr(self, test_name)()
File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
fn()
File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
method(*args, **kwargs)
File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 227, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 4886, in test_ddp_apply_optim_in_backward_grad_as_bucket_view_false
self._test_ddp_apply_optim_in_backward(
File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 4858, in _test_ddp_apply_optim_in_backward
self.assertEqual(
File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 4284, in assertEqual
raise error_metas.pop()[0].to_error( # type: ignore[index]
AssertionError: Tensor-likes are not close!
Mismatched elements: 6 / 3072 (0.2%)
Greatest absolute difference: 0.00250244140625 at index (8, 1) (up to 1e-05 allowed)
Greatest relative difference: 4.14412943428033e-06 at index (8, 1) (up to 1.3e-06 allowed)
Params not equal at iteration 4
To execute this test, run the following from the base repo dir:
PYTORCH_TEST_WITH_ROCM=1 python test/distributed/test_distributed_spawn.py TestDistBackendWithSpawn.test_ddp_apply_optim_in_backward_grad_as_bucket_view_false
This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
Process 1 exited with error code 10 and exception:
Traceback (most recent call last):
File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
getattr(self, test_name)()
File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
fn()
File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
method(*args, **kwargs)
File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 227, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 4886, in test_ddp_apply_optim_in_backward_grad_as_bucket_view_false
self._test_ddp_apply_optim_in_backward(
File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 4858, in _test_ddp_apply_optim_in_backward
self.assertEqual(
File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 4284, in assertEqual
raise error_metas.pop()[0].to_error( # type: ignore[index]
AssertionError: Tensor-likes are not close!
Mismatched elements: 6 / 3072 (0.2%)
Greatest absolute difference: 0.00250244140625 at index (8, 1) (up to 1e-05 allowed)
Greatest relative difference: 4.14412943428033e-06 at index (8, 1) (up to 1.3e-06 allowed)
Params not equal at iteration 4
To execute this test, run the following from the base repo dir:
PYTORCH_TEST_WITH_ROCM=1 python test/distributed/test_distributed_spawn.py TestDistBackendWithSpawn.test_ddp_apply_optim_in_backward_grad_as_bucket_view_false
This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
Stack trace
Traceback (most recent call last):
File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper
self._join_processes(fn)
File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes
self._check_return_codes(fn, elapsed_time)
File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes
raise RuntimeError(error)
RuntimeError: Process 0 exited with error code 10 and exception:
Traceback (most recent call last):
File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
getattr(self, test_name)()
File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
fn()
File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
method(*args, **kwargs)
File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 227, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 4886, in test_ddp_apply_optim_in_backward_grad_as_bucket_view_false
self._test_ddp_apply_optim_in_backward(
File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 4858, in _test_ddp_apply_optim_in_backward
self.assertEqual(
File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 4284, in assertEqual
raise error_metas.pop()[0].to_error( # type: ignore[index]
AssertionError: Tensor-likes are not close!
Mismatched elements: 6 / 3072 (0.2%)
Greatest absolute difference: 0.00250244140625 at index (8, 1) (up to 1e-05 allowed)
Greatest relative difference: 4.14412943428033e-06 at index (8, 1) (up to 1.3e-06 allowed)
Params not equal at iteration 4
To execute this test, run the following from the base repo dir:
PYTORCH_TEST_WITH_ROCM=1 python test/distributed/test_distributed_spawn.py TestDistBackendWithSpawn.test_ddp_apply_optim_in_backward_grad_as_bucket_view_false
This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
Process 1 exited with error code 10 and exception:
Traceback (most recent call last):
File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
getattr(self, test_name)()
File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
fn()
File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
method(*args, **kwargs)
File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 227, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 4886, in test_ddp_apply_optim_in_backward_grad_as_bucket_view_false
self._test_ddp_apply_optim_in_backward(
File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 4858, in _test_ddp_apply_optim_in_backward
self.assertEqual(
File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 4284, in assertEqual
raise error_metas.pop()[0].to_error( # type: ignore[index]
AssertionError: Tensor-likes are not close!
Mismatched elements: 6 / 3072 (0.2%)
Greatest absolute difference: 0.00250244140625 at index (8, 1) (up to 1e-05 allowed)
Greatest relative difference: 4.14412943428033e-06 at index (8, 1) (up to 1.3e-06 allowed)
Params not equal at iteration 4
To execute this test, run the following from the base repo dir:
PYTORCH_TEST_WITH_ROCM=1 python test/distributed/test_distributed_spawn.py TestDistBackendWithSpawn.test_ddp_apply_optim_in_backward_grad_as_bucket_view_false
This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
Standard error
/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/backends/cudnn/__init__.py:173: UserWarning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (Triggered internally at /var/lib/jenkins/pytorch/torch/csrc/cuda/Module.cpp:1965.)
torch._C._cuda_set_cudnn_benchmark_limit(_benchmark_limit)
Standard out
Process 0 terminated with exit code 10, terminating remaining processes.
TestDistBackendWithSpawn.test_ddp_apply_optim_in_backward_grad_as_bucket_view_false
RuntimeError: Process 0 exited with error code 10 and exception:
Traceback (most recent call last):
File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
getattr(self, test_name)()
File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
fn()
File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
method(*args, **kwargs)
File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 227, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 4886, in test_ddp_apply_optim_in_backward_grad_as_bucket_view_false
self._test_ddp_apply_optim_in_backward(
File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 4858, in _test_ddp_apply_optim_in_backward
self.assertEqual(
File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 4284, in assertEqual
raise error_metas.pop()[0].to_error( # type: ignore[index]
AssertionError: Tensor-likes are not close!
Mismatched elements: 3 / 3072 (0.1%)
Greatest absolute difference: 0.006591796875 at index (0, 465) (up to 1e-05 allowed)
Greatest relative difference: 7.1619579102844e-06 at index (0, 465) (up to 1.3e-06 allowed)
Params not equal at iteration 4
To execute this test, run the following from the base repo dir:
PYTORCH_TEST_WITH_ROCM=1 python test/distributed/test_distributed_spawn.py TestDistBackendWithSpawn.test_ddp_apply_optim_in_backward_grad_as_bucket_view_false
This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
Process 2 exited with error code 10 and exception:
Traceback (most recent call last):
File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
getattr(self, test_name)()
File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
fn()
File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
method(*args, **kwargs)
File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 227, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 4886, in test_ddp_apply_optim_in_backward_grad_as_bucket_view_false
self._test_ddp_apply_optim_in_backward(
File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 4858, in _test_ddp_apply_optim_in_backward
self.assertEqual(
File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 4284, in assertEqual
raise error_metas.pop()[0].to_error( # type: ignore[index]
AssertionError: Tensor-likes are not close!
Mismatched elements: 3 / 3072 (0.1%)
Greatest absolute difference: 0.006591796875 at index (0, 465) (up to 1e-05 allowed)
Greatest relative difference: 7.1619579102844e-06 at index (0, 465) (up to 1.3e-06 allowed)
Params not equal at iteration 4
To execute this test, run the following from the base repo dir:
PYTORCH_TEST_WITH_ROCM=1 python test/distributed/test_distributed_spawn.py TestDistBackendWithSpawn.test_ddp_apply_optim_in_backward_grad_as_bucket_view_false
This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
Process 3 exited with error code 10 and exception:
Traceback (most recent call last):
File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
getattr(self, test_name)()
File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
fn()
File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
method(*args, **kwargs)
File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 227, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 4886, in test_ddp_apply_optim_in_backward_grad_as_bucket_view_false
self._test_ddp_apply_optim_in_backward(
File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 4858, in _test_ddp_apply_optim_in_backward
self.assertEqual(
File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 4284, in assertEqual
raise error_metas.pop()[0].to_error( # type: ignore[index]
AssertionError: Tensor-likes are not close!
Mismatched elements: 3 / 3072 (0.1%)
Greatest absolute difference: 0.006591796875 at index (0, 465) (up to 1e-05 allowed)
Greatest relative difference: 7.1619579102844e-06 at index (0, 465) (up to 1.3e-06 allowed)
Params not equal at iteration 4
To execute this test, run the following from the base repo dir:
PYTORCH_TEST_WITH_ROCM=1 python test/distributed/test_distributed_spawn.py TestDistBackendWithSpawn.test_ddp_apply_optim_in_backward_grad_as_bucket_view_false
This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
Stack trace
Traceback (most recent call last):
File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 770, in wrapper
self._join_processes(fn)
File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1039, in _join_processes
self._check_return_codes(fn, elapsed_time)
File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 1079, in _check_return_codes
raise RuntimeError(error)
RuntimeError: Process 0 exited with error code 10 and exception:
Traceback (most recent call last):
File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
getattr(self, test_name)()
File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
fn()
File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
method(*args, **kwargs)
File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 227, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 4886, in test_ddp_apply_optim_in_backward_grad_as_bucket_view_false
self._test_ddp_apply_optim_in_backward(
File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 4858, in _test_ddp_apply_optim_in_backward
self.assertEqual(
File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 4284, in assertEqual
raise error_metas.pop()[0].to_error( # type: ignore[index]
AssertionError: Tensor-likes are not close!
Mismatched elements: 3 / 3072 (0.1%)
Greatest absolute difference: 0.006591796875 at index (0, 465) (up to 1e-05 allowed)
Greatest relative difference: 7.1619579102844e-06 at index (0, 465) (up to 1.3e-06 allowed)
Params not equal at iteration 4
To execute this test, run the following from the base repo dir:
PYTORCH_TEST_WITH_ROCM=1 python test/distributed/test_distributed_spawn.py TestDistBackendWithSpawn.test_ddp_apply_optim_in_backward_grad_as_bucket_view_false
This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
Process 2 exited with error code 10 and exception:
Traceback (most recent call last):
File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
getattr(self, test_name)()
File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
fn()
File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
method(*args, **kwargs)
File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 227, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 4886, in test_ddp_apply_optim_in_backward_grad_as_bucket_view_false
self._test_ddp_apply_optim_in_backward(
File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 4858, in _test_ddp_apply_optim_in_backward
self.assertEqual(
File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 4284, in assertEqual
raise error_metas.pop()[0].to_error( # type: ignore[index]
AssertionError: Tensor-likes are not close!
Mismatched elements: 3 / 3072 (0.1%)
Greatest absolute difference: 0.006591796875 at index (0, 465) (up to 1e-05 allowed)
Greatest relative difference: 7.1619579102844e-06 at index (0, 465) (up to 1.3e-06 allowed)
Params not equal at iteration 4
To execute this test, run the following from the base repo dir:
PYTORCH_TEST_WITH_ROCM=1 python test/distributed/test_distributed_spawn.py TestDistBackendWithSpawn.test_ddp_apply_optim_in_backward_grad_as_bucket_view_false
This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
Process 3 exited with error code 10 and exception:
Traceback (most recent call last):
File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 925, in run_test
getattr(self, test_name)()
File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 772, in wrapper
fn()
File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3329, in wrapper
method(*args, **kwargs)
File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_distributed.py", line 227, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 4886, in test_ddp_apply_optim_in_backward_grad_as_bucket_view_false
self._test_ddp_apply_optim_in_backward(
File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/distributed/distributed_test.py", line 4858, in _test_ddp_apply_optim_in_backward
self.assertEqual(
File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 4284, in assertEqual
raise error_metas.pop()[0].to_error( # type: ignore[index]
AssertionError: Tensor-likes are not close!
Mismatched elements: 3 / 3072 (0.1%)
Greatest absolute difference: 0.006591796875 at index (0, 465) (up to 1e-05 allowed)
Greatest relative difference: 7.1619579102844e-06 at index (0, 465) (up to 1.3e-06 allowed)
Params not equal at iteration 4
To execute this test, run the following from the base repo dir:
PYTORCH_TEST_WITH_ROCM=1 python test/distributed/test_distributed_spawn.py TestDistBackendWithSpawn.test_ddp_apply_optim_in_backward_grad_as_bucket_view_false
This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
Standard error
/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/backends/cudnn/__init__.py:173: UserWarning: cuDNN Benchmark limit is not supported in MIOpen and will have no effect. (Triggered internally at /var/lib/jenkins/pytorch/torch/csrc/cuda/Module.cpp:1965.)
torch._C._cuda_set_cudnn_benchmark_limit(_benchmark_limit)
Standard out
Process 0 terminated with exit code 10, terminating remaining processes.
Loading