Skip to content

[CI] Parity: classify conv2d_backward ROCm skip as PT2.0 - Convolution#3410

Open
ethanwee1 wants to merge 1 commit into
ROCm:developfrom
ethanwee1:ethanwee/parity-conv2d-category
Open

[CI] Parity: classify conv2d_backward ROCm skip as PT2.0 - Convolution#3410
ethanwee1 wants to merge 1 commit into
ROCm:developfrom
ethanwee1:ethanwee/parity-conv2d-category

[CI] Parity: classify conv2d_backward ROCm skip as PT2.0 - Convolution

b213fd9
Select commit
Loading
Failed to load commit list.
ROCm Repo Management API / Jenkins failed Jul 8, 2026 in 4h 0m 33s

Tests/Test Inductor/Run pytorch_inductor_1: error in 'error' step

Tests / Test PyTorch / Test PyTorch / Run pytorch_test_1 / Shell Script

Error in sh step, with arguments rocminfo.

script returned exit code 1
Build log 
[2026-07-08T17:40:35.365Z] + rocminfo
[2026-07-08T17:40:35.365Z] ROCk module version 6.10.5 is loaded
[2026-07-08T17:40:35.365Z] Unable to open /dev/kfd read-write: Invalid argument
[2026-07-08T17:40:35.365Z] jenkins is member of video group

Tests / Test PyTorch / Test PyTorch / Run pytorch_test_1 / Shell Script

Error in sh step, with arguments ./test_pytorch_test.sh.

script returned exit code 1
Build log 
[2026-07-08T17:46:59.766Z] + ./test_pytorch_test.sh
[2026-07-08T17:46:59.766Z] + [[ -z '' ]]
[2026-07-08T17:46:59.766Z] + DEFAULT_EXECUTION_OPTIONS='-e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts         -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip         --copy-whls'
[2026-07-08T17:46:59.766Z] + source prepare_docker_env.sh -e 'EXTENSION_BUILD_GFXARCH=gfx90a;gfx908;gfx942' -e CI=1 -e TEST_CONFIG=default -e SHARD_NUMBER=1 -e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip --copy-whls
[2026-07-08T17:46:59.766Z] ++ set -ex
[2026-07-08T17:46:59.766Z] ++ set -o pipefail
[2026-07-08T17:46:59.766Z] ++ copy_whls=0
[2026-07-08T17:46:59.766Z] ++ envvars=()
[2026-07-08T17:46:59.766Z] ++ [[ 13 -gt 0 ]]
[2026-07-08T17:46:59.766Z] ++ key=-e
[2026-07-08T17:46:59.766Z] ++ case $key in
[2026-07-08T17:46:59.766Z] ++ envvars+=("-e $2")
[2026-07-08T17:46:59.766Z] ++ shift
[2026-07-08T17:46:59.766Z] ++ shift
[2026-07-08T17:46:59.766Z] ++ [[ 11 -gt 0 ]]
[2026-07-08T17:46:59.766Z] ++ key=-e
[2026-07-08T17:46:59.766Z] ++ case $key in
[2026-07-08T17:46:59.766Z] ++ envvars+=("-e $2")
[2026-07-08T17:46:59.766Z] ++ shift
[2026-07-08T17:46:59.766Z] ++ shift
[2026-07-08T17:46:59.766Z] ++ [[ 9 -gt 0 ]]
[2026-07-08T17:46:59.766Z] ++ key=-e
[2026-07-08T17:46:59.766Z] ++ case $key in
[2026-07-08T17:46:59.766Z] ++ envvars+=("-e $2")
[2026-07-08T17:46:59.766Z] ++ shift
[2026-07-08T17:46:59.766Z] ++ shift
[2026-07-08T17:46:59.766Z] ++ [[ 7 -gt 0 ]]
[2026-07-08T17:46:59.766Z] ++ key=-e
[2026-07-08T17:46:59.766Z] ++ case $key in
[2026-07-08T17:46:59.766Z] ++ envvars+=("-e $2")
[2026-07-08T17:46:59.766Z] ++ shift
[2026-07-08T17:46:59.766Z] ++ shift
[2026-07-08T17:46:59.766Z] ++ [[ 5 -gt 0 ]]
[2026-07-08T17:46:59.766Z] ++ key=-e
[2026-07-08T17:46:59.766Z] ++ case $key in
[2026-07-08T17:46:59.766Z] ++ envvars+=("-e $2")
[2026-07-08T17:46:59.766Z] ++ shift
[2026-07-08T17:46:59.766Z] ++ shift
[2026-07-08T17:46:59.766Z] ++ [[ 3 -gt 0 ]]
[2026-07-08T17:46:59.766Z] ++ key=-e
[2026-07-08T17:46:59.766Z] ++ case $key in
[2026-07-08T17:46:59.766Z] ++ envvars+=("-e $2")
[2026-07-08T17:46:59.766Z] ++ shift
[2026-07-08T17:46:59.766Z] ++ shift
[2026-07-08T17:46:59.766Z] ++ [[ 1 -gt 0 ]]
[2026-07-08T17:46:59.766Z] ++ key=--copy-whls
[2026-07-08T17:46:59.766Z] ++ case $key in
[2026-07-08T17:46:59.766Z] ++ copy_whls=1
[2026-07-08T17:46:59.766Z] ++ shift
[2026-07-08T17:46:59.766Z] ++ [[ 0 -gt 0 ]]
[2026-07-08T17:46:59.766Z] ++ envvars+=("-e PREBUILT=${PREBUILT:-false}")
[2026-07-08T17:46:59.766Z] ++ envvars+=("-e TEST_CORE=${TEST_CORE:-false}")
[2026-07-08T17:46:59.766Z] ++ [[ false == \t\r\u\e ]]
[2026-07-08T17:46:59.766Z] ++ [[ true == \t\r\u\e ]]
[2026-07-08T17:46:59.766Z] ++ envvars+=("-e TEST_CORE=${TEST_CORE}")
[2026-07-08T17:46:59.766Z] ++ docker ps -a
[2026-07-08T17:46:59.766Z] CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES
[2026-07-08T17:46:59.766Z] ++ docker rm -f pytorch-ci-container
[2026-07-08T17:46:59.766Z] ++ docker pull rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942
[2026-07-08T17:47:01.146Z] pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942: Pulling from rocm/pytorch-ci-private
[2026-07-08T17:47:02.016Z] 4c8a34755929: Pulling fs layer
[2026-07-08T17:47:02.016Z] cb259a83ac3d: Pulling fs layer
[2026-07-08T17:47:02.016Z] 99e6462d03e0: Pulling fs layer
[2026-07-08T17:47:02.016Z] 1057fba0c7df: Pulling fs layer
[2026-07-08T17:47:02.016Z] 0e6d9b5f5e8d: Pulling fs layer
[2026-07-08T17:47:02.016Z] 65f281ce2e70: Pulling fs layer
[2026-07-08T17:47:02.016Z] 34e29023481f: Pulling fs layer
[2026-07-08T17:47:02.016Z] 8d78d529f04d: Pulling fs layer
[2026-07-08T17:47:02.016Z] 906ac4de0a2b: Pulling fs layer
[2026-07-08T17:47:02.016Z] c608e00b1282: Pulling fs layer
[2026-07-08T17:47:02.016Z] d7df6379f9cc: Pulling fs layer
[2026-07-08T17:47:02.016Z] 3d56742d201d: Pulling fs layer
[2026-07-08T17:47:02.016Z] 5edf52847c35: Pulling fs layer
[2026-07-08T17:47:02.016Z] 9548298aa5f4: Pulling fs layer
[2026-07-08T17:47:02.016Z] 86cc05da26d1: Pulling fs layer
[2026-07-08T17:47:02.016Z] 747272797eac: Pulling fs layer
[2026-07-08T17:47:02.016Z] db0af168a1a4: Pulling fs layer
[2026-07-08T17:47:02.016Z] ed874654c321: Pulling fs layer
[2026-07-08T17:47:02.016Z] 3d56742d201d: Pulling fs layer
[2026-07-08T17:47:02.016Z] 0a0ffadd8b4f: Pulling fs layer
[2026-07-08T17:47:02.016Z] 8b3d007242f8: Pulling fs layer
[2026-07-08T17:47:02.016Z] dedc784bccba: Pulling fs layer
[2026-07-08T17:47:02.016Z] 27fcad4d96a6: Pulling fs layer
[2026-07-08T17:47:02.016Z] bf3e4cf00cd6: Pulling fs layer
[2026-07-08T17:47:02.016Z] 9ae773ccc4b2: Pulling fs layer
[2026-07-08T17:47:02.016Z] 78cb638d6686: Pulling fs layer
[2026-07-08T17:47:02.016Z] f3058198fba5: Pulling fs layer
[2026-07-08T17:47:02.016Z] f5afebc61536: Pulling fs layer
[2026-07-08T17:47:02.016Z] cccff27d82fb: Pulling fs layer
[2026-07-08T17:47:02.016Z] c0664a7365cb: Pulling fs layer
[2026-07-08T17:47:02.016Z] 07b45639fc6f: Pulling fs layer
[2026-07-08T17:47:02.016Z] b96eaed0e751: Pulling fs layer
[2026-07-08T17:47:02.016Z] f790d498bebb: Pulling fs layer
[2026-07-08T17:47:02.016Z] 95d704643610: Pulling fs layer
[2026-07-08T17:47:02.016Z] 423e26ddb517: Pulling fs layer
[2026-07-08T17:47:02.016Z] 4106a853273f: Pulling fs layer
[2026-07-08T17:47:02.016Z] 5f8dd10c9c78: Pulling fs layer
[2026-07-08T17:47:02.016Z] b09b7d04a8d9: Pulling fs layer
[2026-07-08T17:47:02.016Z] 3d56742d201d: Pulling fs layer
[2026-07-08T17:47:02.016Z] 4cfd21f31c8b: Pulling fs layer
[2026-07-08T17:47:02.016Z] 241cbb1c2f26: Pulling fs layer
[2026-07-08T17:47:02.016Z] f0ef294e4f9b: Pulling fs layer
[2026-07-08T17:47:02.016Z] 4f4fb700ef54: Pulling fs layer
[2026-07-08T17:47:02.016Z] 78a3c5370db4: Pulling fs layer
[2026-07-08T17:47:02.016Z] 42239a12ed73: Pulling fs layer
[2026-07-08T17:47:02.016Z] 3d56742d201d: Pulling fs layer
[2026-07-08T17:47:02.016Z] 284e8cbb8a05: Pulling fs layer
[2026-07-08T17:47:02.016Z] 0fcc3b23bd99: Pulling fs layer
[2026-07-08T17:47:02.016Z] d9a8b776bea7: Pulling fs layer
[2026-07-08T17:47:02.016Z] f391428cfe0f: Pulling fs layer
[2026-07-08T17:47:02.016Z] eab367f1ca39: Pulling fs layer
[2026-07-08T17:47:02.016Z] ba1fabf878bd: Pulling fs layer
[2026-07-08T17:47:02.016Z] ed86e5a5d9b1: Pulling fs layer
[2026-07-08T17:47:02.016Z] 4f4fb700ef54: Pulling fs layer
[2026-07-08T17:47:02.016Z] fef65a673dc9: Pulling fs layer
[2026-07-08T17:47:02.016Z] 3d56742d201d: Pulling fs layer
[2026-07-08T17:47:02.016Z] 8d998c79d309: Pulling fs layer
[2026-07-08T17:47:02.016Z] 99e6462d03e0: Download complete
[2026-07-08T17:47:02.016Z] 69145e0460f6: Download complete
[2026-07-08T17:47:02.016Z] 65f281ce2e70: Download complete
[2026-07-08T17:47:02.016Z] c608e00b1282: Download complete
[2026-07-08T17:47:02.016Z] 906ac4de0a2b: Download complete
[2026-07-08T17:47:02.016Z] db0af168a1a4: Download complete
[2026-07-08T17:47:02.016Z] 34e29023481f: Download complete
[2026-07-08T17:47:02.016Z] 8d78d529f04d: Download complete
[2026-07-08T17:47:02.016Z] b96eaed0e751: Download complete
[2026-07-08T17:47:02.442Z] 07b45639fc6f: Download complete
[2026-07-08T17:47:02.442Z] 3d56742d201d: Download complete
[2026-07-08T17:47:02.442Z] f790d498bebb: Download complete
[2026-07-08T17:47:02.442Z] 9ae773ccc4b2: Download complete
[2026-07-08T17:47:02.442Z] bf3e4cf00cd6: Download complete
[2026-07-08T17:47:02.442Z] d7df6379f9cc: Download complete
[2026-07-08T17:47:02.442Z] 8d998c79d309: Download complete
[2026-07-08T17:47:02.442Z] 747272797eac: Download complete
[2026-07-08T17:47:02.442Z] 27fcad4d96a6: Download complete
[2026-07-08T17:47:02.442Z] 42239a12ed73: Download complete
[2026-07-08T17:47:02.442Z] ed86e5a5d9b1: Download complete
[2026-07-08T17:47:02.442Z] c0664a7365cb: Download complete
[2026-07-08T17:47:02.442Z] 241cbb1c2f26: Download complete
[2026-07-08T17:47:02.442Z] 8b3d007242f8: Download complete
[2026-07-08T17:47:02.442Z] 5f8dd10c9c78: Download complete
[2026-07-08T17:47:02.442Z] fef65a673dc9: Download complete
[2026-07-08T17:47:02.442Z] 86cc05da26d1: Download complete
[2026-07-08T17:47:02.442Z] ed874654c321: Download complete
[2026-07-08T17:47:02.442Z] 0a0ffadd8b4f: Download complete
[2026-07-08T17:47:02.442Z] f0ef294e4f9b: Download complete
[2026-07-08T17:47:02.442Z] 4f4fb700ef54: Download complete
[2026-07-08T17:47:02.442Z] 0e6d9b5f5e8d: Download complete
[2026-07-08T17:47:02.442Z] 9548298aa5f4: Download complete
[2026-07-08T17:47:02.442Z] 423e26ddb517: Download complete
[2026-07-08T17:47:02.442Z] 4cfd21f31c8b: Download complete
[2026-07-08T17:47:02.442Z] f3058198fba5: Download complete
[2026-07-08T17:47:02.442Z] f391428cfe0f: Download complete
[2026-07-08T17:47:02.442Z] eab367f1ca39: Download complete
[2026-07-08T17:47:02.442Z] 78a3c5370db4: Download complete
[2026-07-08T17:47:02.442Z] 284e8cbb8a05: Download complete
[2026-07-08T17:47:02.442Z] 0fcc3b23bd99: Download complete
[2026-07-08T17:47:02.442Z] b09b7d04a8d9: Download complete
[2026-07-08T17:47:02.442Z] f5afebc61536: Download complete
[2026-07-08T17:47:02.845Z] 4106a853273f: Download complete
[2026-07-08T17:47:03.263Z] ba1fabf878bd: Download complete
[2026-07-08T17:47:03.263Z] 4c8a34755929: Download complete
[2026-07-08T17:47:03.263Z] cb259a83ac3d: Download complete
[2026-07-08T17:47:03.698Z] 95d704643610: Download complete
[2026-07-08T17:47:04.097Z] cb259a83ac3d: Pull complete
[2026-07-08T17:47:04.097Z] 99e6462d03e0: Pull complete
[2026-07-08T17:47:06.103Z] cccff27d82fb: Download complete
[2026-07-08T17:47:09.443Z] 1057fba0c7df: Download complete
[2026-07-08T17:47:15.990Z] 1057fba0c7df: Pull complete
[2026-07-08T17:47:15.990Z] 65f281ce2e70: Pull complete
[2026-07-08T17:47:15.990Z] c608e00b1282: Pull complete
[2026-07-08T17:47:16.417Z] 906ac4de0a2b: Pull complete
[2026-07-08T17:47:16.417Z] 34e29023481f: Pull complete
[2026-07-08T17:47:16.417Z] 8d78d529f04d: Pull complete
[2026-07-08T17:47:16.417Z] 3d56742d201d: Pull complete
[2026-07-08T17:47:16.417Z] d7df6379f9cc: Pull complete
[2026-07-08T17:47:16.417Z] 0e6d9b5f5e8d: Pull complete
[2026-07-08T17:47:19.889Z] 78cb638d6686: Download complete
[2026-07-08T17:47:48.736Z] 5edf52847c35: Download complete
[2026-07-08T17:48:23.181Z] db0af168a1a4: Pull complete
[2026-07-08T17:48:23.181Z] 747272797eac: Pull complete
[2026-07-08T17:48:23.181Z] 8b3d007242f8: Pull complete
[2026-07-08T17:48:23.181Z] 86cc05da26d1: Pull complete
[2026-07-08T17:48:23.181Z] ed874654c321: Pull complete
[2026-07-08T17:48:23.181Z] 0a0ffadd8b4f: Pull complete
[2026-07-08T17:48:23.181Z] 9548298aa5f4: Pull complete
[2026-07-08T17:48:23.181Z] 5edf52847c35: Pull complete
[2026-07-08T17:48:57.862Z] d9a8b776bea7: Download complete
[2026-07-08T17:49:47.914Z] dedc784bccba: Download complete
[2026-07-08T17:51:12.582Z] 9ae773ccc4b2: Pull complete
[2026-07-08T17:51:12.582Z] bf3e4cf00cd6: Pull complete
[2026-07-08T17:51:12.582Z] 27fcad4d96a6: Pull complete
[2026-07-08T17:51:12.582Z] dedc784bccba: Pull complete
[2026-07-08T17:51:20.409Z] f3058198fba5: Pull complete
[2026-07-08T17:51:20.409Z] f5afebc61536: Pull complete
[2026-07-08T17:51:20.409Z] 78cb638d6686: Pull complete
[2026-07-08T17:51:21.833Z] b96eaed0e751: Pull complete
[2026-07-08T17:51:21.833Z] 07b45639fc6f: Pull complete
[2026-07-08T17:51:21.833Z] f790d498bebb: Pull complete
[2026-07-08T17:51:21.833Z] c0664a7365cb: Pull complete
[2026-07-08T17:51:21.833Z] cccff27d82fb: Pull complete
[2026-07-08T17:51:22.641Z] 95d704643610: Pull complete
[2026-07-08T17:51:22.641Z] 42239a12ed73: Pull complete
[2026-07-08T17:51:22.641Z] 241cbb1c2f26: Pull complete
[2026-07-08T17:51:22.641Z] 5f8dd10c9c78: Pull complete
[2026-07-08T17:51:22.641Z] f0ef294e4f9b: Pull complete
[2026-07-08T17:51:22.641Z] 4f4fb700ef54: Pull complete
[2026-07-08T17:51:22.641Z] 423e26ddb517: Pull complete
[2026-07-08T17:51:22.641Z] 4cfd21f31c8b: Pull complete
[2026-07-08T17:51:22.641Z] 78a3c5370db4: Pull complete
[2026-07-08T17:51:22.641Z] 284e8cbb8a05: Pull complete
[2026-07-08T17:51:22.641Z] 0fcc3b23bd99: Pull complete
[2026-07-08T17:51:22.641Z] b09b7d04a8d9: Pull complete
[2026-07-08T17:51:22.641Z] 4106a853273f: Pull complete
[2026-07-08T17:52:33.195Z] f391428cfe0f: Pull complete
[2026-07-08T17:52:33.195Z] eab367f1ca39: Pull complete
[2026-07-08T17:52:33.195Z] d9a8b776bea7: Pull complete
[2026-07-08T17:52:33.195Z] ba1fabf878bd: Pull complete
[2026-07-08T17:52:33.195Z] 8d998c79d309: Pull complete
[2026-07-08T17:52:33.195Z] ed86e5a5d9b1: Pull complete
[2026-07-08T17:52:33.195Z] fef65a673dc9: Pull complete
[2026-07-08T17:52:33.195Z] 4c8a34755929: Pull complete
[2026-07-08T17:52:33.195Z] Digest: sha256:b836aa0e221ac85a2184ec24a67b9d5089f3caa45d182cf521dee7d38112aa81
[2026-07-08T17:52:33.195Z] Status: Downloaded newer image for rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942
[2026-07-08T17:52:33.195Z] docker.io/rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942
[2026-07-08T17:52:33.195Z] ++ GPU_FLAGS=
[2026-07-08T17:52:33.195Z] ++ echo 'NODE_LABELS: gfx90a mlir k8s-mi250-1-flkn3 gfx90a mlir k8s-mi250-1-flkn3'
[2026-07-08T17:52:33.195Z] NODE_LABELS: gfx90a mlir k8s-mi250-1-flkn3 gfx90a mlir k8s-mi250-1-flkn3
[2026-07-08T17:52:33.195Z] ++ [[ ! gfx90a mlir k8s-mi250-1-flkn3 =~ build-only ]]
[2026-07-08T17:52:33.195Z] ++ GPU_FLAGS='--group-add video --device=/dev/kfd --device=/dev/dri'
[2026-07-08T17:52:33.195Z] ++++ dirname prepare_docker_env.sh
[2026-07-08T17:52:33.195Z] +++ cd .
[2026-07-08T17:52:33.195Z] +++ pwd
[2026-07-08T17:52:33.195Z] ++ script_dir=/home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3410
[2026-07-08T17:52:33.195Z] ++ docker run --name pytorch-ci-container -e 'EXTENSION_BUILD_GFXARCH=gfx90a;gfx908;gfx942' -e CI=1 -e TEST_CONFIG=default -e SHARD_NUMBER=1 -e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip -e PREBUILT=false -e TEST_CORE=true -e TEST_CORE=true -v /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3410:/host_workspace -v /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3410:/scripts --entrypoint bash -d --network=host --group-add video --device=/dev/kfd --device=/dev/dri --ipc=host --pid=host --shm-size 8G --cap-add=SYS_PTRACE --security-opt seccomp=unconfined rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942 -lc 'sleep infinity'
[2026-07-08T17:52:33.195Z] 5227aea7cd2f74d5138a931d7058fa3445b25397a1e4d17f692be1117f5a0d00
[2026-07-08T17:52:33.195Z] ++ env -i python3 /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3410/pytorch/tools/stats/export_test_times.py
[2026-07-08T17:52:33.195Z] Exporting test times from test-infra
[2026-07-08T17:52:33.195Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-times.json to /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3410/pytorch/.additional_ci_files/test-times.json
[2026-07-08T17:52:33.195Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-class-times.json to /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3410/pytorch/.additional_ci_files/test-class-times.json
[2026-07-08T17:52:33.195Z] ++ python3 /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3410/pytorch/tools/stats/export_test_times.py
[2026-07-08T17:52:33.195Z] Exporting test times from test-infra
[2026-07-08T17:52:33.195Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-times.json to /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3410/pytorch/.additional_ci_files/test-times.json
[2026-07-08T17:52:33.195Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-class-times.json to /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3410/pytorch/.additional_ci_files/test-class-times.json
[2026-07-08T17:52:33.195Z] ++ [[ -d /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3410/pytorch/.additional_ci_files ]]
[2026-07-08T17:52:33.195Z] ++ sed -i -E 's/"\S+rocm\S+"/""/g' /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3410/pytorch/.additional_ci_files/test-times.json
[2026-07-08T17:52:33.195Z] ++ [[ false != \t\r\u\e ]]
[2026-07-08T17:52:33.195Z] ++ docker exec -u root pytorch-ci-container bash -c 'rm -rf /var/lib/jenkins/pytorch'
[2026-07-08T17:52:33.195Z] ++ docker cp /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3410/pytorch pytorch-ci-container:/var/lib/jenkins/pytorch
[2026-07-08T17:52:33.195Z] ++ docker exec -u root pytorch-ci-container bash -c 'chown root:root -R /var/lib/jenkins/pytorch'
[2026-07-08T17:52:33.579Z] ++ [[ 1 -eq 1 ]]
[2026-07-08T17:52:33.579Z] ++ docker cp /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3410/rocm_composable_kernel-7.1.1.dev0+g4266f8-py3-none-any.whl /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3410/torch-2.14.0a0+gitb213fd9-cp312-cp312-linux_x86_64.whl pytorch-ci-container:/var/lib/jenkins/pytorch
[2026-07-08T17:52:33.579Z] docker: 'docker cp' requires 2 arguments
[2026-07-08T17:52:33.579Z] 
[2026-07-08T17:52:33.579Z] Usage:  docker cp [OPTIONS] CONTAINER:SRC_PATH DEST_PATH|-
[2026-07-08T17:52:33.579Z] 	docker cp [OPTIONS] SRC_PATH|- CONTAINER:DEST_PATH
[2026-07-08T17:52:33.579Z] 
[2026-07-08T17:52:33.579Z] Run 'docker cp --help' for more information

Tests / Test PyTorch / Test PyTorch / Run pytorch_test_1 / Error signal

Error in error step, with arguments pytorch_test_1 failed.

pytorch_test_1 failed

Tests / Test PyTorch / Test PyTorch / Run pytorch_test_2 / Shell Script

Error in sh step, with arguments rocminfo.

Timeout has been exceeded
Build log 
[2026-07-08T17:52:41.464Z] + rocminfo
[2026-07-08T17:55:36.891Z] _amdgpu_device_initialize: amdgpu_query_gpu_info_init failed
[2026-07-08T17:58:33.875Z] _amdgpu_device_initialize: amdgpu_query_gpu_info_init failed
[2026-07-08T18:01:29.853Z] _amdgpu_device_initialize: amdgpu_query_gpu_info_init failed
[2026-07-08T18:04:10.517Z] _amdgpu_device_initialize: amdgpu_query_gpu_info_init failed
[2026-07-08T18:07:06.310Z] _amdgpu_device_initialize: amdgpu_query_gpu_info_init failed
[2026-07-08T18:12:11.436Z] Sending interrupt signal to process
[2026-07-08T18:12:18.207Z] Terminated
[2026-07-08T18:12:18.319Z] script returned exit code 143

Tests / Test Distributed / Test Distributed / Run pytorch_distributed_1 / Shell Script

Error in sh step, with arguments rocminfo.

script returned exit code 1
Build log 
[2026-07-08T17:40:40.114Z] + rocminfo
[2026-07-08T17:40:40.114Z] ROCk module version 6.10.5 is loaded
[2026-07-08T17:40:40.114Z] Unable to open /dev/kfd read-write: Invalid argument
[2026-07-08T17:40:40.114Z] jenkins is member of video group

Tests / Test Distributed / Test Distributed / Run pytorch_distributed_1 / Shell Script

Error in sh step, with arguments ./test_pytorch_test_distributed.sh.

script returned exit code 1
Build log 
[2026-07-08T19:41:43.461Z] + ./test_pytorch_test_distributed.sh
[2026-07-08T19:41:43.462Z] + [[ -z '' ]]
[2026-07-08T19:41:43.462Z] + DEFAULT_EXECUTION_OPTIONS='-e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts         -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip         --copy-whls'
[2026-07-08T19:41:43.462Z] + source prepare_docker_env.sh -e 'EXTENSION_BUILD_GFXARCH=gfx90a;gfx908;gfx942' -e CI=1 -e TEST_CONFIG=distributed -e SHARD_NUMBER=1 -e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip --copy-whls
[2026-07-08T19:41:43.462Z] ++ set -ex
[2026-07-08T19:41:43.462Z] ++ set -o pipefail
[2026-07-08T19:41:43.462Z] ++ copy_whls=0
[2026-07-08T19:41:43.462Z] ++ envvars=()
[2026-07-08T19:41:43.462Z] ++ [[ 13 -gt 0 ]]
[2026-07-08T19:41:43.462Z] ++ key=-e
[2026-07-08T19:41:43.462Z] ++ case $key in
[2026-07-08T19:41:43.462Z] ++ envvars+=("-e $2")
[2026-07-08T19:41:43.462Z] ++ shift
[2026-07-08T19:41:43.462Z] ++ shift
[2026-07-08T19:41:43.462Z] ++ [[ 11 -gt 0 ]]
[2026-07-08T19:41:43.462Z] ++ key=-e
[2026-07-08T19:41:43.462Z] ++ case $key in
[2026-07-08T19:41:43.462Z] ++ envvars+=("-e $2")
[2026-07-08T19:41:43.462Z] ++ shift
[2026-07-08T19:41:43.462Z] ++ shift
[2026-07-08T19:41:43.462Z] ++ [[ 9 -gt 0 ]]
[2026-07-08T19:41:43.462Z] ++ key=-e
[2026-07-08T19:41:43.462Z] ++ case $key in
[2026-07-08T19:41:43.462Z] ++ envvars+=("-e $2")
[2026-07-08T19:41:43.462Z] ++ shift
[2026-07-08T19:41:43.462Z] ++ shift
[2026-07-08T19:41:43.462Z] ++ [[ 7 -gt 0 ]]
[2026-07-08T19:41:43.462Z] ++ key=-e
[2026-07-08T19:41:43.462Z] ++ case $key in
[2026-07-08T19:41:43.462Z] ++ envvars+=("-e $2")
[2026-07-08T19:41:43.462Z] ++ shift
[2026-07-08T19:41:43.462Z] ++ shift
[2026-07-08T19:41:43.462Z] ++ [[ 5 -gt 0 ]]
[2026-07-08T19:41:43.462Z] ++ key=-e
[2026-07-08T19:41:43.462Z] ++ case $key in
[2026-07-08T19:41:43.462Z] ++ envvars+=("-e $2")
[2026-07-08T19:41:43.462Z] ++ shift
[2026-07-08T19:41:43.462Z] ++ shift
[2026-07-08T19:41:43.462Z] ++ [[ 3 -gt 0 ]]
[2026-07-08T19:41:43.462Z] ++ key=-e
[2026-07-08T19:41:43.462Z] ++ case $key in
[2026-07-08T19:41:43.462Z] ++ envvars+=("-e $2")
[2026-07-08T19:41:43.462Z] ++ shift
[2026-07-08T19:41:43.462Z] ++ shift
[2026-07-08T19:41:43.462Z] ++ [[ 1 -gt 0 ]]
[2026-07-08T19:41:43.462Z] ++ key=--copy-whls
[2026-07-08T19:41:43.462Z] ++ case $key in
[2026-07-08T19:41:43.462Z] ++ copy_whls=1
[2026-07-08T19:41:43.462Z] ++ shift
[2026-07-08T19:41:43.462Z] ++ [[ 0 -gt 0 ]]
[2026-07-08T19:41:43.462Z] ++ envvars+=("-e PREBUILT=${PREBUILT:-false}")
[2026-07-08T19:41:43.462Z] ++ envvars+=("-e TEST_CORE=${TEST_CORE:-false}")
[2026-07-08T19:41:43.462Z] ++ [[ false == \t\r\u\e ]]
[2026-07-08T19:41:43.462Z] ++ [[ true == \t\r\u\e ]]
[2026-07-08T19:41:43.462Z] ++ envvars+=("-e TEST_CORE=${TEST_CORE}")
[2026-07-08T19:41:43.462Z] ++ docker ps -a
[2026-07-08T19:41:43.462Z] CONTAINER ID   IMAGE                                                                                                                        COMMAND                  CREATED       STATUS                   PORTS                                                 NAMES
[2026-07-08T19:41:43.462Z] f44846c58189   rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-1ae06b266a82a9290b48a6f384f86d9bfcc747b2-gfx908_gfx90a_gfx942   "bash -lc 'sleep inf…"   2 hours ago   Up 2 hours                                                                     pytorch-ci-container
[2026-07-08T19:41:43.462Z] 33af8c9e058f   dashboard-dashboard-ui                                                                                                       "/docker-entrypoint.…"   2 weeks ago   Up 19 hours              80/tcp, 0.0.0.0:3000->3000/tcp, [::]:3000->3000/tcp   rocagents-dashboard-ui
[2026-07-08T19:41:43.462Z] 7c6be83a2db1   dashboard-dashboard-api                                                                                                      "uvicorn main:app --…"   2 weeks ago   Up 19 hours (healthy)    0.0.0.0:8080->8080/tcp, [::]:8080->8080/tcp           rocagents-dashboard-api
[2026-07-08T19:41:43.462Z] 780dc8fe5a7b   rocm/eigen-test:ubuntu20.04-rocm5.3.0-eigen                                                                                  "/scripts/run_tests_…"   5 weeks ago   Exited (1) 5 weeks ago                                                         infallible_chatelet
[2026-07-08T19:41:43.462Z] 2d7786cfd066   deployment-rocagents                                                                                                         "python src/multi_re…"   8 weeks ago   Up 19 hours (healthy)                                                          rocagents-orchestrator
[2026-07-08T19:41:43.462Z] ++ docker rm -f pytorch-ci-container
[2026-07-08T19:41:46.750Z] ++ docker pull rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942
[2026-07-08T19:41:46.750Z] pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942: Pulling from rocm/pytorch-ci-private
[2026-07-08T19:41:46.750Z] Digest: sha256:b836aa0e221ac85a2184ec24a67b9d5089f3caa45d182cf521dee7d38112aa81
[2026-07-08T19:41:46.750Z] Status: Image is up to date for rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942
[2026-07-08T19:41:46.750Z] docker.io/rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942
[2026-07-08T19:41:46.750Z] ++ GPU_FLAGS=
[2026-07-08T19:41:46.750Z] ++ echo 'NODE_LABELS: multi_gpu rocm dlmodels gpu_8 gfx90a ixt-hq-ubb4-31 mlir MI250 multi_gpu rocm dlmodels gpu_8 gfx90a ixt-hq-ubb4-31 mlir MI250'
[2026-07-08T19:41:46.750Z] NODE_LABELS: multi_gpu rocm dlmodels gpu_8 gfx90a ixt-hq-ubb4-31 mlir MI250 multi_gpu rocm dlmodels gpu_8 gfx90a ixt-hq-ubb4-31 mlir MI250
[2026-07-08T19:41:46.750Z] ++ [[ ! multi_gpu rocm dlmodels gpu_8 gfx90a ixt-hq-ubb4-31 mlir MI250 =~ build-only ]]
[2026-07-08T19:41:46.750Z] ++ GPU_FLAGS='--group-add video --device=/dev/kfd --device=/dev/dri'
[2026-07-08T19:41:46.750Z] ++++ dirname prepare_docker_env.sh
[2026-07-08T19:41:46.750Z] +++ cd .
[2026-07-08T19:41:46.750Z] +++ pwd
[2026-07-08T19:41:46.750Z] ++ script_dir=/home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3410
[2026-07-08T19:41:46.750Z] ++ docker run --name pytorch-ci-container -e 'EXTENSION_BUILD_GFXARCH=gfx90a;gfx908;gfx942' -e CI=1 -e TEST_CONFIG=distributed -e SHARD_NUMBER=1 -e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip -e PREBUILT=false -e TEST_CORE=true -e TEST_CORE=true -v /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3410:/host_workspace -v /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3410:/scripts --entrypoint bash -d --network=host --group-add video --device=/dev/kfd --device=/dev/dri --ipc=host --pid=host --shm-size 8G --cap-add=SYS_PTRACE --security-opt seccomp=unconfined rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942 -lc 'sleep infinity'
[2026-07-08T19:41:47.284Z] a907f066d33bcef218afb998f0466e0bafac20c9cd1cd7a6abb9d6a95cd7e911
[2026-07-08T19:41:47.284Z] ++ env -i python3 /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3410/pytorch/tools/stats/export_test_times.py
[2026-07-08T19:41:47.827Z] Exporting test times from test-infra
[2026-07-08T19:41:47.828Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-times.json to /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3410/pytorch/.additional_ci_files/test-times.json
[2026-07-08T19:41:47.828Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-class-times.json to /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3410/pytorch/.additional_ci_files/test-class-times.json
[2026-07-08T19:41:47.828Z] ++ python3 /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3410/pytorch/tools/stats/export_test_times.py
[2026-07-08T19:41:47.828Z] Exporting test times from test-infra
[2026-07-08T19:41:47.828Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-times.json to /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3410/pytorch/.additional_ci_files/test-times.json
[2026-07-08T19:41:47.828Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-class-times.json to /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3410/pytorch/.additional_ci_files/test-class-times.json
[2026-07-08T19:41:47.828Z] ++ [[ -d /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3410/pytorch/.additional_ci_files ]]
[2026-07-08T19:41:47.828Z] ++ sed -i -E 's/"\S+rocm\S+"/""/g' /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3410/pytorch/.additional_ci_files/test-times.json
[2026-07-08T19:41:47.828Z] ++ [[ false != \t\r\u\e ]]
[2026-07-08T19:41:47.828Z] ++ docker exec -u root pytorch-ci-container bash -c 'rm -rf /var/lib/jenkins/pytorch'
[2026-07-08T19:41:47.828Z] ++ docker cp /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3410/pytorch pytorch-ci-container:/var/lib/jenkins/pytorch
[2026-07-08T19:41:52.216Z] ++ docker exec -u root pytorch-ci-container bash -c 'chown root:root -R /var/lib/jenkins/pytorch'
[2026-07-08T19:41:52.216Z] ++ [[ 1 -eq 1 ]]
[2026-07-08T19:41:52.216Z] ++ docker cp /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3410/rocm_composable_kernel-7.1.1.dev0+g4266f8-py3-none-any.whl /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3410/torch-2.14.0a0+gitb213fd9-cp312-cp312-linux_x86_64.whl pytorch-ci-container:/var/lib/jenkins/pytorch
[2026-07-08T19:41:52.216Z] docker: 'docker cp' requires 2 arguments
[2026-07-08T19:41:52.216Z] 
[2026-07-08T19:41:52.216Z] Usage:  docker cp [OPTIONS] CONTAINER:SRC_PATH DEST_PATH|-
[2026-07-08T19:41:52.216Z] 	docker cp [OPTIONS] SRC_PATH|- CONTAINER:DEST_PATH
[2026-07-08T19:41:52.216Z] 
[2026-07-08T19:41:52.216Z] Run 'docker cp --help' for more information

Tests / Test Distributed / Test Distributed / Run pytorch_distributed_1 / Error signal

Error in error step, with arguments pytorch_distributed_1 failed.

pytorch_distributed_1 failed

Tests / Test Distributed / Test Distributed / Run pytorch_distributed_2 / Shell Script

Error in sh step, with arguments rocminfo.

Timeout has been exceeded
Build log 
[2026-07-08T19:41:59.799Z] + rocminfo
[2026-07-08T19:44:55.825Z] _amdgpu_device_initialize: amdgpu_query_gpu_info_init failed
[2026-07-08T19:47:37.910Z] _amdgpu_device_initialize: amdgpu_query_gpu_info_init failed
[2026-07-08T19:50:33.486Z] _amdgpu_device_initialize: amdgpu_query_gpu_info_init failed
[2026-07-08T19:55:59.765Z] Sending interrupt signal to process
[2026-07-08T19:56:06.560Z] Terminated
[2026-07-08T19:56:06.649Z] script returned exit code 143

Tests / Test Inductor / Test Inductor / Run pytorch_inductor_1 / Shell Script

Error in sh step, with arguments rocminfo.

script returned exit code 1
Build log 
[2026-07-08T17:40:45.281Z] + rocminfo
[2026-07-08T17:40:59.241Z] ROCk module version 6.10.5 is loaded
[2026-07-08T17:40:59.241Z] Unable to open /dev/kfd read-write: Invalid argument
[2026-07-08T17:40:59.241Z] jenkins is member of video group

Tests / Test Inductor / Test Inductor / Run pytorch_inductor_1 / Shell Script

Error in sh step, with arguments ./test_pytorch_inductor.sh.

script returned exit code 1
Build log 
[2026-07-08T19:26:38.370Z] + ./test_pytorch_inductor.sh
[2026-07-08T19:26:38.370Z] + [[ -z '' ]]
[2026-07-08T19:26:38.370Z] + DEFAULT_EXECUTION_OPTIONS='-e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts         -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip         --copy-whls'
[2026-07-08T19:26:38.370Z] + source prepare_docker_env.sh -e 'EXTENSION_BUILD_GFXARCH=gfx90a;gfx908;gfx942' -e CI=1 -e TEST_CONFIG=inductor -e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip --copy-whls
[2026-07-08T19:26:38.370Z] ++ set -ex
[2026-07-08T19:26:38.370Z] ++ set -o pipefail
[2026-07-08T19:26:38.370Z] ++ copy_whls=0
[2026-07-08T19:26:38.370Z] ++ envvars=()
[2026-07-08T19:26:38.370Z] ++ [[ 11 -gt 0 ]]
[2026-07-08T19:26:38.370Z] ++ key=-e
[2026-07-08T19:26:38.370Z] ++ case $key in
[2026-07-08T19:26:38.370Z] ++ envvars+=("-e $2")
[2026-07-08T19:26:38.370Z] ++ shift
[2026-07-08T19:26:38.370Z] ++ shift
[2026-07-08T19:26:38.370Z] ++ [[ 9 -gt 0 ]]
[2026-07-08T19:26:38.370Z] ++ key=-e
[2026-07-08T19:26:38.370Z] ++ case $key in
[2026-07-08T19:26:38.370Z] ++ envvars+=("-e $2")
[2026-07-08T19:26:38.370Z] ++ shift
[2026-07-08T19:26:38.370Z] ++ shift
[2026-07-08T19:26:38.370Z] ++ [[ 7 -gt 0 ]]
[2026-07-08T19:26:38.370Z] ++ key=-e
[2026-07-08T19:26:38.370Z] ++ case $key in
[2026-07-08T19:26:38.370Z] ++ envvars+=("-e $2")
[2026-07-08T19:26:38.370Z] ++ shift
[2026-07-08T19:26:38.370Z] ++ shift
[2026-07-08T19:26:38.370Z] ++ [[ 5 -gt 0 ]]
[2026-07-08T19:26:38.370Z] ++ key=-e
[2026-07-08T19:26:38.370Z] ++ case $key in
[2026-07-08T19:26:38.370Z] ++ envvars+=("-e $2")
[2026-07-08T19:26:38.370Z] ++ shift
[2026-07-08T19:26:38.370Z] ++ shift
[2026-07-08T19:26:38.370Z] ++ [[ 3 -gt 0 ]]
[2026-07-08T19:26:38.370Z] ++ key=-e
[2026-07-08T19:26:38.370Z] ++ case $key in
[2026-07-08T19:26:38.370Z] ++ envvars+=("-e $2")
[2026-07-08T19:26:38.370Z] ++ shift
[2026-07-08T19:26:38.370Z] ++ shift
[2026-07-08T19:26:38.370Z] ++ [[ 1 -gt 0 ]]
[2026-07-08T19:26:38.370Z] ++ key=--copy-whls
[2026-07-08T19:26:38.370Z] ++ case $key in
[2026-07-08T19:26:38.370Z] ++ copy_whls=1
[2026-07-08T19:26:38.370Z] ++ shift
[2026-07-08T19:26:38.370Z] ++ [[ 0 -gt 0 ]]
[2026-07-08T19:26:38.370Z] ++ envvars+=("-e PREBUILT=${PREBUILT:-false}")
[2026-07-08T19:26:38.370Z] ++ envvars+=("-e TEST_CORE=${TEST_CORE:-false}")
[2026-07-08T19:26:38.370Z] ++ [[ false == \t\r\u\e ]]
[2026-07-08T19:26:38.370Z] ++ [[ true == \t\r\u\e ]]
[2026-07-08T19:26:38.370Z] ++ envvars+=("-e TEST_CORE=${TEST_CORE}")
[2026-07-08T19:26:38.370Z] ++ docker ps -a
[2026-07-08T19:26:38.370Z] CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES
[2026-07-08T19:26:38.370Z] ++ docker rm -f pytorch-ci-container
[2026-07-08T19:26:38.370Z] ++ docker pull rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942
[2026-07-08T19:26:39.304Z] pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942: Pulling from rocm/pytorch-ci-private
[2026-07-08T19:26:40.230Z] cb259a83ac3d: Pulling fs layer
[2026-07-08T19:26:40.230Z] 1057fba0c7df: Pulling fs layer
[2026-07-08T19:26:40.230Z] 3d56742d201d: Pulling fs layer
[2026-07-08T19:26:40.230Z] db0af168a1a4: Pulling fs layer
[2026-07-08T19:26:40.230Z] 5edf52847c35: Pulling fs layer
[2026-07-08T19:26:40.230Z] 9ae773ccc4b2: Pulling fs layer
[2026-07-08T19:26:40.230Z] 747272797eac: Pulling fs layer
[2026-07-08T19:26:40.230Z] eab367f1ca39: Pulling fs layer
[2026-07-08T19:26:40.230Z] cccff27d82fb: Pulling fs layer
[2026-07-08T19:26:40.230Z] 9548298aa5f4: Pulling fs layer
[2026-07-08T19:26:40.230Z] 4cfd21f31c8b: Pulling fs layer
[2026-07-08T19:26:40.230Z] 78cb638d6686: Pulling fs layer
[2026-07-08T19:26:40.230Z] 42239a12ed73: Pulling fs layer
[2026-07-08T19:26:40.230Z] dedc784bccba: Pulling fs layer
[2026-07-08T19:26:40.230Z] d9a8b776bea7: Pulling fs layer
[2026-07-08T19:26:40.230Z] f790d498bebb: Pulling fs layer
[2026-07-08T19:26:40.230Z] 4f4fb700ef54: Pulling fs layer
[2026-07-08T19:26:40.230Z] 284e8cbb8a05: Pulling fs layer
[2026-07-08T19:26:40.230Z] 99e6462d03e0: Pulling fs layer
[2026-07-08T19:26:40.230Z] 4c8a34755929: Pulling fs layer
[2026-07-08T19:26:40.230Z] ed874654c321: Pulling fs layer
[2026-07-08T19:26:40.230Z] 3d56742d201d: Pulling fs layer
[2026-07-08T19:26:40.230Z] 0fcc3b23bd99: Pulling fs layer
[2026-07-08T19:26:40.230Z] f0ef294e4f9b: Pulling fs layer
[2026-07-08T19:26:40.230Z] 34e29023481f: Pulling fs layer
[2026-07-08T19:26:40.230Z] 65f281ce2e70: Pulling fs layer
[2026-07-08T19:26:40.230Z] d7df6379f9cc: Pulling fs layer
[2026-07-08T19:26:40.230Z] 0e6d9b5f5e8d: Pulling fs layer
[2026-07-08T19:26:40.230Z] 4106a853273f: Pulling fs layer
[2026-07-08T19:26:40.230Z] 95d704643610: Pulling fs layer
[2026-07-08T19:26:40.230Z] 8b3d007242f8: Pulling fs layer
[2026-07-08T19:26:40.230Z] 3d56742d201d: Pulling fs layer
[2026-07-08T19:26:40.230Z] 27fcad4d96a6: Pulling fs layer
[2026-07-08T19:26:40.230Z] 86cc05da26d1: Pulling fs layer
[2026-07-08T19:26:40.230Z] 8d78d529f04d: Pulling fs layer
[2026-07-08T19:26:40.230Z] 0a0ffadd8b4f: Pulling fs layer
[2026-07-08T19:26:40.230Z] bf3e4cf00cd6: Pulling fs layer
[2026-07-08T19:26:40.230Z] ed86e5a5d9b1: Pulling fs layer
[2026-07-08T19:26:40.230Z] fef65a673dc9: Pulling fs layer
[2026-07-08T19:26:40.230Z] b09b7d04a8d9: Pulling fs layer
[2026-07-08T19:26:40.230Z] f391428cfe0f: Pulling fs layer
[2026-07-08T19:26:40.230Z] 8d998c79d309: Pulling fs layer
[2026-07-08T19:26:40.230Z] 3d56742d201d: Pulling fs layer
[2026-07-08T19:26:40.230Z] c0664a7365cb: Pulling fs layer
[2026-07-08T19:26:40.230Z] 07b45639fc6f: Pulling fs layer
[2026-07-08T19:26:40.230Z] f3058198fba5: Pulling fs layer
[2026-07-08T19:26:40.230Z] c608e00b1282: Pulling fs layer
[2026-07-08T19:26:40.230Z] b96eaed0e751: Pulling fs layer
[2026-07-08T19:26:40.230Z] 3d56742d201d: Pulling fs layer
[2026-07-08T19:26:40.230Z] ba1fabf878bd: Pulling fs layer
[2026-07-08T19:26:40.230Z] f5afebc61536: Pulling fs layer
[2026-07-08T19:26:40.230Z] 241cbb1c2f26: Pulling fs layer
[2026-07-08T19:26:40.230Z] 78a3c5370db4: Pulling fs layer
[2026-07-08T19:26:40.230Z] 4f4fb700ef54: Pulling fs layer
[2026-07-08T19:26:40.230Z] 5f8dd10c9c78: Pulling fs layer
[2026-07-08T19:26:40.230Z] 906ac4de0a2b: Pulling fs layer
[2026-07-08T19:26:40.230Z] 423e26ddb517: Pulling fs layer
[2026-07-08T19:26:40.230Z] 69145e0460f6: Download complete
[2026-07-08T19:26:40.662Z] 9ae773ccc4b2: Download complete
[2026-07-08T19:26:40.662Z] eab367f1ca39: Download complete
[2026-07-08T19:26:40.662Z] db0af168a1a4: Download complete
[2026-07-08T19:26:40.662Z] 3d56742d201d: Download complete
[2026-07-08T19:26:40.662Z] 747272797eac: Download complete
[2026-07-08T19:26:40.662Z] f5afebc61536: Download complete
[2026-07-08T19:26:40.662Z] 86cc05da26d1: Download complete
[2026-07-08T19:26:40.662Z] b96eaed0e751: Download complete
[2026-07-08T19:26:40.662Z] f391428cfe0f: Download complete
[2026-07-08T19:26:40.662Z] f790d498bebb: Download complete
[2026-07-08T19:26:40.662Z] 42239a12ed73: Download complete
[2026-07-08T19:26:40.662Z] 78a3c5370db4: Download complete
[2026-07-08T19:26:40.662Z] bf3e4cf00cd6: Download complete
[2026-07-08T19:26:40.662Z] 4cfd21f31c8b: Download complete
[2026-07-08T19:26:40.662Z] c608e00b1282: Download complete
[2026-07-08T19:26:40.662Z] 27fcad4d96a6: Download complete
[2026-07-08T19:26:40.662Z] fef65a673dc9: Download complete
[2026-07-08T19:26:40.662Z] 07b45639fc6f: Download complete
[2026-07-08T19:26:40.662Z] 4f4fb700ef54: Download complete
[2026-07-08T19:26:41.106Z] 9548298aa5f4: Download complete
[2026-07-08T19:26:41.106Z] b09b7d04a8d9: Download complete
[2026-07-08T19:26:41.106Z] f0ef294e4f9b: Download complete
[2026-07-08T19:26:41.106Z] 284e8cbb8a05: Download complete
[2026-07-08T19:26:41.106Z] f3058198fba5: Download complete
[2026-07-08T19:26:41.106Z] 241cbb1c2f26: Download complete
[2026-07-08T19:26:41.106Z] 0fcc3b23bd99: Download complete
[2026-07-08T19:26:41.106Z] 423e26ddb517: Download complete
[2026-07-08T19:26:41.106Z] 8b3d007242f8: Download complete
[2026-07-08T19:26:41.106Z] ed874654c321: Download complete
[2026-07-08T19:26:41.106Z] 0a0ffadd8b4f: Download complete
[2026-07-08T19:26:41.106Z] 5f8dd10c9c78: Download complete
[2026-07-08T19:26:41.106Z] 34e29023481f: Download complete
[2026-07-08T19:26:41.106Z] 8d78d529f04d: Download complete
[2026-07-08T19:26:41.106Z] c0664a7365cb: Download complete
[2026-07-08T19:26:41.106Z] 8d998c79d309: Download complete
[2026-07-08T19:26:41.106Z] ed86e5a5d9b1: Download complete
[2026-07-08T19:26:41.106Z] 906ac4de0a2b: Download complete
[2026-07-08T19:26:41.106Z] d7df6379f9cc: Download complete
[2026-07-08T19:26:41.106Z] 99e6462d03e0: Download complete
[2026-07-08T19:26:41.106Z] 65f281ce2e70: Download complete
[2026-07-08T19:26:41.106Z] 0e6d9b5f5e8d: Download complete
[2026-07-08T19:26:41.106Z] 4106a853273f: Download complete
[2026-07-08T19:26:41.106Z] cb259a83ac3d: Download complete
[2026-07-08T19:26:41.531Z] ba1fabf878bd: Download complete
[2026-07-08T19:26:41.941Z] cb259a83ac3d: Pull complete
[2026-07-08T19:26:41.941Z] 4c8a34755929: Download complete
[2026-07-08T19:26:41.941Z] 99e6462d03e0: Pull complete
[2026-07-08T19:26:42.405Z] 95d704643610: Download complete
[2026-07-08T19:26:43.924Z] cccff27d82fb: Download complete
[2026-07-08T19:26:48.062Z] 1057fba0c7df: Download complete
[2026-07-08T19:26:55.626Z] 1057fba0c7df: Pull complete
[2026-07-08T19:26:55.626Z] 3d56742d201d: Pull complete
[2026-07-08T19:26:55.626Z] c608e00b1282: Pull complete
[2026-07-08T19:26:55.626Z] 34e29023481f: Pull complete
[2026-07-08T19:26:55.626Z] 8d78d529f04d: Pull complete
[2026-07-08T19:26:55.626Z] 906ac4de0a2b: Pull complete
[2026-07-08T19:26:55.626Z] d7df6379f9cc: Pull complete
[2026-07-08T19:26:55.626Z] 65f281ce2e70: Pull complete
[2026-07-08T19:26:55.626Z] 0e6d9b5f5e8d: Pull complete
[2026-07-08T19:26:59.306Z] 78cb638d6686: Download complete
[2026-07-08T19:27:30.403Z] 5edf52847c35: Download complete
[2026-07-08T19:28:06.823Z] db0af168a1a4: Pull complete
[2026-07-08T19:28:06.823Z] 747272797eac: Pull complete
[2026-07-08T19:28:06.823Z] 86cc05da26d1: Pull complete
[2026-07-08T19:28:06.823Z] 9548298aa5f4: Pull complete
[2026-07-08T19:28:06.823Z] 8b3d007242f8: Pull complete
[2026-07-08T19:28:06.823Z] ed874654c321: Pull complete
[2026-07-08T19:28:06.823Z] 0a0ffadd8b4f: Pull complete
[2026-07-08T19:28:06.823Z] 5edf52847c35: Pull complete
[2026-07-08T19:28:40.744Z] d9a8b776bea7: Download complete
[2026-07-08T19:29:32.755Z] dedc784bccba: Download complete
[2026-07-08T19:31:00.667Z] 9ae773ccc4b2: Pull complete
[2026-07-08T19:31:00.667Z] bf3e4cf00cd6: Pull complete
[2026-07-08T19:31:00.667Z] 27fcad4d96a6: Pull complete
[2026-07-08T19:31:00.667Z] dedc784bccba: Pull complete
[2026-07-08T19:31:00.667Z] f5afebc61536: Pull complete
[2026-07-08T19:31:00.667Z] f3058198fba5: Pull complete
[2026-07-08T19:31:00.667Z] 78cb638d6686: Pull complete
[2026-07-08T19:31:01.885Z] b96eaed0e751: Pull complete
[2026-07-08T19:31:01.885Z] f790d498bebb: Pull complete
[2026-07-08T19:31:01.885Z] 07b45639fc6f: Pull complete
[2026-07-08T19:31:01.885Z] c0664a7365cb: Pull complete
[2026-07-08T19:31:01.885Z] cccff27d82fb: Pull complete
[2026-07-08T19:31:02.342Z] 95d704643610: Pull complete
[2026-07-08T19:31:02.814Z] 42239a12ed73: Pull complete
[2026-07-08T19:31:02.814Z] 78a3c5370db4: Pull complete
[2026-07-08T19:31:02.814Z] 4cfd21f31c8b: Pull complete
[2026-07-08T19:31:02.814Z] 4f4fb700ef54: Pull complete
[2026-07-08T19:31:02.814Z] b09b7d04a8d9: Pull complete
[2026-07-08T19:31:02.814Z] f0ef294e4f9b: Pull complete
[2026-07-08T19:31:02.814Z] 284e8cbb8a05: Pull complete
[2026-07-08T19:31:02.814Z] 241cbb1c2f26: Pull complete
[2026-07-08T19:31:02.814Z] 0fcc3b23bd99: Pull complete
[2026-07-08T19:31:02.814Z] 423e26ddb517: Pull complete
[2026-07-08T19:31:02.814Z] 5f8dd10c9c78: Pull complete
[2026-07-08T19:31:02.814Z] 4106a853273f: Pull complete
[2026-07-08T19:32:08.416Z] eab367f1ca39: Pull complete
[2026-07-08T19:32:08.416Z] f391428cfe0f: Pull complete
[2026-07-08T19:32:08.416Z] d9a8b776bea7: Pull complete
[2026-07-08T19:32:08.416Z] fef65a673dc9: Pull complete
[2026-07-08T19:32:08.416Z] 8d998c79d309: Pull complete
[2026-07-08T19:32:08.416Z] ed86e5a5d9b1: Pull complete
[2026-07-08T19:32:08.416Z] ba1fabf878bd: Pull complete
[2026-07-08T19:32:08.416Z] 4c8a34755929: Pull complete
[2026-07-08T19:32:08.416Z] Digest: sha256:b836aa0e221ac85a2184ec24a67b9d5089f3caa45d182cf521dee7d38112aa81
[2026-07-08T19:32:08.416Z] Status: Downloaded newer image for rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942
[2026-07-08T19:32:08.416Z] docker.io/rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942
[2026-07-08T19:32:08.416Z] ++ GPU_FLAGS=
[2026-07-08T19:32:08.416Z] ++ echo 'NODE_LABELS: gfx90a k8s-mi250-1-t3wl9 mlir gfx90a k8s-mi250-1-t3wl9 mlir'
[2026-07-08T19:32:08.416Z] NODE_LABELS: gfx90a k8s-mi250-1-t3wl9 mlir gfx90a k8s-mi250-1-t3wl9 mlir
[2026-07-08T19:32:08.416Z] ++ [[ ! gfx90a k8s-mi250-1-t3wl9 mlir =~ build-only ]]
[2026-07-08T19:32:08.416Z] ++ GPU_FLAGS='--group-add video --device=/dev/kfd --device=/dev/dri'
[2026-07-08T19:32:08.416Z] ++++ dirname prepare_docker_env.sh
[2026-07-08T19:32:08.416Z] +++ cd .
[2026-07-08T19:32:08.416Z] +++ pwd
[2026-07-08T19:32:08.416Z] ++ script_dir=/home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3410
[2026-07-08T19:32:08.416Z] ++ docker run --name pytorch-ci-container -e 'EXTENSION_BUILD_GFXARCH=gfx90a;gfx908;gfx942' -e CI=1 -e TEST_CONFIG=inductor -e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip -e PREBUILT=false -e TEST_CORE=true -e TEST_CORE=true -v /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3410:/host_workspace -v /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3410:/scripts --entrypoint bash -d --network=host --group-add video --device=/dev/kfd --device=/dev/dri --ipc=host --pid=host --shm-size 8G --cap-add=SYS_PTRACE --security-opt seccomp=unconfined rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942 -lc 'sleep infinity'
[2026-07-08T19:32:11.517Z] e356bf72c84b1f40852ffcea2778a979c795d215a6897603ef582a868df3bf20
[2026-07-08T19:32:12.612Z] ++ env -i python3 /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3410/pytorch/tools/stats/export_test_times.py
[2026-07-08T19:32:18.305Z] Exporting test times from test-infra
[2026-07-08T19:32:18.305Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-times.json to /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3410/pytorch/.additional_ci_files/test-times.json
[2026-07-08T19:32:18.305Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-class-times.json to /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3410/pytorch/.additional_ci_files/test-class-times.json
[2026-07-08T19:32:18.305Z] ++ python3 /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3410/pytorch/tools/stats/export_test_times.py
[2026-07-08T19:32:18.305Z] Exporting test times from test-infra
[2026-07-08T19:32:18.305Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-times.json to /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3410/pytorch/.additional_ci_files/test-times.json
[2026-07-08T19:32:18.305Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-class-times.json to /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3410/pytorch/.additional_ci_files/test-class-times.json
[2026-07-08T19:32:18.305Z] ++ [[ -d /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3410/pytorch/.additional_ci_files ]]
[2026-07-08T19:32:18.305Z] ++ sed -i -E 's/"\S+rocm\S+"/""/g' /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3410/pytorch/.additional_ci_files/test-times.json
[2026-07-08T19:32:18.305Z] ++ [[ false != \t\r\u\e ]]
[2026-07-08T19:32:18.305Z] ++ docker exec -u root pytorch-ci-container bash -c 'rm -rf /var/lib/jenkins/pytorch'
[2026-07-08T19:32:18.305Z] ++ docker cp /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3410/pytorch pytorch-ci-container:/var/lib/jenkins/pytorch
[2026-07-08T19:32:22.408Z] ++ docker exec -u root pytorch-ci-container bash -c 'chown root:root -R /var/lib/jenkins/pytorch'
[2026-07-08T19:32:22.408Z] ++ [[ 1 -eq 1 ]]
[2026-07-08T19:32:22.408Z] ++ docker cp /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3410/rocm_composable_kernel-7.1.1.dev0+g4266f8-py3-none-any.whl /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3410/torch-2.14.0a0+gitb213fd9-cp312-cp312-linux_x86_64.whl pytorch-ci-container:/var/lib/jenkins/pytorch
[2026-07-08T19:32:22.408Z] docker: 'docker cp' requires 2 arguments
[2026-07-08T19:32:22.408Z] 
[2026-07-08T19:32:22.408Z] Usage:  docker cp [OPTIONS] CONTAINER:SRC_PATH DEST_PATH|-
[2026-07-08T19:32:22.408Z] 	docker cp [OPTIONS] SRC_PATH|- CONTAINER:DEST_PATH
[2026-07-08T19:32:22.408Z] 
[2026-07-08T19:32:22.408Z] Run 'docker cp --help' for more information

Tests / Test Inductor / Test Inductor / Run pytorch_inductor_1 / Error signal

Error in error step, with arguments pytorch_inductor_1 failed.

pytorch_inductor_1 failed

Details

  • Kill older PR Builds (1.9 sec)
  • Initialize (1 hr 44 min)
    • Download CI scripts (36 sec)
    • Checkout Pytorch (4 min 26 sec)
    • Check base Docker image existence (6.3 sec)
    • Pull Docker Image (6 min 9 sec)
    • Build PyTorch (1 hr 31 min)
  • Tests (2 hr 16 min)
    • Test PyTorch (9 ms)
      • Test PyTorch (32 min)
        • Run pytorch_test_1 (12 min)
          Error: script returned exit code 1 - logs
          Error: script returned exit code 1 - logs
          Error: pytorch_test_1 failed - logs
        • Run pytorch_test_2 (19 min)
          Error: Timeout has been exceeded - logs
    • Test Distributed (8 ms)
      • Test Distributed (2 hr 15 min)
        • Run pytorch_distributed_1 (2 hr 1 min)
          Error: script returned exit code 1 - logs
          Error: script returned exit code 1 - logs
          Error: pytorch_distributed_1 failed - logs
        • Run pytorch_distributed_2 (14 min)
          Error: Timeout has been exceeded - logs
    • Test Inductor (7 ms)
      • Test Inductor (1 hr 52 min)
        • Run pytorch_inductor_1 (1 hr 52 min)
          Error: script returned exit code 1 - logs
          Error: script returned exit code 1 - logs
          Error: pytorch_inductor_1 failed - logs
    • Test PyTorch Slow (8 ms)
      • Test PyTorch Slow (8.3 sec)
    • Microbenchmark (17 sec)
      • Microbenchmark (8.3 sec)
  • Post Build (1.9 sec)
  • Declarative: Post Actions (4.3 sec)
View more details on ROCm Repo Management API