Skip to content

[CI] Parity: enable auto_classify by default#3413

Open
ethanwee1 wants to merge 1 commit into
ROCm:developfrom
ethanwee1:ethanwee/parity-autoclassify-default
Open

[CI] Parity: enable auto_classify by default#3413
ethanwee1 wants to merge 1 commit into
ROCm:developfrom
ethanwee1:ethanwee/parity-autoclassify-default

[CI] Parity: enable auto_classify by default

c3556ff
Select commit
Loading
Failed to load commit list.
ROCm Repo Management API / Jenkins failed Jul 8, 2026 in 2h 6m 39s

Tests/Test Inductor/Run pytorch_inductor_1: error in 'error' step

Tests / Test PyTorch / Test PyTorch / Run pytorch_test_1 / Shell Script

Error in sh step, with arguments ./test_pytorch_test.sh.

script returned exit code 1
Build log 
[2026-07-08T21:27:56.623Z] + ./test_pytorch_test.sh
[2026-07-08T21:27:56.623Z] + [[ -z '' ]]
[2026-07-08T21:27:56.623Z] + DEFAULT_EXECUTION_OPTIONS='-e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts         -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip         --copy-whls'
[2026-07-08T21:27:56.623Z] + source prepare_docker_env.sh -e 'EXTENSION_BUILD_GFXARCH=gfx90a;gfx908;gfx942' -e CI=1 -e TEST_CONFIG=default -e SHARD_NUMBER=1 -e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip --copy-whls
[2026-07-08T21:27:56.623Z] ++ set -ex
[2026-07-08T21:27:56.623Z] ++ set -o pipefail
[2026-07-08T21:27:56.623Z] ++ copy_whls=0
[2026-07-08T21:27:56.623Z] ++ envvars=()
[2026-07-08T21:27:56.623Z] ++ [[ 13 -gt 0 ]]
[2026-07-08T21:27:56.623Z] ++ key=-e
[2026-07-08T21:27:56.623Z] ++ case $key in
[2026-07-08T21:27:56.623Z] ++ envvars+=("-e $2")
[2026-07-08T21:27:56.623Z] ++ shift
[2026-07-08T21:27:56.623Z] ++ shift
[2026-07-08T21:27:56.623Z] ++ [[ 11 -gt 0 ]]
[2026-07-08T21:27:56.623Z] ++ key=-e
[2026-07-08T21:27:56.623Z] ++ case $key in
[2026-07-08T21:27:56.623Z] ++ envvars+=("-e $2")
[2026-07-08T21:27:56.623Z] ++ shift
[2026-07-08T21:27:56.623Z] ++ shift
[2026-07-08T21:27:56.623Z] ++ [[ 9 -gt 0 ]]
[2026-07-08T21:27:56.623Z] ++ key=-e
[2026-07-08T21:27:56.623Z] ++ case $key in
[2026-07-08T21:27:56.623Z] ++ envvars+=("-e $2")
[2026-07-08T21:27:56.623Z] ++ shift
[2026-07-08T21:27:56.623Z] ++ shift
[2026-07-08T21:27:56.623Z] ++ [[ 7 -gt 0 ]]
[2026-07-08T21:27:56.623Z] ++ key=-e
[2026-07-08T21:27:56.623Z] ++ case $key in
[2026-07-08T21:27:56.623Z] ++ envvars+=("-e $2")
[2026-07-08T21:27:56.623Z] ++ shift
[2026-07-08T21:27:56.623Z] ++ shift
[2026-07-08T21:27:56.623Z] ++ [[ 5 -gt 0 ]]
[2026-07-08T21:27:56.623Z] ++ key=-e
[2026-07-08T21:27:56.623Z] ++ case $key in
[2026-07-08T21:27:56.623Z] ++ envvars+=("-e $2")
[2026-07-08T21:27:56.623Z] ++ shift
[2026-07-08T21:27:56.623Z] ++ shift
[2026-07-08T21:27:56.623Z] ++ [[ 3 -gt 0 ]]
[2026-07-08T21:27:56.623Z] ++ key=-e
[2026-07-08T21:27:56.623Z] ++ case $key in
[2026-07-08T21:27:56.623Z] ++ envvars+=("-e $2")
[2026-07-08T21:27:56.623Z] ++ shift
[2026-07-08T21:27:56.623Z] ++ shift
[2026-07-08T21:27:56.623Z] ++ [[ 1 -gt 0 ]]
[2026-07-08T21:27:56.623Z] ++ key=--copy-whls
[2026-07-08T21:27:56.623Z] ++ case $key in
[2026-07-08T21:27:56.623Z] ++ copy_whls=1
[2026-07-08T21:27:56.623Z] ++ shift
[2026-07-08T21:27:56.623Z] ++ [[ 0 -gt 0 ]]
[2026-07-08T21:27:56.623Z] ++ envvars+=("-e PREBUILT=${PREBUILT:-false}")
[2026-07-08T21:27:56.623Z] ++ envvars+=("-e TEST_CORE=${TEST_CORE:-false}")
[2026-07-08T21:27:56.623Z] ++ [[ false == \t\r\u\e ]]
[2026-07-08T21:27:56.623Z] ++ [[ true == \t\r\u\e ]]
[2026-07-08T21:27:56.623Z] ++ envvars+=("-e TEST_CORE=${TEST_CORE}")
[2026-07-08T21:27:56.623Z] ++ docker ps -a
[2026-07-08T21:27:56.623Z] CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES
[2026-07-08T21:27:56.623Z] ++ docker rm -f pytorch-ci-container
[2026-07-08T21:27:56.623Z] ++ docker pull rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942
[2026-07-08T21:27:57.926Z] pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942: Pulling from rocm/pytorch-ci-private
[2026-07-08T21:27:58.730Z] 747272797eac: Pulling fs layer
[2026-07-08T21:27:58.730Z] db0af168a1a4: Pulling fs layer
[2026-07-08T21:27:58.730Z] f3058198fba5: Pulling fs layer
[2026-07-08T21:27:58.730Z] 4c8a34755929: Pulling fs layer
[2026-07-08T21:27:58.730Z] bf3e4cf00cd6: Pulling fs layer
[2026-07-08T21:27:58.730Z] 3d56742d201d: Pulling fs layer
[2026-07-08T21:27:58.730Z] 86cc05da26d1: Pulling fs layer
[2026-07-08T21:27:58.730Z] 8b3d007242f8: Pulling fs layer
[2026-07-08T21:27:58.730Z] d7df6379f9cc: Pulling fs layer
[2026-07-08T21:27:58.730Z] 78cb638d6686: Pulling fs layer
[2026-07-08T21:27:58.730Z] ed874654c321: Pulling fs layer
[2026-07-08T21:27:58.730Z] 65f281ce2e70: Pulling fs layer
[2026-07-08T21:27:58.730Z] 95d704643610: Pulling fs layer
[2026-07-08T21:27:58.730Z] 423e26ddb517: Pulling fs layer
[2026-07-08T21:27:58.730Z] 1057fba0c7df: Pulling fs layer
[2026-07-08T21:27:58.730Z] eab367f1ca39: Pulling fs layer
[2026-07-08T21:27:58.730Z] 5edf52847c35: Pulling fs layer
[2026-07-08T21:27:58.730Z] ba1fabf878bd: Pulling fs layer
[2026-07-08T21:27:58.730Z] 8d998c79d309: Pulling fs layer
[2026-07-08T21:27:58.730Z] 0fcc3b23bd99: Pulling fs layer
[2026-07-08T21:27:58.730Z] 4106a853273f: Pulling fs layer
[2026-07-08T21:27:58.730Z] 34e29023481f: Pulling fs layer
[2026-07-08T21:27:58.730Z] b09b7d04a8d9: Pulling fs layer
[2026-07-08T21:27:58.730Z] 3d56742d201d: Pulling fs layer
[2026-07-08T21:27:58.730Z] 5f8dd10c9c78: Pulling fs layer
[2026-07-08T21:27:58.730Z] 9548298aa5f4: Pulling fs layer
[2026-07-08T21:27:58.730Z] cb259a83ac3d: Pulling fs layer
[2026-07-08T21:27:58.730Z] 42239a12ed73: Pulling fs layer
[2026-07-08T21:27:58.730Z] 3d56742d201d: Pulling fs layer
[2026-07-08T21:27:58.730Z] 99e6462d03e0: Pulling fs layer
[2026-07-08T21:27:58.730Z] c0664a7365cb: Pulling fs layer
[2026-07-08T21:27:58.730Z] d9a8b776bea7: Pulling fs layer
[2026-07-08T21:27:58.730Z] fef65a673dc9: Pulling fs layer
[2026-07-08T21:27:58.730Z] f5afebc61536: Pulling fs layer
[2026-07-08T21:27:58.730Z] 9ae773ccc4b2: Pulling fs layer
[2026-07-08T21:27:58.730Z] 906ac4de0a2b: Pulling fs layer
[2026-07-08T21:27:58.730Z] c608e00b1282: Pulling fs layer
[2026-07-08T21:27:58.730Z] 3d56742d201d: Pulling fs layer
[2026-07-08T21:27:58.730Z] cccff27d82fb: Pulling fs layer
[2026-07-08T21:27:58.730Z] 0e6d9b5f5e8d: Pulling fs layer
[2026-07-08T21:27:58.730Z] dedc784bccba: Pulling fs layer
[2026-07-08T21:27:58.730Z] 07b45639fc6f: Pulling fs layer
[2026-07-08T21:27:58.730Z] 27fcad4d96a6: Pulling fs layer
[2026-07-08T21:27:58.730Z] b96eaed0e751: Pulling fs layer
[2026-07-08T21:27:58.730Z] 0a0ffadd8b4f: Pulling fs layer
[2026-07-08T21:27:58.730Z] 284e8cbb8a05: Pulling fs layer
[2026-07-08T21:27:58.730Z] 3d56742d201d: Pulling fs layer
[2026-07-08T21:27:58.730Z] f790d498bebb: Pulling fs layer
[2026-07-08T21:27:58.730Z] 4cfd21f31c8b: Pulling fs layer
[2026-07-08T21:27:58.730Z] 241cbb1c2f26: Pulling fs layer
[2026-07-08T21:27:58.730Z] f0ef294e4f9b: Pulling fs layer
[2026-07-08T21:27:58.730Z] f391428cfe0f: Pulling fs layer
[2026-07-08T21:27:58.730Z] 4f4fb700ef54: Pulling fs layer
[2026-07-08T21:27:58.730Z] 4f4fb700ef54: Pulling fs layer
[2026-07-08T21:27:58.730Z] 8d78d529f04d: Pulling fs layer
[2026-07-08T21:27:58.730Z] ed86e5a5d9b1: Pulling fs layer
[2026-07-08T21:27:58.730Z] 78a3c5370db4: Pulling fs layer
[2026-07-08T21:27:58.730Z] 69145e0460f6: Download complete
[2026-07-08T21:27:58.730Z] f3058198fba5: Download complete
[2026-07-08T21:27:58.730Z] 3d56742d201d: Download complete
[2026-07-08T21:27:58.730Z] 747272797eac: Download complete
[2026-07-08T21:27:58.730Z] 27fcad4d96a6: Download complete
[2026-07-08T21:27:58.730Z] db0af168a1a4: Download complete
[2026-07-08T21:27:58.730Z] 86cc05da26d1: Download complete
[2026-07-08T21:27:59.113Z] 8b3d007242f8: Download complete
[2026-07-08T21:27:59.113Z] bf3e4cf00cd6: Download complete
[2026-07-08T21:27:59.113Z] ed874654c321: Download complete
[2026-07-08T21:27:59.113Z] 4cfd21f31c8b: Download complete
[2026-07-08T21:27:59.113Z] d7df6379f9cc: Download complete
[2026-07-08T21:27:59.113Z] 65f281ce2e70: Download complete
[2026-07-08T21:27:59.113Z] 423e26ddb517: Download complete
[2026-07-08T21:27:59.113Z] 8d998c79d309: Download complete
[2026-07-08T21:27:59.113Z] 241cbb1c2f26: Download complete
[2026-07-08T21:27:59.113Z] 9548298aa5f4: Download complete
[2026-07-08T21:27:59.113Z] f391428cfe0f: Download complete
[2026-07-08T21:27:59.113Z] f790d498bebb: Download complete
[2026-07-08T21:27:59.113Z] 0fcc3b23bd99: Download complete
[2026-07-08T21:27:59.113Z] eab367f1ca39: Download complete
[2026-07-08T21:27:59.113Z] c608e00b1282: Download complete
[2026-07-08T21:27:59.113Z] 9ae773ccc4b2: Download complete
[2026-07-08T21:27:59.113Z] b96eaed0e751: Download complete
[2026-07-08T21:27:59.113Z] 07b45639fc6f: Download complete
[2026-07-08T21:27:59.113Z] 34e29023481f: Download complete
[2026-07-08T21:27:59.113Z] c0664a7365cb: Download complete
[2026-07-08T21:27:59.113Z] f5afebc61536: Download complete
[2026-07-08T21:27:59.113Z] b09b7d04a8d9: Download complete
[2026-07-08T21:27:59.113Z] 42239a12ed73: Download complete
[2026-07-08T21:27:59.496Z] 4f4fb700ef54: Download complete
[2026-07-08T21:27:59.496Z] f0ef294e4f9b: Download complete
[2026-07-08T21:27:59.496Z] 5f8dd10c9c78: Download complete
[2026-07-08T21:27:59.496Z] 0a0ffadd8b4f: Download complete
[2026-07-08T21:27:59.496Z] 78a3c5370db4: Download complete
[2026-07-08T21:27:59.496Z] fef65a673dc9: Download complete
[2026-07-08T21:27:59.496Z] 8d78d529f04d: Download complete
[2026-07-08T21:27:59.496Z] 99e6462d03e0: Download complete
[2026-07-08T21:27:59.496Z] 906ac4de0a2b: Download complete
[2026-07-08T21:27:59.496Z] 284e8cbb8a05: Download complete
[2026-07-08T21:27:59.496Z] ed86e5a5d9b1: Download complete
[2026-07-08T21:27:59.496Z] 4106a853273f: Download complete
[2026-07-08T21:27:59.496Z] 0e6d9b5f5e8d: Download complete
[2026-07-08T21:27:59.893Z] ba1fabf878bd: Download complete
[2026-07-08T21:27:59.893Z] 4c8a34755929: Download complete
[2026-07-08T21:27:59.893Z] cb259a83ac3d: Download complete
[2026-07-08T21:28:00.292Z] 95d704643610: Download complete
[2026-07-08T21:28:00.686Z] 99e6462d03e0: Pull complete
[2026-07-08T21:28:00.686Z] cb259a83ac3d: Pull complete
[2026-07-08T21:28:02.560Z] cccff27d82fb: Download complete
[2026-07-08T21:28:06.707Z] 1057fba0c7df: Download complete
[2026-07-08T21:28:14.449Z] 1057fba0c7df: Pull complete
[2026-07-08T21:28:16.315Z] 0e6d9b5f5e8d: Pull complete
[2026-07-08T21:28:16.699Z] 3d56742d201d: Pull complete
[2026-07-08T21:28:16.699Z] d7df6379f9cc: Pull complete
[2026-07-08T21:28:16.699Z] 65f281ce2e70: Pull complete
[2026-07-08T21:28:16.699Z] c608e00b1282: Pull complete
[2026-07-08T21:28:16.699Z] 34e29023481f: Pull complete
[2026-07-08T21:28:16.699Z] 8d78d529f04d: Pull complete
[2026-07-08T21:28:16.699Z] 906ac4de0a2b: Pull complete
[2026-07-08T21:28:18.019Z] 78cb638d6686: Download complete
[2026-07-08T21:28:52.384Z] 5edf52847c35: Download complete
[2026-07-08T21:29:26.868Z] 747272797eac: Pull complete
[2026-07-08T21:29:26.868Z] db0af168a1a4: Pull complete
[2026-07-08T21:29:26.868Z] 86cc05da26d1: Pull complete
[2026-07-08T21:29:26.868Z] ed874654c321: Pull complete
[2026-07-08T21:29:26.868Z] 9548298aa5f4: Pull complete
[2026-07-08T21:29:26.868Z] 0a0ffadd8b4f: Pull complete
[2026-07-08T21:29:26.868Z] 5edf52847c35: Pull complete
[2026-07-08T21:29:26.868Z] 8b3d007242f8: Pull complete
[2026-07-08T21:30:01.222Z] d9a8b776bea7: Download complete
[2026-07-08T21:30:50.570Z] dedc784bccba: Download complete
[2026-07-08T21:32:14.749Z] dedc784bccba: Pull complete
[2026-07-08T21:32:14.749Z] 27fcad4d96a6: Pull complete
[2026-07-08T21:32:14.749Z] bf3e4cf00cd6: Pull complete
[2026-07-08T21:32:14.749Z] 9ae773ccc4b2: Pull complete
[2026-07-08T21:32:14.749Z] f3058198fba5: Pull complete
[2026-07-08T21:32:14.749Z] f5afebc61536: Pull complete
[2026-07-08T21:32:14.749Z] 78cb638d6686: Pull complete
[2026-07-08T21:32:14.749Z] f790d498bebb: Pull complete
[2026-07-08T21:32:14.749Z] b96eaed0e751: Pull complete
[2026-07-08T21:32:14.749Z] 07b45639fc6f: Pull complete
[2026-07-08T21:32:14.750Z] c0664a7365cb: Pull complete
[2026-07-08T21:32:14.750Z] cccff27d82fb: Pull complete
[2026-07-08T21:32:14.750Z] 95d704643610: Pull complete
[2026-07-08T21:32:14.750Z] 4cfd21f31c8b: Pull complete
[2026-07-08T21:32:14.750Z] 423e26ddb517: Pull complete
[2026-07-08T21:32:14.750Z] 241cbb1c2f26: Pull complete
[2026-07-08T21:32:14.750Z] 0fcc3b23bd99: Pull complete
[2026-07-08T21:32:14.750Z] b09b7d04a8d9: Pull complete
[2026-07-08T21:32:14.750Z] 42239a12ed73: Pull complete
[2026-07-08T21:32:14.750Z] 4f4fb700ef54: Pull complete
[2026-07-08T21:32:14.750Z] f0ef294e4f9b: Pull complete
[2026-07-08T21:32:14.750Z] 5f8dd10c9c78: Pull complete
[2026-07-08T21:32:14.750Z] 78a3c5370db4: Pull complete
[2026-07-08T21:32:14.750Z] 284e8cbb8a05: Pull complete
[2026-07-08T21:32:14.750Z] 4106a853273f: Pull complete
[2026-07-08T21:33:13.673Z] f391428cfe0f: Pull complete
[2026-07-08T21:33:13.673Z] eab367f1ca39: Pull complete
[2026-07-08T21:33:13.673Z] d9a8b776bea7: Pull complete
[2026-07-08T21:33:13.673Z] 8d998c79d309: Pull complete
[2026-07-08T21:33:13.673Z] fef65a673dc9: Pull complete
[2026-07-08T21:33:13.673Z] ed86e5a5d9b1: Pull complete
[2026-07-08T21:33:13.673Z] ba1fabf878bd: Pull complete
[2026-07-08T21:33:13.673Z] 4c8a34755929: Pull complete
[2026-07-08T21:33:13.673Z] Digest: sha256:b836aa0e221ac85a2184ec24a67b9d5089f3caa45d182cf521dee7d38112aa81
[2026-07-08T21:33:13.673Z] Status: Downloaded newer image for rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942
[2026-07-08T21:33:13.673Z] docker.io/rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942
[2026-07-08T21:33:13.673Z] ++ GPU_FLAGS=
[2026-07-08T21:33:13.673Z] ++ echo 'NODE_LABELS: k8s-mi250-1-z3q7c gfx90a mlir k8s-mi250-1-z3q7c gfx90a mlir'
[2026-07-08T21:33:13.673Z] NODE_LABELS: k8s-mi250-1-z3q7c gfx90a mlir k8s-mi250-1-z3q7c gfx90a mlir
[2026-07-08T21:33:13.673Z] ++ [[ ! k8s-mi250-1-z3q7c gfx90a mlir =~ build-only ]]
[2026-07-08T21:33:13.673Z] ++ GPU_FLAGS='--group-add video --device=/dev/kfd --device=/dev/dri'
[2026-07-08T21:33:13.673Z] ++++ dirname prepare_docker_env.sh
[2026-07-08T21:33:13.673Z] +++ cd .
[2026-07-08T21:33:13.673Z] +++ pwd
[2026-07-08T21:33:13.673Z] ++ script_dir=/home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3413
[2026-07-08T21:33:13.673Z] ++ docker run --name pytorch-ci-container -e 'EXTENSION_BUILD_GFXARCH=gfx90a;gfx908;gfx942' -e CI=1 -e TEST_CONFIG=default -e SHARD_NUMBER=1 -e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip -e PREBUILT=false -e TEST_CORE=true -e TEST_CORE=true -v /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3413:/host_workspace -v /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3413:/scripts --entrypoint bash -d --network=host --group-add video --device=/dev/kfd --device=/dev/dri --ipc=host --pid=host --shm-size 8G --cap-add=SYS_PTRACE --security-opt seccomp=unconfined rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942 -lc 'sleep infinity'
[2026-07-08T21:33:16.944Z] b3f63a122a03bf1a22998a88fad3ea67bca8666865d1dd08f7f2d11d40e8633d
[2026-07-08T21:33:17.321Z] ++ env -i python3 /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3413/pytorch/tools/stats/export_test_times.py
[2026-07-08T21:33:25.315Z] Exporting test times from test-infra
[2026-07-08T21:33:25.315Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-times.json to /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3413/pytorch/.additional_ci_files/test-times.json
[2026-07-08T21:33:25.315Z] Could not download https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-times.json because: The read operation timed out.
[2026-07-08T21:33:25.315Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-class-times.json to /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3413/pytorch/.additional_ci_files/test-class-times.json
[2026-07-08T21:33:25.315Z] ++ python3 /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3413/pytorch/tools/stats/export_test_times.py
[2026-07-08T21:33:25.315Z] Exporting test times from test-infra
[2026-07-08T21:33:25.315Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-times.json to /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3413/pytorch/.additional_ci_files/test-times.json
[2026-07-08T21:33:25.315Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-class-times.json to /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3413/pytorch/.additional_ci_files/test-class-times.json
[2026-07-08T21:33:25.315Z] ++ [[ -d /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3413/pytorch/.additional_ci_files ]]
[2026-07-08T21:33:25.315Z] ++ sed -i -E 's/"\S+rocm\S+"/""/g' /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3413/pytorch/.additional_ci_files/test-times.json
[2026-07-08T21:33:25.315Z] ++ [[ false != \t\r\u\e ]]
[2026-07-08T21:33:25.315Z] ++ docker exec -u root pytorch-ci-container bash -c 'rm -rf /var/lib/jenkins/pytorch'
[2026-07-08T21:33:25.698Z] ++ docker cp /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3413/pytorch pytorch-ci-container:/var/lib/jenkins/pytorch
[2026-07-08T21:33:29.016Z] ++ docker exec -u root pytorch-ci-container bash -c 'chown root:root -R /var/lib/jenkins/pytorch'
[2026-07-08T21:33:29.016Z] ++ [[ 1 -eq 1 ]]
[2026-07-08T21:33:29.016Z] ++ docker cp /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3413/rocm_composable_kernel-7.1.1.dev0+g4266f8-py3-none-any.whl /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3413/torch-2.14.0a0+gitc3556ff-cp312-cp312-linux_x86_64.whl pytorch-ci-container:/var/lib/jenkins/pytorch
[2026-07-08T21:33:29.016Z] docker: 'docker cp' requires 2 arguments
[2026-07-08T21:33:29.016Z] 
[2026-07-08T21:33:29.016Z] Usage:  docker cp [OPTIONS] CONTAINER:SRC_PATH DEST_PATH|-
[2026-07-08T21:33:29.016Z] 	docker cp [OPTIONS] SRC_PATH|- CONTAINER:DEST_PATH
[2026-07-08T21:33:29.016Z] 
[2026-07-08T21:33:29.016Z] Run 'docker cp --help' for more information

Tests / Test PyTorch / Test PyTorch / Run pytorch_test_1 / Error signal

Error in error step, with arguments pytorch_test_1 failed.

pytorch_test_1 failed

Tests / Test PyTorch / Test PyTorch / Run pytorch_test_2 / Shell Script

Error in sh step, with arguments rocminfo.

script returned exit code 1
Build log 
[2026-07-08T21:33:38.792Z] + rocminfo
[2026-07-08T21:33:38.792Z] ROCk module version 6.10.5 is loaded
[2026-07-08T21:33:38.792Z] Unable to open /dev/kfd read-write: Invalid argument
[2026-07-08T21:33:38.792Z] jenkins is member of video group

Tests / Test PyTorch / Test PyTorch / Run pytorch_test_2 / Shell Script

Error in sh step, with arguments ./test_pytorch_test.sh.

script returned exit code 1
Build log 
[2026-07-08T22:11:16.648Z] + ./test_pytorch_test.sh
[2026-07-08T22:11:16.648Z] + [[ -z '' ]]
[2026-07-08T22:11:16.648Z] + DEFAULT_EXECUTION_OPTIONS='-e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts         -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip         --copy-whls'
[2026-07-08T22:11:16.648Z] + source prepare_docker_env.sh -e 'EXTENSION_BUILD_GFXARCH=gfx90a;gfx908;gfx942' -e CI=1 -e TEST_CONFIG=default -e SHARD_NUMBER=2 -e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip --copy-whls
[2026-07-08T22:11:16.648Z] ++ set -ex
[2026-07-08T22:11:16.648Z] ++ set -o pipefail
[2026-07-08T22:11:16.648Z] ++ copy_whls=0
[2026-07-08T22:11:16.648Z] ++ envvars=()
[2026-07-08T22:11:16.648Z] ++ [[ 13 -gt 0 ]]
[2026-07-08T22:11:16.648Z] ++ key=-e
[2026-07-08T22:11:16.648Z] ++ case $key in
[2026-07-08T22:11:16.648Z] ++ envvars+=("-e $2")
[2026-07-08T22:11:16.648Z] ++ shift
[2026-07-08T22:11:16.648Z] ++ shift
[2026-07-08T22:11:16.648Z] ++ [[ 11 -gt 0 ]]
[2026-07-08T22:11:16.648Z] ++ key=-e
[2026-07-08T22:11:16.648Z] ++ case $key in
[2026-07-08T22:11:16.648Z] ++ envvars+=("-e $2")
[2026-07-08T22:11:16.648Z] ++ shift
[2026-07-08T22:11:16.648Z] ++ shift
[2026-07-08T22:11:16.648Z] ++ [[ 9 -gt 0 ]]
[2026-07-08T22:11:16.648Z] ++ key=-e
[2026-07-08T22:11:16.648Z] ++ case $key in
[2026-07-08T22:11:16.648Z] ++ envvars+=("-e $2")
[2026-07-08T22:11:16.648Z] ++ shift
[2026-07-08T22:11:16.648Z] ++ shift
[2026-07-08T22:11:16.648Z] ++ [[ 7 -gt 0 ]]
[2026-07-08T22:11:16.648Z] ++ key=-e
[2026-07-08T22:11:16.648Z] ++ case $key in
[2026-07-08T22:11:16.648Z] ++ envvars+=("-e $2")
[2026-07-08T22:11:16.648Z] ++ shift
[2026-07-08T22:11:16.648Z] ++ shift
[2026-07-08T22:11:16.648Z] ++ [[ 5 -gt 0 ]]
[2026-07-08T22:11:16.648Z] ++ key=-e
[2026-07-08T22:11:16.648Z] ++ case $key in
[2026-07-08T22:11:16.648Z] ++ envvars+=("-e $2")
[2026-07-08T22:11:16.648Z] ++ shift
[2026-07-08T22:11:16.648Z] ++ shift
[2026-07-08T22:11:16.648Z] ++ [[ 3 -gt 0 ]]
[2026-07-08T22:11:16.648Z] ++ key=-e
[2026-07-08T22:11:16.648Z] ++ case $key in
[2026-07-08T22:11:16.648Z] ++ envvars+=("-e $2")
[2026-07-08T22:11:16.648Z] ++ shift
[2026-07-08T22:11:16.648Z] ++ shift
[2026-07-08T22:11:16.648Z] ++ [[ 1 -gt 0 ]]
[2026-07-08T22:11:16.648Z] ++ key=--copy-whls
[2026-07-08T22:11:16.648Z] ++ case $key in
[2026-07-08T22:11:16.648Z] ++ copy_whls=1
[2026-07-08T22:11:16.648Z] ++ shift
[2026-07-08T22:11:16.648Z] ++ [[ 0 -gt 0 ]]
[2026-07-08T22:11:16.648Z] ++ envvars+=("-e PREBUILT=${PREBUILT:-false}")
[2026-07-08T22:11:16.648Z] ++ envvars+=("-e TEST_CORE=${TEST_CORE:-false}")
[2026-07-08T22:11:16.648Z] ++ [[ false == \t\r\u\e ]]
[2026-07-08T22:11:16.648Z] ++ [[ true == \t\r\u\e ]]
[2026-07-08T22:11:16.648Z] ++ envvars+=("-e TEST_CORE=${TEST_CORE}")
[2026-07-08T22:11:16.648Z] ++ docker ps -a
[2026-07-08T22:11:16.648Z] CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES
[2026-07-08T22:11:16.648Z] ++ docker rm -f pytorch-ci-container
[2026-07-08T22:11:16.648Z] ++ docker pull rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942
[2026-07-08T22:11:17.970Z] pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942: Pulling from rocm/pytorch-ci-private
[2026-07-08T22:11:18.351Z] 86cc05da26d1: Pulling fs layer
[2026-07-08T22:11:18.351Z] 4c8a34755929: Pulling fs layer
[2026-07-08T22:11:18.351Z] 9ae773ccc4b2: Pulling fs layer
[2026-07-08T22:11:18.351Z] 8d78d529f04d: Pulling fs layer
[2026-07-08T22:11:18.351Z] 78cb638d6686: Pulling fs layer
[2026-07-08T22:11:18.351Z] 1057fba0c7df: Pulling fs layer
[2026-07-08T22:11:18.351Z] f0ef294e4f9b: Pulling fs layer
[2026-07-08T22:11:18.351Z] 95d704643610: Pulling fs layer
[2026-07-08T22:11:18.351Z] f391428cfe0f: Pulling fs layer
[2026-07-08T22:11:18.351Z] 4cfd21f31c8b: Pulling fs layer
[2026-07-08T22:11:18.351Z] 3d56742d201d: Pulling fs layer
[2026-07-08T22:11:18.351Z] 8b3d007242f8: Pulling fs layer
[2026-07-08T22:11:18.351Z] cb259a83ac3d: Pulling fs layer
[2026-07-08T22:11:18.351Z] 284e8cbb8a05: Pulling fs layer
[2026-07-08T22:11:18.351Z] f5afebc61536: Pulling fs layer
[2026-07-08T22:11:18.351Z] 241cbb1c2f26: Pulling fs layer
[2026-07-08T22:11:18.351Z] 27fcad4d96a6: Pulling fs layer
[2026-07-08T22:11:18.351Z] 34e29023481f: Pulling fs layer
[2026-07-08T22:11:18.351Z] 4106a853273f: Pulling fs layer
[2026-07-08T22:11:18.351Z] 9548298aa5f4: Pulling fs layer
[2026-07-08T22:11:18.351Z] 0fcc3b23bd99: Pulling fs layer
[2026-07-08T22:11:18.351Z] cccff27d82fb: Pulling fs layer
[2026-07-08T22:11:18.351Z] 07b45639fc6f: Pulling fs layer
[2026-07-08T22:11:18.351Z] ed874654c321: Pulling fs layer
[2026-07-08T22:11:18.351Z] eab367f1ca39: Pulling fs layer
[2026-07-08T22:11:18.351Z] 3d56742d201d: Pulling fs layer
[2026-07-08T22:11:18.351Z] 5edf52847c35: Pulling fs layer
[2026-07-08T22:11:18.351Z] d7df6379f9cc: Pulling fs layer
[2026-07-08T22:11:18.351Z] 4f4fb700ef54: Pulling fs layer
[2026-07-08T22:11:18.351Z] 8d998c79d309: Pulling fs layer
[2026-07-08T22:11:18.351Z] c608e00b1282: Pulling fs layer
[2026-07-08T22:11:18.351Z] f3058198fba5: Pulling fs layer
[2026-07-08T22:11:18.351Z] 78a3c5370db4: Pulling fs layer
[2026-07-08T22:11:18.351Z] 423e26ddb517: Pulling fs layer
[2026-07-08T22:11:18.351Z] 5f8dd10c9c78: Pulling fs layer
[2026-07-08T22:11:18.351Z] ba1fabf878bd: Pulling fs layer
[2026-07-08T22:11:18.351Z] b96eaed0e751: Pulling fs layer
[2026-07-08T22:11:18.351Z] b09b7d04a8d9: Pulling fs layer
[2026-07-08T22:11:18.351Z] 3d56742d201d: Pulling fs layer
[2026-07-08T22:11:18.351Z] bf3e4cf00cd6: Pulling fs layer
[2026-07-08T22:11:18.351Z] 0a0ffadd8b4f: Pulling fs layer
[2026-07-08T22:11:18.351Z] 906ac4de0a2b: Pulling fs layer
[2026-07-08T22:11:18.351Z] 747272797eac: Pulling fs layer
[2026-07-08T22:11:18.351Z] ed86e5a5d9b1: Pulling fs layer
[2026-07-08T22:11:18.351Z] db0af168a1a4: Pulling fs layer
[2026-07-08T22:11:18.351Z] 65f281ce2e70: Pulling fs layer
[2026-07-08T22:11:18.351Z] 42239a12ed73: Pulling fs layer
[2026-07-08T22:11:18.351Z] 99e6462d03e0: Pulling fs layer
[2026-07-08T22:11:18.351Z] d9a8b776bea7: Pulling fs layer
[2026-07-08T22:11:18.351Z] 3d56742d201d: Pulling fs layer
[2026-07-08T22:11:18.351Z] 3d56742d201d: Pulling fs layer
[2026-07-08T22:11:18.351Z] dedc784bccba: Pulling fs layer
[2026-07-08T22:11:18.351Z] 4f4fb700ef54: Pulling fs layer
[2026-07-08T22:11:18.351Z] fef65a673dc9: Pulling fs layer
[2026-07-08T22:11:18.351Z] c0664a7365cb: Pulling fs layer
[2026-07-08T22:11:18.351Z] 0e6d9b5f5e8d: Pulling fs layer
[2026-07-08T22:11:18.351Z] f790d498bebb: Pulling fs layer
[2026-07-08T22:11:18.743Z] 69145e0460f6: Download complete
[2026-07-08T22:11:18.743Z] 3d56742d201d: Download complete
[2026-07-08T22:11:18.743Z] f0ef294e4f9b: Download complete
[2026-07-08T22:11:18.743Z] f391428cfe0f: Download complete
[2026-07-08T22:11:18.743Z] 07b45639fc6f: Download complete
[2026-07-08T22:11:18.743Z] bf3e4cf00cd6: Download complete
[2026-07-08T22:11:18.743Z] 9ae773ccc4b2: Download complete
[2026-07-08T22:11:18.743Z] 8d78d529f04d: Download complete
[2026-07-08T22:11:18.743Z] 4cfd21f31c8b: Download complete
[2026-07-08T22:11:19.120Z] d7df6379f9cc: Download complete
[2026-07-08T22:11:19.120Z] 5f8dd10c9c78: Download complete
[2026-07-08T22:11:19.120Z] f790d498bebb: Download complete
[2026-07-08T22:11:19.120Z] 0a0ffadd8b4f: Download complete
[2026-07-08T22:11:19.120Z] 9548298aa5f4: Download complete
[2026-07-08T22:11:19.120Z] 34e29023481f: Download complete
[2026-07-08T22:11:19.120Z] eab367f1ca39: Download complete
[2026-07-08T22:11:19.120Z] 0fcc3b23bd99: Download complete
[2026-07-08T22:11:19.120Z] 99e6462d03e0: Download complete
[2026-07-08T22:11:19.120Z] ed874654c321: Download complete
[2026-07-08T22:11:19.120Z] 86cc05da26d1: Download complete
[2026-07-08T22:11:19.120Z] 241cbb1c2f26: Download complete
[2026-07-08T22:11:19.120Z] 906ac4de0a2b: Download complete
[2026-07-08T22:11:19.120Z] 284e8cbb8a05: Download complete
[2026-07-08T22:11:19.120Z] 423e26ddb517: Download complete
[2026-07-08T22:11:19.120Z] f5afebc61536: Download complete
[2026-07-08T22:11:19.120Z] 4f4fb700ef54: Download complete
[2026-07-08T22:11:19.120Z] 8b3d007242f8: Download complete
[2026-07-08T22:11:19.120Z] 27fcad4d96a6: Download complete
[2026-07-08T22:11:19.120Z] fef65a673dc9: Download complete
[2026-07-08T22:11:19.120Z] f3058198fba5: Download complete
[2026-07-08T22:11:19.120Z] c608e00b1282: Download complete
[2026-07-08T22:11:19.120Z] 8d998c79d309: Download complete
[2026-07-08T22:11:19.120Z] 78a3c5370db4: Download complete
[2026-07-08T22:11:19.120Z] 65f281ce2e70: Download complete
[2026-07-08T22:11:19.120Z] 42239a12ed73: Download complete
[2026-07-08T22:11:19.120Z] b96eaed0e751: Download complete
[2026-07-08T22:11:19.120Z] ed86e5a5d9b1: Download complete
[2026-07-08T22:11:19.120Z] c0664a7365cb: Download complete
[2026-07-08T22:11:19.120Z] 747272797eac: Download complete
[2026-07-08T22:11:19.120Z] db0af168a1a4: Download complete
[2026-07-08T22:11:19.120Z] b09b7d04a8d9: Download complete
[2026-07-08T22:11:19.508Z] 4106a853273f: Download complete
[2026-07-08T22:11:19.508Z] 0e6d9b5f5e8d: Download complete
[2026-07-08T22:11:19.888Z] 4c8a34755929: Download complete
[2026-07-08T22:11:19.888Z] ba1fabf878bd: Download complete
[2026-07-08T22:11:19.888Z] cb259a83ac3d: Download complete
[2026-07-08T22:11:20.722Z] 95d704643610: Download complete
[2026-07-08T22:11:20.722Z] 99e6462d03e0: Pull complete
[2026-07-08T22:11:20.722Z] cb259a83ac3d: Pull complete
[2026-07-08T22:11:22.590Z] cccff27d82fb: Download complete
[2026-07-08T22:11:26.729Z] 1057fba0c7df: Download complete
[2026-07-08T22:11:33.247Z] 65f281ce2e70: Pull complete
[2026-07-08T22:11:33.247Z] 0e6d9b5f5e8d: Pull complete
[2026-07-08T22:11:33.247Z] 1057fba0c7df: Pull complete
[2026-07-08T22:11:33.247Z] 8d78d529f04d: Pull complete
[2026-07-08T22:11:33.247Z] d7df6379f9cc: Pull complete
[2026-07-08T22:11:33.247Z] 34e29023481f: Pull complete
[2026-07-08T22:11:33.247Z] 906ac4de0a2b: Pull complete
[2026-07-08T22:11:33.247Z] c608e00b1282: Pull complete
[2026-07-08T22:11:37.392Z] 78cb638d6686: Download complete
[2026-07-08T22:12:05.969Z] 5edf52847c35: Download complete
[2026-07-08T22:12:46.971Z] 3d56742d201d: Pull complete
[2026-07-08T22:12:46.971Z] 0a0ffadd8b4f: Pull complete
[2026-07-08T22:12:46.971Z] 9548298aa5f4: Pull complete
[2026-07-08T22:12:46.971Z] ed874654c321: Pull complete
[2026-07-08T22:12:46.971Z] 86cc05da26d1: Pull complete
[2026-07-08T22:12:46.971Z] 8b3d007242f8: Pull complete
[2026-07-08T22:12:46.971Z] 747272797eac: Pull complete
[2026-07-08T22:12:46.971Z] db0af168a1a4: Pull complete
[2026-07-08T22:12:46.971Z] 5edf52847c35: Pull complete
[2026-07-08T22:13:21.182Z] d9a8b776bea7: Download complete
[2026-07-08T22:14:02.171Z] dedc784bccba: Download complete
[2026-07-08T22:15:41.505Z] bf3e4cf00cd6: Pull complete
[2026-07-08T22:15:41.505Z] 9ae773ccc4b2: Pull complete
[2026-07-08T22:15:41.505Z] 27fcad4d96a6: Pull complete
[2026-07-08T22:15:41.505Z] dedc784bccba: Pull complete
[2026-07-08T22:15:41.505Z] 78cb638d6686: Pull complete
[2026-07-08T22:15:41.505Z] f5afebc61536: Pull complete
[2026-07-08T22:15:41.505Z] f3058198fba5: Pull complete
[2026-07-08T22:15:41.505Z] 07b45639fc6f: Pull complete
[2026-07-08T22:15:41.505Z] f790d498bebb: Pull complete
[2026-07-08T22:15:41.505Z] b96eaed0e751: Pull complete
[2026-07-08T22:15:41.505Z] c0664a7365cb: Pull complete
[2026-07-08T22:15:41.505Z] cccff27d82fb: Pull complete
[2026-07-08T22:15:41.505Z] 423e26ddb517: Pull complete
[2026-07-08T22:15:41.505Z] 4106a853273f: Pull complete
[2026-07-08T22:15:41.505Z] 95d704643610: Pull complete
[2026-07-08T22:15:41.505Z] f0ef294e4f9b: Pull complete
[2026-07-08T22:15:41.505Z] 4cfd21f31c8b: Pull complete
[2026-07-08T22:15:41.505Z] 5f8dd10c9c78: Pull complete
[2026-07-08T22:15:41.505Z] 0fcc3b23bd99: Pull complete
[2026-07-08T22:15:41.505Z] 241cbb1c2f26: Pull complete
[2026-07-08T22:15:41.505Z] 284e8cbb8a05: Pull complete
[2026-07-08T22:15:41.505Z] 78a3c5370db4: Pull complete
[2026-07-08T22:15:41.505Z] 42239a12ed73: Pull complete
[2026-07-08T22:15:41.505Z] b09b7d04a8d9: Pull complete
[2026-07-08T22:16:40.316Z] f391428cfe0f: Pull complete
[2026-07-08T22:16:40.316Z] eab367f1ca39: Pull complete
[2026-07-08T22:16:40.316Z] d9a8b776bea7: Pull complete
[2026-07-08T22:16:40.316Z] 4f4fb700ef54: Pull complete
[2026-07-08T22:16:40.316Z] fef65a673dc9: Pull complete
[2026-07-08T22:16:40.316Z] 8d998c79d309: Pull complete
[2026-07-08T22:16:40.316Z] ed86e5a5d9b1: Pull complete
[2026-07-08T22:16:40.316Z] ba1fabf878bd: Pull complete
[2026-07-08T22:16:40.316Z] 4c8a34755929: Pull complete
[2026-07-08T22:16:40.316Z] Digest: sha256:b836aa0e221ac85a2184ec24a67b9d5089f3caa45d182cf521dee7d38112aa81
[2026-07-08T22:16:40.316Z] Status: Downloaded newer image for rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942
[2026-07-08T22:16:40.316Z] docker.io/rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942
[2026-07-08T22:16:40.316Z] ++ GPU_FLAGS=
[2026-07-08T22:16:40.316Z] ++ echo 'NODE_LABELS: k8s-mi250-1-5n7vt gfx90a mlir k8s-mi250-1-5n7vt gfx90a mlir'
[2026-07-08T22:16:40.316Z] NODE_LABELS: k8s-mi250-1-5n7vt gfx90a mlir k8s-mi250-1-5n7vt gfx90a mlir
[2026-07-08T22:16:40.316Z] ++ [[ ! k8s-mi250-1-5n7vt gfx90a mlir =~ build-only ]]
[2026-07-08T22:16:40.316Z] ++ GPU_FLAGS='--group-add video --device=/dev/kfd --device=/dev/dri'
[2026-07-08T22:16:40.316Z] ++++ dirname prepare_docker_env.sh
[2026-07-08T22:16:40.316Z] +++ cd .
[2026-07-08T22:16:40.316Z] +++ pwd
[2026-07-08T22:16:40.316Z] ++ script_dir=/home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3413
[2026-07-08T22:16:40.316Z] ++ docker run --name pytorch-ci-container -e 'EXTENSION_BUILD_GFXARCH=gfx90a;gfx908;gfx942' -e CI=1 -e TEST_CONFIG=default -e SHARD_NUMBER=2 -e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip -e PREBUILT=false -e TEST_CORE=true -e TEST_CORE=true -v /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3413:/host_workspace -v /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3413:/scripts --entrypoint bash -d --network=host --group-add video --device=/dev/kfd --device=/dev/dri --ipc=host --pid=host --shm-size 8G --cap-add=SYS_PTRACE --security-opt seccomp=unconfined rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942 -lc 'sleep infinity'
[2026-07-08T22:16:46.650Z] 37c4d09d05879e8b7170ba8bc79bf83a8c6e3c952efa8efd418825a91118faff
[2026-07-08T22:16:46.650Z] ++ env -i python3 /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3413/pytorch/tools/stats/export_test_times.py
[2026-07-08T22:17:00.549Z] Exporting test times from test-infra
[2026-07-08T22:17:00.549Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-times.json to /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3413/pytorch/.additional_ci_files/test-times.json
[2026-07-08T22:17:00.549Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-class-times.json to /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3413/pytorch/.additional_ci_files/test-class-times.json
[2026-07-08T22:17:00.549Z] ++ python3 /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3413/pytorch/tools/stats/export_test_times.py
[2026-07-08T22:17:00.549Z] Exporting test times from test-infra
[2026-07-08T22:17:00.549Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-times.json to /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3413/pytorch/.additional_ci_files/test-times.json
[2026-07-08T22:17:00.549Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-class-times.json to /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3413/pytorch/.additional_ci_files/test-class-times.json
[2026-07-08T22:17:00.549Z] ++ [[ -d /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3413/pytorch/.additional_ci_files ]]
[2026-07-08T22:17:00.549Z] ++ sed -i -E 's/"\S+rocm\S+"/""/g' /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3413/pytorch/.additional_ci_files/test-times.json
[2026-07-08T22:17:00.549Z] ++ [[ false != \t\r\u\e ]]
[2026-07-08T22:17:00.549Z] ++ docker exec -u root pytorch-ci-container bash -c 'rm -rf /var/lib/jenkins/pytorch'
[2026-07-08T22:17:00.549Z] ++ docker cp /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3413/pytorch pytorch-ci-container:/var/lib/jenkins/pytorch
[2026-07-08T22:17:03.784Z] ++ docker exec -u root pytorch-ci-container bash -c 'chown root:root -R /var/lib/jenkins/pytorch'
[2026-07-08T22:17:03.784Z] ++ [[ 1 -eq 1 ]]
[2026-07-08T22:17:03.784Z] ++ docker cp /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3413/rocm_composable_kernel-7.1.1.dev0+g4266f8-py3-none-any.whl /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3413/torch-2.14.0a0+gitc3556ff-cp312-cp312-linux_x86_64.whl pytorch-ci-container:/var/lib/jenkins/pytorch
[2026-07-08T22:17:03.784Z] docker: 'docker cp' requires 2 arguments
[2026-07-08T22:17:03.784Z] 
[2026-07-08T22:17:03.784Z] Usage:  docker cp [OPTIONS] CONTAINER:SRC_PATH DEST_PATH|-
[2026-07-08T22:17:03.784Z] 	docker cp [OPTIONS] SRC_PATH|- CONTAINER:DEST_PATH
[2026-07-08T22:17:03.784Z] 
[2026-07-08T22:17:03.784Z] Run 'docker cp --help' for more information

Tests / Test PyTorch / Test PyTorch / Run pytorch_test_2 / Error signal

Error in error step, with arguments pytorch_test_2 failed.

pytorch_test_2 failed

Tests / Test Distributed / Test Distributed / Run pytorch_distributed_1 / Shell Script

Error in sh step, with arguments ./test_pytorch_test_distributed.sh.

script returned exit code 1
Build log 
[2026-07-08T21:24:55.219Z] + ./test_pytorch_test_distributed.sh
[2026-07-08T21:24:55.219Z] + [[ -z '' ]]
[2026-07-08T21:24:55.219Z] + DEFAULT_EXECUTION_OPTIONS='-e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts         -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip         --copy-whls'
[2026-07-08T21:24:55.219Z] + source prepare_docker_env.sh -e 'EXTENSION_BUILD_GFXARCH=gfx90a;gfx908;gfx942' -e CI=1 -e TEST_CONFIG=distributed -e SHARD_NUMBER=1 -e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip --copy-whls
[2026-07-08T21:24:55.219Z] ++ set -ex
[2026-07-08T21:24:55.219Z] ++ set -o pipefail
[2026-07-08T21:24:55.219Z] ++ copy_whls=0
[2026-07-08T21:24:55.219Z] ++ envvars=()
[2026-07-08T21:24:55.219Z] ++ [[ 13 -gt 0 ]]
[2026-07-08T21:24:55.219Z] ++ key=-e
[2026-07-08T21:24:55.219Z] ++ case $key in
[2026-07-08T21:24:55.219Z] ++ envvars+=("-e $2")
[2026-07-08T21:24:55.219Z] ++ shift
[2026-07-08T21:24:55.219Z] ++ shift
[2026-07-08T21:24:55.219Z] ++ [[ 11 -gt 0 ]]
[2026-07-08T21:24:55.219Z] ++ key=-e
[2026-07-08T21:24:55.219Z] ++ case $key in
[2026-07-08T21:24:55.219Z] ++ envvars+=("-e $2")
[2026-07-08T21:24:55.219Z] ++ shift
[2026-07-08T21:24:55.219Z] ++ shift
[2026-07-08T21:24:55.219Z] ++ [[ 9 -gt 0 ]]
[2026-07-08T21:24:55.219Z] ++ key=-e
[2026-07-08T21:24:55.219Z] ++ case $key in
[2026-07-08T21:24:55.219Z] ++ envvars+=("-e $2")
[2026-07-08T21:24:55.219Z] ++ shift
[2026-07-08T21:24:55.219Z] ++ shift
[2026-07-08T21:24:55.219Z] ++ [[ 7 -gt 0 ]]
[2026-07-08T21:24:55.219Z] ++ key=-e
[2026-07-08T21:24:55.219Z] ++ case $key in
[2026-07-08T21:24:55.219Z] ++ envvars+=("-e $2")
[2026-07-08T21:24:55.219Z] ++ shift
[2026-07-08T21:24:55.219Z] ++ shift
[2026-07-08T21:24:55.219Z] ++ [[ 5 -gt 0 ]]
[2026-07-08T21:24:55.219Z] ++ key=-e
[2026-07-08T21:24:55.219Z] ++ case $key in
[2026-07-08T21:24:55.219Z] ++ envvars+=("-e $2")
[2026-07-08T21:24:55.219Z] ++ shift
[2026-07-08T21:24:55.219Z] ++ shift
[2026-07-08T21:24:55.219Z] ++ [[ 3 -gt 0 ]]
[2026-07-08T21:24:55.219Z] ++ key=-e
[2026-07-08T21:24:55.219Z] ++ case $key in
[2026-07-08T21:24:55.219Z] ++ envvars+=("-e $2")
[2026-07-08T21:24:55.219Z] ++ shift
[2026-07-08T21:24:55.219Z] ++ shift
[2026-07-08T21:24:55.219Z] ++ [[ 1 -gt 0 ]]
[2026-07-08T21:24:55.219Z] ++ key=--copy-whls
[2026-07-08T21:24:55.219Z] ++ case $key in
[2026-07-08T21:24:55.219Z] ++ copy_whls=1
[2026-07-08T21:24:55.219Z] ++ shift
[2026-07-08T21:24:55.219Z] ++ [[ 0 -gt 0 ]]
[2026-07-08T21:24:55.219Z] ++ envvars+=("-e PREBUILT=${PREBUILT:-false}")
[2026-07-08T21:24:55.219Z] ++ envvars+=("-e TEST_CORE=${TEST_CORE:-false}")
[2026-07-08T21:24:55.219Z] ++ [[ false == \t\r\u\e ]]
[2026-07-08T21:24:55.219Z] ++ [[ true == \t\r\u\e ]]
[2026-07-08T21:24:55.219Z] ++ envvars+=("-e TEST_CORE=${TEST_CORE}")
[2026-07-08T21:24:55.219Z] ++ docker ps -a
[2026-07-08T21:24:55.219Z] CONTAINER ID   IMAGE                                                                                                                        COMMAND                  CREATED             STATUS             PORTS     NAMES
[2026-07-08T21:24:55.219Z] 412a7a7fc457   rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-fee2bb08afdd079f6a1cf65a81a7074f0198d83d-gfx908_gfx90a_gfx942   "bash -lc 'sleep inf…"   About an hour ago   Up About an hour             pytorch-ci-container
[2026-07-08T21:24:55.219Z] ++ docker rm -f pytorch-ci-container
[2026-07-08T21:25:01.589Z] ++ docker pull rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942
[2026-07-08T21:25:01.589Z] pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942: Pulling from rocm/pytorch-ci-private
[2026-07-08T21:25:01.589Z] Digest: sha256:b836aa0e221ac85a2184ec24a67b9d5089f3caa45d182cf521dee7d38112aa81
[2026-07-08T21:25:01.589Z] Status: Image is up to date for rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942
[2026-07-08T21:25:01.589Z] docker.io/rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942
[2026-07-08T21:25:01.589Z] ++ GPU_FLAGS=
[2026-07-08T21:25:01.589Z] ++ echo 'NODE_LABELS: multi_gpu rocm pytorch-nightly selftest ixt-hq-15 gfx908 multi_gpu rocm pytorch-nightly selftest ixt-hq-15 gfx908'
[2026-07-08T21:25:01.589Z] NODE_LABELS: multi_gpu rocm pytorch-nightly selftest ixt-hq-15 gfx908 multi_gpu rocm pytorch-nightly selftest ixt-hq-15 gfx908
[2026-07-08T21:25:01.589Z] ++ [[ ! multi_gpu rocm pytorch-nightly selftest ixt-hq-15 gfx908 =~ build-only ]]
[2026-07-08T21:25:01.589Z] ++ GPU_FLAGS='--group-add video --device=/dev/kfd --device=/dev/dri'
[2026-07-08T21:25:01.589Z] ++++ dirname prepare_docker_env.sh
[2026-07-08T21:25:01.589Z] +++ cd .
[2026-07-08T21:25:01.589Z] +++ pwd
[2026-07-08T21:25:01.589Z] ++ script_dir=/home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3413
[2026-07-08T21:25:01.589Z] ++ docker run --name pytorch-ci-container -e 'EXTENSION_BUILD_GFXARCH=gfx90a;gfx908;gfx942' -e CI=1 -e TEST_CONFIG=distributed -e SHARD_NUMBER=1 -e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip -e PREBUILT=false -e TEST_CORE=true -e TEST_CORE=true -v /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3413:/host_workspace -v /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3413:/scripts --entrypoint bash -d --network=host --group-add video --device=/dev/kfd --device=/dev/dri --ipc=host --pid=host --shm-size 8G --cap-add=SYS_PTRACE --security-opt seccomp=unconfined rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942 -lc 'sleep infinity'
[2026-07-08T21:25:01.589Z] c5e8e9d9083875b1c7958b7e8c4f684c26be56733869302a7bcdbaf8cc4f55de
[2026-07-08T21:25:02.121Z] ++ env -i python3 /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3413/pytorch/tools/stats/export_test_times.py
[2026-07-08T21:25:03.247Z] Exporting test times from test-infra
[2026-07-08T21:25:03.247Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-times.json to /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3413/pytorch/.additional_ci_files/test-times.json
[2026-07-08T21:25:03.247Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-class-times.json to /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3413/pytorch/.additional_ci_files/test-class-times.json
[2026-07-08T21:25:03.247Z] ++ python3 /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3413/pytorch/tools/stats/export_test_times.py
[2026-07-08T21:25:03.247Z] Exporting test times from test-infra
[2026-07-08T21:25:03.247Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-times.json to /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3413/pytorch/.additional_ci_files/test-times.json
[2026-07-08T21:25:03.247Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-class-times.json to /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3413/pytorch/.additional_ci_files/test-class-times.json
[2026-07-08T21:25:03.772Z] ++ [[ -d /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3413/pytorch/.additional_ci_files ]]
[2026-07-08T21:25:03.772Z] ++ sed -i -E 's/"\S+rocm\S+"/""/g' /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3413/pytorch/.additional_ci_files/test-times.json
[2026-07-08T21:25:03.772Z] ++ [[ false != \t\r\u\e ]]
[2026-07-08T21:25:03.772Z] ++ docker exec -u root pytorch-ci-container bash -c 'rm -rf /var/lib/jenkins/pytorch'
[2026-07-08T21:25:03.772Z] ++ docker cp /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3413/pytorch pytorch-ci-container:/var/lib/jenkins/pytorch
[2026-07-08T21:25:10.122Z] ++ docker exec -u root pytorch-ci-container bash -c 'chown root:root -R /var/lib/jenkins/pytorch'
[2026-07-08T21:25:10.122Z] ++ [[ 1 -eq 1 ]]
[2026-07-08T21:25:10.122Z] ++ docker cp /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3413/rocm_composable_kernel-7.1.1.dev0+g4266f8-py3-none-any.whl /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3413/torch-2.14.0a0+gitc3556ff-cp312-cp312-linux_x86_64.whl pytorch-ci-container:/var/lib/jenkins/pytorch
[2026-07-08T21:25:10.325Z] "docker cp" requires exactly 2 arguments.
[2026-07-08T21:25:10.325Z] See 'docker cp --help'.
[2026-07-08T21:25:10.325Z] 
[2026-07-08T21:25:10.325Z] Usage:  docker cp [OPTIONS] CONTAINER:SRC_PATH DEST_PATH|-
[2026-07-08T21:25:10.325Z] 	docker cp [OPTIONS] SRC_PATH|- CONTAINER:DEST_PATH
[2026-07-08T21:25:10.325Z] 
[2026-07-08T21:25:10.325Z] Copy files/folders between a container and the local filesystem

Tests / Test Distributed / Test Distributed / Run pytorch_distributed_1 / Error signal

Error in error step, with arguments pytorch_distributed_1 failed.

pytorch_distributed_1 failed

Tests / Test Distributed / Test Distributed / Run pytorch_distributed_2 / Shell Script

Error in sh step, with arguments ./test_pytorch_test_distributed.sh.

script returned exit code 1
Build log 
[2026-07-08T21:30:17.816Z] + ./test_pytorch_test_distributed.sh
[2026-07-08T21:30:17.816Z] + [[ -z '' ]]
[2026-07-08T21:30:17.816Z] + DEFAULT_EXECUTION_OPTIONS='-e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts         -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip         --copy-whls'
[2026-07-08T21:30:17.816Z] + source prepare_docker_env.sh -e 'EXTENSION_BUILD_GFXARCH=gfx90a;gfx908;gfx942' -e CI=1 -e TEST_CONFIG=distributed -e SHARD_NUMBER=2 -e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip --copy-whls
[2026-07-08T21:30:17.816Z] ++ set -ex
[2026-07-08T21:30:17.816Z] ++ set -o pipefail
[2026-07-08T21:30:17.816Z] ++ copy_whls=0
[2026-07-08T21:30:17.816Z] ++ envvars=()
[2026-07-08T21:30:17.816Z] ++ [[ 13 -gt 0 ]]
[2026-07-08T21:30:17.816Z] ++ key=-e
[2026-07-08T21:30:17.816Z] ++ case $key in
[2026-07-08T21:30:17.816Z] ++ envvars+=("-e $2")
[2026-07-08T21:30:17.816Z] ++ shift
[2026-07-08T21:30:17.816Z] ++ shift
[2026-07-08T21:30:17.816Z] ++ [[ 11 -gt 0 ]]
[2026-07-08T21:30:17.816Z] ++ key=-e
[2026-07-08T21:30:17.816Z] ++ case $key in
[2026-07-08T21:30:17.816Z] ++ envvars+=("-e $2")
[2026-07-08T21:30:17.816Z] ++ shift
[2026-07-08T21:30:17.816Z] ++ shift
[2026-07-08T21:30:17.816Z] ++ [[ 9 -gt 0 ]]
[2026-07-08T21:30:17.816Z] ++ key=-e
[2026-07-08T21:30:17.816Z] ++ case $key in
[2026-07-08T21:30:17.816Z] ++ envvars+=("-e $2")
[2026-07-08T21:30:17.816Z] ++ shift
[2026-07-08T21:30:17.816Z] ++ shift
[2026-07-08T21:30:17.816Z] ++ [[ 7 -gt 0 ]]
[2026-07-08T21:30:17.816Z] ++ key=-e
[2026-07-08T21:30:17.816Z] ++ case $key in
[2026-07-08T21:30:17.816Z] ++ envvars+=("-e $2")
[2026-07-08T21:30:17.816Z] ++ shift
[2026-07-08T21:30:17.816Z] ++ shift
[2026-07-08T21:30:17.816Z] ++ [[ 5 -gt 0 ]]
[2026-07-08T21:30:17.816Z] ++ key=-e
[2026-07-08T21:30:17.816Z] ++ case $key in
[2026-07-08T21:30:17.816Z] ++ envvars+=("-e $2")
[2026-07-08T21:30:17.816Z] ++ shift
[2026-07-08T21:30:17.816Z] ++ shift
[2026-07-08T21:30:17.816Z] ++ [[ 3 -gt 0 ]]
[2026-07-08T21:30:17.816Z] ++ key=-e
[2026-07-08T21:30:17.816Z] ++ case $key in
[2026-07-08T21:30:17.816Z] ++ envvars+=("-e $2")
[2026-07-08T21:30:17.816Z] ++ shift
[2026-07-08T21:30:17.816Z] ++ shift
[2026-07-08T21:30:17.816Z] ++ [[ 1 -gt 0 ]]
[2026-07-08T21:30:17.816Z] ++ key=--copy-whls
[2026-07-08T21:30:17.816Z] ++ case $key in
[2026-07-08T21:30:17.816Z] ++ copy_whls=1
[2026-07-08T21:30:17.816Z] ++ shift
[2026-07-08T21:30:17.816Z] ++ [[ 0 -gt 0 ]]
[2026-07-08T21:30:17.816Z] ++ envvars+=("-e PREBUILT=${PREBUILT:-false}")
[2026-07-08T21:30:17.816Z] ++ envvars+=("-e TEST_CORE=${TEST_CORE:-false}")
[2026-07-08T21:30:17.816Z] ++ [[ false == \t\r\u\e ]]
[2026-07-08T21:30:17.816Z] ++ [[ true == \t\r\u\e ]]
[2026-07-08T21:30:17.816Z] ++ envvars+=("-e TEST_CORE=${TEST_CORE}")
[2026-07-08T21:30:17.816Z] ++ docker ps -a
[2026-07-08T21:30:17.816Z] CONTAINER ID   IMAGE                                                                                                                        COMMAND                  CREATED         STATUS         PORTS     NAMES
[2026-07-08T21:30:17.816Z] c5e8e9d90838   rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942   "bash -lc 'sleep inf…"   5 minutes ago   Up 5 minutes             pytorch-ci-container
[2026-07-08T21:30:17.816Z] ++ docker rm -f pytorch-ci-container
[2026-07-08T21:30:18.962Z] ++ docker pull rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942
[2026-07-08T21:30:19.502Z] pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942: Pulling from rocm/pytorch-ci-private
[2026-07-08T21:30:19.502Z] Digest: sha256:b836aa0e221ac85a2184ec24a67b9d5089f3caa45d182cf521dee7d38112aa81
[2026-07-08T21:30:19.502Z] Status: Image is up to date for rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942
[2026-07-08T21:30:19.502Z] docker.io/rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942
[2026-07-08T21:30:19.502Z] ++ GPU_FLAGS=
[2026-07-08T21:30:19.502Z] ++ echo 'NODE_LABELS: multi_gpu rocm pytorch-nightly selftest ixt-hq-15 gfx908 multi_gpu rocm pytorch-nightly selftest ixt-hq-15 gfx908'
[2026-07-08T21:30:19.502Z] NODE_LABELS: multi_gpu rocm pytorch-nightly selftest ixt-hq-15 gfx908 multi_gpu rocm pytorch-nightly selftest ixt-hq-15 gfx908
[2026-07-08T21:30:19.502Z] ++ [[ ! multi_gpu rocm pytorch-nightly selftest ixt-hq-15 gfx908 =~ build-only ]]
[2026-07-08T21:30:19.502Z] ++ GPU_FLAGS='--group-add video --device=/dev/kfd --device=/dev/dri'
[2026-07-08T21:30:19.502Z] ++++ dirname prepare_docker_env.sh
[2026-07-08T21:30:19.502Z] +++ cd .
[2026-07-08T21:30:19.502Z] +++ pwd
[2026-07-08T21:30:19.502Z] ++ script_dir=/home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3413
[2026-07-08T21:30:19.502Z] ++ docker run --name pytorch-ci-container -e 'EXTENSION_BUILD_GFXARCH=gfx90a;gfx908;gfx942' -e CI=1 -e TEST_CONFIG=distributed -e SHARD_NUMBER=2 -e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip -e PREBUILT=false -e TEST_CORE=true -e TEST_CORE=true -v /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3413:/host_workspace -v /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3413:/scripts --entrypoint bash -d --network=host --group-add video --device=/dev/kfd --device=/dev/dri --ipc=host --pid=host --shm-size 8G --cap-add=SYS_PTRACE --security-opt seccomp=unconfined rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942 -lc 'sleep infinity'
[2026-07-08T21:30:20.035Z] f2bbcc5adcb5db1eb78be0b3d3802de13fc9b16d6eaaaf3341e482107e3368e5
[2026-07-08T21:30:20.035Z] ++ env -i python3 /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3413/pytorch/tools/stats/export_test_times.py
[2026-07-08T21:30:21.153Z] Exporting test times from test-infra
[2026-07-08T21:30:21.153Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-times.json to /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3413/pytorch/.additional_ci_files/test-times.json
[2026-07-08T21:30:21.153Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-class-times.json to /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3413/pytorch/.additional_ci_files/test-class-times.json
[2026-07-08T21:30:21.153Z] ++ python3 /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3413/pytorch/tools/stats/export_test_times.py
[2026-07-08T21:30:21.153Z] Exporting test times from test-infra
[2026-07-08T21:30:21.153Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-times.json to /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3413/pytorch/.additional_ci_files/test-times.json
[2026-07-08T21:30:21.153Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-class-times.json to /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3413/pytorch/.additional_ci_files/test-class-times.json
[2026-07-08T21:30:21.153Z] ++ [[ -d /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3413/pytorch/.additional_ci_files ]]
[2026-07-08T21:30:21.153Z] ++ sed -i -E 's/"\S+rocm\S+"/""/g' /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3413/pytorch/.additional_ci_files/test-times.json
[2026-07-08T21:30:21.153Z] ++ [[ false != \t\r\u\e ]]
[2026-07-08T21:30:21.153Z] ++ docker exec -u root pytorch-ci-container bash -c 'rm -rf /var/lib/jenkins/pytorch'
[2026-07-08T21:30:21.696Z] ++ docker cp /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3413/pytorch pytorch-ci-container:/var/lib/jenkins/pytorch
[2026-07-08T21:30:28.076Z] ++ docker exec -u root pytorch-ci-container bash -c 'chown root:root -R /var/lib/jenkins/pytorch'
[2026-07-08T21:30:28.076Z] ++ [[ 1 -eq 1 ]]
[2026-07-08T21:30:28.076Z] ++ docker cp /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3413/rocm_composable_kernel-7.1.1.dev0+g4266f8-py3-none-any.whl /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3413/torch-2.14.0a0+gitc3556ff-cp312-cp312-linux_x86_64.whl pytorch-ci-container:/var/lib/jenkins/pytorch
[2026-07-08T21:30:28.076Z] "docker cp" requires exactly 2 arguments.
[2026-07-08T21:30:28.076Z] See 'docker cp --help'.
[2026-07-08T21:30:28.076Z] 
[2026-07-08T21:30:28.076Z] Usage:  docker cp [OPTIONS] CONTAINER:SRC_PATH DEST_PATH|-
[2026-07-08T21:30:28.076Z] 	docker cp [OPTIONS] SRC_PATH|- CONTAINER:DEST_PATH
[2026-07-08T21:30:28.076Z] 
[2026-07-08T21:30:28.076Z] Copy files/folders between a container and the local filesystem

Tests / Test Distributed / Test Distributed / Run pytorch_distributed_2 / Error signal

Error in error step, with arguments pytorch_distributed_2 failed.

pytorch_distributed_2 failed

Tests / Test Inductor / Test Inductor / Run pytorch_inductor_1 / Shell Script

Error in sh step, with arguments ./test_pytorch_inductor.sh.

script returned exit code 1
Build log 
[2026-07-08T21:33:42.435Z] + ./test_pytorch_inductor.sh
[2026-07-08T21:33:42.435Z] + [[ -z '' ]]
[2026-07-08T21:33:42.435Z] + DEFAULT_EXECUTION_OPTIONS='-e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts         -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip         --copy-whls'
[2026-07-08T21:33:42.435Z] + source prepare_docker_env.sh -e 'EXTENSION_BUILD_GFXARCH=gfx90a;gfx908;gfx942' -e CI=1 -e TEST_CONFIG=inductor -e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip --copy-whls
[2026-07-08T21:33:42.435Z] ++ set -ex
[2026-07-08T21:33:42.435Z] ++ set -o pipefail
[2026-07-08T21:33:42.435Z] ++ copy_whls=0
[2026-07-08T21:33:42.435Z] ++ envvars=()
[2026-07-08T21:33:42.435Z] ++ [[ 11 -gt 0 ]]
[2026-07-08T21:33:42.435Z] ++ key=-e
[2026-07-08T21:33:42.435Z] ++ case $key in
[2026-07-08T21:33:42.435Z] ++ envvars+=("-e $2")
[2026-07-08T21:33:42.435Z] ++ shift
[2026-07-08T21:33:42.435Z] ++ shift
[2026-07-08T21:33:42.435Z] ++ [[ 9 -gt 0 ]]
[2026-07-08T21:33:42.435Z] ++ key=-e
[2026-07-08T21:33:42.435Z] ++ case $key in
[2026-07-08T21:33:42.435Z] ++ envvars+=("-e $2")
[2026-07-08T21:33:42.435Z] ++ shift
[2026-07-08T21:33:42.435Z] ++ shift
[2026-07-08T21:33:42.436Z] ++ [[ 7 -gt 0 ]]
[2026-07-08T21:33:42.436Z] ++ key=-e
[2026-07-08T21:33:42.436Z] ++ case $key in
[2026-07-08T21:33:42.436Z] ++ envvars+=("-e $2")
[2026-07-08T21:33:42.436Z] ++ shift
[2026-07-08T21:33:42.436Z] ++ shift
[2026-07-08T21:33:42.436Z] ++ [[ 5 -gt 0 ]]
[2026-07-08T21:33:42.436Z] ++ key=-e
[2026-07-08T21:33:42.436Z] ++ case $key in
[2026-07-08T21:33:42.436Z] ++ envvars+=("-e $2")
[2026-07-08T21:33:42.436Z] ++ shift
[2026-07-08T21:33:42.436Z] ++ shift
[2026-07-08T21:33:42.436Z] ++ [[ 3 -gt 0 ]]
[2026-07-08T21:33:42.436Z] ++ key=-e
[2026-07-08T21:33:42.436Z] ++ case $key in
[2026-07-08T21:33:42.436Z] ++ envvars+=("-e $2")
[2026-07-08T21:33:42.436Z] ++ shift
[2026-07-08T21:33:42.436Z] ++ shift
[2026-07-08T21:33:42.436Z] ++ [[ 1 -gt 0 ]]
[2026-07-08T21:33:42.436Z] ++ key=--copy-whls
[2026-07-08T21:33:42.436Z] ++ case $key in
[2026-07-08T21:33:42.436Z] ++ copy_whls=1
[2026-07-08T21:33:42.436Z] ++ shift
[2026-07-08T21:33:42.436Z] ++ [[ 0 -gt 0 ]]
[2026-07-08T21:33:42.436Z] ++ envvars+=("-e PREBUILT=${PREBUILT:-false}")
[2026-07-08T21:33:42.436Z] ++ envvars+=("-e TEST_CORE=${TEST_CORE:-false}")
[2026-07-08T21:33:42.436Z] ++ [[ false == \t\r\u\e ]]
[2026-07-08T21:33:42.436Z] ++ [[ true == \t\r\u\e ]]
[2026-07-08T21:33:42.436Z] ++ envvars+=("-e TEST_CORE=${TEST_CORE}")
[2026-07-08T21:33:42.436Z] ++ docker ps -a
[2026-07-08T21:33:42.436Z] CONTAINER ID   IMAGE                                                                                                                        COMMAND                  CREATED       STATUS                   PORTS                                                 NAMES
[2026-07-08T21:33:42.436Z] 003d6c0f642a   rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-1ae06b266a82a9290b48a6f384f86d9bfcc747b2-gfx908_gfx90a_gfx942   "bash -lc 'sleep inf…"   2 hours ago   Up 2 hours                                                                     pytorch-ci-container
[2026-07-08T21:33:42.436Z] 33af8c9e058f   dashboard-dashboard-ui                                                                                                       "/docker-entrypoint.…"   2 weeks ago   Up 21 hours              80/tcp, 0.0.0.0:3000->3000/tcp, [::]:3000->3000/tcp   rocagents-dashboard-ui
[2026-07-08T21:33:42.436Z] 7c6be83a2db1   dashboard-dashboard-api                                                                                                      "uvicorn main:app --…"   2 weeks ago   Up 21 hours (healthy)    0.0.0.0:8080->8080/tcp, [::]:8080->8080/tcp           rocagents-dashboard-api
[2026-07-08T21:33:42.436Z] 780dc8fe5a7b   rocm/eigen-test:ubuntu20.04-rocm5.3.0-eigen                                                                                  "/scripts/run_tests_…"   5 weeks ago   Exited (1) 5 weeks ago                                                         infallible_chatelet
[2026-07-08T21:33:42.436Z] 2d7786cfd066   deployment-rocagents                                                                                                         "python src/multi_re…"   8 weeks ago   Up 21 hours (healthy)                                                          rocagents-orchestrator
[2026-07-08T21:33:42.436Z] ++ docker rm -f pytorch-ci-container
[2026-07-08T21:33:45.777Z] ++ docker pull rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942
[2026-07-08T21:33:45.777Z] pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942: Pulling from rocm/pytorch-ci-private
[2026-07-08T21:33:45.777Z] Digest: sha256:b836aa0e221ac85a2184ec24a67b9d5089f3caa45d182cf521dee7d38112aa81
[2026-07-08T21:33:45.777Z] Status: Image is up to date for rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942
[2026-07-08T21:33:45.777Z] docker.io/rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942
[2026-07-08T21:33:45.777Z] ++ GPU_FLAGS=
[2026-07-08T21:33:45.777Z] ++ echo 'NODE_LABELS: multi_gpu rocm dlmodels gpu_8 gfx90a ixt-hq-ubb4-31 mlir MI250 multi_gpu rocm dlmodels gpu_8 gfx90a ixt-hq-ubb4-31 mlir MI250'
[2026-07-08T21:33:45.777Z] NODE_LABELS: multi_gpu rocm dlmodels gpu_8 gfx90a ixt-hq-ubb4-31 mlir MI250 multi_gpu rocm dlmodels gpu_8 gfx90a ixt-hq-ubb4-31 mlir MI250
[2026-07-08T21:33:45.777Z] ++ [[ ! multi_gpu rocm dlmodels gpu_8 gfx90a ixt-hq-ubb4-31 mlir MI250 =~ build-only ]]
[2026-07-08T21:33:45.777Z] ++ GPU_FLAGS='--group-add video --device=/dev/kfd --device=/dev/dri'
[2026-07-08T21:33:45.777Z] ++++ dirname prepare_docker_env.sh
[2026-07-08T21:33:45.777Z] +++ cd .
[2026-07-08T21:33:45.777Z] +++ pwd
[2026-07-08T21:33:45.777Z] ++ script_dir=/home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3413
[2026-07-08T21:33:45.777Z] ++ docker run --name pytorch-ci-container -e 'EXTENSION_BUILD_GFXARCH=gfx90a;gfx908;gfx942' -e CI=1 -e TEST_CONFIG=inductor -e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip -e PREBUILT=false -e TEST_CORE=true -e TEST_CORE=true -v /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3413:/host_workspace -v /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3413:/scripts --entrypoint bash -d --network=host --group-add video --device=/dev/kfd --device=/dev/dri --ipc=host --pid=host --shm-size 8G --cap-add=SYS_PTRACE --security-opt seccomp=unconfined rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942 -lc 'sleep infinity'
[2026-07-08T21:33:46.307Z] 2483ba88d4407adc59a58420e7f73840cc5a170165cda25dad008ae2d2be362d
[2026-07-08T21:33:46.307Z] ++ env -i python3 /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3413/pytorch/tools/stats/export_test_times.py
[2026-07-08T21:33:46.845Z] Exporting test times from test-infra
[2026-07-08T21:33:46.845Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-times.json to /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3413/pytorch/.additional_ci_files/test-times.json
[2026-07-08T21:33:46.845Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-class-times.json to /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3413/pytorch/.additional_ci_files/test-class-times.json
[2026-07-08T21:33:46.845Z] ++ python3 /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3413/pytorch/tools/stats/export_test_times.py
[2026-07-08T21:33:46.845Z] Exporting test times from test-infra
[2026-07-08T21:33:46.845Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-times.json to /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3413/pytorch/.additional_ci_files/test-times.json
[2026-07-08T21:33:46.845Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-class-times.json to /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3413/pytorch/.additional_ci_files/test-class-times.json
[2026-07-08T21:33:46.845Z] ++ [[ -d /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3413/pytorch/.additional_ci_files ]]
[2026-07-08T21:33:46.845Z] ++ sed -i -E 's/"\S+rocm\S+"/""/g' /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3413/pytorch/.additional_ci_files/test-times.json
[2026-07-08T21:33:46.845Z] ++ [[ false != \t\r\u\e ]]
[2026-07-08T21:33:46.845Z] ++ docker exec -u root pytorch-ci-container bash -c 'rm -rf /var/lib/jenkins/pytorch'
[2026-07-08T21:33:46.845Z] ++ docker cp /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3413/pytorch pytorch-ci-container:/var/lib/jenkins/pytorch
[2026-07-08T21:33:51.006Z] ++ docker exec -u root pytorch-ci-container bash -c 'chown root:root -R /var/lib/jenkins/pytorch'
[2026-07-08T21:33:51.532Z] ++ [[ 1 -eq 1 ]]
[2026-07-08T21:33:51.532Z] ++ docker cp /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3413/rocm_composable_kernel-7.1.1.dev0+g4266f8-py3-none-any.whl /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3413/torch-2.14.0a0+gitc3556ff-cp312-cp312-linux_x86_64.whl pytorch-ci-container:/var/lib/jenkins/pytorch
[2026-07-08T21:33:51.532Z] docker: 'docker cp' requires 2 arguments
[2026-07-08T21:33:51.532Z] 
[2026-07-08T21:33:51.532Z] Usage:  docker cp [OPTIONS] CONTAINER:SRC_PATH DEST_PATH|-
[2026-07-08T21:33:51.532Z] 	docker cp [OPTIONS] SRC_PATH|- CONTAINER:DEST_PATH
[2026-07-08T21:33:51.532Z] 
[2026-07-08T21:33:51.532Z] Run 'docker cp --help' for more information

Tests / Test Inductor / Test Inductor / Run pytorch_inductor_1 / Error signal

Output truncated.

Details

  • Kill older PR Builds (1.9 sec)
  • Initialize (1 hr 9 min)
    • Download CI scripts (35 sec)
    • Checkout Pytorch (2 min 33 sec)
    • Check base Docker image existence (10 sec)
    • Pull Docker Image (9 min 52 sec)
    • Build PyTorch (54 min)
  • Tests (56 min)
    • Test PyTorch (15 ms)
      • Test PyTorch (56 min)
        • Run pytorch_test_1 (12 min)
          Error: script returned exit code 1 - logs
          Error: pytorch_test_1 failed - logs
        • Run pytorch_test_2 (43 min)
          Error: script returned exit code 1 - logs
          Error: script returned exit code 1 - logs
          Error: pytorch_test_2 failed - logs
    • Test Distributed (9 ms)
      • Test Distributed (10 min)
        • Run pytorch_distributed_1 (4 min 29 sec)
          Error: script returned exit code 1 - logs
          Error: pytorch_distributed_1 failed - logs
        • Run pytorch_distributed_2 (5 min 17 sec)
          Error: script returned exit code 1 - logs
          Error: pytorch_distributed_2 failed - logs
    • Test Inductor (7 ms)
      • Test Inductor (13 min)
        • Run pytorch_inductor_1 (13 min)
          Error: script returned exit code 1 - logs
          Error: pytorch_inductor_1 failed - logs
    • Test PyTorch Slow (8 ms)
      • Test PyTorch Slow (13 sec)
    • Microbenchmark (26 sec)
      • Microbenchmark (13 sec)
  • Post Build (3.3 sec)
  • Declarative: Post Actions (5.6 sec)
View more details on ROCm Repo Management API