[CI] Parity: fold CUDA distributed-test split (pytorch#189232) into distributed config#3432
Open
ethanwee1 wants to merge 12 commits into
Open
[CI] Parity: fold CUDA distributed-test split (pytorch#189232) into distributed config#3432ethanwee1 wants to merge 12 commits into
ethanwee1 wants to merge 12 commits into
ROCm Repo Management API / Jenkins
failed
Jul 14, 2026 in 3h 46m 41s
Tests/Test Inductor/Run pytorch_inductor_1: error in 'error' step
Tests / Test PyTorch / Test PyTorch / Run pytorch_test_1 / Shell Script
Error in sh step, with arguments ./test_pytorch_test.sh.
script returned exit code 1
Build log
[2026-07-14T16:56:59.952Z] + ./test_pytorch_test.sh
[2026-07-14T16:56:59.952Z] + [[ -z '' ]]
[2026-07-14T16:56:59.952Z] + DEFAULT_EXECUTION_OPTIONS='-e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip --copy-whls'
[2026-07-14T16:56:59.952Z] + source prepare_docker_env.sh -e 'EXTENSION_BUILD_GFXARCH=gfx90a;gfx908;gfx942' -e CI=1 -e TEST_CONFIG=default -e SHARD_NUMBER=1 -e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip --copy-whls
[2026-07-14T16:56:59.952Z] ++ set -ex
[2026-07-14T16:56:59.952Z] ++ set -o pipefail
[2026-07-14T16:56:59.952Z] ++ copy_whls=0
[2026-07-14T16:56:59.952Z] ++ envvars=()
[2026-07-14T16:56:59.952Z] ++ [[ 13 -gt 0 ]]
[2026-07-14T16:56:59.952Z] ++ key=-e
[2026-07-14T16:56:59.952Z] ++ case $key in
[2026-07-14T16:56:59.952Z] ++ envvars+=("-e $2")
[2026-07-14T16:56:59.952Z] ++ shift
[2026-07-14T16:56:59.952Z] ++ shift
[2026-07-14T16:56:59.952Z] ++ [[ 11 -gt 0 ]]
[2026-07-14T16:56:59.952Z] ++ key=-e
[2026-07-14T16:56:59.952Z] ++ case $key in
[2026-07-14T16:56:59.952Z] ++ envvars+=("-e $2")
[2026-07-14T16:56:59.952Z] ++ shift
[2026-07-14T16:56:59.952Z] ++ shift
[2026-07-14T16:56:59.952Z] ++ [[ 9 -gt 0 ]]
[2026-07-14T16:56:59.952Z] ++ key=-e
[2026-07-14T16:56:59.952Z] ++ case $key in
[2026-07-14T16:56:59.952Z] ++ envvars+=("-e $2")
[2026-07-14T16:56:59.952Z] ++ shift
[2026-07-14T16:56:59.952Z] ++ shift
[2026-07-14T16:56:59.952Z] ++ [[ 7 -gt 0 ]]
[2026-07-14T16:56:59.952Z] ++ key=-e
[2026-07-14T16:56:59.952Z] ++ case $key in
[2026-07-14T16:56:59.952Z] ++ envvars+=("-e $2")
[2026-07-14T16:56:59.952Z] ++ shift
[2026-07-14T16:56:59.952Z] ++ shift
[2026-07-14T16:56:59.952Z] ++ [[ 5 -gt 0 ]]
[2026-07-14T16:56:59.952Z] ++ key=-e
[2026-07-14T16:56:59.952Z] ++ case $key in
[2026-07-14T16:56:59.952Z] ++ envvars+=("-e $2")
[2026-07-14T16:56:59.952Z] ++ shift
[2026-07-14T16:56:59.952Z] ++ shift
[2026-07-14T16:56:59.952Z] ++ [[ 3 -gt 0 ]]
[2026-07-14T16:56:59.952Z] ++ key=-e
[2026-07-14T16:56:59.952Z] ++ case $key in
[2026-07-14T16:56:59.952Z] ++ envvars+=("-e $2")
[2026-07-14T16:56:59.952Z] ++ shift
[2026-07-14T16:56:59.952Z] ++ shift
[2026-07-14T16:56:59.952Z] ++ [[ 1 -gt 0 ]]
[2026-07-14T16:56:59.952Z] ++ key=--copy-whls
[2026-07-14T16:56:59.952Z] ++ case $key in
[2026-07-14T16:56:59.952Z] ++ copy_whls=1
[2026-07-14T16:56:59.952Z] ++ shift
[2026-07-14T16:56:59.952Z] ++ [[ 0 -gt 0 ]]
[2026-07-14T16:56:59.952Z] ++ envvars+=("-e PREBUILT=${PREBUILT:-false}")
[2026-07-14T16:56:59.952Z] ++ envvars+=("-e TEST_CORE=${TEST_CORE:-false}")
[2026-07-14T16:56:59.952Z] ++ [[ false == \t\r\u\e ]]
[2026-07-14T16:56:59.952Z] ++ [[ true == \t\r\u\e ]]
[2026-07-14T16:56:59.952Z] ++ envvars+=("-e TEST_CORE=${TEST_CORE}")
[2026-07-14T16:56:59.952Z] ++ docker ps -a
[2026-07-14T16:56:59.952Z] CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
[2026-07-14T16:56:59.952Z] ++ docker rm -f pytorch-ci-container
[2026-07-14T16:56:59.952Z] ++ docker pull rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942
[2026-07-14T16:57:01.249Z] pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942: Pulling from rocm/pytorch-ci-private
[2026-07-14T16:57:01.635Z] cb259a83ac3d: Pulling fs layer
[2026-07-14T16:57:01.635Z] 8d78d529f04d: Pulling fs layer
[2026-07-14T16:57:01.635Z] 0e6d9b5f5e8d: Pulling fs layer
[2026-07-14T16:57:01.635Z] 99e6462d03e0: Pulling fs layer
[2026-07-14T16:57:01.635Z] db0af168a1a4: Pulling fs layer
[2026-07-14T16:57:01.635Z] 0a0ffadd8b4f: Pulling fs layer
[2026-07-14T16:57:01.635Z] 747272797eac: Pulling fs layer
[2026-07-14T16:57:01.635Z] bf3e4cf00cd6: Pulling fs layer
[2026-07-14T16:57:01.635Z] f0ef294e4f9b: Pulling fs layer
[2026-07-14T16:57:01.635Z] cccff27d82fb: Pulling fs layer
[2026-07-14T16:57:01.635Z] 42239a12ed73: Pulling fs layer
[2026-07-14T16:57:01.635Z] d9a8b776bea7: Pulling fs layer
[2026-07-14T16:57:01.635Z] 4106a853273f: Pulling fs layer
[2026-07-14T16:57:01.635Z] 4c8a34755929: Pulling fs layer
[2026-07-14T16:57:01.635Z] 0fcc3b23bd99: Pulling fs layer
[2026-07-14T16:57:01.635Z] 8b3d007242f8: Pulling fs layer
[2026-07-14T16:57:01.635Z] 5edf52847c35: Pulling fs layer
[2026-07-14T16:57:01.635Z] 3d56742d201d: Pulling fs layer
[2026-07-14T16:57:01.635Z] 906ac4de0a2b: Pulling fs layer
[2026-07-14T16:57:01.635Z] dedc784bccba: Pulling fs layer
[2026-07-14T16:57:01.635Z] 4cfd21f31c8b: Pulling fs layer
[2026-07-14T16:57:01.635Z] 423e26ddb517: Pulling fs layer
[2026-07-14T16:57:01.635Z] 3d56742d201d: Pulling fs layer
[2026-07-14T16:57:01.635Z] 78cb638d6686: Pulling fs layer
[2026-07-14T16:57:01.635Z] 65f281ce2e70: Pulling fs layer
[2026-07-14T16:57:01.635Z] 1057fba0c7df: Pulling fs layer
[2026-07-14T16:57:01.635Z] eab367f1ca39: Pulling fs layer
[2026-07-14T16:57:01.635Z] 27fcad4d96a6: Pulling fs layer
[2026-07-14T16:57:01.635Z] b09b7d04a8d9: Pulling fs layer
[2026-07-14T16:57:01.635Z] 86cc05da26d1: Pulling fs layer
[2026-07-14T16:57:01.635Z] 3d56742d201d: Pulling fs layer
[2026-07-14T16:57:01.635Z] d7df6379f9cc: Pulling fs layer
[2026-07-14T16:57:01.635Z] 95d704643610: Pulling fs layer
[2026-07-14T16:57:01.635Z] c0664a7365cb: Pulling fs layer
[2026-07-14T16:57:01.635Z] 241cbb1c2f26: Pulling fs layer
[2026-07-14T16:57:01.635Z] b96eaed0e751: Pulling fs layer
[2026-07-14T16:57:01.635Z] ed874654c321: Pulling fs layer
[2026-07-14T16:57:01.635Z] ba1fabf878bd: Pulling fs layer
[2026-07-14T16:57:01.635Z] ed86e5a5d9b1: Pulling fs layer
[2026-07-14T16:57:01.635Z] 9548298aa5f4: Pulling fs layer
[2026-07-14T16:57:01.635Z] 8d998c79d309: Pulling fs layer
[2026-07-14T16:57:01.635Z] f391428cfe0f: Pulling fs layer
[2026-07-14T16:57:01.635Z] 284e8cbb8a05: Pulling fs layer
[2026-07-14T16:57:01.635Z] 3d56742d201d: Pulling fs layer
[2026-07-14T16:57:01.635Z] 78a3c5370db4: Pulling fs layer
[2026-07-14T16:57:01.635Z] f5afebc61536: Pulling fs layer
[2026-07-14T16:57:01.635Z] 3d56742d201d: Pulling fs layer
[2026-07-14T16:57:01.635Z] 34e29023481f: Pulling fs layer
[2026-07-14T16:57:01.635Z] 07b45639fc6f: Pulling fs layer
[2026-07-14T16:57:01.635Z] f790d498bebb: Pulling fs layer
[2026-07-14T16:57:01.635Z] c608e00b1282: Pulling fs layer
[2026-07-14T16:57:01.635Z] f3058198fba5: Pulling fs layer
[2026-07-14T16:57:01.635Z] 5f8dd10c9c78: Pulling fs layer
[2026-07-14T16:57:01.635Z] 4f4fb700ef54: Pulling fs layer
[2026-07-14T16:57:01.635Z] fef65a673dc9: Pulling fs layer
[2026-07-14T16:57:01.635Z] 9ae773ccc4b2: Pulling fs layer
[2026-07-14T16:57:01.635Z] 4f4fb700ef54: Pulling fs layer
[2026-07-14T16:57:02.014Z] 69145e0460f6: Download complete
[2026-07-14T16:57:02.014Z] 8d78d529f04d: Download complete
[2026-07-14T16:57:02.393Z] bf3e4cf00cd6: Download complete
[2026-07-14T16:57:02.393Z] 99e6462d03e0: Download complete
[2026-07-14T16:57:02.393Z] 8b3d007242f8: Download complete
[2026-07-14T16:57:02.393Z] 5f8dd10c9c78: Download complete
[2026-07-14T16:57:02.393Z] db0af168a1a4: Download complete
[2026-07-14T16:57:02.393Z] 4cfd21f31c8b: Download complete
[2026-07-14T16:57:02.393Z] f0ef294e4f9b: Download complete
[2026-07-14T16:57:02.393Z] c608e00b1282: Download complete
[2026-07-14T16:57:02.393Z] 0a0ffadd8b4f: Download complete
[2026-07-14T16:57:02.393Z] 747272797eac: Download complete
[2026-07-14T16:57:02.393Z] 3d56742d201d: Download complete
[2026-07-14T16:57:02.393Z] 906ac4de0a2b: Download complete
[2026-07-14T16:57:02.393Z] 8d998c79d309: Download complete
[2026-07-14T16:57:02.393Z] b96eaed0e751: Download complete
[2026-07-14T16:57:02.393Z] 0fcc3b23bd99: Download complete
[2026-07-14T16:57:02.393Z] f790d498bebb: Download complete
[2026-07-14T16:57:02.393Z] f391428cfe0f: Download complete
[2026-07-14T16:57:02.393Z] 0e6d9b5f5e8d: Download complete
[2026-07-14T16:57:02.393Z] 27fcad4d96a6: Download complete
[2026-07-14T16:57:02.393Z] 9ae773ccc4b2: Download complete
[2026-07-14T16:57:02.393Z] 86cc05da26d1: Download complete
[2026-07-14T16:57:02.393Z] 4f4fb700ef54: Download complete
[2026-07-14T16:57:02.393Z] 9548298aa5f4: Download complete
[2026-07-14T16:57:02.393Z] d7df6379f9cc: Download complete
[2026-07-14T16:57:02.393Z] 241cbb1c2f26: Download complete
[2026-07-14T16:57:02.393Z] 07b45639fc6f: Download complete
[2026-07-14T16:57:02.393Z] ed874654c321: Download complete
[2026-07-14T16:57:02.393Z] b09b7d04a8d9: Download complete
[2026-07-14T16:57:02.393Z] fef65a673dc9: Download complete
[2026-07-14T16:57:02.393Z] f5afebc61536: Download complete
[2026-07-14T16:57:02.393Z] c0664a7365cb: Download complete
[2026-07-14T16:57:02.393Z] eab367f1ca39: Download complete
[2026-07-14T16:57:02.393Z] ed86e5a5d9b1: Download complete
[2026-07-14T16:57:02.393Z] f3058198fba5: Download complete
[2026-07-14T16:57:02.393Z] 65f281ce2e70: Download complete
[2026-07-14T16:57:02.393Z] 284e8cbb8a05: Download complete
[2026-07-14T16:57:02.393Z] 42239a12ed73: Download complete
[2026-07-14T16:57:02.393Z] 78a3c5370db4: Download complete
[2026-07-14T16:57:02.393Z] 423e26ddb517: Download complete
[2026-07-14T16:57:02.393Z] 34e29023481f: Download complete
[2026-07-14T16:57:02.771Z] 4106a853273f: Download complete
[2026-07-14T16:57:02.771Z] cb259a83ac3d: Download complete
[2026-07-14T16:57:03.149Z] ba1fabf878bd: Download complete
[2026-07-14T16:57:03.149Z] 4c8a34755929: Download complete
[2026-07-14T16:57:03.958Z] 99e6462d03e0: Pull complete
[2026-07-14T16:57:03.958Z] cb259a83ac3d: Pull complete
[2026-07-14T16:57:03.958Z] 95d704643610: Download complete
[2026-07-14T16:57:05.851Z] cccff27d82fb: Download complete
[2026-07-14T16:57:10.084Z] 1057fba0c7df: Download complete
[2026-07-14T16:57:16.644Z] 65f281ce2e70: Pull complete
[2026-07-14T16:57:16.644Z] 1057fba0c7df: Pull complete
[2026-07-14T16:57:16.644Z] 8d78d529f04d: Pull complete
[2026-07-14T16:57:16.644Z] c608e00b1282: Pull complete
[2026-07-14T16:57:16.644Z] 906ac4de0a2b: Pull complete
[2026-07-14T16:57:16.644Z] 0e6d9b5f5e8d: Pull complete
[2026-07-14T16:57:16.644Z] d7df6379f9cc: Pull complete
[2026-07-14T16:57:16.644Z] 34e29023481f: Pull complete
[2026-07-14T16:57:20.824Z] 78cb638d6686: Download complete
[2026-07-14T16:57:49.632Z] 5edf52847c35: Download complete
[2026-07-14T16:58:30.799Z] 5edf52847c35: Pull complete
[2026-07-14T16:58:30.799Z] 9548298aa5f4: Pull complete
[2026-07-14T16:58:30.799Z] db0af168a1a4: Pull complete
[2026-07-14T16:58:30.799Z] 0a0ffadd8b4f: Pull complete
[2026-07-14T16:58:30.799Z] 747272797eac: Pull complete
[2026-07-14T16:58:30.799Z] 86cc05da26d1: Pull complete
[2026-07-14T16:58:30.799Z] ed874654c321: Pull complete
[2026-07-14T16:58:30.799Z] 8b3d007242f8: Pull complete
[2026-07-14T16:59:01.308Z] d9a8b776bea7: Download complete
[2026-07-14T16:59:50.522Z] dedc784bccba: Download complete
[2026-07-14T17:01:15.148Z] dedc784bccba: Pull complete
[2026-07-14T17:01:15.148Z] bf3e4cf00cd6: Pull complete
[2026-07-14T17:01:15.148Z] 27fcad4d96a6: Pull complete
[2026-07-14T17:01:15.148Z] 9ae773ccc4b2: Pull complete
[2026-07-14T17:01:15.148Z] f3058198fba5: Pull complete
[2026-07-14T17:01:15.148Z] 78cb638d6686: Pull complete
[2026-07-14T17:01:15.148Z] f5afebc61536: Pull complete
[2026-07-14T17:01:15.978Z] cccff27d82fb: Pull complete
[2026-07-14T17:01:16.801Z] 07b45639fc6f: Pull complete
[2026-07-14T17:01:16.801Z] c0664a7365cb: Pull complete
[2026-07-14T17:01:16.801Z] b96eaed0e751: Pull complete
[2026-07-14T17:01:16.801Z] f790d498bebb: Pull complete
[2026-07-14T17:01:17.618Z] 5f8dd10c9c78: Pull complete
[2026-07-14T17:01:17.618Z] 4cfd21f31c8b: Pull complete
[2026-07-14T17:01:17.618Z] f0ef294e4f9b: Pull complete
[2026-07-14T17:01:17.618Z] 241cbb1c2f26: Pull complete
[2026-07-14T17:01:17.618Z] b09b7d04a8d9: Pull complete
[2026-07-14T17:01:17.618Z] 78a3c5370db4: Pull complete
[2026-07-14T17:01:17.618Z] 423e26ddb517: Pull complete
[2026-07-14T17:01:17.618Z] 4106a853273f: Pull complete
[2026-07-14T17:01:17.618Z] 95d704643610: Pull complete
[2026-07-14T17:01:17.618Z] 3d56742d201d: Pull complete
[2026-07-14T17:01:17.618Z] 0fcc3b23bd99: Pull complete
[2026-07-14T17:01:17.618Z] 4f4fb700ef54: Pull complete
[2026-07-14T17:01:17.618Z] 284e8cbb8a05: Pull complete
[2026-07-14T17:01:17.618Z] 42239a12ed73: Pull complete
[2026-07-14T17:02:16.613Z] f391428cfe0f: Pull complete
[2026-07-14T17:02:16.614Z] eab367f1ca39: Pull complete
[2026-07-14T17:02:16.614Z] d9a8b776bea7: Pull complete
[2026-07-14T17:02:16.614Z] 8d998c79d309: Pull complete
[2026-07-14T17:02:16.614Z] fef65a673dc9: Pull complete
[2026-07-14T17:02:16.614Z] ed86e5a5d9b1: Pull complete
[2026-07-14T17:02:16.614Z] ba1fabf878bd: Pull complete
[2026-07-14T17:02:16.614Z] 4c8a34755929: Pull complete
[2026-07-14T17:02:16.614Z] Digest: sha256:b836aa0e221ac85a2184ec24a67b9d5089f3caa45d182cf521dee7d38112aa81
[2026-07-14T17:02:16.614Z] Status: Downloaded newer image for rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942
[2026-07-14T17:02:16.614Z] docker.io/rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942
[2026-07-14T17:02:16.614Z] ++ GPU_FLAGS=
[2026-07-14T17:02:16.614Z] ++ echo 'NODE_LABELS: gfx90a mlir k8s-mi250-1-zc2mq gfx90a mlir k8s-mi250-1-zc2mq'
[2026-07-14T17:02:16.614Z] NODE_LABELS: gfx90a mlir k8s-mi250-1-zc2mq gfx90a mlir k8s-mi250-1-zc2mq
[2026-07-14T17:02:16.614Z] ++ [[ ! gfx90a mlir k8s-mi250-1-zc2mq =~ build-only ]]
[2026-07-14T17:02:16.614Z] ++ GPU_FLAGS='--group-add video --device=/dev/kfd --device=/dev/dri'
[2026-07-14T17:02:16.614Z] ++++ dirname prepare_docker_env.sh
[2026-07-14T17:02:16.614Z] +++ cd .
[2026-07-14T17:02:16.614Z] +++ pwd
[2026-07-14T17:02:16.614Z] ++ script_dir=/home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3432
[2026-07-14T17:02:16.614Z] ++ docker run --name pytorch-ci-container -e 'EXTENSION_BUILD_GFXARCH=gfx90a;gfx908;gfx942' -e CI=1 -e TEST_CONFIG=default -e SHARD_NUMBER=1 -e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip -e PREBUILT=false -e TEST_CORE=true -e TEST_CORE=true -v /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3432:/host_workspace -v /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3432:/scripts --entrypoint bash -d --network=host --group-add video --device=/dev/kfd --device=/dev/dri --ipc=host --pid=host --shm-size 8G --cap-add=SYS_PTRACE --security-opt seccomp=unconfined rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942 -lc 'sleep infinity'
[2026-07-14T17:02:26.323Z] 1df7f5443d01b5c0ec6b1207b2a2db182a7d90b9e1f45f5d63afe3f6e56f93b8
[2026-07-14T17:02:26.323Z] ++ env -i python3 /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3432/pytorch/tools/stats/export_test_times.py
[2026-07-14T17:02:34.257Z] Exporting test times from test-infra
[2026-07-14T17:02:34.257Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-times.json to /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3432/pytorch/.additional_ci_files/test-times.json
[2026-07-14T17:02:34.257Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-class-times.json to /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3432/pytorch/.additional_ci_files/test-class-times.json
[2026-07-14T17:02:34.257Z] ++ python3 /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3432/pytorch/tools/stats/export_test_times.py
[2026-07-14T17:02:34.257Z] Exporting test times from test-infra
[2026-07-14T17:02:34.257Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-times.json to /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3432/pytorch/.additional_ci_files/test-times.json
[2026-07-14T17:02:34.257Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-class-times.json to /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3432/pytorch/.additional_ci_files/test-class-times.json
[2026-07-14T17:02:34.257Z] ++ [[ -d /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3432/pytorch/.additional_ci_files ]]
[2026-07-14T17:02:34.257Z] ++ sed -i -E 's/"\S+rocm\S+"/""/g' /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3432/pytorch/.additional_ci_files/test-times.json
[2026-07-14T17:02:34.257Z] ++ [[ false != \t\r\u\e ]]
[2026-07-14T17:02:34.257Z] ++ docker exec -u root pytorch-ci-container bash -c 'rm -rf /var/lib/jenkins/pytorch'
[2026-07-14T17:02:34.257Z] ++ docker cp /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3432/pytorch pytorch-ci-container:/var/lib/jenkins/pytorch
[2026-07-14T17:02:37.598Z] ++ docker exec -u root pytorch-ci-container bash -c 'chown root:root -R /var/lib/jenkins/pytorch'
[2026-07-14T17:02:37.598Z] ++ [[ 1 -eq 1 ]]
[2026-07-14T17:02:37.598Z] ++ docker cp /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3432/rocm_composable_kernel-7.1.1.dev0+g4266f8-py3-none-any.whl /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3432/torch-2.14.0a0+git2614239-cp312-cp312-linux_x86_64.whl pytorch-ci-container:/var/lib/jenkins/pytorch
[2026-07-14T17:02:37.598Z] docker: 'docker cp' requires 2 arguments
[2026-07-14T17:02:37.598Z]
[2026-07-14T17:02:37.598Z] Usage: docker cp [OPTIONS] CONTAINER:SRC_PATH DEST_PATH|-
[2026-07-14T17:02:37.598Z] docker cp [OPTIONS] SRC_PATH|- CONTAINER:DEST_PATH
[2026-07-14T17:02:37.598Z]
[2026-07-14T17:02:37.598Z] Run 'docker cp --help' for more information
Tests / Test PyTorch / Test PyTorch / Run pytorch_test_1 / Error signal
Error in error step, with arguments pytorch_test_1 failed.
pytorch_test_1 failed
Tests / Test PyTorch / Test PyTorch / Run pytorch_test_2 / Check out from version control
Error in checkout step.
Error cloning remote repo 'origin'
Build log
Build log truncated.
[2026-07-14T18:32:32.378Z] fetch-pack: unexpected disconnect while reading sideband packet
[2026-07-14T18:32:32.378Z] fatal: early EOF
[2026-07-14T18:32:32.378Z] fatal: fetch-pack: invalid index-pack output
[2026-07-14T18:32:32.378Z]
[2026-07-14T18:32:32.378Z] at PluginClassLoader for git-client//org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2848)
[2026-07-14T18:32:32.378Z] at PluginClassLoader for git-client//org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:2189)
[2026-07-14T18:32:32.378Z] at PluginClassLoader for git-client//org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:638)
[2026-07-14T18:32:32.378Z] at PluginClassLoader for git-client//org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:880)
[2026-07-14T18:32:32.378Z] at PluginClassLoader for git-client//org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:173)
[2026-07-14T18:32:32.378Z] at PluginClassLoader for git-client//org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:164)
[2026-07-14T18:32:32.378Z] at hudson.remoting.UserRequest.perform(UserRequest.java:225)
[2026-07-14T18:32:32.378Z] at hudson.remoting.UserRequest.perform(UserRequest.java:50)
[2026-07-14T18:32:32.378Z] at hudson.remoting.Request$2.run(Request.java:391)
[2026-07-14T18:32:32.378Z] at hudson.remoting.InterceptingExecutorService.lambda$wrap$0(InterceptingExecutorService.java:81)
[2026-07-14T18:32:32.378Z] at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
[2026-07-14T18:32:32.378Z] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
[2026-07-14T18:32:32.378Z] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
[2026-07-14T18:32:32.378Z] at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:138)
[2026-07-14T18:32:32.378Z] at java.base/java.lang.Thread.run(Unknown Source)
[2026-07-14T18:32:32.378Z] Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to JNLP4-connect connection from 10.194.132.130/10.194.132.130:38085
[2026-07-14T18:32:32.378Z] at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1916)
[2026-07-14T18:32:32.378Z] at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:384)
[2026-07-14T18:32:32.378Z] at hudson.remoting.Channel.call(Channel.java:1108)
[2026-07-14T18:32:32.378Z] at PluginClassLoader for git-client//org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:155)
[2026-07-14T18:32:32.378Z] at jdk.internal.reflect.GeneratedMethodAccessor1263.invoke(Unknown Source)
[2026-07-14T18:32:32.378Z] at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
[2026-07-14T18:32:32.378Z] at java.base/java.lang.reflect.Method.invoke(Unknown Source)
[2026-07-14T18:32:32.378Z] at PluginClassLoader for git-client//org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:140)
[2026-07-14T18:32:32.378Z] at PluginClassLoader for git-client/jdk.proxy22/jdk.proxy22.$Proxy115.execute(Unknown Source)
[2026-07-14T18:32:32.378Z] at PluginClassLoader for git//hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1220)
[2026-07-14T18:32:32.378Z] at PluginClassLoader for git//hudson.plugins.git.GitSCM._checkout(GitSCM.java:1310)
[2026-07-14T18:32:32.378Z] at PluginClassLoader for git//hudson.plugins.git.GitSCM.checkout(GitSCM.java:1277)
[2026-07-14T18:32:32.378Z] at PluginClassLoader for workflow-scm-step//org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:136)
[2026-07-14T18:32:32.378Z] at PluginClassLoader for workflow-scm-step//org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:101)
[2026-07-14T18:32:32.378Z] at PluginClassLoader for workflow-scm-step//org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:88)
[2026-07-14T18:32:32.378Z] at PluginClassLoader for workflow-step-api//org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:49)
[2026-07-14T18:32:32.378Z] at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
[2026-07-14T18:32:32.378Z] at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
[2026-07-14T18:32:32.378Z] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
[2026-07-14T18:32:32.378Z] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
[2026-07-14T18:32:32.378Z] ... 1 more
[2026-07-14T18:32:32.378Z] ERROR: Error cloning remote repo 'origin'
[2026-07-14T18:32:32.378Z] ERROR: Maximum checkout retry attempts reached, aborting
Tests / Test PyTorch / Test PyTorch / Run pytorch_test_2 / Shell Script
Error in sh step, with arguments ./test_pytorch_test.sh.
script returned exit code 1
Build log
[2026-07-14T18:36:42.376Z] + ./test_pytorch_test.sh
[2026-07-14T18:36:42.377Z] + [[ -z '' ]]
[2026-07-14T18:36:42.377Z] + DEFAULT_EXECUTION_OPTIONS='-e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip --copy-whls'
[2026-07-14T18:36:42.377Z] + source prepare_docker_env.sh -e 'EXTENSION_BUILD_GFXARCH=gfx90a;gfx908;gfx942' -e CI=1 -e TEST_CONFIG=default -e SHARD_NUMBER=2 -e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip --copy-whls
[2026-07-14T18:36:42.377Z] ++ set -ex
[2026-07-14T18:36:42.377Z] ++ set -o pipefail
[2026-07-14T18:36:42.377Z] ++ copy_whls=0
[2026-07-14T18:36:42.377Z] ++ envvars=()
[2026-07-14T18:36:42.377Z] ++ [[ 13 -gt 0 ]]
[2026-07-14T18:36:42.377Z] ++ key=-e
[2026-07-14T18:36:42.377Z] ++ case $key in
[2026-07-14T18:36:42.377Z] ++ envvars+=("-e $2")
[2026-07-14T18:36:42.377Z] ++ shift
[2026-07-14T18:36:42.377Z] ++ shift
[2026-07-14T18:36:42.377Z] ++ [[ 11 -gt 0 ]]
[2026-07-14T18:36:42.377Z] ++ key=-e
[2026-07-14T18:36:42.377Z] ++ case $key in
[2026-07-14T18:36:42.377Z] ++ envvars+=("-e $2")
[2026-07-14T18:36:42.377Z] ++ shift
[2026-07-14T18:36:42.377Z] ++ shift
[2026-07-14T18:36:42.377Z] ++ [[ 9 -gt 0 ]]
[2026-07-14T18:36:42.377Z] ++ key=-e
[2026-07-14T18:36:42.377Z] ++ case $key in
[2026-07-14T18:36:42.377Z] ++ envvars+=("-e $2")
[2026-07-14T18:36:42.377Z] ++ shift
[2026-07-14T18:36:42.377Z] ++ shift
[2026-07-14T18:36:42.377Z] ++ [[ 7 -gt 0 ]]
[2026-07-14T18:36:42.377Z] ++ key=-e
[2026-07-14T18:36:42.377Z] ++ case $key in
[2026-07-14T18:36:42.377Z] ++ envvars+=("-e $2")
[2026-07-14T18:36:42.377Z] ++ shift
[2026-07-14T18:36:42.377Z] ++ shift
[2026-07-14T18:36:42.377Z] ++ [[ 5 -gt 0 ]]
[2026-07-14T18:36:42.377Z] ++ key=-e
[2026-07-14T18:36:42.377Z] ++ case $key in
[2026-07-14T18:36:42.377Z] ++ envvars+=("-e $2")
[2026-07-14T18:36:42.377Z] ++ shift
[2026-07-14T18:36:42.377Z] ++ shift
[2026-07-14T18:36:42.377Z] ++ [[ 3 -gt 0 ]]
[2026-07-14T18:36:42.377Z] ++ key=-e
[2026-07-14T18:36:42.377Z] ++ case $key in
[2026-07-14T18:36:42.377Z] ++ envvars+=("-e $2")
[2026-07-14T18:36:42.377Z] ++ shift
[2026-07-14T18:36:42.377Z] ++ shift
[2026-07-14T18:36:42.377Z] ++ [[ 1 -gt 0 ]]
[2026-07-14T18:36:42.377Z] ++ key=--copy-whls
[2026-07-14T18:36:42.377Z] ++ case $key in
[2026-07-14T18:36:42.377Z] ++ copy_whls=1
[2026-07-14T18:36:42.377Z] ++ shift
[2026-07-14T18:36:42.377Z] ++ [[ 0 -gt 0 ]]
[2026-07-14T18:36:42.377Z] ++ envvars+=("-e PREBUILT=${PREBUILT:-false}")
[2026-07-14T18:36:42.377Z] ++ envvars+=("-e TEST_CORE=${TEST_CORE:-false}")
[2026-07-14T18:36:42.377Z] ++ [[ false == \t\r\u\e ]]
[2026-07-14T18:36:42.377Z] ++ [[ true == \t\r\u\e ]]
[2026-07-14T18:36:42.377Z] ++ envvars+=("-e TEST_CORE=${TEST_CORE}")
[2026-07-14T18:36:42.377Z] ++ docker ps -a
[2026-07-14T18:36:42.377Z] CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
[2026-07-14T18:36:42.377Z] ++ docker rm -f pytorch-ci-container
[2026-07-14T18:36:42.377Z] ++ docker pull rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942
[2026-07-14T18:36:45.118Z] pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942: Pulling from rocm/pytorch-ci-private
[2026-07-14T18:36:51.614Z] 0e6d9b5f5e8d: Pulling fs layer
[2026-07-14T18:36:51.614Z] 1057fba0c7df: Pulling fs layer
[2026-07-14T18:36:51.614Z] 27fcad4d96a6: Pulling fs layer
[2026-07-14T18:36:51.614Z] bf3e4cf00cd6: Pulling fs layer
[2026-07-14T18:36:51.614Z] 9ae773ccc4b2: Pulling fs layer
[2026-07-14T18:36:51.614Z] 99e6462d03e0: Pulling fs layer
[2026-07-14T18:36:51.614Z] 0a0ffadd8b4f: Pulling fs layer
[2026-07-14T18:36:51.614Z] 747272797eac: Pulling fs layer
[2026-07-14T18:36:51.614Z] f790d498bebb: Pulling fs layer
[2026-07-14T18:36:51.614Z] 4f4fb700ef54: Pulling fs layer
[2026-07-14T18:36:51.614Z] f391428cfe0f: Pulling fs layer
[2026-07-14T18:36:51.614Z] dedc784bccba: Pulling fs layer
[2026-07-14T18:36:51.614Z] f5afebc61536: Pulling fs layer
[2026-07-14T18:36:51.614Z] 3d56742d201d: Pulling fs layer
[2026-07-14T18:36:51.614Z] eab367f1ca39: Pulling fs layer
[2026-07-14T18:36:51.614Z] 241cbb1c2f26: Pulling fs layer
[2026-07-14T18:36:51.614Z] 4106a853273f: Pulling fs layer
[2026-07-14T18:36:51.614Z] 3d56742d201d: Pulling fs layer
[2026-07-14T18:36:51.614Z] 8d78d529f04d: Pulling fs layer
[2026-07-14T18:36:51.614Z] 3d56742d201d: Pulling fs layer
[2026-07-14T18:36:51.614Z] 86cc05da26d1: Pulling fs layer
[2026-07-14T18:36:51.614Z] 4f4fb700ef54: Pulling fs layer
[2026-07-14T18:36:51.614Z] 0fcc3b23bd99: Pulling fs layer
[2026-07-14T18:36:51.614Z] 423e26ddb517: Pulling fs layer
[2026-07-14T18:36:51.614Z] f0ef294e4f9b: Pulling fs layer
[2026-07-14T18:36:51.614Z] 34e29023481f: Pulling fs layer
[2026-07-14T18:36:51.614Z] f3058198fba5: Pulling fs layer
[2026-07-14T18:36:51.614Z] cccff27d82fb: Pulling fs layer
[2026-07-14T18:36:51.615Z] ed874654c321: Pulling fs layer
[2026-07-14T18:36:51.615Z] 9548298aa5f4: Pulling fs layer
[2026-07-14T18:36:51.615Z] d9a8b776bea7: Pulling fs layer
[2026-07-14T18:36:51.615Z] 284e8cbb8a05: Pulling fs layer
[2026-07-14T18:36:51.615Z] 4c8a34755929: Pulling fs layer
[2026-07-14T18:36:51.615Z] 78cb638d6686: Pulling fs layer
[2026-07-14T18:36:51.615Z] c608e00b1282: Pulling fs layer
[2026-07-14T18:36:51.615Z] 906ac4de0a2b: Pulling fs layer
[2026-07-14T18:36:51.615Z] 3d56742d201d: Pulling fs layer
[2026-07-14T18:36:51.615Z] c0664a7365cb: Pulling fs layer
[2026-07-14T18:36:51.615Z] d7df6379f9cc: Pulling fs layer
[2026-07-14T18:36:51.615Z] 07b45639fc6f: Pulling fs layer
[2026-07-14T18:36:51.615Z] 78a3c5370db4: Pulling fs layer
[2026-07-14T18:36:51.615Z] 42239a12ed73: Pulling fs layer
[2026-07-14T18:36:51.615Z] 3d56742d201d: Pulling fs layer
[2026-07-14T18:36:51.615Z] 8b3d007242f8: Pulling fs layer
[2026-07-14T18:36:51.615Z] db0af168a1a4: Pulling fs layer
[2026-07-14T18:36:51.615Z] cb259a83ac3d: Pulling fs layer
[2026-07-14T18:36:51.615Z] 95d704643610: Pulling fs layer
[2026-07-14T18:36:51.615Z] 5edf52847c35: Pulling fs layer
[2026-07-14T18:36:51.615Z] fef65a673dc9: Pulling fs layer
[2026-07-14T18:36:51.615Z] 8d998c79d309: Pulling fs layer
[2026-07-14T18:36:51.615Z] 4cfd21f31c8b: Pulling fs layer
[2026-07-14T18:36:51.615Z] 5f8dd10c9c78: Pulling fs layer
[2026-07-14T18:36:51.615Z] ba1fabf878bd: Pulling fs layer
[2026-07-14T18:36:51.615Z] b09b7d04a8d9: Pulling fs layer
[2026-07-14T18:36:51.615Z] 65f281ce2e70: Pulling fs layer
[2026-07-14T18:36:51.615Z] ed86e5a5d9b1: Pulling fs layer
[2026-07-14T18:36:51.615Z] b96eaed0e751: Pulling fs layer
[2026-07-14T18:36:51.615Z] 69145e0460f6: Download complete
[2026-07-14T18:36:51.615Z] 27fcad4d96a6: Download complete
[2026-07-14T18:36:52.015Z] fef65a673dc9: Download complete
[2026-07-14T18:36:52.015Z] 99e6462d03e0: Download complete
[2026-07-14T18:36:52.015Z] db0af168a1a4: Download complete
[2026-07-14T18:36:52.015Z] 241cbb1c2f26: Download complete
[2026-07-14T18:36:52.015Z] bf3e4cf00cd6: Download complete
[2026-07-14T18:36:52.015Z] 4f4fb700ef54: Download complete
[2026-07-14T18:36:52.015Z] f5afebc61536: Download complete
[2026-07-14T18:36:52.015Z] 0fcc3b23bd99: Download complete
[2026-07-14T18:36:52.015Z] 747272797eac: Download complete
[2026-07-14T18:36:52.015Z] 86cc05da26d1: Download complete
[2026-07-14T18:36:52.015Z] f790d498bebb: Download complete
[2026-07-14T18:36:52.015Z] 9ae773ccc4b2: Download complete
[2026-07-14T18:36:52.015Z] 0a0ffadd8b4f: Download complete
[2026-07-14T18:36:52.015Z] 65f281ce2e70: Download complete
[2026-07-14T18:36:52.015Z] 8d998c79d309: Download complete
[2026-07-14T18:36:52.015Z] 5f8dd10c9c78: Download complete
[2026-07-14T18:36:52.015Z] b09b7d04a8d9: Download complete
[2026-07-14T18:36:52.015Z] 8d78d529f04d: Download complete
[2026-07-14T18:36:52.015Z] 34e29023481f: Download complete
[2026-07-14T18:36:52.015Z] 3d56742d201d: Download complete
[2026-07-14T18:36:52.015Z] 423e26ddb517: Download complete
[2026-07-14T18:36:52.015Z] 284e8cbb8a05: Download complete
[2026-07-14T18:36:52.413Z] 9548298aa5f4: Download complete
[2026-07-14T18:36:52.413Z] c608e00b1282: Download complete
[2026-07-14T18:36:52.413Z] 0e6d9b5f5e8d: Download complete
[2026-07-14T18:36:52.413Z] f3058198fba5: Download complete
[2026-07-14T18:36:52.413Z] f391428cfe0f: Download complete
[2026-07-14T18:36:52.413Z] eab367f1ca39: Download complete
[2026-07-14T18:36:52.413Z] c0664a7365cb: Download complete
[2026-07-14T18:36:52.413Z] 8b3d007242f8: Download complete
[2026-07-14T18:36:52.413Z] f0ef294e4f9b: Download complete
[2026-07-14T18:36:52.413Z] 906ac4de0a2b: Download complete
[2026-07-14T18:36:52.413Z] d7df6379f9cc: Download complete
[2026-07-14T18:36:52.413Z] 4106a853273f: Download complete
[2026-07-14T18:36:52.413Z] 07b45639fc6f: Download complete
[2026-07-14T18:36:52.413Z] 78a3c5370db4: Download complete
[2026-07-14T18:36:52.413Z] b96eaed0e751: Download complete
[2026-07-14T18:36:52.413Z] 42239a12ed73: Download complete
[2026-07-14T18:36:52.413Z] ed874654c321: Download complete
[2026-07-14T18:36:52.413Z] ed86e5a5d9b1: Download complete
[2026-07-14T18:36:52.413Z] 4cfd21f31c8b: Download complete
[2026-07-14T18:36:53.253Z] ba1fabf878bd: Download complete
[2026-07-14T18:36:53.253Z] cb259a83ac3d: Download complete
[2026-07-14T18:36:53.253Z] 4c8a34755929: Download complete
[2026-07-14T18:36:53.645Z] 95d704643610: Download complete
[2026-07-14T18:36:53.645Z] cb259a83ac3d: Pull complete
[2026-07-14T18:36:54.042Z] 99e6462d03e0: Pull complete
[2026-07-14T18:36:55.384Z] cccff27d82fb: Download complete
[2026-07-14T18:36:58.859Z] 1057fba0c7df: Download complete
[2026-07-14T18:37:06.788Z] 65f281ce2e70: Pull complete
[2026-07-14T18:37:06.788Z] 8d78d529f04d: Pull complete
[2026-07-14T18:37:06.788Z] 34e29023481f: Pull complete
[2026-07-14T18:37:06.788Z] 3d56742d201d: Pull complete
[2026-07-14T18:37:06.788Z] c608e00b1282: Pull complete
[2026-07-14T18:37:06.788Z] 0e6d9b5f5e8d: Pull complete
[2026-07-14T18:37:06.788Z] 906ac4de0a2b: Pull complete
[2026-07-14T18:37:06.788Z] d7df6379f9cc: Pull complete
[2026-07-14T18:37:06.788Z] 1057fba0c7df: Pull complete
[2026-07-14T18:37:10.131Z] 78cb638d6686: Download complete
[2026-07-14T18:37:39.194Z] 5edf52847c35: Download complete
[2026-07-14T18:38:14.231Z] 747272797eac: Pull complete
[2026-07-14T18:38:14.231Z] 86cc05da26d1: Pull complete
[2026-07-14T18:38:14.231Z] 9548298aa5f4: Pull complete
[2026-07-14T18:38:14.231Z] ed874654c321: Pull complete
[2026-07-14T18:38:14.231Z] 5edf52847c35: Pull complete
[2026-07-14T18:38:14.231Z] db0af168a1a4: Pull complete
[2026-07-14T18:38:14.231Z] 0a0ffadd8b4f: Pull complete
[2026-07-14T18:38:14.231Z] 8b3d007242f8: Pull complete
[2026-07-14T18:38:49.074Z] d9a8b776bea7: Download complete
[2026-07-14T18:39:38.790Z] dedc784bccba: Download complete
[2026-07-14T18:41:03.668Z] 27fcad4d96a6: Pull complete
[2026-07-14T18:41:03.668Z] bf3e4cf00cd6: Pull complete
[2026-07-14T18:41:03.668Z] 9ae773ccc4b2: Pull complete
[2026-07-14T18:41:03.668Z] dedc784bccba: Pull complete
[2026-07-14T18:41:07.002Z] f5afebc61536: Pull complete
[2026-07-14T18:41:07.002Z] f3058198fba5: Pull complete
[2026-07-14T18:41:07.002Z] 78cb638d6686: Pull complete
[2026-07-14T18:41:07.882Z] f790d498bebb: Pull complete
[2026-07-14T18:41:07.882Z] c0664a7365cb: Pull complete
[2026-07-14T18:41:07.882Z] 07b45639fc6f: Pull complete
[2026-07-14T18:41:07.882Z] b96eaed0e751: Pull complete
[2026-07-14T18:41:07.882Z] cccff27d82fb: Pull complete
[2026-07-14T18:41:08.340Z] 423e26ddb517: Pull complete
[2026-07-14T18:41:08.340Z] 95d704643610: Pull complete
[2026-07-14T18:41:08.340Z] 241cbb1c2f26: Pull complete
[2026-07-14T18:41:08.340Z] 0fcc3b23bd99: Pull complete
[2026-07-14T18:41:08.340Z] 5f8dd10c9c78: Pull complete
[2026-07-14T18:41:08.340Z] b09b7d04a8d9: Pull complete
[2026-07-14T18:41:08.340Z] 284e8cbb8a05: Pull complete
[2026-07-14T18:41:08.340Z] f0ef294e4f9b: Pull complete
[2026-07-14T18:41:08.340Z] 4106a853273f: Pull complete
[2026-07-14T18:41:08.340Z] 78a3c5370db4: Pull complete
[2026-07-14T18:41:08.340Z] 42239a12ed73: Pull complete
[2026-07-14T18:41:08.340Z] 4cfd21f31c8b: Pull complete
[2026-07-14T18:42:08.265Z] f391428cfe0f: Pull complete
[2026-07-14T18:42:08.265Z] eab367f1ca39: Pull complete
[2026-07-14T18:42:08.265Z] d9a8b776bea7: Pull complete
[2026-07-14T18:42:08.723Z] fef65a673dc9: Pull complete
[2026-07-14T18:42:08.723Z] 4f4fb700ef54: Pull complete
[2026-07-14T18:42:08.723Z] 8d998c79d309: Pull complete
[2026-07-14T18:42:08.723Z] ed86e5a5d9b1: Pull complete
[2026-07-14T18:42:08.723Z] ba1fabf878bd: Pull complete
[2026-07-14T18:42:08.723Z] 4c8a34755929: Pull complete
[2026-07-14T18:42:08.723Z] Digest: sha256:b836aa0e221ac85a2184ec24a67b9d5089f3caa45d182cf521dee7d38112aa81
[2026-07-14T18:42:08.723Z] Status: Downloaded newer image for rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942
[2026-07-14T18:42:08.723Z] docker.io/rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942
[2026-07-14T18:42:08.723Z] ++ GPU_FLAGS=
[2026-07-14T18:42:08.723Z] ++ echo 'NODE_LABELS: k8s-mi250-1-xgz2s gfx90a mlir k8s-mi250-1-xgz2s gfx90a mlir'
[2026-07-14T18:42:08.723Z] NODE_LABELS: k8s-mi250-1-xgz2s gfx90a mlir k8s-mi250-1-xgz2s gfx90a mlir
[2026-07-14T18:42:08.723Z] ++ [[ ! k8s-mi250-1-xgz2s gfx90a mlir =~ build-only ]]
[2026-07-14T18:42:08.723Z] ++ GPU_FLAGS='--group-add video --device=/dev/kfd --device=/dev/dri'
[2026-07-14T18:42:08.723Z] ++++ dirname prepare_docker_env.sh
[2026-07-14T18:42:08.723Z] +++ cd .
[2026-07-14T18:42:08.723Z] +++ pwd
[2026-07-14T18:42:08.723Z] ++ script_dir=/home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3432
[2026-07-14T18:42:08.723Z] ++ docker run --name pytorch-ci-container -e 'EXTENSION_BUILD_GFXARCH=gfx90a;gfx908;gfx942' -e CI=1 -e TEST_CONFIG=default -e SHARD_NUMBER=2 -e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip -e PREBUILT=false -e TEST_CORE=true -e TEST_CORE=true -v /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3432:/host_workspace -v /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3432:/scripts --entrypoint bash -d --network=host --group-add video --device=/dev/kfd --device=/dev/dri --ipc=host --pid=host --shm-size 8G --cap-add=SYS_PTRACE --security-opt seccomp=unconfined rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942 -lc 'sleep infinity'
[2026-07-14T18:42:12.206Z] 67d5cf4218f3567d4fb56385dc8f293945336939c9d9e3a83b6686341e520200
[2026-07-14T18:42:12.206Z] ++ env -i python3 /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3432/pytorch/tools/stats/export_test_times.py
[2026-07-14T18:42:13.109Z] Exporting test times from test-infra
[2026-07-14T18:42:13.109Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-times.json to /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3432/pytorch/.additional_ci_files/test-times.json
[2026-07-14T18:42:13.109Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-class-times.json to /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3432/pytorch/.additional_ci_files/test-class-times.json
[2026-07-14T18:42:13.109Z] ++ python3 /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3432/pytorch/tools/stats/export_test_times.py
[2026-07-14T18:42:13.527Z] Exporting test times from test-infra
[2026-07-14T18:42:13.527Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-times.json to /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3432/pytorch/.additional_ci_files/test-times.json
[2026-07-14T18:42:13.527Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-class-times.json to /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3432/pytorch/.additional_ci_files/test-class-times.json
[2026-07-14T18:42:13.527Z] ++ [[ -d /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3432/pytorch/.additional_ci_files ]]
[2026-07-14T18:42:13.527Z] ++ sed -i -E 's/"\S+rocm\S+"/""/g' /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3432/pytorch/.additional_ci_files/test-times.json
[2026-07-14T18:42:13.527Z] ++ [[ false != \t\r\u\e ]]
[2026-07-14T18:42:13.527Z] ++ docker exec -u root pytorch-ci-container bash -c 'rm -rf /var/lib/jenkins/pytorch'
[2026-07-14T18:42:13.527Z] ++ docker cp /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3432/pytorch pytorch-ci-container:/var/lib/jenkins/pytorch
[2026-07-14T18:42:17.061Z] ++ docker exec -u root pytorch-ci-container bash -c 'chown root:root -R /var/lib/jenkins/pytorch'
[2026-07-14T18:42:17.481Z] ++ [[ 1 -eq 1 ]]
[2026-07-14T18:42:17.481Z] ++ docker cp /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3432/rocm_composable_kernel-7.1.1.dev0+g4266f8-py3-none-any.whl /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3432/torch-2.14.0a0+git2614239-cp312-cp312-linux_x86_64.whl pytorch-ci-container:/var/lib/jenkins/pytorch
[2026-07-14T18:42:17.481Z] docker: 'docker cp' requires 2 arguments
[2026-07-14T18:42:17.481Z]
[2026-07-14T18:42:17.481Z] Usage: docker cp [OPTIONS] CONTAINER:SRC_PATH DEST_PATH|-
[2026-07-14T18:42:17.481Z] docker cp [OPTIONS] SRC_PATH|- CONTAINER:DEST_PATH
[2026-07-14T18:42:17.481Z]
[2026-07-14T18:42:17.481Z] Run 'docker cp --help' for more information
Tests / Test PyTorch / Test PyTorch / Run pytorch_test_2 / Error signal
Error in error step, with arguments pytorch_test_2 failed.
pytorch_test_2 failed
Tests / Test Distributed / Test Distributed / Run pytorch_distributed_1 / Shell Script
Error in sh step, with arguments ./test_pytorch_test_distributed.sh.
script returned exit code 1
Build log
[2026-07-14T16:10:16.184Z] + ./test_pytorch_test_distributed.sh
[2026-07-14T16:10:16.184Z] + [[ -z '' ]]
[2026-07-14T16:10:16.184Z] + DEFAULT_EXECUTION_OPTIONS='-e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip --copy-whls'
[2026-07-14T16:10:16.184Z] + source prepare_docker_env.sh -e 'EXTENSION_BUILD_GFXARCH=gfx90a;gfx908;gfx942' -e CI=1 -e TEST_CONFIG=distributed -e SHARD_NUMBER=1 -e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip --copy-whls
[2026-07-14T16:10:16.184Z] ++ set -ex
[2026-07-14T16:10:16.184Z] ++ set -o pipefail
[2026-07-14T16:10:16.184Z] ++ copy_whls=0
[2026-07-14T16:10:16.184Z] ++ envvars=()
[2026-07-14T16:10:16.184Z] ++ [[ 13 -gt 0 ]]
[2026-07-14T16:10:16.184Z] ++ key=-e
[2026-07-14T16:10:16.184Z] ++ case $key in
[2026-07-14T16:10:16.184Z] ++ envvars+=("-e $2")
[2026-07-14T16:10:16.184Z] ++ shift
[2026-07-14T16:10:16.184Z] ++ shift
[2026-07-14T16:10:16.184Z] ++ [[ 11 -gt 0 ]]
[2026-07-14T16:10:16.184Z] ++ key=-e
[2026-07-14T16:10:16.184Z] ++ case $key in
[2026-07-14T16:10:16.184Z] ++ envvars+=("-e $2")
[2026-07-14T16:10:16.184Z] ++ shift
[2026-07-14T16:10:16.184Z] ++ shift
[2026-07-14T16:10:16.184Z] ++ [[ 9 -gt 0 ]]
[2026-07-14T16:10:16.184Z] ++ key=-e
[2026-07-14T16:10:16.184Z] ++ case $key in
[2026-07-14T16:10:16.184Z] ++ envvars+=("-e $2")
[2026-07-14T16:10:16.184Z] ++ shift
[2026-07-14T16:10:16.184Z] ++ shift
[2026-07-14T16:10:16.184Z] ++ [[ 7 -gt 0 ]]
[2026-07-14T16:10:16.184Z] ++ key=-e
[2026-07-14T16:10:16.184Z] ++ case $key in
[2026-07-14T16:10:16.184Z] ++ envvars+=("-e $2")
[2026-07-14T16:10:16.184Z] ++ shift
[2026-07-14T16:10:16.184Z] ++ shift
[2026-07-14T16:10:16.184Z] ++ [[ 5 -gt 0 ]]
[2026-07-14T16:10:16.184Z] ++ key=-e
[2026-07-14T16:10:16.184Z] ++ case $key in
[2026-07-14T16:10:16.184Z] ++ envvars+=("-e $2")
[2026-07-14T16:10:16.184Z] ++ shift
[2026-07-14T16:10:16.184Z] ++ shift
[2026-07-14T16:10:16.184Z] ++ [[ 3 -gt 0 ]]
[2026-07-14T16:10:16.184Z] ++ key=-e
[2026-07-14T16:10:16.184Z] ++ case $key in
[2026-07-14T16:10:16.184Z] ++ envvars+=("-e $2")
[2026-07-14T16:10:16.184Z] ++ shift
[2026-07-14T16:10:16.184Z] ++ shift
[2026-07-14T16:10:16.184Z] ++ [[ 1 -gt 0 ]]
[2026-07-14T16:10:16.184Z] ++ key=--copy-whls
[2026-07-14T16:10:16.184Z] ++ case $key in
[2026-07-14T16:10:16.184Z] ++ copy_whls=1
[2026-07-14T16:10:16.184Z] ++ shift
[2026-07-14T16:10:16.184Z] ++ [[ 0 -gt 0 ]]
[2026-07-14T16:10:16.184Z] ++ envvars+=("-e PREBUILT=${PREBUILT:-false}")
[2026-07-14T16:10:16.184Z] ++ envvars+=("-e TEST_CORE=${TEST_CORE:-false}")
[2026-07-14T16:10:16.184Z] ++ [[ false == \t\r\u\e ]]
[2026-07-14T16:10:16.184Z] ++ [[ true == \t\r\u\e ]]
[2026-07-14T16:10:16.184Z] ++ envvars+=("-e TEST_CORE=${TEST_CORE}")
[2026-07-14T16:10:16.184Z] ++ docker ps -a
[2026-07-14T16:10:16.184Z] CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
[2026-07-14T16:10:16.184Z] f72db091c5a2 rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942 "bash -lc 'sleep inf…" 13 hours ago Up 13 hours pytorch-ci-container
[2026-07-14T16:10:16.184Z] ++ docker rm -f pytorch-ci-container
[2026-07-14T16:10:17.302Z] ++ docker pull rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942
[2026-07-14T16:10:17.854Z] pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942: Pulling from rocm/pytorch-ci-private
[2026-07-14T16:10:17.854Z] Digest: sha256:b836aa0e221ac85a2184ec24a67b9d5089f3caa45d182cf521dee7d38112aa81
[2026-07-14T16:10:17.854Z] Status: Image is up to date for rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942
[2026-07-14T16:10:17.854Z] docker.io/rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942
[2026-07-14T16:10:17.854Z] ++ GPU_FLAGS=
[2026-07-14T16:10:17.854Z] ++ echo 'NODE_LABELS: multi_gpu rocm pytorch-nightly selftest ixt-hq-15 gfx908 multi_gpu rocm pytorch-nightly selftest ixt-hq-15 gfx908'
[2026-07-14T16:10:17.854Z] NODE_LABELS: multi_gpu rocm pytorch-nightly selftest ixt-hq-15 gfx908 multi_gpu rocm pytorch-nightly selftest ixt-hq-15 gfx908
[2026-07-14T16:10:17.854Z] ++ [[ ! multi_gpu rocm pytorch-nightly selftest ixt-hq-15 gfx908 =~ build-only ]]
[2026-07-14T16:10:17.854Z] ++ GPU_FLAGS='--group-add video --device=/dev/kfd --device=/dev/dri'
[2026-07-14T16:10:17.854Z] ++++ dirname prepare_docker_env.sh
[2026-07-14T16:10:17.854Z] +++ cd .
[2026-07-14T16:10:17.854Z] +++ pwd
[2026-07-14T16:10:17.854Z] ++ script_dir=/home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3432
[2026-07-14T16:10:17.854Z] ++ docker run --name pytorch-ci-container -e 'EXTENSION_BUILD_GFXARCH=gfx90a;gfx908;gfx942' -e CI=1 -e TEST_CONFIG=distributed -e SHARD_NUMBER=1 -e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip -e PREBUILT=false -e TEST_CORE=true -e TEST_CORE=true -v /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3432:/host_workspace -v /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3432:/scripts --entrypoint bash -d --network=host --group-add video --device=/dev/kfd --device=/dev/dri --ipc=host --pid=host --shm-size 8G --cap-add=SYS_PTRACE --security-opt seccomp=unconfined rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942 -lc 'sleep infinity'
[2026-07-14T16:10:18.396Z] 576e7e5cab9910f78dda9e303a65067c2ace051c090e31676a9451bbd74c1e96
[2026-07-14T16:10:18.396Z] ++ env -i python3 /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3432/pytorch/tools/stats/export_test_times.py
[2026-07-14T16:10:19.503Z] Exporting test times from test-infra
[2026-07-14T16:10:19.503Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-times.json to /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3432/pytorch/.additional_ci_files/test-times.json
[2026-07-14T16:10:19.503Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-class-times.json to /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3432/pytorch/.additional_ci_files/test-class-times.json
[2026-07-14T16:10:19.503Z] ++ python3 /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3432/pytorch/tools/stats/export_test_times.py
[2026-07-14T16:10:20.024Z] Exporting test times from test-infra
[2026-07-14T16:10:20.024Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-times.json to /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3432/pytorch/.additional_ci_files/test-times.json
[2026-07-14T16:10:20.024Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-class-times.json to /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3432/pytorch/.additional_ci_files/test-class-times.json
[2026-07-14T16:10:20.024Z] ++ [[ -d /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3432/pytorch/.additional_ci_files ]]
[2026-07-14T16:10:20.024Z] ++ sed -i -E 's/"\S+rocm\S+"/""/g' /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3432/pytorch/.additional_ci_files/test-times.json
[2026-07-14T16:10:20.024Z] ++ [[ false != \t\r\u\e ]]
[2026-07-14T16:10:20.024Z] ++ docker exec -u root pytorch-ci-container bash -c 'rm -rf /var/lib/jenkins/pytorch'
[2026-07-14T16:10:20.024Z] ++ docker cp /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3432/pytorch pytorch-ci-container:/var/lib/jenkins/pytorch
[2026-07-14T16:10:26.449Z] ++ docker exec -u root pytorch-ci-container bash -c 'chown root:root -R /var/lib/jenkins/pytorch'
[2026-07-14T16:10:26.964Z] ++ [[ 1 -eq 1 ]]
[2026-07-14T16:10:26.964Z] ++ docker cp /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3432/rocm_composable_kernel-7.1.1.dev0+g4266f8-py3-none-any.whl /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3432/torch-2.14.0a0+git2614239-cp312-cp312-linux_x86_64.whl pytorch-ci-container:/var/lib/jenkins/pytorch
[2026-07-14T16:10:26.964Z] "docker cp" requires exactly 2 arguments.
[2026-07-14T16:10:26.964Z] See 'docker cp --help'.
[2026-07-14T16:10:26.964Z]
[2026-07-14T16:10:26.964Z] Usage: docker cp [OPTIONS] CONTAINER:SRC_PATH DEST_PATH|-
[2026-07-14T16:10:26.964Z] docker cp [OPTIONS] SRC_PATH|- CONTAINER:DEST_PATH
[2026-07-14T16:10:26.964Z]
[2026-07-14T16:10:26.964Z] Copy files/folders between a container and the local filesystem
Tests / Test Distributed / Test Distributed / Run pytorch_distributed_1 / Error signal
Error in error step, with arguments pytorch_distributed_1 failed.
pytorch_distributed_1 failed
Tests / Test Distributed / Test Distributed / Run pytorch_distributed_2 / Shell Script
Error in sh step, with arguments ./test_pytorch_test_distributed.sh.
script returned exit code 1
Build log
[2026-07-14T16:15:30.436Z] + ./test_pytorch_test_distributed.sh
[2026-07-14T16:15:30.436Z] + [[ -z '' ]]
[2026-07-14T16:15:30.436Z] + DEFAULT_EXECUTION_OPTIONS='-e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip --copy-whls'
[2026-07-14T16:15:30.436Z] + source prepare_docker_env.sh -e 'EXTENSION_BUILD_GFXARCH=gfx90a;gfx908;gfx942' -e CI=1 -e TEST_CONFIG=distributed -e SHARD_NUMBER=2 -e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip --copy-whls
[2026-07-14T16:15:30.436Z] ++ set -ex
[2026-07-14T16:15:30.436Z] ++ set -o pipefail
[2026-07-14T16:15:30.436Z] ++ copy_whls=0
[2026-07-14T16:15:30.436Z] ++ envvars=()
[2026-07-14T16:15:30.436Z] ++ [[ 13 -gt 0 ]]
[2026-07-14T16:15:30.436Z] ++ key=-e
[2026-07-14T16:15:30.436Z] ++ case $key in
[2026-07-14T16:15:30.436Z] ++ envvars+=("-e $2")
[2026-07-14T16:15:30.436Z] ++ shift
[2026-07-14T16:15:30.436Z] ++ shift
[2026-07-14T16:15:30.436Z] ++ [[ 11 -gt 0 ]]
[2026-07-14T16:15:30.436Z] ++ key=-e
[2026-07-14T16:15:30.436Z] ++ case $key in
[2026-07-14T16:15:30.436Z] ++ envvars+=("-e $2")
[2026-07-14T16:15:30.436Z] ++ shift
[2026-07-14T16:15:30.436Z] ++ shift
[2026-07-14T16:15:30.436Z] ++ [[ 9 -gt 0 ]]
[2026-07-14T16:15:30.436Z] ++ key=-e
[2026-07-14T16:15:30.436Z] ++ case $key in
[2026-07-14T16:15:30.436Z] ++ envvars+=("-e $2")
[2026-07-14T16:15:30.436Z] ++ shift
[2026-07-14T16:15:30.436Z] ++ shift
[2026-07-14T16:15:30.436Z] ++ [[ 7 -gt 0 ]]
[2026-07-14T16:15:30.436Z] ++ key=-e
[2026-07-14T16:15:30.436Z] ++ case $key in
[2026-07-14T16:15:30.436Z] ++ envvars+=("-e $2")
[2026-07-14T16:15:30.436Z] ++ shift
[2026-07-14T16:15:30.436Z] ++ shift
[2026-07-14T16:15:30.436Z] ++ [[ 5 -gt 0 ]]
[2026-07-14T16:15:30.436Z] ++ key=-e
[2026-07-14T16:15:30.436Z] ++ case $key in
[2026-07-14T16:15:30.436Z] ++ envvars+=("-e $2")
[2026-07-14T16:15:30.436Z] ++ shift
[2026-07-14T16:15:30.436Z] ++ shift
[2026-07-14T16:15:30.436Z] ++ [[ 3 -gt 0 ]]
[2026-07-14T16:15:30.436Z] ++ key=-e
[2026-07-14T16:15:30.436Z] ++ case $key in
[2026-07-14T16:15:30.436Z] ++ envvars+=("-e $2")
[2026-07-14T16:15:30.436Z] ++ shift
[2026-07-14T16:15:30.436Z] ++ shift
[2026-07-14T16:15:30.436Z] ++ [[ 1 -gt 0 ]]
[2026-07-14T16:15:30.436Z] ++ key=--copy-whls
[2026-07-14T16:15:30.436Z] ++ case $key in
[2026-07-14T16:15:30.436Z] ++ copy_whls=1
[2026-07-14T16:15:30.436Z] ++ shift
[2026-07-14T16:15:30.436Z] ++ [[ 0 -gt 0 ]]
[2026-07-14T16:15:30.436Z] ++ envvars+=("-e PREBUILT=${PREBUILT:-false}")
[2026-07-14T16:15:30.436Z] ++ envvars+=("-e TEST_CORE=${TEST_CORE:-false}")
[2026-07-14T16:15:30.436Z] ++ [[ false == \t\r\u\e ]]
[2026-07-14T16:15:30.436Z] ++ [[ true == \t\r\u\e ]]
[2026-07-14T16:15:30.436Z] ++ envvars+=("-e TEST_CORE=${TEST_CORE}")
[2026-07-14T16:15:30.436Z] ++ docker ps -a
[2026-07-14T16:15:30.436Z] CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
[2026-07-14T16:15:30.436Z] 576e7e5cab99 rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942 "bash -lc 'sleep inf…" 5 minutes ago Up 5 minutes pytorch-ci-container
[2026-07-14T16:15:30.436Z] ++ docker rm -f pytorch-ci-container
[2026-07-14T16:15:31.558Z] ++ docker pull rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942
[2026-07-14T16:15:32.090Z] pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942: Pulling from rocm/pytorch-ci-private
[2026-07-14T16:15:32.090Z] Digest: sha256:b836aa0e221ac85a2184ec24a67b9d5089f3caa45d182cf521dee7d38112aa81
[2026-07-14T16:15:32.090Z] Status: Image is up to date for rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942
[2026-07-14T16:15:32.090Z] docker.io/rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942
[2026-07-14T16:15:32.090Z] ++ GPU_FLAGS=
[2026-07-14T16:15:32.090Z] ++ echo 'NODE_LABELS: multi_gpu rocm pytorch-nightly selftest ixt-hq-15 gfx908 multi_gpu rocm pytorch-nightly selftest ixt-hq-15 gfx908'
[2026-07-14T16:15:32.090Z] NODE_LABELS: multi_gpu rocm pytorch-nightly selftest ixt-hq-15 gfx908 multi_gpu rocm pytorch-nightly selftest ixt-hq-15 gfx908
[2026-07-14T16:15:32.090Z] ++ [[ ! multi_gpu rocm pytorch-nightly selftest ixt-hq-15 gfx908 =~ build-only ]]
[2026-07-14T16:15:32.090Z] ++ GPU_FLAGS='--group-add video --device=/dev/kfd --device=/dev/dri'
[2026-07-14T16:15:32.090Z] ++++ dirname prepare_docker_env.sh
[2026-07-14T16:15:32.090Z] +++ cd .
[2026-07-14T16:15:32.090Z] +++ pwd
[2026-07-14T16:15:32.090Z] ++ script_dir=/home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3432
[2026-07-14T16:15:32.090Z] ++ docker run --name pytorch-ci-container -e 'EXTENSION_BUILD_GFXARCH=gfx90a;gfx908;gfx942' -e CI=1 -e TEST_CONFIG=distributed -e SHARD_NUMBER=2 -e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip -e PREBUILT=false -e TEST_CORE=true -e TEST_CORE=true -v /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3432:/host_workspace -v /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3432:/scripts --entrypoint bash -d --network=host --group-add video --device=/dev/kfd --device=/dev/dri --ipc=host --pid=host --shm-size 8G --cap-add=SYS_PTRACE --security-opt seccomp=unconfined rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942 -lc 'sleep infinity'
[2026-07-14T16:15:32.090Z] cba74b87bea4107c9f7217e86d96494b99fa920249cc4e86174e20c7b65599f4
[2026-07-14T16:15:32.707Z] ++ env -i python3 /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3432/pytorch/tools/stats/export_test_times.py
[2026-07-14T16:15:33.243Z] Exporting test times from test-infra
[2026-07-14T16:15:33.243Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-times.json to /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3432/pytorch/.additional_ci_files/test-times.json
[2026-07-14T16:15:33.243Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-class-times.json to /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3432/pytorch/.additional_ci_files/test-class-times.json
[2026-07-14T16:15:33.243Z] ++ python3 /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3432/pytorch/tools/stats/export_test_times.py
[2026-07-14T16:15:33.771Z] Exporting test times from test-infra
[2026-07-14T16:15:33.771Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-times.json to /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3432/pytorch/.additional_ci_files/test-times.json
[2026-07-14T16:15:33.771Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-class-times.json to /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3432/pytorch/.additional_ci_files/test-class-times.json
[2026-07-14T16:15:33.771Z] ++ [[ -d /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3432/pytorch/.additional_ci_files ]]
[2026-07-14T16:15:33.771Z] ++ sed -i -E 's/"\S+rocm\S+"/""/g' /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3432/pytorch/.additional_ci_files/test-times.json
[2026-07-14T16:15:33.771Z] ++ [[ false != \t\r\u\e ]]
[2026-07-14T16:15:33.771Z] ++ docker exec -u root pytorch-ci-container bash -c 'rm -rf /var/lib/jenkins/pytorch'
[2026-07-14T16:15:33.771Z] ++ docker cp /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3432/pytorch pytorch-ci-container:/var/lib/jenkins/pytorch
[2026-07-14T16:15:40.304Z] ++ docker exec -u root pytorch-ci-container bash -c 'chown root:root -R /var/lib/jenkins/pytorch'
[2026-07-14T16:15:40.304Z] ++ [[ 1 -eq 1 ]]
[2026-07-14T16:15:40.304Z] ++ docker cp /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3432/rocm_composable_kernel-7.1.1.dev0+g4266f8-py3-none-any.whl /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3432/torch-2.14.0a0+git2614239-cp312-cp312-linux_x86_64.whl pytorch-ci-container:/var/lib/jenkins/pytorch
[2026-07-14T16:15:40.514Z] "docker cp" requires exactly 2 arguments.
[2026-07-14T16:15:40.514Z] See 'docker cp --help'.
[2026-07-14T16:15:40.514Z]
[2026-07-14T16:15:40.514Z] Usage: docker cp [OPTIONS] CONTAINER:SRC_PATH DEST_PATH|-
[2026-07-14T16:15:40.514Z] docker cp [OPTIONS] SRC_PATH|- CONTAINER:DEST_PATH
[2026-07-14T16:15:40.514Z]
[2026-07-14T16:15:40.514Z] Copy files/folders between a container and the local filesystem
Tests / Test Distributed / Test Distributed / Run pytorch_distributed_2 / Error signal
Error in error step, with arguments pytorch_distributed_2 failed.
pytorch_distributed_2 failed
Tests / Test Inductor / Test Inductor / Run pytorch_inductor_1 / Shell Script
Error in sh step, with arguments ./test_pytorch_inductor.sh.
script returned exit code 1
Build log
[2026-07-14T16:57:16.526Z] + ./test_pytorch_inductor.sh
[2026-07-14T16:57:16.526Z] + [[ -z '' ]]
[2026-07-14T16:57:16.526Z] + DEFAULT_EXECUTION_OPTIONS='-e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip --copy-whls'
[2026-07-14T16:57:16.526Z] + source prepare_docker_env.sh -e 'EXTENSION_BUILD_GFXARCH=gfx90a;gfx908;gfx942' -e CI=1 -e TEST_CONFIG=inductor -e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip --copy-whls
[2026-07-14T16:57:16.526Z] ++ set -ex
[2026-07-14T16:57:16.526Z] ++ set -o pipefail
[2026-07-14T16:57:16.526Z] ++ copy_whls=0
[2026-07-14T16:57:16.526Z] ++ envvars=()
[2026-07-14T16:57:16.526Z] ++ [[ 11 -gt 0 ]]
[2026-07-14T16:57:16.526Z] ++ key=-e
[2026-07-14T16:57:16.526Z] ++ case $key in
[2026-07-14T16:57:16.526Z] ++ envvars+=("-e $2")
[2026-07-14T16:57:16.526Z] ++ shift
[2026-07-14T16:57:16.526Z] ++ shift
[2026-07-14T16:57:16.526Z] ++ [[ 9 -gt 0 ]]
[2026-07-14T16:57:16.526Z] ++ key=-e
[2026-07-14T16:57:16.526Z] ++ case $key in
[2026-07-14T16:57:16.526Z] ++ envvars+=("-e $2")
[2026-07-14T16:57:16.526Z] ++ shift
[2026-07-14T16:57:16.526Z] ++ shift
[2026-07-14T16:57:16.526Z] ++ [[ 7 -gt 0 ]]
[2026-07-14T16:57:16.526Z] ++ key=-e
[2026-07-14T16:57:16.526Z] ++ case $key in
[2026-07-14T16:57:16.526Z] ++ envvars+=("-e $2")
[2026-07-14T16:57:16.526Z] ++ shift
[2026-07-14T16:57:16.526Z] ++ shift
[2026-07-14T16:57:16.526Z] ++ [[ 5 -gt 0 ]]
[2026-07-14T16:57:16.526Z] ++ key=-e
[2026-07-14T16:57:16.526Z] ++ case $key in
[2026-07-14T16:57:16.526Z] ++ envvars+=("-e $2")
[2026-07-14T16:57:16.526Z] ++ shift
[2026-07-14T16:57:16.526Z] ++ shift
[2026-07-14T16:57:16.526Z] ++ [[ 3 -gt 0 ]]
[2026-07-14T16:57:16.526Z] ++ key=-e
[2026-07-14T16:57:16.526Z] ++ case $key in
[2026-07-14T16:57:16.526Z] ++ envvars+=("-e $2")
[2026-07-14T16:57:16.526Z] ++ shift
[2026-07-14T16:57:16.526Z] ++ shift
[2026-07-14T16:57:16.526Z] ++ [[ 1 -gt 0 ]]
[2026-07-14T16:57:16.526Z] ++ key=--copy-whls
[2026-07-14T16:57:16.526Z] ++ case $key in
[2026-07-14T16:57:16.526Z] ++ copy_whls=1
[2026-07-14T16:57:16.526Z] ++ shift
[2026-07-14T16:57:16.526Z] ++ [[ 0 -gt 0 ]]
[2026-07-14T16:57:16.526Z] ++ envvars+=("-e PREBUILT=${PREBUILT:-false}")
[2026-07-14T16:57:16.526Z] ++ envvars+=("-e TEST_CORE=${TEST_CORE:-false}")
[2026-07-14T16:57:16.526Z] ++ [[ false == \t\r\u\e ]]
[2026-07-14T16:57:16.526Z] ++ [[ true == \t\r\u\e ]]
[2026-07-14T16:57:16.526Z] ++ envvars+=("-e TEST_CORE=${TEST_CORE}")
[2026-07-14T16:57:16.526Z] ++ docker ps -a
[2026-07-14T16:57:16.526Z] CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
[2026-07-14T16:57:16.526Z] ++ docker rm -f pytorch-ci-container
[2026-07-14T16:57:16.526Z] ++ docker pull rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942
[2026-07-14T16:57:17.835Z] pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942: Pulling from rocm/pytorch-ci-private
[2026-07-14T16:57:18.217Z] 1057fba0c7df: Pulling fs layer
[2026-07-14T16:57:18.217Z] cb259a83ac3d: Pulling fs layer
[2026-07-14T16:57:18.217Z] b96eaed0e751: Pulling fs layer
[2026-07-14T16:57:18.217Z] dedc784bccba: Pulling fs layer
[2026-07-14T16:57:18.217Z] 0a0ffadd8b4f: Pulling fs layer
[2026-07-14T16:57:18.217Z] ed874654c321: Pulling fs layer
[2026-07-14T16:57:18.217Z] 4c8a34755929: Pulling fs layer
[2026-07-14T16:57:18.217Z] 34e29023481f: Pulling fs layer
[2026-07-14T16:57:18.217Z] 284e8cbb8a05: Pulling fs layer
[2026-07-14T16:57:18.217Z] 4106a853273f: Pulling fs layer
[2026-07-14T16:57:18.217Z] cccff27d82fb: Pulling fs layer
[2026-07-14T16:57:18.217Z] 747272797eac: Pulling fs layer
[2026-07-14T16:57:18.217Z] 0e6d9b5f5e8d: Pulling fs layer
[2026-07-14T16:57:18.217Z] 95d704643610: Pulling fs layer
[2026-07-14T16:57:18.217Z] 9ae773ccc4b2: Pulling fs layer
[2026-07-14T16:57:18.217Z] 5f8dd10c9c78: Pulling fs layer
[2026-07-14T16:57:18.217Z] 78a3c5370db4: Pulling fs layer
[2026-07-14T16:57:18.217Z] 86cc05da26d1: Pulling fs layer
[2026-07-14T16:57:18.217Z] f3058198fba5: Pulling fs layer
[2026-07-14T16:57:18.217Z] 5edf52847c35: Pulling fs layer
[2026-07-14T16:57:18.217Z] 3d56742d201d: Pulling fs layer
[2026-07-14T16:57:18.217Z] bf3e4cf00cd6: Pulling fs layer
[2026-07-14T16:57:18.217Z] c608e00b1282: Pulling fs layer
[2026-07-14T16:57:18.217Z] f391428cfe0f: Pulling fs layer
[2026-07-14T16:57:18.217Z] 27fcad4d96a6: Pulling fs layer
[2026-07-14T16:57:18.217Z] ed86e5a5d9b1: Pulling fs layer
[2026-07-14T16:57:18.217Z] d7df6379f9cc: Pulling fs layer
[2026-07-14T16:57:18.217Z] 42239a12ed73: Pulling fs layer
[2026-07-14T16:57:18.217Z] 65f281ce2e70: Pulling fs layer
[2026-07-14T16:57:18.217Z] eab367f1ca39: Pulling fs layer
[2026-07-14T16:57:18.217Z] 78cb638d6686: Pulling fs layer
[2026-07-14T16:57:18.217Z] 3d56742d201d: Pulling fs layer
[2026-07-14T16:57:18.217Z] 3d56742d201d: Pulling fs layer
[2026-07-14T16:57:18.217Z] fef65a673dc9: Pulling fs layer
[2026-07-14T16:57:18.217Z] 8d78d529f04d: Pulling fs layer
[2026-07-14T16:57:18.217Z] 3d56742d201d: Pulling fs layer
[2026-07-14T16:57:18.217Z] 8d998c79d309: Pulling fs layer
[2026-07-14T16:57:18.217Z] 8b3d007242f8: Pulling fs layer
[2026-07-14T16:57:18.217Z] 4cfd21f31c8b: Pulling fs layer
[2026-07-14T16:57:18.217Z] 99e6462d03e0: Pulling fs layer
[2026-07-14T16:57:18.217Z] d9a8b776bea7: Pulling fs layer
Output truncated.
Details
- Kill older PR Builds (2.9 sec)
- Initialize (1 hr 9 min)
- Download CI scripts (41 sec)
- Checkout Pytorch (1 min 31 sec)
- Check base Docker image existence (13 sec)
- Pull Docker Image (10 min)
- Build PyTorch (54 min)
- Tests (2 hr 36 min)
- Test PyTorch (12 ms)
- Test Distributed (11 ms)
- Test Inductor (10 ms)
- Test PyTorch Slow (7 ms)
- Test PyTorch Slow (13 sec)
- Microbenchmark (26 sec)
- Microbenchmark (13 sec)
- Post Build (1.9 sec)
- Declarative: Post Actions (3.8 sec)
Loading