[CI] Parity: fix flaky-test misattribution in log-based detection#3431
Open
ethanwee1 wants to merge 1 commit into
Open
[CI] Parity: fix flaky-test misattribution in log-based detection#3431ethanwee1 wants to merge 1 commit into
ethanwee1 wants to merge 1 commit into
ROCm Repo Management API / Jenkins
failed
Jul 14, 2026 in 4h 47m 40s
Tests/Test Inductor/Run pytorch_inductor_1: error in 'error' step
Tests / Test PyTorch / Test PyTorch / Run pytorch_test_1 / Shell Script
Error in sh step, with arguments ./test_pytorch_test.sh.
script returned exit code 1
Build log
[2026-07-14T00:00:08.368Z] + ./test_pytorch_test.sh
[2026-07-14T00:00:08.368Z] + [[ -z '' ]]
[2026-07-14T00:00:08.368Z] + DEFAULT_EXECUTION_OPTIONS='-e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip --copy-whls'
[2026-07-14T00:00:08.368Z] + source prepare_docker_env.sh -e 'EXTENSION_BUILD_GFXARCH=gfx90a;gfx908;gfx942' -e CI=1 -e TEST_CONFIG=default -e SHARD_NUMBER=1 -e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip --copy-whls
[2026-07-14T00:00:08.368Z] ++ set -ex
[2026-07-14T00:00:08.368Z] ++ set -o pipefail
[2026-07-14T00:00:08.368Z] ++ copy_whls=0
[2026-07-14T00:00:08.368Z] ++ envvars=()
[2026-07-14T00:00:08.368Z] ++ [[ 13 -gt 0 ]]
[2026-07-14T00:00:08.368Z] ++ key=-e
[2026-07-14T00:00:08.368Z] ++ case $key in
[2026-07-14T00:00:08.368Z] ++ envvars+=("-e $2")
[2026-07-14T00:00:08.368Z] ++ shift
[2026-07-14T00:00:08.368Z] ++ shift
[2026-07-14T00:00:08.368Z] ++ [[ 11 -gt 0 ]]
[2026-07-14T00:00:08.368Z] ++ key=-e
[2026-07-14T00:00:08.368Z] ++ case $key in
[2026-07-14T00:00:08.368Z] ++ envvars+=("-e $2")
[2026-07-14T00:00:08.369Z] ++ shift
[2026-07-14T00:00:08.369Z] ++ shift
[2026-07-14T00:00:08.369Z] ++ [[ 9 -gt 0 ]]
[2026-07-14T00:00:08.369Z] ++ key=-e
[2026-07-14T00:00:08.369Z] ++ case $key in
[2026-07-14T00:00:08.369Z] ++ envvars+=("-e $2")
[2026-07-14T00:00:08.369Z] ++ shift
[2026-07-14T00:00:08.369Z] ++ shift
[2026-07-14T00:00:08.369Z] ++ [[ 7 -gt 0 ]]
[2026-07-14T00:00:08.369Z] ++ key=-e
[2026-07-14T00:00:08.369Z] ++ case $key in
[2026-07-14T00:00:08.369Z] ++ envvars+=("-e $2")
[2026-07-14T00:00:08.369Z] ++ shift
[2026-07-14T00:00:08.369Z] ++ shift
[2026-07-14T00:00:08.369Z] ++ [[ 5 -gt 0 ]]
[2026-07-14T00:00:08.369Z] ++ key=-e
[2026-07-14T00:00:08.369Z] ++ case $key in
[2026-07-14T00:00:08.369Z] ++ envvars+=("-e $2")
[2026-07-14T00:00:08.369Z] ++ shift
[2026-07-14T00:00:08.369Z] ++ shift
[2026-07-14T00:00:08.369Z] ++ [[ 3 -gt 0 ]]
[2026-07-14T00:00:08.369Z] ++ key=-e
[2026-07-14T00:00:08.369Z] ++ case $key in
[2026-07-14T00:00:08.369Z] ++ envvars+=("-e $2")
[2026-07-14T00:00:08.369Z] ++ shift
[2026-07-14T00:00:08.369Z] ++ shift
[2026-07-14T00:00:08.369Z] ++ [[ 1 -gt 0 ]]
[2026-07-14T00:00:08.369Z] ++ key=--copy-whls
[2026-07-14T00:00:08.369Z] ++ case $key in
[2026-07-14T00:00:08.369Z] ++ copy_whls=1
[2026-07-14T00:00:08.369Z] ++ shift
[2026-07-14T00:00:08.369Z] ++ [[ 0 -gt 0 ]]
[2026-07-14T00:00:08.369Z] ++ envvars+=("-e PREBUILT=${PREBUILT:-false}")
[2026-07-14T00:00:08.369Z] ++ envvars+=("-e TEST_CORE=${TEST_CORE:-false}")
[2026-07-14T00:00:08.369Z] ++ [[ false == \t\r\u\e ]]
[2026-07-14T00:00:08.369Z] ++ [[ true == \t\r\u\e ]]
[2026-07-14T00:00:08.369Z] ++ envvars+=("-e TEST_CORE=${TEST_CORE}")
[2026-07-14T00:00:08.369Z] ++ docker ps -a
[2026-07-14T00:00:08.369Z] CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
[2026-07-14T00:00:08.369Z] ++ docker rm -f pytorch-ci-container
[2026-07-14T00:00:08.369Z] ++ docker pull rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942
[2026-07-14T00:00:09.659Z] pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942: Pulling from rocm/pytorch-ci-private
[2026-07-14T00:00:10.960Z] cb259a83ac3d: Pulling fs layer
[2026-07-14T00:00:10.960Z] 07b45639fc6f: Pulling fs layer
[2026-07-14T00:00:10.960Z] 4c8a34755929: Pulling fs layer
[2026-07-14T00:00:10.960Z] 0e6d9b5f5e8d: Pulling fs layer
[2026-07-14T00:00:10.960Z] 65f281ce2e70: Pulling fs layer
[2026-07-14T00:00:10.960Z] b96eaed0e751: Pulling fs layer
[2026-07-14T00:00:10.960Z] 99e6462d03e0: Pulling fs layer
[2026-07-14T00:00:10.960Z] f790d498bebb: Pulling fs layer
[2026-07-14T00:00:10.960Z] 8d78d529f04d: Pulling fs layer
[2026-07-14T00:00:10.960Z] 1057fba0c7df: Pulling fs layer
[2026-07-14T00:00:10.960Z] 0a0ffadd8b4f: Pulling fs layer
[2026-07-14T00:00:10.960Z] 95d704643610: Pulling fs layer
[2026-07-14T00:00:10.960Z] 8b3d007242f8: Pulling fs layer
[2026-07-14T00:00:10.960Z] 906ac4de0a2b: Pulling fs layer
[2026-07-14T00:00:10.960Z] 423e26ddb517: Pulling fs layer
[2026-07-14T00:00:10.960Z] 78cb638d6686: Pulling fs layer
[2026-07-14T00:00:10.960Z] f3058198fba5: Pulling fs layer
[2026-07-14T00:00:10.960Z] c608e00b1282: Pulling fs layer
[2026-07-14T00:00:10.960Z] 4106a853273f: Pulling fs layer
[2026-07-14T00:00:10.960Z] dedc784bccba: Pulling fs layer
[2026-07-14T00:00:10.960Z] f5afebc61536: Pulling fs layer
[2026-07-14T00:00:10.960Z] d7df6379f9cc: Pulling fs layer
[2026-07-14T00:00:10.960Z] 5f8dd10c9c78: Pulling fs layer
[2026-07-14T00:00:10.960Z] 27fcad4d96a6: Pulling fs layer
[2026-07-14T00:00:10.960Z] cccff27d82fb: Pulling fs layer
[2026-07-14T00:00:10.960Z] b09b7d04a8d9: Pulling fs layer
[2026-07-14T00:00:10.960Z] 3d56742d201d: Pulling fs layer
[2026-07-14T00:00:10.960Z] bf3e4cf00cd6: Pulling fs layer
[2026-07-14T00:00:10.960Z] c0664a7365cb: Pulling fs layer
[2026-07-14T00:00:10.960Z] 3d56742d201d: Pulling fs layer
[2026-07-14T00:00:10.960Z] 5edf52847c35: Pulling fs layer
[2026-07-14T00:00:10.960Z] 9ae773ccc4b2: Pulling fs layer
[2026-07-14T00:00:10.960Z] 4cfd21f31c8b: Pulling fs layer
[2026-07-14T00:00:10.960Z] 9548298aa5f4: Pulling fs layer
[2026-07-14T00:00:10.960Z] d9a8b776bea7: Pulling fs layer
[2026-07-14T00:00:10.960Z] f391428cfe0f: Pulling fs layer
[2026-07-14T00:00:10.960Z] 86cc05da26d1: Pulling fs layer
[2026-07-14T00:00:10.960Z] 241cbb1c2f26: Pulling fs layer
[2026-07-14T00:00:10.960Z] eab367f1ca39: Pulling fs layer
[2026-07-14T00:00:10.960Z] 747272797eac: Pulling fs layer
[2026-07-14T00:00:10.960Z] ed874654c321: Pulling fs layer
[2026-07-14T00:00:10.960Z] f0ef294e4f9b: Pulling fs layer
[2026-07-14T00:00:10.960Z] 3d56742d201d: Pulling fs layer
[2026-07-14T00:00:10.960Z] ba1fabf878bd: Pulling fs layer
[2026-07-14T00:00:10.960Z] db0af168a1a4: Pulling fs layer
[2026-07-14T00:00:10.960Z] 4f4fb700ef54: Pulling fs layer
[2026-07-14T00:00:10.960Z] 3d56742d201d: Pulling fs layer
[2026-07-14T00:00:10.960Z] ed86e5a5d9b1: Pulling fs layer
[2026-07-14T00:00:10.960Z] 0fcc3b23bd99: Pulling fs layer
[2026-07-14T00:00:10.960Z] 78a3c5370db4: Pulling fs layer
[2026-07-14T00:00:10.960Z] 4f4fb700ef54: Pulling fs layer
[2026-07-14T00:00:10.960Z] 284e8cbb8a05: Pulling fs layer
[2026-07-14T00:00:10.960Z] 3d56742d201d: Pulling fs layer
[2026-07-14T00:00:10.960Z] 42239a12ed73: Pulling fs layer
[2026-07-14T00:00:10.960Z] 8d998c79d309: Pulling fs layer
[2026-07-14T00:00:10.960Z] fef65a673dc9: Pulling fs layer
[2026-07-14T00:00:10.960Z] 34e29023481f: Pulling fs layer
[2026-07-14T00:00:10.960Z] 69145e0460f6: Download complete
[2026-07-14T00:00:10.960Z] 34e29023481f: Download complete
[2026-07-14T00:00:11.337Z] 65f281ce2e70: Download complete
[2026-07-14T00:00:11.337Z] 8b3d007242f8: Download complete
[2026-07-14T00:00:11.337Z] c608e00b1282: Download complete
[2026-07-14T00:00:11.337Z] f5afebc61536: Download complete
[2026-07-14T00:00:11.337Z] 27fcad4d96a6: Download complete
[2026-07-14T00:00:11.337Z] f0ef294e4f9b: Download complete
[2026-07-14T00:00:11.337Z] b96eaed0e751: Download complete
[2026-07-14T00:00:11.337Z] 3d56742d201d: Download complete
[2026-07-14T00:00:11.338Z] 4cfd21f31c8b: Download complete
[2026-07-14T00:00:11.338Z] 747272797eac: Download complete
[2026-07-14T00:00:11.338Z] 4f4fb700ef54: Download complete
[2026-07-14T00:00:11.338Z] 0fcc3b23bd99: Download complete
[2026-07-14T00:00:11.338Z] 42239a12ed73: Download complete
[2026-07-14T00:00:11.338Z] d7df6379f9cc: Download complete
[2026-07-14T00:00:11.338Z] 906ac4de0a2b: Download complete
[2026-07-14T00:00:11.338Z] f3058198fba5: Download complete
[2026-07-14T00:00:11.338Z] f391428cfe0f: Download complete
[2026-07-14T00:00:11.338Z] 284e8cbb8a05: Download complete
[2026-07-14T00:00:11.338Z] 07b45639fc6f: Download complete
[2026-07-14T00:00:11.338Z] 8d78d529f04d: Download complete
[2026-07-14T00:00:11.338Z] 0a0ffadd8b4f: Download complete
[2026-07-14T00:00:11.338Z] eab367f1ca39: Download complete
[2026-07-14T00:00:11.338Z] 78a3c5370db4: Download complete
[2026-07-14T00:00:11.338Z] 9ae773ccc4b2: Download complete
[2026-07-14T00:00:11.338Z] 86cc05da26d1: Download complete
[2026-07-14T00:00:11.338Z] 241cbb1c2f26: Download complete
[2026-07-14T00:00:11.338Z] ed874654c321: Download complete
[2026-07-14T00:00:11.338Z] ed86e5a5d9b1: Download complete
[2026-07-14T00:00:11.338Z] 8d998c79d309: Download complete
[2026-07-14T00:00:11.338Z] fef65a673dc9: Download complete
[2026-07-14T00:00:11.338Z] 5f8dd10c9c78: Download complete
[2026-07-14T00:00:11.338Z] b09b7d04a8d9: Download complete
[2026-07-14T00:00:11.338Z] f790d498bebb: Download complete
[2026-07-14T00:00:11.338Z] 423e26ddb517: Download complete
[2026-07-14T00:00:11.338Z] bf3e4cf00cd6: Download complete
[2026-07-14T00:00:11.338Z] c0664a7365cb: Download complete
[2026-07-14T00:00:11.338Z] 99e6462d03e0: Download complete
[2026-07-14T00:00:11.716Z] db0af168a1a4: Download complete
[2026-07-14T00:00:11.716Z] 9548298aa5f4: Download complete
[2026-07-14T00:00:11.716Z] 0e6d9b5f5e8d: Download complete
[2026-07-14T00:00:11.717Z] 4106a853273f: Download complete
[2026-07-14T00:00:12.099Z] ba1fabf878bd: Download complete
[2026-07-14T00:00:12.477Z] cb259a83ac3d: Download complete
[2026-07-14T00:00:12.477Z] 4c8a34755929: Download complete
[2026-07-14T00:00:12.859Z] 95d704643610: Download complete
[2026-07-14T00:00:12.859Z] 99e6462d03e0: Pull complete
[2026-07-14T00:00:12.859Z] cb259a83ac3d: Pull complete
[2026-07-14T00:00:14.736Z] cccff27d82fb: Download complete
[2026-07-14T00:00:18.860Z] 1057fba0c7df: Download complete
[2026-07-14T00:00:24.028Z] 0e6d9b5f5e8d: Pull complete
[2026-07-14T00:00:24.028Z] 1057fba0c7df: Pull complete
[2026-07-14T00:00:24.028Z] 34e29023481f: Pull complete
[2026-07-14T00:00:24.028Z] 65f281ce2e70: Pull complete
[2026-07-14T00:00:24.028Z] c608e00b1282: Pull complete
[2026-07-14T00:00:24.028Z] 3d56742d201d: Pull complete
[2026-07-14T00:00:24.028Z] d7df6379f9cc: Pull complete
[2026-07-14T00:00:24.028Z] 906ac4de0a2b: Pull complete
[2026-07-14T00:00:24.028Z] 8d78d529f04d: Pull complete
[2026-07-14T00:00:28.192Z] 78cb638d6686: Download complete
[2026-07-14T00:00:56.848Z] 5edf52847c35: Download complete
[2026-07-14T00:01:31.073Z] 8b3d007242f8: Pull complete
[2026-07-14T00:01:31.073Z] 747272797eac: Pull complete
[2026-07-14T00:01:31.073Z] 0a0ffadd8b4f: Pull complete
[2026-07-14T00:01:31.073Z] 86cc05da26d1: Pull complete
[2026-07-14T00:01:31.073Z] ed874654c321: Pull complete
[2026-07-14T00:01:31.073Z] db0af168a1a4: Pull complete
[2026-07-14T00:01:31.073Z] 9548298aa5f4: Pull complete
[2026-07-14T00:01:31.073Z] 5edf52847c35: Pull complete
[2026-07-14T00:02:12.050Z] d9a8b776bea7: Download complete
[2026-07-14T00:02:53.008Z] dedc784bccba: Download complete
[2026-07-14T00:04:17.291Z] 27fcad4d96a6: Pull complete
[2026-07-14T00:04:17.291Z] 9ae773ccc4b2: Pull complete
[2026-07-14T00:04:17.291Z] bf3e4cf00cd6: Pull complete
[2026-07-14T00:04:17.291Z] dedc784bccba: Pull complete
[2026-07-14T00:04:25.122Z] 78cb638d6686: Pull complete
[2026-07-14T00:04:25.943Z] f3058198fba5: Pull complete
[2026-07-14T00:04:27.825Z] f5afebc61536: Pull complete
[2026-07-14T00:04:29.711Z] cccff27d82fb: Pull complete
[2026-07-14T00:04:30.529Z] c0664a7365cb: Pull complete
[2026-07-14T00:04:31.348Z] b96eaed0e751: Pull complete
[2026-07-14T00:04:31.348Z] 07b45639fc6f: Pull complete
[2026-07-14T00:04:31.732Z] f790d498bebb: Pull complete
[2026-07-14T00:04:32.117Z] f0ef294e4f9b: Pull complete
[2026-07-14T00:04:32.117Z] 4cfd21f31c8b: Pull complete
[2026-07-14T00:04:32.117Z] 4f4fb700ef54: Pull complete
[2026-07-14T00:04:32.117Z] 42239a12ed73: Pull complete
[2026-07-14T00:04:32.117Z] 78a3c5370db4: Pull complete
[2026-07-14T00:04:32.117Z] 241cbb1c2f26: Pull complete
[2026-07-14T00:04:32.117Z] 5f8dd10c9c78: Pull complete
[2026-07-14T00:04:32.117Z] b09b7d04a8d9: Pull complete
[2026-07-14T00:04:32.117Z] 423e26ddb517: Pull complete
[2026-07-14T00:04:32.117Z] 4106a853273f: Pull complete
[2026-07-14T00:04:32.117Z] 95d704643610: Pull complete
[2026-07-14T00:04:32.117Z] 0fcc3b23bd99: Pull complete
[2026-07-14T00:04:32.117Z] 284e8cbb8a05: Pull complete
[2026-07-14T00:05:31.059Z] f391428cfe0f: Pull complete
[2026-07-14T00:05:31.059Z] eab367f1ca39: Pull complete
[2026-07-14T00:05:31.059Z] d9a8b776bea7: Pull complete
[2026-07-14T00:05:31.059Z] ed86e5a5d9b1: Pull complete
[2026-07-14T00:05:31.059Z] 8d998c79d309: Pull complete
[2026-07-14T00:05:31.059Z] fef65a673dc9: Pull complete
[2026-07-14T00:05:31.059Z] ba1fabf878bd: Pull complete
[2026-07-14T00:05:31.059Z] 4c8a34755929: Pull complete
[2026-07-14T00:05:31.059Z] Digest: sha256:b836aa0e221ac85a2184ec24a67b9d5089f3caa45d182cf521dee7d38112aa81
[2026-07-14T00:05:31.059Z] Status: Downloaded newer image for rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942
[2026-07-14T00:05:31.059Z] docker.io/rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942
[2026-07-14T00:05:31.059Z] ++ GPU_FLAGS=
[2026-07-14T00:05:31.059Z] ++ echo 'NODE_LABELS: k8s-mi250-1-1mzrg gfx90a mlir k8s-mi250-1-1mzrg gfx90a mlir'
[2026-07-14T00:05:31.059Z] NODE_LABELS: k8s-mi250-1-1mzrg gfx90a mlir k8s-mi250-1-1mzrg gfx90a mlir
[2026-07-14T00:05:31.059Z] ++ [[ ! k8s-mi250-1-1mzrg gfx90a mlir =~ build-only ]]
[2026-07-14T00:05:31.059Z] ++ GPU_FLAGS='--group-add video --device=/dev/kfd --device=/dev/dri'
[2026-07-14T00:05:31.059Z] ++++ dirname prepare_docker_env.sh
[2026-07-14T00:05:31.059Z] +++ cd .
[2026-07-14T00:05:31.059Z] +++ pwd
[2026-07-14T00:05:31.059Z] ++ script_dir=/home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3431
[2026-07-14T00:05:31.059Z] ++ docker run --name pytorch-ci-container -e 'EXTENSION_BUILD_GFXARCH=gfx90a;gfx908;gfx942' -e CI=1 -e TEST_CONFIG=default -e SHARD_NUMBER=1 -e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip -e PREBUILT=false -e TEST_CORE=true -e TEST_CORE=true -v /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3431:/host_workspace -v /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3431:/scripts --entrypoint bash -d --network=host --group-add video --device=/dev/kfd --device=/dev/dri --ipc=host --pid=host --shm-size 8G --cap-add=SYS_PTRACE --security-opt seccomp=unconfined rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942 -lc 'sleep infinity'
[2026-07-14T00:05:33.617Z] 5deab213db41b95d73897a73720cf1f0d134fcab8ea2d7eba9f5d3d57ebec315
[2026-07-14T00:05:34.011Z] ++ env -i python3 /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3431/pytorch/tools/stats/export_test_times.py
[2026-07-14T00:05:34.829Z] Exporting test times from test-infra
[2026-07-14T00:05:34.829Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-times.json to /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3431/pytorch/.additional_ci_files/test-times.json
[2026-07-14T00:05:34.829Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-class-times.json to /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3431/pytorch/.additional_ci_files/test-class-times.json
[2026-07-14T00:05:34.829Z] ++ python3 /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3431/pytorch/tools/stats/export_test_times.py
[2026-07-14T00:05:34.829Z] Exporting test times from test-infra
[2026-07-14T00:05:34.829Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-times.json to /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3431/pytorch/.additional_ci_files/test-times.json
[2026-07-14T00:05:34.829Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-class-times.json to /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3431/pytorch/.additional_ci_files/test-class-times.json
[2026-07-14T00:05:34.829Z] ++ [[ -d /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3431/pytorch/.additional_ci_files ]]
[2026-07-14T00:05:34.829Z] ++ sed -i -E 's/"\S+rocm\S+"/""/g' /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3431/pytorch/.additional_ci_files/test-times.json
[2026-07-14T00:05:34.829Z] ++ [[ false != \t\r\u\e ]]
[2026-07-14T00:05:34.829Z] ++ docker exec -u root pytorch-ci-container bash -c 'rm -rf /var/lib/jenkins/pytorch'
[2026-07-14T00:05:35.215Z] ++ docker cp /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3431/pytorch pytorch-ci-container:/var/lib/jenkins/pytorch
[2026-07-14T00:05:38.519Z] ++ docker exec -u root pytorch-ci-container bash -c 'chown root:root -R /var/lib/jenkins/pytorch'
[2026-07-14T00:05:38.519Z] ++ [[ 1 -eq 1 ]]
[2026-07-14T00:05:38.519Z] ++ docker cp /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3431/rocm_composable_kernel-7.1.1.dev0+g4266f8-py3-none-any.whl /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3431/torch-2.14.0a0+gitd4754e6-cp312-cp312-linux_x86_64.whl pytorch-ci-container:/var/lib/jenkins/pytorch
[2026-07-14T00:05:38.519Z] docker: 'docker cp' requires 2 arguments
[2026-07-14T00:05:38.519Z]
[2026-07-14T00:05:38.519Z] Usage: docker cp [OPTIONS] CONTAINER:SRC_PATH DEST_PATH|-
[2026-07-14T00:05:38.519Z] docker cp [OPTIONS] SRC_PATH|- CONTAINER:DEST_PATH
[2026-07-14T00:05:38.519Z]
[2026-07-14T00:05:38.519Z] Run 'docker cp --help' for more information
Tests / Test PyTorch / Test PyTorch / Run pytorch_test_1 / Error signal
Error in error step, with arguments pytorch_test_1 failed.
pytorch_test_1 failed
Tests / Test PyTorch / Test PyTorch / Run pytorch_test_2 / Shell Script
Error in sh step, with arguments ./test_pytorch_test.sh.
script returned exit code 1
Build log
[2026-07-14T00:21:34.348Z] + ./test_pytorch_test.sh
[2026-07-14T00:21:34.348Z] + [[ -z '' ]]
[2026-07-14T00:21:34.348Z] + DEFAULT_EXECUTION_OPTIONS='-e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip --copy-whls'
[2026-07-14T00:21:34.348Z] + source prepare_docker_env.sh -e 'EXTENSION_BUILD_GFXARCH=gfx90a;gfx908;gfx942' -e CI=1 -e TEST_CONFIG=default -e SHARD_NUMBER=2 -e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip --copy-whls
[2026-07-14T00:21:34.348Z] ++ set -ex
[2026-07-14T00:21:34.348Z] ++ set -o pipefail
[2026-07-14T00:21:34.348Z] ++ copy_whls=0
[2026-07-14T00:21:34.348Z] ++ envvars=()
[2026-07-14T00:21:34.348Z] ++ [[ 13 -gt 0 ]]
[2026-07-14T00:21:34.348Z] ++ key=-e
[2026-07-14T00:21:34.348Z] ++ case $key in
[2026-07-14T00:21:34.348Z] ++ envvars+=("-e $2")
[2026-07-14T00:21:34.348Z] ++ shift
[2026-07-14T00:21:34.348Z] ++ shift
[2026-07-14T00:21:34.348Z] ++ [[ 11 -gt 0 ]]
[2026-07-14T00:21:34.348Z] ++ key=-e
[2026-07-14T00:21:34.348Z] ++ case $key in
[2026-07-14T00:21:34.348Z] ++ envvars+=("-e $2")
[2026-07-14T00:21:34.348Z] ++ shift
[2026-07-14T00:21:34.348Z] ++ shift
[2026-07-14T00:21:34.348Z] ++ [[ 9 -gt 0 ]]
[2026-07-14T00:21:34.348Z] ++ key=-e
[2026-07-14T00:21:34.348Z] ++ case $key in
[2026-07-14T00:21:34.348Z] ++ envvars+=("-e $2")
[2026-07-14T00:21:34.348Z] ++ shift
[2026-07-14T00:21:34.348Z] ++ shift
[2026-07-14T00:21:34.348Z] ++ [[ 7 -gt 0 ]]
[2026-07-14T00:21:34.348Z] ++ key=-e
[2026-07-14T00:21:34.348Z] ++ case $key in
[2026-07-14T00:21:34.348Z] ++ envvars+=("-e $2")
[2026-07-14T00:21:34.348Z] ++ shift
[2026-07-14T00:21:34.348Z] ++ shift
[2026-07-14T00:21:34.348Z] ++ [[ 5 -gt 0 ]]
[2026-07-14T00:21:34.348Z] ++ key=-e
[2026-07-14T00:21:34.348Z] ++ case $key in
[2026-07-14T00:21:34.348Z] ++ envvars+=("-e $2")
[2026-07-14T00:21:34.348Z] ++ shift
[2026-07-14T00:21:34.348Z] ++ shift
[2026-07-14T00:21:34.348Z] ++ [[ 3 -gt 0 ]]
[2026-07-14T00:21:34.348Z] ++ key=-e
[2026-07-14T00:21:34.348Z] ++ case $key in
[2026-07-14T00:21:34.348Z] ++ envvars+=("-e $2")
[2026-07-14T00:21:34.348Z] ++ shift
[2026-07-14T00:21:34.348Z] ++ shift
[2026-07-14T00:21:34.348Z] ++ [[ 1 -gt 0 ]]
[2026-07-14T00:21:34.348Z] ++ key=--copy-whls
[2026-07-14T00:21:34.348Z] ++ case $key in
[2026-07-14T00:21:34.348Z] ++ copy_whls=1
[2026-07-14T00:21:34.348Z] ++ shift
[2026-07-14T00:21:34.348Z] ++ [[ 0 -gt 0 ]]
[2026-07-14T00:21:34.348Z] ++ envvars+=("-e PREBUILT=${PREBUILT:-false}")
[2026-07-14T00:21:34.348Z] ++ envvars+=("-e TEST_CORE=${TEST_CORE:-false}")
[2026-07-14T00:21:34.348Z] ++ [[ false == \t\r\u\e ]]
[2026-07-14T00:21:34.348Z] ++ [[ true == \t\r\u\e ]]
[2026-07-14T00:21:34.348Z] ++ envvars+=("-e TEST_CORE=${TEST_CORE}")
[2026-07-14T00:21:34.348Z] ++ docker ps -a
[2026-07-14T00:21:34.348Z] CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
[2026-07-14T00:21:34.348Z] ++ docker rm -f pytorch-ci-container
[2026-07-14T00:21:34.348Z] ++ docker pull rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942
[2026-07-14T00:21:35.640Z] pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942: Pulling from rocm/pytorch-ci-private
[2026-07-14T00:21:36.440Z] cb259a83ac3d: Pulling fs layer
[2026-07-14T00:21:36.440Z] 0e6d9b5f5e8d: Pulling fs layer
[2026-07-14T00:21:36.440Z] c608e00b1282: Pulling fs layer
[2026-07-14T00:21:36.440Z] 34e29023481f: Pulling fs layer
[2026-07-14T00:21:36.440Z] 4c8a34755929: Pulling fs layer
[2026-07-14T00:21:36.440Z] 07b45639fc6f: Pulling fs layer
[2026-07-14T00:21:36.440Z] 3d56742d201d: Pulling fs layer
[2026-07-14T00:21:36.440Z] 1057fba0c7df: Pulling fs layer
[2026-07-14T00:21:36.440Z] 9ae773ccc4b2: Pulling fs layer
[2026-07-14T00:21:36.440Z] db0af168a1a4: Pulling fs layer
[2026-07-14T00:21:36.440Z] ed874654c321: Pulling fs layer
[2026-07-14T00:21:36.440Z] dedc784bccba: Pulling fs layer
[2026-07-14T00:21:36.440Z] 4cfd21f31c8b: Pulling fs layer
[2026-07-14T00:21:36.440Z] fef65a673dc9: Pulling fs layer
[2026-07-14T00:21:36.440Z] 3d56742d201d: Pulling fs layer
[2026-07-14T00:21:36.440Z] 95d704643610: Pulling fs layer
[2026-07-14T00:21:36.440Z] 241cbb1c2f26: Pulling fs layer
[2026-07-14T00:21:36.440Z] f0ef294e4f9b: Pulling fs layer
[2026-07-14T00:21:36.440Z] ba1fabf878bd: Pulling fs layer
[2026-07-14T00:21:36.440Z] d9a8b776bea7: Pulling fs layer
[2026-07-14T00:21:36.440Z] 99e6462d03e0: Pulling fs layer
[2026-07-14T00:21:36.440Z] 0fcc3b23bd99: Pulling fs layer
[2026-07-14T00:21:36.440Z] 3d56742d201d: Pulling fs layer
[2026-07-14T00:21:36.440Z] 8b3d007242f8: Pulling fs layer
[2026-07-14T00:21:36.440Z] 86cc05da26d1: Pulling fs layer
[2026-07-14T00:21:36.440Z] 5f8dd10c9c78: Pulling fs layer
[2026-07-14T00:21:36.440Z] 8d78d529f04d: Pulling fs layer
[2026-07-14T00:21:36.440Z] 42239a12ed73: Pulling fs layer
[2026-07-14T00:21:36.440Z] 423e26ddb517: Pulling fs layer
[2026-07-14T00:21:36.440Z] 3d56742d201d: Pulling fs layer
[2026-07-14T00:21:36.440Z] 3d56742d201d: Pulling fs layer
[2026-07-14T00:21:36.440Z] bf3e4cf00cd6: Pulling fs layer
[2026-07-14T00:21:36.440Z] b96eaed0e751: Pulling fs layer
[2026-07-14T00:21:36.440Z] 9548298aa5f4: Pulling fs layer
[2026-07-14T00:21:36.440Z] 78cb638d6686: Pulling fs layer
[2026-07-14T00:21:36.440Z] b09b7d04a8d9: Pulling fs layer
[2026-07-14T00:21:36.440Z] d7df6379f9cc: Pulling fs layer
[2026-07-14T00:21:36.440Z] 747272797eac: Pulling fs layer
[2026-07-14T00:21:36.440Z] f790d498bebb: Pulling fs layer
[2026-07-14T00:21:36.440Z] f3058198fba5: Pulling fs layer
[2026-07-14T00:21:36.440Z] 5edf52847c35: Pulling fs layer
[2026-07-14T00:21:36.440Z] 906ac4de0a2b: Pulling fs layer
[2026-07-14T00:21:36.440Z] 65f281ce2e70: Pulling fs layer
[2026-07-14T00:21:36.440Z] f391428cfe0f: Pulling fs layer
[2026-07-14T00:21:36.440Z] 78a3c5370db4: Pulling fs layer
[2026-07-14T00:21:36.440Z] 4f4fb700ef54: Pulling fs layer
[2026-07-14T00:21:36.440Z] eab367f1ca39: Pulling fs layer
[2026-07-14T00:21:36.440Z] 0a0ffadd8b4f: Pulling fs layer
[2026-07-14T00:21:36.440Z] f5afebc61536: Pulling fs layer
[2026-07-14T00:21:36.440Z] c0664a7365cb: Pulling fs layer
[2026-07-14T00:21:36.440Z] 4106a853273f: Pulling fs layer
[2026-07-14T00:21:36.440Z] 27fcad4d96a6: Pulling fs layer
[2026-07-14T00:21:36.440Z] cccff27d82fb: Pulling fs layer
[2026-07-14T00:21:36.440Z] 8d998c79d309: Pulling fs layer
[2026-07-14T00:21:36.440Z] 284e8cbb8a05: Pulling fs layer
[2026-07-14T00:21:36.440Z] 4f4fb700ef54: Pulling fs layer
[2026-07-14T00:21:36.440Z] ed86e5a5d9b1: Pulling fs layer
[2026-07-14T00:21:36.814Z] 69145e0460f6: Download complete
[2026-07-14T00:21:36.814Z] ed874654c321: Download complete
[2026-07-14T00:21:36.814Z] 07b45639fc6f: Download complete
[2026-07-14T00:21:36.814Z] 42239a12ed73: Download complete
[2026-07-14T00:21:36.814Z] 3d56742d201d: Download complete
[2026-07-14T00:21:36.814Z] 8b3d007242f8: Download complete
[2026-07-14T00:21:36.814Z] 86cc05da26d1: Download complete
[2026-07-14T00:21:36.814Z] 9ae773ccc4b2: Download complete
[2026-07-14T00:21:36.814Z] db0af168a1a4: Download complete
[2026-07-14T00:21:36.814Z] 241cbb1c2f26: Download complete
[2026-07-14T00:21:36.814Z] c608e00b1282: Download complete
[2026-07-14T00:21:36.814Z] 34e29023481f: Download complete
[2026-07-14T00:21:36.814Z] f0ef294e4f9b: Download complete
[2026-07-14T00:21:37.188Z] 4f4fb700ef54: Download complete
[2026-07-14T00:21:37.188Z] f5afebc61536: Download complete
[2026-07-14T00:21:37.188Z] 65f281ce2e70: Download complete
[2026-07-14T00:21:37.188Z] 906ac4de0a2b: Download complete
[2026-07-14T00:21:37.188Z] 0a0ffadd8b4f: Download complete
[2026-07-14T00:21:37.188Z] fef65a673dc9: Download complete
[2026-07-14T00:21:37.188Z] 99e6462d03e0: Download complete
[2026-07-14T00:21:37.188Z] 4cfd21f31c8b: Download complete
[2026-07-14T00:21:37.188Z] 8d998c79d309: Download complete
[2026-07-14T00:21:37.188Z] 0fcc3b23bd99: Download complete
[2026-07-14T00:21:37.188Z] 423e26ddb517: Download complete
[2026-07-14T00:21:37.188Z] b96eaed0e751: Download complete
[2026-07-14T00:21:37.188Z] f790d498bebb: Download complete
[2026-07-14T00:21:37.188Z] 9548298aa5f4: Download complete
[2026-07-14T00:21:37.188Z] f3058198fba5: Download complete
[2026-07-14T00:21:37.188Z] 78a3c5370db4: Download complete
[2026-07-14T00:21:37.188Z] 27fcad4d96a6: Download complete
[2026-07-14T00:21:37.188Z] bf3e4cf00cd6: Download complete
[2026-07-14T00:21:37.188Z] c0664a7365cb: Download complete
[2026-07-14T00:21:37.188Z] d7df6379f9cc: Download complete
[2026-07-14T00:21:37.188Z] 747272797eac: Download complete
[2026-07-14T00:21:37.188Z] f391428cfe0f: Download complete
[2026-07-14T00:21:37.188Z] 8d78d529f04d: Download complete
[2026-07-14T00:21:37.188Z] ed86e5a5d9b1: Download complete
[2026-07-14T00:21:37.188Z] 284e8cbb8a05: Download complete
[2026-07-14T00:21:37.188Z] b09b7d04a8d9: Download complete
[2026-07-14T00:21:37.188Z] eab367f1ca39: Download complete
[2026-07-14T00:21:37.188Z] 5f8dd10c9c78: Download complete
[2026-07-14T00:21:37.188Z] 4106a853273f: Download complete
[2026-07-14T00:21:37.563Z] cb259a83ac3d: Download complete
[2026-07-14T00:21:37.563Z] 0e6d9b5f5e8d: Download complete
[2026-07-14T00:21:37.563Z] ba1fabf878bd: Download complete
[2026-07-14T00:21:37.939Z] 4c8a34755929: Download complete
[2026-07-14T00:21:37.940Z] 99e6462d03e0: Pull complete
[2026-07-14T00:21:37.940Z] cb259a83ac3d: Pull complete
[2026-07-14T00:21:38.315Z] 95d704643610: Download complete
[2026-07-14T00:21:40.154Z] cccff27d82fb: Download complete
[2026-07-14T00:21:44.252Z] 1057fba0c7df: Download complete
[2026-07-14T00:21:49.379Z] 65f281ce2e70: Pull complete
[2026-07-14T00:21:49.379Z] 8d78d529f04d: Pull complete
[2026-07-14T00:21:49.379Z] 0e6d9b5f5e8d: Pull complete
[2026-07-14T00:21:49.379Z] 1057fba0c7df: Pull complete
[2026-07-14T00:21:49.379Z] c608e00b1282: Pull complete
[2026-07-14T00:21:49.379Z] 34e29023481f: Pull complete
[2026-07-14T00:21:49.379Z] 906ac4de0a2b: Pull complete
[2026-07-14T00:21:49.379Z] d7df6379f9cc: Pull complete
[2026-07-14T00:21:54.503Z] 78cb638d6686: Download complete
[2026-07-14T00:22:23.156Z] 5edf52847c35: Download complete
[2026-07-14T00:22:57.390Z] ed874654c321: Pull complete
[2026-07-14T00:22:57.390Z] 8b3d007242f8: Pull complete
[2026-07-14T00:22:57.390Z] 86cc05da26d1: Pull complete
[2026-07-14T00:22:57.390Z] db0af168a1a4: Pull complete
[2026-07-14T00:22:57.390Z] 0a0ffadd8b4f: Pull complete
[2026-07-14T00:22:57.390Z] 9548298aa5f4: Pull complete
[2026-07-14T00:22:57.390Z] 747272797eac: Pull complete
[2026-07-14T00:22:57.390Z] 5edf52847c35: Pull complete
[2026-07-14T00:23:38.347Z] d9a8b776bea7: Download complete
[2026-07-14T00:24:12.536Z] dedc784bccba: Download complete
[2026-07-14T00:25:51.903Z] 9ae773ccc4b2: Pull complete
[2026-07-14T00:25:51.903Z] 27fcad4d96a6: Pull complete
[2026-07-14T00:25:51.903Z] bf3e4cf00cd6: Pull complete
[2026-07-14T00:25:51.903Z] dedc784bccba: Pull complete
[2026-07-14T00:25:51.903Z] f5afebc61536: Pull complete
[2026-07-14T00:25:51.903Z] f3058198fba5: Pull complete
[2026-07-14T00:25:51.903Z] 78cb638d6686: Pull complete
[2026-07-14T00:25:51.903Z] 07b45639fc6f: Pull complete
[2026-07-14T00:25:51.903Z] b96eaed0e751: Pull complete
[2026-07-14T00:25:51.903Z] f790d498bebb: Pull complete
[2026-07-14T00:25:51.903Z] c0664a7365cb: Pull complete
[2026-07-14T00:25:51.903Z] cccff27d82fb: Pull complete
[2026-07-14T00:25:51.903Z] 423e26ddb517: Pull complete
[2026-07-14T00:25:51.903Z] 95d704643610: Pull complete
[2026-07-14T00:25:51.903Z] 42239a12ed73: Pull complete
[2026-07-14T00:25:51.903Z] 3d56742d201d: Pull complete
[2026-07-14T00:25:51.903Z] 241cbb1c2f26: Pull complete
[2026-07-14T00:25:51.903Z] f0ef294e4f9b: Pull complete
[2026-07-14T00:25:51.903Z] 4cfd21f31c8b: Pull complete
[2026-07-14T00:25:51.903Z] 0fcc3b23bd99: Pull complete
[2026-07-14T00:25:51.903Z] 78a3c5370db4: Pull complete
[2026-07-14T00:25:51.903Z] 284e8cbb8a05: Pull complete
[2026-07-14T00:25:51.903Z] b09b7d04a8d9: Pull complete
[2026-07-14T00:25:51.903Z] 5f8dd10c9c78: Pull complete
[2026-07-14T00:25:51.903Z] 4106a853273f: Pull complete
[2026-07-14T00:27:02.405Z] d9a8b776bea7: Pull complete
[2026-07-14T00:27:02.405Z] f391428cfe0f: Pull complete
[2026-07-14T00:27:02.405Z] eab367f1ca39: Pull complete
[2026-07-14T00:27:02.405Z] ba1fabf878bd: Pull complete
[2026-07-14T00:27:02.405Z] ed86e5a5d9b1: Pull complete
[2026-07-14T00:27:02.405Z] 4f4fb700ef54: Pull complete
[2026-07-14T00:27:02.405Z] fef65a673dc9: Pull complete
[2026-07-14T00:27:02.405Z] 8d998c79d309: Pull complete
[2026-07-14T00:27:02.405Z] 4c8a34755929: Pull complete
[2026-07-14T00:27:02.405Z] Digest: sha256:b836aa0e221ac85a2184ec24a67b9d5089f3caa45d182cf521dee7d38112aa81
[2026-07-14T00:27:02.405Z] Status: Downloaded newer image for rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942
[2026-07-14T00:27:02.405Z] docker.io/rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942
[2026-07-14T00:27:02.405Z] ++ GPU_FLAGS=
[2026-07-14T00:27:02.405Z] ++ echo 'NODE_LABELS: gfx90a mlir k8s-mi250-1-2nqgl gfx90a mlir k8s-mi250-1-2nqgl'
[2026-07-14T00:27:02.405Z] NODE_LABELS: gfx90a mlir k8s-mi250-1-2nqgl gfx90a mlir k8s-mi250-1-2nqgl
[2026-07-14T00:27:02.405Z] ++ [[ ! gfx90a mlir k8s-mi250-1-2nqgl =~ build-only ]]
[2026-07-14T00:27:02.405Z] ++ GPU_FLAGS='--group-add video --device=/dev/kfd --device=/dev/dri'
[2026-07-14T00:27:02.405Z] ++++ dirname prepare_docker_env.sh
[2026-07-14T00:27:02.405Z] +++ cd .
[2026-07-14T00:27:02.405Z] +++ pwd
[2026-07-14T00:27:02.405Z] ++ script_dir=/home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3431
[2026-07-14T00:27:02.405Z] ++ docker run --name pytorch-ci-container -e 'EXTENSION_BUILD_GFXARCH=gfx90a;gfx908;gfx942' -e CI=1 -e TEST_CONFIG=default -e SHARD_NUMBER=2 -e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip -e PREBUILT=false -e TEST_CORE=true -e TEST_CORE=true -v /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3431:/host_workspace -v /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3431:/scripts --entrypoint bash -d --network=host --group-add video --device=/dev/kfd --device=/dev/dri --ipc=host --pid=host --shm-size 8G --cap-add=SYS_PTRACE --security-opt seccomp=unconfined rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942 -lc 'sleep infinity'
[2026-07-14T00:27:02.406Z] 771be9c0543c2dee0d21c3e8b1fb648b1c800cd013d39c94d23476805a866cd8
[2026-07-14T00:27:02.406Z] ++ env -i python3 /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3431/pytorch/tools/stats/export_test_times.py
[2026-07-14T00:27:02.406Z] Exporting test times from test-infra
[2026-07-14T00:27:02.406Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-times.json to /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3431/pytorch/.additional_ci_files/test-times.json
[2026-07-14T00:27:02.406Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-class-times.json to /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3431/pytorch/.additional_ci_files/test-class-times.json
[2026-07-14T00:27:02.406Z] ++ python3 /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3431/pytorch/tools/stats/export_test_times.py
[2026-07-14T00:27:02.406Z] Exporting test times from test-infra
[2026-07-14T00:27:02.406Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-times.json to /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3431/pytorch/.additional_ci_files/test-times.json
[2026-07-14T00:27:02.406Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-class-times.json to /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3431/pytorch/.additional_ci_files/test-class-times.json
[2026-07-14T00:27:02.406Z] ++ [[ -d /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3431/pytorch/.additional_ci_files ]]
[2026-07-14T00:27:02.406Z] ++ sed -i -E 's/"\S+rocm\S+"/""/g' /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3431/pytorch/.additional_ci_files/test-times.json
[2026-07-14T00:27:02.406Z] ++ [[ false != \t\r\u\e ]]
[2026-07-14T00:27:02.406Z] ++ docker exec -u root pytorch-ci-container bash -c 'rm -rf /var/lib/jenkins/pytorch'
[2026-07-14T00:27:02.406Z] ++ docker cp /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3431/pytorch pytorch-ci-container:/var/lib/jenkins/pytorch
[2026-07-14T00:27:04.940Z] ++ docker exec -u root pytorch-ci-container bash -c 'chown root:root -R /var/lib/jenkins/pytorch'
[2026-07-14T00:27:05.325Z] ++ [[ 1 -eq 1 ]]
[2026-07-14T00:27:05.325Z] ++ docker cp /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3431/rocm_composable_kernel-7.1.1.dev0+g4266f8-py3-none-any.whl /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3431/torch-2.14.0a0+gitd4754e6-cp312-cp312-linux_x86_64.whl pytorch-ci-container:/var/lib/jenkins/pytorch
[2026-07-14T00:27:05.325Z] docker: 'docker cp' requires 2 arguments
[2026-07-14T00:27:05.325Z]
[2026-07-14T00:27:05.325Z] Usage: docker cp [OPTIONS] CONTAINER:SRC_PATH DEST_PATH|-
[2026-07-14T00:27:05.325Z] docker cp [OPTIONS] SRC_PATH|- CONTAINER:DEST_PATH
[2026-07-14T00:27:05.325Z]
[2026-07-14T00:27:05.325Z] Run 'docker cp --help' for more information
Tests / Test PyTorch / Test PyTorch / Run pytorch_test_2 / Error signal
Error in error step, with arguments pytorch_test_2 failed.
pytorch_test_2 failed
Tests / Test Distributed / Test Distributed / Run pytorch_distributed_1 / Shell Script
Error in sh step, with arguments ./test_pytorch_test_distributed.sh.
script returned exit code 1
Build log
[2026-07-14T02:06:12.957Z] + ./test_pytorch_test_distributed.sh
[2026-07-14T02:06:12.957Z] + [[ -z '' ]]
[2026-07-14T02:06:12.957Z] + DEFAULT_EXECUTION_OPTIONS='-e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip --copy-whls'
[2026-07-14T02:06:12.957Z] + source prepare_docker_env.sh -e 'EXTENSION_BUILD_GFXARCH=gfx90a;gfx908;gfx942' -e CI=1 -e TEST_CONFIG=distributed -e SHARD_NUMBER=1 -e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip --copy-whls
[2026-07-14T02:06:12.957Z] ++ set -ex
[2026-07-14T02:06:12.957Z] ++ set -o pipefail
[2026-07-14T02:06:12.957Z] ++ copy_whls=0
[2026-07-14T02:06:12.957Z] ++ envvars=()
[2026-07-14T02:06:12.957Z] ++ [[ 13 -gt 0 ]]
[2026-07-14T02:06:12.957Z] ++ key=-e
[2026-07-14T02:06:12.957Z] ++ case $key in
[2026-07-14T02:06:12.957Z] ++ envvars+=("-e $2")
[2026-07-14T02:06:12.957Z] ++ shift
[2026-07-14T02:06:12.957Z] ++ shift
[2026-07-14T02:06:12.957Z] ++ [[ 11 -gt 0 ]]
[2026-07-14T02:06:12.957Z] ++ key=-e
[2026-07-14T02:06:12.957Z] ++ case $key in
[2026-07-14T02:06:12.957Z] ++ envvars+=("-e $2")
[2026-07-14T02:06:12.957Z] ++ shift
[2026-07-14T02:06:12.957Z] ++ shift
[2026-07-14T02:06:12.957Z] ++ [[ 9 -gt 0 ]]
[2026-07-14T02:06:12.957Z] ++ key=-e
[2026-07-14T02:06:12.957Z] ++ case $key in
[2026-07-14T02:06:12.957Z] ++ envvars+=("-e $2")
[2026-07-14T02:06:12.957Z] ++ shift
[2026-07-14T02:06:12.957Z] ++ shift
[2026-07-14T02:06:12.957Z] ++ [[ 7 -gt 0 ]]
[2026-07-14T02:06:12.957Z] ++ key=-e
[2026-07-14T02:06:12.957Z] ++ case $key in
[2026-07-14T02:06:12.957Z] ++ envvars+=("-e $2")
[2026-07-14T02:06:12.957Z] ++ shift
[2026-07-14T02:06:12.957Z] ++ shift
[2026-07-14T02:06:12.957Z] ++ [[ 5 -gt 0 ]]
[2026-07-14T02:06:12.957Z] ++ key=-e
[2026-07-14T02:06:12.957Z] ++ case $key in
[2026-07-14T02:06:12.957Z] ++ envvars+=("-e $2")
[2026-07-14T02:06:12.957Z] ++ shift
[2026-07-14T02:06:12.957Z] ++ shift
[2026-07-14T02:06:12.957Z] ++ [[ 3 -gt 0 ]]
[2026-07-14T02:06:12.957Z] ++ key=-e
[2026-07-14T02:06:12.957Z] ++ case $key in
[2026-07-14T02:06:12.957Z] ++ envvars+=("-e $2")
[2026-07-14T02:06:12.957Z] ++ shift
[2026-07-14T02:06:12.957Z] ++ shift
[2026-07-14T02:06:12.957Z] ++ [[ 1 -gt 0 ]]
[2026-07-14T02:06:12.957Z] ++ key=--copy-whls
[2026-07-14T02:06:12.957Z] ++ case $key in
[2026-07-14T02:06:12.957Z] ++ copy_whls=1
[2026-07-14T02:06:12.957Z] ++ shift
[2026-07-14T02:06:12.957Z] ++ [[ 0 -gt 0 ]]
[2026-07-14T02:06:12.957Z] ++ envvars+=("-e PREBUILT=${PREBUILT:-false}")
[2026-07-14T02:06:12.957Z] ++ envvars+=("-e TEST_CORE=${TEST_CORE:-false}")
[2026-07-14T02:06:12.957Z] ++ [[ false == \t\r\u\e ]]
[2026-07-14T02:06:12.957Z] ++ [[ true == \t\r\u\e ]]
[2026-07-14T02:06:12.957Z] ++ envvars+=("-e TEST_CORE=${TEST_CORE}")
[2026-07-14T02:06:12.957Z] ++ docker ps -a
[2026-07-14T02:06:12.957Z] CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
[2026-07-14T02:06:12.957Z] 9208476f0134 rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-4830fa265c7b75a523cafd75ad7d03ccc6adefb1-gfx908_gfx90a_gfx942 "bash -lc 'sleep inf…" 3 hours ago Up 3 hours pytorch-ci-container
[2026-07-14T02:06:12.957Z] ++ docker rm -f pytorch-ci-container
[2026-07-14T02:06:18.352Z] ++ docker pull rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942
[2026-07-14T02:06:19.460Z] pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942: Pulling from rocm/pytorch-ci-private
[2026-07-14T02:06:19.990Z] cb259a83ac3d: Pulling fs layer
[2026-07-14T02:06:19.990Z] 99e6462d03e0: Pulling fs layer
[2026-07-14T02:06:19.990Z] 1057fba0c7df: Pulling fs layer
[2026-07-14T02:06:19.990Z] 0e6d9b5f5e8d: Pulling fs layer
[2026-07-14T02:06:19.990Z] 65f281ce2e70: Pulling fs layer
[2026-07-14T02:06:19.990Z] 34e29023481f: Pulling fs layer
[2026-07-14T02:06:19.990Z] 8d78d529f04d: Pulling fs layer
[2026-07-14T02:06:19.990Z] 906ac4de0a2b: Pulling fs layer
[2026-07-14T02:06:19.990Z] c608e00b1282: Pulling fs layer
[2026-07-14T02:06:19.990Z] d7df6379f9cc: Pulling fs layer
[2026-07-14T02:06:19.990Z] 3d56742d201d: Pulling fs layer
[2026-07-14T02:06:19.990Z] 5edf52847c35: Pulling fs layer
[2026-07-14T02:06:19.990Z] 9548298aa5f4: Pulling fs layer
[2026-07-14T02:06:19.990Z] 86cc05da26d1: Pulling fs layer
[2026-07-14T02:06:19.990Z] 747272797eac: Pulling fs layer
[2026-07-14T02:06:19.990Z] db0af168a1a4: Pulling fs layer
[2026-07-14T02:06:19.990Z] ed874654c321: Pulling fs layer
[2026-07-14T02:06:19.990Z] 0a0ffadd8b4f: Pulling fs layer
[2026-07-14T02:06:19.990Z] 8b3d007242f8: Pulling fs layer
[2026-07-14T02:06:19.990Z] dedc784bccba: Pulling fs layer
[2026-07-14T02:06:19.990Z] 65f281ce2e70: Waiting
[2026-07-14T02:06:19.990Z] 27fcad4d96a6: Pulling fs layer
[2026-07-14T02:06:19.990Z] bf3e4cf00cd6: Pulling fs layer
[2026-07-14T02:06:19.990Z] 34e29023481f: Waiting
[2026-07-14T02:06:19.990Z] 9ae773ccc4b2: Pulling fs layer
[2026-07-14T02:06:19.990Z] 8d78d529f04d: Waiting
[2026-07-14T02:06:19.990Z] 78cb638d6686: Pulling fs layer
[2026-07-14T02:06:19.990Z] 906ac4de0a2b: Waiting
[2026-07-14T02:06:19.990Z] f3058198fba5: Pulling fs layer
[2026-07-14T02:06:19.990Z] c608e00b1282: Waiting
[2026-07-14T02:06:19.990Z] f5afebc61536: Pulling fs layer
[2026-07-14T02:06:19.990Z] d7df6379f9cc: Waiting
[2026-07-14T02:06:19.990Z] cccff27d82fb: Pulling fs layer
[2026-07-14T02:06:19.990Z] c0664a7365cb: Pulling fs layer
[2026-07-14T02:06:19.990Z] 3d56742d201d: Waiting
[2026-07-14T02:06:19.990Z] 5edf52847c35: Waiting
[2026-07-14T02:06:19.990Z] 07b45639fc6f: Pulling fs layer
[2026-07-14T02:06:19.990Z] 9548298aa5f4: Waiting
[2026-07-14T02:06:19.990Z] b96eaed0e751: Pulling fs layer
[2026-07-14T02:06:19.990Z] f790d498bebb: Pulling fs layer
[2026-07-14T02:06:19.990Z] 95d704643610: Pulling fs layer
[2026-07-14T02:06:19.990Z] 86cc05da26d1: Waiting
[2026-07-14T02:06:19.990Z] 423e26ddb517: Pulling fs layer
[2026-07-14T02:06:19.990Z] 4106a853273f: Pulling fs layer
[2026-07-14T02:06:19.990Z] 5f8dd10c9c78: Pulling fs layer
[2026-07-14T02:06:19.990Z] 747272797eac: Waiting
[2026-07-14T02:06:19.990Z] b09b7d04a8d9: Pulling fs layer
[2026-07-14T02:06:19.990Z] 0e6d9b5f5e8d: Waiting
[2026-07-14T02:06:19.990Z] 8b3d007242f8: Waiting
[2026-07-14T02:06:19.990Z] 0a0ffadd8b4f: Waiting
[2026-07-14T02:06:19.990Z] 4cfd21f31c8b: Pulling fs layer
[2026-07-14T02:06:19.990Z] db0af168a1a4: Waiting
[2026-07-14T02:06:19.990Z] 241cbb1c2f26: Pulling fs layer
[2026-07-14T02:06:19.990Z] ed874654c321: Waiting
[2026-07-14T02:06:19.990Z] f0ef294e4f9b: Pulling fs layer
[2026-07-14T02:06:19.990Z] dedc784bccba: Waiting
[2026-07-14T02:06:19.990Z] 27fcad4d96a6: Waiting
[2026-07-14T02:06:19.990Z] 4f4fb700ef54: Pulling fs layer
[2026-07-14T02:06:19.990Z] 9ae773ccc4b2: Waiting
[2026-07-14T02:06:19.990Z] 78a3c5370db4: Pulling fs layer
[2026-07-14T02:06:19.990Z] 42239a12ed73: Pulling fs layer
[2026-07-14T02:06:19.990Z] f3058198fba5: Waiting
[2026-07-14T02:06:19.990Z] 284e8cbb8a05: Pulling fs layer
[2026-07-14T02:06:19.990Z] f5afebc61536: Waiting
[2026-07-14T02:06:19.990Z] 0fcc3b23bd99: Pulling fs layer
[2026-07-14T02:06:19.990Z] cccff27d82fb: Waiting
[2026-07-14T02:06:19.990Z] f0ef294e4f9b: Waiting
[2026-07-14T02:06:19.990Z] 4cfd21f31c8b: Waiting
[2026-07-14T02:06:19.990Z] 5f8dd10c9c78: Waiting
[2026-07-14T02:06:19.990Z] 423e26ddb517: Waiting
[2026-07-14T02:06:19.990Z] b09b7d04a8d9: Waiting
[2026-07-14T02:06:19.990Z] d9a8b776bea7: Pulling fs layer
[2026-07-14T02:06:19.990Z] f790d498bebb: Waiting
[2026-07-14T02:06:19.990Z] 241cbb1c2f26: Waiting
[2026-07-14T02:06:19.990Z] f391428cfe0f: Pulling fs layer
[2026-07-14T02:06:19.990Z] c0664a7365cb: Waiting
[2026-07-14T02:06:19.990Z] 95d704643610: Waiting
[2026-07-14T02:06:19.990Z] 4106a853273f: Waiting
[2026-07-14T02:06:19.990Z] eab367f1ca39: Pulling fs layer
[2026-07-14T02:06:19.990Z] b96eaed0e751: Waiting
[2026-07-14T02:06:19.990Z] 284e8cbb8a05: Waiting
[2026-07-14T02:06:19.990Z] ba1fabf878bd: Pulling fs layer
[2026-07-14T02:06:19.990Z] 07b45639fc6f: Waiting
[2026-07-14T02:06:19.990Z] 4f4fb700ef54: Waiting
[2026-07-14T02:06:19.990Z] ed86e5a5d9b1: Pulling fs layer
[2026-07-14T02:06:19.990Z] 0fcc3b23bd99: Waiting
[2026-07-14T02:06:19.990Z] 42239a12ed73: Waiting
[2026-07-14T02:06:19.990Z] d9a8b776bea7: Waiting
[2026-07-14T02:06:19.990Z] 78a3c5370db4: Waiting
[2026-07-14T02:06:19.990Z] fef65a673dc9: Pulling fs layer
[2026-07-14T02:06:19.990Z] f391428cfe0f: Waiting
[2026-07-14T02:06:19.990Z] ba1fabf878bd: Waiting
[2026-07-14T02:06:19.990Z] 8d998c79d309: Pulling fs layer
[2026-07-14T02:06:19.990Z] ed86e5a5d9b1: Waiting
[2026-07-14T02:06:19.990Z] 4c8a34755929: Pulling fs layer
[2026-07-14T02:06:19.990Z] fef65a673dc9: Waiting
[2026-07-14T02:06:19.991Z] 8d998c79d309: Waiting
[2026-07-14T02:06:19.991Z] 99e6462d03e0: Download complete
[2026-07-14T02:06:20.693Z] cb259a83ac3d: Verifying Checksum
[2026-07-14T02:06:20.693Z] cb259a83ac3d: Download complete
[2026-07-14T02:06:20.693Z] 0e6d9b5f5e8d: Verifying Checksum
[2026-07-14T02:06:20.693Z] 0e6d9b5f5e8d: Download complete
[2026-07-14T02:06:20.693Z] 65f281ce2e70: Verifying Checksum
[2026-07-14T02:06:20.693Z] 65f281ce2e70: Download complete
[2026-07-14T02:06:20.693Z] 34e29023481f: Verifying Checksum
[2026-07-14T02:06:20.693Z] 34e29023481f: Download complete
[2026-07-14T02:06:20.693Z] 8d78d529f04d: Verifying Checksum
[2026-07-14T02:06:20.693Z] 8d78d529f04d: Download complete
[2026-07-14T02:06:21.260Z] 906ac4de0a2b: Download complete
[2026-07-14T02:06:21.260Z] c608e00b1282: Download complete
[2026-07-14T02:06:21.260Z] d7df6379f9cc: Verifying Checksum
[2026-07-14T02:06:21.260Z] d7df6379f9cc: Download complete
[2026-07-14T02:06:21.813Z] 3d56742d201d: Verifying Checksum
[2026-07-14T02:06:21.813Z] 3d56742d201d: Download complete
[2026-07-14T02:06:21.813Z] 9548298aa5f4: Download complete
[2026-07-14T02:06:22.362Z] 86cc05da26d1: Verifying Checksum
[2026-07-14T02:06:22.362Z] 86cc05da26d1: Download complete
[2026-07-14T02:06:22.362Z] 747272797eac: Verifying Checksum
[2026-07-14T02:06:22.362Z] 747272797eac: Download complete
[2026-07-14T02:06:22.362Z] cb259a83ac3d: Pull complete
[2026-07-14T02:06:22.362Z] 99e6462d03e0: Pull complete
[2026-07-14T02:06:22.362Z] db0af168a1a4: Verifying Checksum
[2026-07-14T02:06:22.362Z] db0af168a1a4: Download complete
[2026-07-14T02:06:23.210Z] ed874654c321: Verifying Checksum
[2026-07-14T02:06:23.210Z] ed874654c321: Download complete
[2026-07-14T02:06:23.210Z] 0a0ffadd8b4f: Download complete
[2026-07-14T02:06:23.210Z] 8b3d007242f8: Download complete
[2026-07-14T02:06:33.362Z] 1057fba0c7df: Verifying Checksum
[2026-07-14T02:06:33.362Z] 1057fba0c7df: Download complete
[2026-07-14T02:06:33.362Z] 27fcad4d96a6: Verifying Checksum
[2026-07-14T02:06:33.362Z] 27fcad4d96a6: Download complete
[2026-07-14T02:06:33.362Z] bf3e4cf00cd6: Download complete
[2026-07-14T02:06:33.362Z] 9ae773ccc4b2: Verifying Checksum
[2026-07-14T02:06:33.362Z] 9ae773ccc4b2: Download complete
[2026-07-14T02:06:58.141Z] 1057fba0c7df: Pull complete
[2026-07-14T02:06:58.141Z] 0e6d9b5f5e8d: Pull complete
[2026-07-14T02:06:58.141Z] 65f281ce2e70: Pull complete
[2026-07-14T02:06:58.141Z] 34e29023481f: Pull complete
[2026-07-14T02:06:58.141Z] 8d78d529f04d: Pull complete
[2026-07-14T02:06:58.141Z] 906ac4de0a2b: Pull complete
[2026-07-14T02:06:58.141Z] c608e00b1282: Pull complete
[2026-07-14T02:06:58.141Z] d7df6379f9cc: Pull complete
[2026-07-14T02:06:58.141Z] 3d56742d201d: Pull complete
[2026-07-14T02:07:55.141Z] 78cb638d6686: Verifying Checksum
[2026-07-14T02:07:55.141Z] 78cb638d6686: Download complete
[2026-07-14T02:07:55.141Z] f3058198fba5: Download complete
[2026-07-14T02:07:55.141Z] f5afebc61536: Download complete
[2026-07-14T02:08:03.672Z] cccff27d82fb: Verifying Checksum
[2026-07-14T02:08:03.672Z] cccff27d82fb: Download complete
[2026-07-14T02:08:03.672Z] c0664a7365cb: Verifying Checksum
[2026-07-14T02:08:03.672Z] c0664a7365cb: Download complete
[2026-07-14T02:08:04.309Z] 07b45639fc6f: Verifying Checksum
[2026-07-14T02:08:04.309Z] 07b45639fc6f: Download complete
[2026-07-14T02:08:04.309Z] b96eaed0e751: Download complete
[2026-07-14T02:08:05.221Z] f790d498bebb: Verifying Checksum
[2026-07-14T02:08:05.221Z] f790d498bebb: Download complete
[2026-07-14T02:08:07.790Z] 95d704643610: Verifying Checksum
[2026-07-14T02:08:07.790Z] 95d704643610: Download complete
[2026-07-14T02:08:08.394Z] 423e26ddb517: Download complete
[2026-07-14T02:08:09.019Z] 4106a853273f: Verifying Checksum
[2026-07-14T02:08:09.019Z] 4106a853273f: Download complete
[2026-07-14T02:08:09.626Z] 5f8dd10c9c78: Verifying Checksum
[2026-07-14T02:08:09.626Z] 5f8dd10c9c78: Download complete
[2026-07-14T02:08:09.626Z] b09b7d04a8d9: Download complete
[2026-07-14T02:08:10.158Z] 4cfd21f31c8b: Download complete
[2026-07-14T02:08:10.158Z] 5edf52847c35: Verifying Checksum
[2026-07-14T02:08:10.158Z] 5edf52847c35: Download complete
[2026-07-14T02:08:10.158Z] 241cbb1c2f26: Verifying Checksum
[2026-07-14T02:08:10.158Z] 241cbb1c2f26: Download complete
[2026-07-14T02:08:10.158Z] f0ef294e4f9b: Download complete
[2026-07-14T02:08:10.700Z] 4f4fb700ef54: Verifying Checksum
[2026-07-14T02:08:10.700Z] 4f4fb700ef54: Download complete
[2026-07-14T02:08:10.700Z] 78a3c5370db4: Download complete
[2026-07-14T02:08:10.700Z] 42239a12ed73: Verifying Checksum
[2026-07-14T02:08:10.700Z] 42239a12ed73: Download complete
[2026-07-14T02:08:10.700Z] 284e8cbb8a05: Download complete
[2026-07-14T02:08:11.231Z] 0fcc3b23bd99: Verifying Checksum
[2026-07-14T02:08:11.761Z] f391428cfe0f: Verifying Checksum
[2026-07-14T02:08:11.761Z] f391428cfe0f: Download complete
[2026-07-14T02:08:11.761Z] eab367f1ca39: Download complete
[2026-07-14T02:08:13.555Z] ba1fabf878bd: Verifying Checksum
[2026-07-14T02:08:13.555Z] ba1fabf878bd: Download complete
[2026-07-14T02:08:13.555Z] ed86e5a5d9b1: Download complete
[2026-07-14T02:08:13.555Z] fef65a673dc9: Download complete
[2026-07-14T02:08:14.597Z] 8d998c79d309: Verifying Checksum
[2026-07-14T02:08:14.597Z] 8d998c79d309: Download complete
[2026-07-14T02:08:15.864Z] 4c8a34755929: Verifying Checksum
[2026-07-14T02:08:15.864Z] 4c8a34755929: Download complete
[2026-07-14T02:10:17.925Z] 5edf52847c35: Pull complete
[2026-07-14T02:10:17.925Z] 9548298aa5f4: Pull complete
[2026-07-14T02:10:17.925Z] 86cc05da26d1: Pull complete
[2026-07-14T02:10:17.925Z] 747272797eac: Pull complete
[2026-07-14T02:10:17.925Z] db0af168a1a4: Pull complete
[2026-07-14T02:10:17.925Z] ed874654c321: Pull complete
[2026-07-14T02:10:17.925Z] 0a0ffadd8b4f: Pull complete
[2026-07-14T02:10:17.925Z] 8b3d007242f8: Pull complete
[2026-07-14T02:12:17.436Z] dedc784bccba: Verifying Checksum
[2026-07-14T02:12:17.436Z] dedc784bccba: Download complete
[2026-07-14T02:13:01.620Z] d9a8b776bea7: Verifying Checksum
[2026-07-14T02:13:01.620Z] d9a8b776bea7: Download complete
[2026-07-14T02:18:33.834Z] dedc784bccba: Pull complete
[2026-07-14T02:18:33.834Z] 27fcad4d96a6: Pull complete
[2026-07-14T02:18:33.834Z] bf3e4cf00cd6: Pull complete
[2026-07-14T02:18:33.834Z] 9ae773ccc4b2: Pull complete
[2026-07-14T02:19:17.417Z] 78cb638d6686: Pull complete
[2026-07-14T02:19:17.417Z] f3058198fba5: Pull complete
[2026-07-14T02:19:17.417Z] f5afebc61536: Pull complete
[2026-07-14T02:19:23.654Z] cccff27d82fb: Pull complete
[2026-07-14T02:19:23.654Z] c0664a7365cb: Pull complete
[2026-07-14T02:19:23.654Z] 07b45639fc6f: Pull complete
[2026-07-14T02:19:23.654Z] b96eaed0e751: Pull complete
[2026-07-14T02:19:23.654Z] f790d498bebb: Pull complete
[2026-07-14T02:19:25.427Z] 95d704643610: Pull complete
[2026-07-14T02:19:25.427Z] 423e26ddb517: Pull complete
[2026-07-14T02:19:25.956Z] 4106a853273f: Pull complete
[2026-07-14T02:19:26.475Z] 5f8dd10c9c78: Pull complete
[2026-07-14T02:19:26.475Z] b09b7d04a8d9: Pull complete
[2026-07-14T02:19:26.475Z] 4cfd21f31c8b: Pull complete
[2026-07-14T02:19:26.475Z] 241cbb1c2f26: Pull complete
[2026-07-14T02:19:26.475Z] f0ef294e4f9b: Pull complete
[2026-07-14T02:19:26.992Z] 4f4fb700ef54: Pull complete
[2026-07-14T02:19:26.992Z] 78a3c5370db4: Pull complete
[2026-07-14T02:19:26.992Z] 42239a12ed73: Pull complete
[2026-07-14T02:19:26.992Z] 284e8cbb8a05: Pull complete
[2026-07-14T02:19:26.992Z] 0fcc3b23bd99: Pull complete
[2026-07-14T02:23:43.033Z] d9a8b776bea7: Pull complete
[2026-07-14T02:23:43.033Z] f391428cfe0f: Pull complete
[2026-07-14T02:23:43.033Z] eab367f1ca39: Pull complete
[2026-07-14T02:23:43.033Z] ba1fabf878bd: Pull complete
[2026-07-14T02:23:43.033Z] ed86e5a5d9b1: Pull complete
[2026-07-14T02:23:43.033Z] fef65a673dc9: Pull complete
[2026-07-14T02:23:43.033Z] 8d998c79d309: Pull complete
[2026-07-14T02:23:43.033Z] 4c8a34755929: Pull complete
[2026-07-14T02:23:43.033Z] Digest: sha256:b836aa0e221ac85a2184ec24a67b9d5089f3caa45d182cf521dee7d38112aa81
[2026-07-14T02:23:43.033Z] Status: Downloaded newer image for rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942
[2026-07-14T02:23:43.033Z] docker.io/rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942
[2026-07-14T02:23:43.033Z] ++ GPU_FLAGS=
[2026-07-14T02:23:43.033Z] ++ echo 'NODE_LABELS: multi_gpu rocm pytorch-nightly selftest ixt-hq-15 gfx908 multi_gpu rocm pytorch-nightly selftest ixt-hq-15 gfx908'
[2026-07-14T02:23:43.033Z] NODE_LABELS: multi_gpu rocm pytorch-nightly selftest ixt-hq-15 gfx908 multi_gpu rocm pytorch-nightly selftest ixt-hq-15 gfx908
[2026-07-14T02:23:43.033Z] ++ [[ ! multi_gpu rocm pytorch-nightly selftest ixt-hq-15 gfx908 =~ build-only ]]
[2026-07-14T02:23:43.033Z] ++ GPU_FLAGS='--group-add video --device=/dev/kfd --device=/dev/dri'
[2026-07-14T02:23:43.033Z] ++++ dirname prepare_docker_env.sh
[2026-07-14T02:23:43.033Z] +++ cd .
[2026-07-14T02:23:43.033Z] +++ pwd
[2026-07-14T02:23:43.033Z] ++ script_dir=/home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3431
[2026-07-14T02:23:43.033Z] ++ docker run --name pytorch-ci-container -e 'EXTENSION_BUILD_GFXARCH=gfx90a;gfx908;gfx942' -e CI=1 -e TEST_CONFIG=distributed -e SHARD_NUMBER=1 -e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip -e PREBUILT=false -e TEST_CORE=true -e TEST_CORE=true -v /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3431:/host_workspace -v /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3431:/scripts --entrypoint bash -d --network=host --group-add video --device=/dev/kfd --device=/dev/dri --ipc=host --pid=host --shm-size 8G --cap-add=SYS_PTRACE --security-opt seccomp=unconfined rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942 -lc 'sleep infinity'
[2026-07-14T02:23:43.548Z] 943cd5f66bc88a78855b43798fe0e00c27ce8d9621652874ba59287d58cdc987
[2026-07-14T02:23:43.548Z] ++ env -i python3 /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3431/pytorch/tools/stats/export_test_times.py
[2026-07-14T02:23:44.646Z] Exporting test times from test-infra
[2026-07-14T02:23:44.646Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-times.json to /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3431/pytorch/.additional_ci_files/test-times.json
[2026-07-14T02:23:44.646Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-class-times.json to /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3431/pytorch/.additional_ci_files/test-class-times.json
[2026-07-14T02:23:44.646Z] ++ python3 /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3431/pytorch/tools/stats/export_test_times.py
[2026-07-14T02:23:45.167Z] Exporting test times from test-infra
[2026-07-14T02:23:45.167Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-times.json to /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3431/pytorch/.additional_ci_files/test-times.json
[2026-07-14T02:23:45.167Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-class-times.json to /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3431/pytorch/.additional_ci_files/test-class-times.json
[2026-07-14T02:23:45.167Z] ++ [[ -d /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3431/pytorch/.additional_ci_files ]]
[2026-07-14T02:23:45.167Z] ++ sed -i -E 's/"\S+rocm\S+"/""/g' /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3431/pytorch/.additional_ci_files/test-times.json
[2026-07-14T02:23:45.167Z] ++ [[ false != \t\r\u\e ]]
[2026-07-14T02:23:45.167Z] ++ docker exec -u root pytorch-ci-container bash -c 'rm -rf /var/lib/jenkins/pytorch'
[2026-07-14T02:23:45.167Z] ++ docker cp /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3431/pytorch pytorch-ci-container:/var/lib/jenkins/pytorch
[2026-07-14T02:23:52.168Z] ++ docker exec -u root pytorch-ci-container bash -c 'chown root:root -R /var/lib/jenkins/pytorch'
[2026-07-14T02:23:52.168Z] ++ [[ 1 -eq 1 ]]
[2026-07-14T02:23:52.168Z] ++ docker cp /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3431/rocm_composable_kernel-7.1.1.dev0+g4266f8-py3-none-any.whl /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3431/torch-2.14.0a0+gitd4754e6-cp312-cp312-linux_x86_64.whl pytorch-ci-container:/var/lib/jenkins/pytorch
[2026-07-14T02:23:52.168Z] "docker cp" requires exactly 2 arguments.
[2026-07-14T02:23:52.168Z] See 'docker cp --help'.
[2026-07-14T02:23:52.168Z]
[2026-07-14T02:23:52.168Z] Usage: docker cp [OPTIONS] CONTAINER:SRC_PATH DEST_PATH|-
[2026-07-14T02:23:52.168Z] docker cp [OPTIONS] SRC_PATH|- CONTAINER:DEST_PATH
[2026-07-14T02:23:52.168Z]
[2026-07-14T02:23:52.168Z] Copy files/folders between a container and the local filesystem
Tests / Test Distributed / Test Distributed / Run pytorch_distributed_1 / Error signal
Error in error step, with arguments pytorch_distributed_1 failed.
pytorch_distributed_1 failed
Tests / Test Distributed / Test Distributed / Run pytorch_distributed_2 / Shell Script
Error in sh step, with arguments ./test_pytorch_test_distributed.sh.
script returned exit code 1
Build log
[2026-07-14T03:28:14.520Z] + ./test_pytorch_test_distributed.sh
[2026-07-14T03:28:14.520Z] + [[ -z '' ]]
[2026-07-14T03:28:14.520Z] + DEFAULT_EXECUTION_OPTIONS='-e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip --copy-whls'
[2026-07-14T03:28:14.520Z] + source prepare_docker_env.sh -e 'EXTENSION_BUILD_GFXARCH=gfx90a;gfx908;gfx942' -e CI=1 -e TEST_CONFIG=distributed -e SHARD_NUMBER=2 -e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip --copy-whls
[2026-07-14T03:28:14.520Z] ++ set -ex
[2026-07-14T03:28:14.520Z] ++ set -o pipefail
[2026-07-14T03:28:14.520Z] ++ copy_whls=0
[2026-07-14T03:28:14.520Z] ++ envvars=()
[2026-07-14T03:28:14.520Z] ++ [[ 13 -gt 0 ]]
[2026-07-14T03:28:14.520Z] ++ key=-e
[2026-07-14T03:28:14.520Z] ++ case $key in
[2026-07-14T03:28:14.520Z] ++ envvars+=("-e $2")
[2026-07-14T03:28:14.520Z] ++ shift
[2026-07-14T03:28:14.520Z] ++ shift
[2026-07-14T03:28:14.520Z] ++ [[ 11 -gt 0 ]]
[2026-07-14T03:28:14.520Z] ++ key=-e
[2026-07-14T03:28:14.520Z] ++ case $key in
[2026-07-14T03:28:14.520Z] ++ envvars+=("-e $2")
[2026-07-14T03:28:14.520Z] ++ shift
[2026-07-14T03:28:14.520Z] ++ shift
[2026-07-14T03:28:14.520Z] ++ [[ 9 -gt 0 ]]
[2026-07-14T03:28:14.520Z] ++ key=-e
[2026-07-14T03:28:14.520Z] ++ case $key in
[2026-07-14T03:28:14.520Z] ++ envvars+=("-e $2")
[2026-07-14T03:28:14.520Z] ++ shift
[2026-07-14T03:28:14.520Z] ++ shift
[2026-07-14T03:28:14.520Z] ++ [[ 7 -gt 0 ]]
[2026-07-14T03:28:14.520Z] ++ key=-e
[2026-07-14T03:28:14.520Z] ++ case $key in
[2026-07-14T03:28:14.520Z] ++ envvars+=("-e $2")
[2026-07-14T03:28:14.520Z] ++ shift
[2026-07-14T03:28:14.520Z] ++ shift
[2026-07-14T03:28:14.520Z] ++ [[ 5 -gt 0 ]]
[2026-07-14T03:28:14.520Z] ++ key=-e
[2026-07-14T03:28:14.520Z] ++ case $key in
[2026-07-14T03:28:14.520Z] ++ envvars+=("-e $2")
[2026-07-14T03:28:14.520Z] ++ shift
[2026-07-14T03:28:14.520Z] ++ shift
[2026-07-14T03:28:14.520Z] ++ [[ 3 -gt 0 ]]
[2026-07-14T03:28:14.520Z] ++ key=-e
[2026-07-14T03:28:14.520Z] ++ case $key in
[2026-07-14T03:28:14.520Z] ++ envvars+=("-e $2")
[2026-07-14T03:28:14.520Z] ++ shift
[2026-07-14T03:28:14.520Z] ++ shift
[2026-07-14T03:28:14.520Z] ++ [[ 1 -gt 0 ]]
[2026-07-14T03:28:14.520Z] ++ key=--copy-whls
[2026-07-14T03:28:14.520Z] ++ case $key in
[2026-07-14T03:28:14.520Z] ++ copy_whls=1
[2026-07-14T03:28:14.520Z] ++ shift
[2026-07-14T03:28:14.520Z] ++ [[ 0 -gt 0 ]]
[2026-07-14T03:28:14.520Z] ++ envvars+=("-e PREBUILT=${PREBUILT:-false}")
[2026-07-14T03:28:14.520Z] ++ envvars+=("-e TEST_CORE=${TEST_CORE:-false}")
[2026-07-14T03:28:14.520Z] ++ [[ false == \t\r\u\e ]]
[2026-07-14T03:28:14.520Z] ++ [[ true == \t\r\u\e ]]
[2026-07-14T03:28:14.520Z] ++ envvars+=("-e TEST_CORE=${TEST_CORE}")
[2026-07-14T03:28:14.520Z] ++ docker ps -a
[2026-07-14T03:28:14.520Z] CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
[2026-07-14T03:28:14.520Z] 488b7efcbc27 rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942 "bash -lc 'sleep inf…" 3 minutes ago Up 3 minutes pytorch-ci-container
[2026-07-14T03:28:14.520Z] ++ docker rm -f pytorch-ci-container
[2026-07-14T03:28:15.914Z] ++ docker pull rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942
[2026-07-14T03:28:16.574Z] pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942: Pulling from rocm/pytorch-ci-private
[2026-07-14T03:28:16.574Z] Digest: sha256:b836aa0e221ac85a2184ec24a67b9d5089f3caa45d182cf521dee7d38112aa81
[2026-07-14T03:28:16.574Z] Status: Image is up to date for rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942
[2026-07-14T03:28:16.574Z] docker.io/rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942
[2026-07-14T03:28:16.574Z] ++ GPU_FLAGS=
[2026-07-14T03:28:16.574Z] ++ echo 'NODE_LABELS: multi_gpu rocm pytorch-nightly selftest ixt-hq-15 gfx908 multi_gpu rocm pytorch-nightly selftest ixt-hq-15 gfx908'
[2026-07-14T03:28:16.574Z] NODE_LABELS: multi_gpu rocm pytorch-nightly selftest ixt-hq-15 gfx908 multi_gpu rocm pytorch-nightly selftest ixt-hq-15 gfx908
[2026-07-14T03:28:16.574Z] ++ [[ ! multi_gpu rocm pytorch-nightly selftest ixt-hq-15 gfx908 =~ build-only ]]
[2026-07-14T03:28:16.574Z] ++ GPU_FLAGS='--group-add video --device=/dev/kfd --device=/dev/dri'
[2026-07-14T03:28:16.574Z] ++++ dirname prepare_docker_env.sh
[2026-07-14T03:28:16.574Z] +++ cd .
[2026-07-14T03:28:16.574Z] +++ pwd
[2026-07-14T03:28:16.574Z] ++ script_dir=/home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3431
[2026-07-14T03:28:16.574Z] ++ docker run --name pytorch-ci-container -e 'EXTENSION_BUILD_GFXARCH=gfx90a;gfx908;gfx942' -e CI=1 -e TEST_CONFIG=distributed -e SHARD_NUMBER=2 -e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip -e PREBUILT=false -e TEST_CORE=true -e TEST_CORE=true -v /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3431:/host_workspace -v /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3431:/scripts --entrypoint bash -d --network=host --group-add video --device=/dev/kfd --device=/dev/dri --ipc=host --pid=host --shm-size 8G --cap-add=SYS_PTRACE --security-opt seccomp=unconfined rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942 -lc 'sleep infinity'
[2026-07-14T03:28:16.574Z] f72db091c5a23f6ffd3da35d2f548dd62c34a1577eb9a0a9a3a36ec83391d9c5
[2026-07-14T03:28:16.574Z] ++ env -i python3 /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3431/pytorch/tools/stats/export_test_times.py
[2026-07-14T03:28:17.250Z] Exporting test times from test-infra
[2026-07-14T03:28:17.250Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-times.json to /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3431/pytorch/.additional_ci_files/test-times.json
[2026-07-14T03:28:17.250Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-class-times.json to /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3431/pytorch/.additional_ci_files/test-class-times.json
[2026-07-14T03:28:17.250Z] ++ python3 /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3431/pytorch/tools/stats/export_test_times.py
[2026-07-14T03:28:17.250Z] Exporting test times from test-infra
Output truncated.
Details
- Kill older PR Builds (2.4 sec)
- Initialize (1 hr 14 min)
- Download CI scripts (24 sec)
- Checkout Pytorch (1 min 38 sec)
- Check base Docker image existence (9.4 sec)
- Pull Docker Image (5 min 0 sec)
- Build PyTorch (1 hr 5 min)
- Tests (3 hr 33 min)
- Test PyTorch (12 ms)
- Test Distributed (11 ms)
- Test Inductor (12 ms)
- Test PyTorch Slow (9 ms)
- Test PyTorch Slow (14 sec)
- Microbenchmark (29 sec)
- Microbenchmark (14 sec)
- Post Build (1.7 sec)
- Declarative: Post Actions (3.6 sec)
Loading