[CI] Parity: fix mi350 inductor reading rerun_disabled trunk run (50k phantom MISSED)#3447
Open
ethanwee1 wants to merge 17 commits into
Open
[CI] Parity: fix mi350 inductor reading rerun_disabled trunk run (50k phantom MISSED)#3447ethanwee1 wants to merge 17 commits into
ethanwee1 wants to merge 17 commits into
ROCm Repo Management API / Jenkins
failed
Jul 15, 2026 in 1h 42m 4s
Tests/Test Inductor/Run pytorch_inductor_1: error in 'error' step
Tests / Test PyTorch / Test PyTorch / Run pytorch_test_1 / Shell Script
Error in sh step, with arguments ./test_pytorch_test.sh.
script returned exit code 1
Build log
[2026-07-15T21:55:02.182Z] + ./test_pytorch_test.sh
[2026-07-15T21:55:02.182Z] + [[ -z '' ]]
[2026-07-15T21:55:02.182Z] + DEFAULT_EXECUTION_OPTIONS='-e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip --copy-whls'
[2026-07-15T21:55:02.182Z] + source prepare_docker_env.sh -e 'EXTENSION_BUILD_GFXARCH=gfx90a;gfx908;gfx942' -e CI=1 -e TEST_CONFIG=default -e SHARD_NUMBER=1 -e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip --copy-whls
[2026-07-15T21:55:02.182Z] ++ set -ex
[2026-07-15T21:55:02.182Z] ++ set -o pipefail
[2026-07-15T21:55:02.182Z] ++ copy_whls=0
[2026-07-15T21:55:02.182Z] ++ envvars=()
[2026-07-15T21:55:02.182Z] ++ [[ 13 -gt 0 ]]
[2026-07-15T21:55:02.182Z] ++ key=-e
[2026-07-15T21:55:02.182Z] ++ case $key in
[2026-07-15T21:55:02.182Z] ++ envvars+=("-e $2")
[2026-07-15T21:55:02.182Z] ++ shift
[2026-07-15T21:55:02.182Z] ++ shift
[2026-07-15T21:55:02.182Z] ++ [[ 11 -gt 0 ]]
[2026-07-15T21:55:02.182Z] ++ key=-e
[2026-07-15T21:55:02.182Z] ++ case $key in
[2026-07-15T21:55:02.182Z] ++ envvars+=("-e $2")
[2026-07-15T21:55:02.182Z] ++ shift
[2026-07-15T21:55:02.182Z] ++ shift
[2026-07-15T21:55:02.182Z] ++ [[ 9 -gt 0 ]]
[2026-07-15T21:55:02.182Z] ++ key=-e
[2026-07-15T21:55:02.182Z] ++ case $key in
[2026-07-15T21:55:02.182Z] ++ envvars+=("-e $2")
[2026-07-15T21:55:02.182Z] ++ shift
[2026-07-15T21:55:02.182Z] ++ shift
[2026-07-15T21:55:02.182Z] ++ [[ 7 -gt 0 ]]
[2026-07-15T21:55:02.182Z] ++ key=-e
[2026-07-15T21:55:02.182Z] ++ case $key in
[2026-07-15T21:55:02.182Z] ++ envvars+=("-e $2")
[2026-07-15T21:55:02.182Z] ++ shift
[2026-07-15T21:55:02.182Z] ++ shift
[2026-07-15T21:55:02.182Z] ++ [[ 5 -gt 0 ]]
[2026-07-15T21:55:02.182Z] ++ key=-e
[2026-07-15T21:55:02.182Z] ++ case $key in
[2026-07-15T21:55:02.182Z] ++ envvars+=("-e $2")
[2026-07-15T21:55:02.182Z] ++ shift
[2026-07-15T21:55:02.182Z] ++ shift
[2026-07-15T21:55:02.182Z] ++ [[ 3 -gt 0 ]]
[2026-07-15T21:55:02.182Z] ++ key=-e
[2026-07-15T21:55:02.182Z] ++ case $key in
[2026-07-15T21:55:02.182Z] ++ envvars+=("-e $2")
[2026-07-15T21:55:02.182Z] ++ shift
[2026-07-15T21:55:02.182Z] ++ shift
[2026-07-15T21:55:02.182Z] ++ [[ 1 -gt 0 ]]
[2026-07-15T21:55:02.182Z] ++ key=--copy-whls
[2026-07-15T21:55:02.182Z] ++ case $key in
[2026-07-15T21:55:02.182Z] ++ copy_whls=1
[2026-07-15T21:55:02.182Z] ++ shift
[2026-07-15T21:55:02.182Z] ++ [[ 0 -gt 0 ]]
[2026-07-15T21:55:02.182Z] ++ envvars+=("-e PREBUILT=${PREBUILT:-false}")
[2026-07-15T21:55:02.182Z] ++ envvars+=("-e TEST_CORE=${TEST_CORE:-false}")
[2026-07-15T21:55:02.182Z] ++ [[ false == \t\r\u\e ]]
[2026-07-15T21:55:02.182Z] ++ [[ true == \t\r\u\e ]]
[2026-07-15T21:55:02.182Z] ++ envvars+=("-e TEST_CORE=${TEST_CORE}")
[2026-07-15T21:55:02.182Z] ++ docker ps -a
[2026-07-15T21:55:02.182Z] CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
[2026-07-15T21:55:02.182Z] 1cf9b630e2ad rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-27d7487d6141807234c885bc3b76f82d055522fd-gfx908_gfx90a_gfx942 "bash -lc 'sleep inf…" 3 hours ago Exited (143) 34 minutes ago pytorch-ci-container
[2026-07-15T21:55:02.182Z] 33af8c9e058f dashboard-dashboard-ui "/docker-entrypoint.…" 3 weeks ago Up 28 minutes 80/tcp, 0.0.0.0:3000->3000/tcp, [::]:3000->3000/tcp rocagents-dashboard-ui
[2026-07-15T21:55:02.182Z] 7c6be83a2db1 dashboard-dashboard-api "uvicorn main:app --…" 3 weeks ago Up 28 minutes (healthy) 0.0.0.0:8080->8080/tcp, [::]:8080->8080/tcp rocagents-dashboard-api
[2026-07-15T21:55:02.182Z] 780dc8fe5a7b rocm/eigen-test:ubuntu20.04-rocm5.3.0-eigen "/scripts/run_tests_…" 6 weeks ago Exited (1) 6 weeks ago infallible_chatelet
[2026-07-15T21:55:02.182Z] 2d7786cfd066 deployment-rocagents "python src/multi_re…" 2 months ago Up 28 minutes (healthy) rocagents-orchestrator
[2026-07-15T21:55:02.182Z] ++ docker rm -f pytorch-ci-container
[2026-07-15T21:55:02.182Z] ++ docker pull rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942
[2026-07-15T21:55:03.290Z] pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942: Pulling from rocm/pytorch-ci-private
[2026-07-15T21:55:03.290Z] cb259a83ac3d: Already exists
[2026-07-15T21:55:03.290Z] 99e6462d03e0: Pulling fs layer
[2026-07-15T21:55:03.290Z] 1057fba0c7df: Pulling fs layer
[2026-07-15T21:55:03.290Z] 0e6d9b5f5e8d: Pulling fs layer
[2026-07-15T21:55:03.290Z] 65f281ce2e70: Pulling fs layer
[2026-07-15T21:55:03.290Z] 34e29023481f: Pulling fs layer
[2026-07-15T21:55:03.290Z] 8d78d529f04d: Pulling fs layer
[2026-07-15T21:55:03.290Z] 906ac4de0a2b: Pulling fs layer
[2026-07-15T21:55:03.290Z] c608e00b1282: Pulling fs layer
[2026-07-15T21:55:03.290Z] d7df6379f9cc: Pulling fs layer
[2026-07-15T21:55:03.290Z] 3d56742d201d: Pulling fs layer
[2026-07-15T21:55:03.290Z] 5edf52847c35: Pulling fs layer
[2026-07-15T21:55:03.290Z] 9548298aa5f4: Pulling fs layer
[2026-07-15T21:55:03.290Z] 86cc05da26d1: Pulling fs layer
[2026-07-15T21:55:03.290Z] 747272797eac: Pulling fs layer
[2026-07-15T21:55:03.290Z] db0af168a1a4: Pulling fs layer
[2026-07-15T21:55:03.290Z] ed874654c321: Pulling fs layer
[2026-07-15T21:55:03.290Z] 0a0ffadd8b4f: Pulling fs layer
[2026-07-15T21:55:03.290Z] 8b3d007242f8: Pulling fs layer
[2026-07-15T21:55:03.290Z] 34e29023481f: Waiting
[2026-07-15T21:55:03.290Z] dedc784bccba: Pulling fs layer
[2026-07-15T21:55:03.290Z] 27fcad4d96a6: Pulling fs layer
[2026-07-15T21:55:03.290Z] 8d78d529f04d: Waiting
[2026-07-15T21:55:03.290Z] bf3e4cf00cd6: Pulling fs layer
[2026-07-15T21:55:03.290Z] 906ac4de0a2b: Waiting
[2026-07-15T21:55:03.290Z] 9ae773ccc4b2: Pulling fs layer
[2026-07-15T21:55:03.290Z] 3d56742d201d: Waiting
[2026-07-15T21:55:03.290Z] c608e00b1282: Waiting
[2026-07-15T21:55:03.290Z] 78cb638d6686: Pulling fs layer
[2026-07-15T21:55:03.290Z] 5edf52847c35: Waiting
[2026-07-15T21:55:03.290Z] f3058198fba5: Pulling fs layer
[2026-07-15T21:55:03.290Z] d7df6379f9cc: Waiting
[2026-07-15T21:55:03.290Z] f5afebc61536: Pulling fs layer
[2026-07-15T21:55:03.290Z] 9548298aa5f4: Waiting
[2026-07-15T21:55:03.290Z] cccff27d82fb: Pulling fs layer
[2026-07-15T21:55:03.290Z] 86cc05da26d1: Waiting
[2026-07-15T21:55:03.290Z] 747272797eac: Waiting
[2026-07-15T21:55:03.290Z] 65f281ce2e70: Waiting
[2026-07-15T21:55:03.290Z] c0664a7365cb: Pulling fs layer
[2026-07-15T21:55:03.290Z] 07b45639fc6f: Pulling fs layer
[2026-07-15T21:55:03.290Z] b96eaed0e751: Pulling fs layer
[2026-07-15T21:55:03.290Z] f790d498bebb: Pulling fs layer
[2026-07-15T21:55:03.290Z] bf3e4cf00cd6: Waiting
[2026-07-15T21:55:03.290Z] 9ae773ccc4b2: Waiting
[2026-07-15T21:55:03.290Z] 95d704643610: Pulling fs layer
[2026-07-15T21:55:03.290Z] db0af168a1a4: Waiting
[2026-07-15T21:55:03.290Z] 423e26ddb517: Pulling fs layer
[2026-07-15T21:55:03.290Z] 4106a853273f: Pulling fs layer
[2026-07-15T21:55:03.290Z] 78cb638d6686: Waiting
[2026-07-15T21:55:03.290Z] ed874654c321: Waiting
[2026-07-15T21:55:03.290Z] 5f8dd10c9c78: Pulling fs layer
[2026-07-15T21:55:03.290Z] f3058198fba5: Waiting
[2026-07-15T21:55:03.290Z] b09b7d04a8d9: Pulling fs layer
[2026-07-15T21:55:03.290Z] 4cfd21f31c8b: Pulling fs layer
[2026-07-15T21:55:03.290Z] 241cbb1c2f26: Pulling fs layer
[2026-07-15T21:55:03.290Z] f0ef294e4f9b: Pulling fs layer
[2026-07-15T21:55:03.290Z] 4f4fb700ef54: Pulling fs layer
[2026-07-15T21:55:03.290Z] 78a3c5370db4: Pulling fs layer
[2026-07-15T21:55:03.290Z] f5afebc61536: Waiting
[2026-07-15T21:55:03.290Z] 42239a12ed73: Pulling fs layer
[2026-07-15T21:55:03.290Z] 95d704643610: Waiting
[2026-07-15T21:55:03.290Z] cccff27d82fb: Waiting
[2026-07-15T21:55:03.290Z] 423e26ddb517: Waiting
[2026-07-15T21:55:03.290Z] 284e8cbb8a05: Pulling fs layer
[2026-07-15T21:55:03.290Z] c0664a7365cb: Waiting
[2026-07-15T21:55:03.290Z] b09b7d04a8d9: Waiting
[2026-07-15T21:55:03.290Z] dedc784bccba: Waiting
[2026-07-15T21:55:03.290Z] 4106a853273f: Waiting
[2026-07-15T21:55:03.290Z] 0fcc3b23bd99: Pulling fs layer
[2026-07-15T21:55:03.290Z] 5f8dd10c9c78: Waiting
[2026-07-15T21:55:03.290Z] d9a8b776bea7: Pulling fs layer
[2026-07-15T21:55:03.290Z] 27fcad4d96a6: Waiting
[2026-07-15T21:55:03.290Z] f391428cfe0f: Pulling fs layer
[2026-07-15T21:55:03.290Z] eab367f1ca39: Pulling fs layer
[2026-07-15T21:55:03.290Z] ba1fabf878bd: Pulling fs layer
[2026-07-15T21:55:03.290Z] 0a0ffadd8b4f: Waiting
[2026-07-15T21:55:03.290Z] 07b45639fc6f: Waiting
[2026-07-15T21:55:03.290Z] ed86e5a5d9b1: Pulling fs layer
[2026-07-15T21:55:03.290Z] b96eaed0e751: Waiting
[2026-07-15T21:55:03.290Z] f790d498bebb: Waiting
[2026-07-15T21:55:03.290Z] fef65a673dc9: Pulling fs layer
[2026-07-15T21:55:03.290Z] 78a3c5370db4: Waiting
[2026-07-15T21:55:03.290Z] 241cbb1c2f26: Waiting
[2026-07-15T21:55:03.290Z] ba1fabf878bd: Waiting
[2026-07-15T21:55:03.290Z] f0ef294e4f9b: Waiting
[2026-07-15T21:55:03.290Z] 8d998c79d309: Pulling fs layer
[2026-07-15T21:55:03.290Z] 4c8a34755929: Pulling fs layer
[2026-07-15T21:55:03.290Z] 8b3d007242f8: Waiting
[2026-07-15T21:55:03.290Z] 4f4fb700ef54: Waiting
[2026-07-15T21:55:03.290Z] ed86e5a5d9b1: Waiting
[2026-07-15T21:55:03.290Z] f391428cfe0f: Waiting
[2026-07-15T21:55:03.290Z] d9a8b776bea7: Waiting
[2026-07-15T21:55:03.290Z] 0fcc3b23bd99: Waiting
[2026-07-15T21:55:03.290Z] 8d998c79d309: Waiting
[2026-07-15T21:55:03.290Z] fef65a673dc9: Waiting
[2026-07-15T21:55:03.290Z] 4c8a34755929: Waiting
[2026-07-15T21:55:03.290Z] 284e8cbb8a05: Waiting
[2026-07-15T21:55:03.290Z] 4cfd21f31c8b: Waiting
[2026-07-15T21:55:03.290Z] 42239a12ed73: Waiting
[2026-07-15T21:55:03.290Z] eab367f1ca39: Waiting
[2026-07-15T21:55:04.101Z] 0e6d9b5f5e8d: Verifying Checksum
[2026-07-15T21:55:04.101Z] 0e6d9b5f5e8d: Download complete
[2026-07-15T21:55:04.101Z] 99e6462d03e0: Verifying Checksum
[2026-07-15T21:55:04.101Z] 99e6462d03e0: Download complete
[2026-07-15T21:55:04.101Z] 99e6462d03e0: Pull complete
[2026-07-15T21:55:04.101Z] 65f281ce2e70: Verifying Checksum
[2026-07-15T21:55:04.101Z] 65f281ce2e70: Download complete
[2026-07-15T21:55:04.101Z] 34e29023481f: Download complete
[2026-07-15T21:55:04.101Z] 8d78d529f04d: Verifying Checksum
[2026-07-15T21:55:04.101Z] 8d78d529f04d: Download complete
[2026-07-15T21:55:04.101Z] 906ac4de0a2b: Verifying Checksum
[2026-07-15T21:55:04.101Z] 906ac4de0a2b: Download complete
[2026-07-15T21:55:04.101Z] c608e00b1282: Verifying Checksum
[2026-07-15T21:55:04.101Z] c608e00b1282: Download complete
[2026-07-15T21:55:04.101Z] d7df6379f9cc: Verifying Checksum
[2026-07-15T21:55:04.101Z] d7df6379f9cc: Download complete
[2026-07-15T21:55:04.101Z] 3d56742d201d: Verifying Checksum
[2026-07-15T21:55:04.101Z] 3d56742d201d: Download complete
[2026-07-15T21:55:04.101Z] 9548298aa5f4: Verifying Checksum
[2026-07-15T21:55:04.101Z] 9548298aa5f4: Download complete
[2026-07-15T21:55:04.101Z] 86cc05da26d1: Download complete
[2026-07-15T21:55:04.101Z] 747272797eac: Download complete
[2026-07-15T21:55:04.101Z] db0af168a1a4: Download complete
[2026-07-15T21:55:04.101Z] ed874654c321: Verifying Checksum
[2026-07-15T21:55:04.101Z] ed874654c321: Download complete
[2026-07-15T21:55:04.639Z] 0a0ffadd8b4f: Verifying Checksum
[2026-07-15T21:55:04.639Z] 0a0ffadd8b4f: Download complete
[2026-07-15T21:55:04.639Z] 8b3d007242f8: Download complete
[2026-07-15T21:55:09.944Z] 1057fba0c7df: Verifying Checksum
[2026-07-15T21:55:09.944Z] 1057fba0c7df: Download complete
[2026-07-15T21:55:09.944Z] 27fcad4d96a6: Download complete
[2026-07-15T21:55:10.492Z] bf3e4cf00cd6: Verifying Checksum
[2026-07-15T21:55:10.492Z] bf3e4cf00cd6: Download complete
[2026-07-15T21:55:10.492Z] 9ae773ccc4b2: Verifying Checksum
[2026-07-15T21:55:10.492Z] 9ae773ccc4b2: Download complete
[2026-07-15T21:55:21.340Z] 1057fba0c7df: Pull complete
[2026-07-15T21:55:21.340Z] 0e6d9b5f5e8d: Pull complete
[2026-07-15T21:55:21.340Z] 65f281ce2e70: Pull complete
[2026-07-15T21:55:21.340Z] 34e29023481f: Pull complete
[2026-07-15T21:55:21.340Z] 8d78d529f04d: Pull complete
[2026-07-15T21:55:21.340Z] 906ac4de0a2b: Pull complete
[2026-07-15T21:55:21.340Z] c608e00b1282: Pull complete
[2026-07-15T21:55:21.340Z] d7df6379f9cc: Pull complete
[2026-07-15T21:55:21.340Z] 3d56742d201d: Pull complete
[2026-07-15T21:55:34.555Z] 78cb638d6686: Verifying Checksum
[2026-07-15T21:55:34.555Z] 78cb638d6686: Download complete
[2026-07-15T21:55:34.555Z] f3058198fba5: Download complete
[2026-07-15T21:55:34.555Z] f5afebc61536: Download complete
[2026-07-15T21:55:38.997Z] cccff27d82fb: Verifying Checksum
[2026-07-15T21:55:38.997Z] cccff27d82fb: Download complete
[2026-07-15T21:55:38.997Z] c0664a7365cb: Verifying Checksum
[2026-07-15T21:55:38.997Z] c0664a7365cb: Download complete
[2026-07-15T21:55:38.997Z] 07b45639fc6f: Verifying Checksum
[2026-07-15T21:55:38.997Z] 07b45639fc6f: Download complete
[2026-07-15T21:55:38.997Z] b96eaed0e751: Verifying Checksum
[2026-07-15T21:55:38.997Z] b96eaed0e751: Download complete
[2026-07-15T21:55:38.997Z] f790d498bebb: Download complete
[2026-07-15T21:55:41.115Z] 95d704643610: Verifying Checksum
[2026-07-15T21:55:41.115Z] 95d704643610: Download complete
[2026-07-15T21:55:41.115Z] 423e26ddb517: Download complete
[2026-07-15T21:55:41.657Z] 4106a853273f: Verifying Checksum
[2026-07-15T21:55:41.657Z] 4106a853273f: Download complete
[2026-07-15T21:55:41.657Z] 5f8dd10c9c78: Verifying Checksum
[2026-07-15T21:55:41.657Z] 5f8dd10c9c78: Download complete
[2026-07-15T21:55:41.657Z] b09b7d04a8d9: Download complete
[2026-07-15T21:55:41.657Z] 4cfd21f31c8b: Verifying Checksum
[2026-07-15T21:55:41.657Z] 4cfd21f31c8b: Download complete
[2026-07-15T21:55:41.657Z] 241cbb1c2f26: Verifying Checksum
[2026-07-15T21:55:41.657Z] 241cbb1c2f26: Download complete
[2026-07-15T21:55:41.657Z] f0ef294e4f9b: Verifying Checksum
[2026-07-15T21:55:41.657Z] f0ef294e4f9b: Download complete
[2026-07-15T21:55:41.657Z] 4f4fb700ef54: Verifying Checksum
[2026-07-15T21:55:41.657Z] 4f4fb700ef54: Download complete
[2026-07-15T21:55:42.456Z] 78a3c5370db4: Download complete
[2026-07-15T21:55:42.456Z] 42239a12ed73: Verifying Checksum
[2026-07-15T21:55:42.456Z] 42239a12ed73: Download complete
[2026-07-15T21:55:42.456Z] 284e8cbb8a05: Verifying Checksum
[2026-07-15T21:55:42.456Z] 284e8cbb8a05: Download complete
[2026-07-15T21:55:42.456Z] 0fcc3b23bd99: Download complete
[2026-07-15T21:56:09.940Z] 5edf52847c35: Verifying Checksum
[2026-07-15T21:56:09.940Z] 5edf52847c35: Download complete
[2026-07-15T21:56:09.940Z] f391428cfe0f: Verifying Checksum
[2026-07-15T21:56:09.940Z] f391428cfe0f: Download complete
[2026-07-15T21:56:09.940Z] eab367f1ca39: Download complete
[2026-07-15T21:56:09.940Z] ba1fabf878bd: Verifying Checksum
[2026-07-15T21:56:09.940Z] ba1fabf878bd: Download complete
[2026-07-15T21:56:09.940Z] ed86e5a5d9b1: Download complete
[2026-07-15T21:56:09.940Z] fef65a673dc9: Verifying Checksum
[2026-07-15T21:56:09.940Z] fef65a673dc9: Download complete
[2026-07-15T21:56:09.940Z] 8d998c79d309: Verifying Checksum
[2026-07-15T21:56:09.940Z] 8d998c79d309: Download complete
[2026-07-15T21:56:09.940Z] 4c8a34755929: Download complete
[2026-07-15T21:57:04.236Z] 5edf52847c35: Pull complete
[2026-07-15T21:57:04.236Z] 9548298aa5f4: Pull complete
[2026-07-15T21:57:04.236Z] 86cc05da26d1: Pull complete
[2026-07-15T21:57:04.236Z] 747272797eac: Pull complete
[2026-07-15T21:57:04.236Z] db0af168a1a4: Pull complete
[2026-07-15T21:57:04.236Z] ed874654c321: Pull complete
[2026-07-15T21:57:04.236Z] 0a0ffadd8b4f: Pull complete
[2026-07-15T21:57:04.236Z] 8b3d007242f8: Pull complete
[2026-07-15T21:57:49.931Z] d9a8b776bea7: Verifying Checksum
[2026-07-15T21:57:49.931Z] d9a8b776bea7: Download complete
[2026-07-15T21:57:51.689Z] dedc784bccba: Download complete
[2026-07-15T22:00:05.094Z] dedc784bccba: Pull complete
[2026-07-15T22:00:05.095Z] 27fcad4d96a6: Pull complete
[2026-07-15T22:00:05.095Z] bf3e4cf00cd6: Pull complete
[2026-07-15T22:00:05.095Z] 9ae773ccc4b2: Pull complete
[2026-07-15T22:00:16.157Z] 78cb638d6686: Pull complete
[2026-07-15T22:00:16.157Z] f3058198fba5: Pull complete
[2026-07-15T22:00:16.157Z] f5afebc61536: Pull complete
[2026-07-15T22:00:17.253Z] cccff27d82fb: Pull complete
[2026-07-15T22:00:17.253Z] c0664a7365cb: Pull complete
[2026-07-15T22:00:17.253Z] 07b45639fc6f: Pull complete
[2026-07-15T22:00:17.253Z] b96eaed0e751: Pull complete
[2026-07-15T22:00:17.253Z] f790d498bebb: Pull complete
[2026-07-15T22:00:18.348Z] 95d704643610: Pull complete
[2026-07-15T22:00:18.348Z] 423e26ddb517: Pull complete
[2026-07-15T22:00:18.348Z] 4106a853273f: Pull complete
[2026-07-15T22:00:18.348Z] 5f8dd10c9c78: Pull complete
[2026-07-15T22:00:18.348Z] b09b7d04a8d9: Pull complete
[2026-07-15T22:00:18.348Z] 4cfd21f31c8b: Pull complete
[2026-07-15T22:00:18.348Z] 241cbb1c2f26: Pull complete
[2026-07-15T22:00:18.348Z] f0ef294e4f9b: Pull complete
[2026-07-15T22:00:18.348Z] 4f4fb700ef54: Pull complete
[2026-07-15T22:00:18.348Z] 78a3c5370db4: Pull complete
[2026-07-15T22:00:18.348Z] 42239a12ed73: Pull complete
[2026-07-15T22:00:18.348Z] 284e8cbb8a05: Pull complete
[2026-07-15T22:00:18.348Z] 0fcc3b23bd99: Pull complete
[2026-07-15T22:01:45.941Z] d9a8b776bea7: Pull complete
[2026-07-15T22:01:45.941Z] f391428cfe0f: Pull complete
[2026-07-15T22:01:45.941Z] eab367f1ca39: Pull complete
[2026-07-15T22:01:45.941Z] ba1fabf878bd: Pull complete
[2026-07-15T22:01:45.941Z] ed86e5a5d9b1: Pull complete
[2026-07-15T22:01:45.941Z] fef65a673dc9: Pull complete
[2026-07-15T22:01:45.941Z] 8d998c79d309: Pull complete
[2026-07-15T22:01:45.941Z] 4c8a34755929: Pull complete
[2026-07-15T22:01:45.941Z] Digest: sha256:b836aa0e221ac85a2184ec24a67b9d5089f3caa45d182cf521dee7d38112aa81
[2026-07-15T22:01:45.941Z] Status: Downloaded newer image for rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942
[2026-07-15T22:01:45.941Z] docker.io/rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942
[2026-07-15T22:01:45.941Z] ++ GPU_FLAGS=
[2026-07-15T22:01:45.941Z] ++ echo 'NODE_LABELS: multi_gpu rocm dlmodels gpu_8 gfx90a ixt-hq-ubb4-31 mlir MI250 multi_gpu rocm dlmodels gpu_8 gfx90a ixt-hq-ubb4-31 mlir MI250'
[2026-07-15T22:01:45.941Z] NODE_LABELS: multi_gpu rocm dlmodels gpu_8 gfx90a ixt-hq-ubb4-31 mlir MI250 multi_gpu rocm dlmodels gpu_8 gfx90a ixt-hq-ubb4-31 mlir MI250
[2026-07-15T22:01:45.941Z] ++ [[ ! multi_gpu rocm dlmodels gpu_8 gfx90a ixt-hq-ubb4-31 mlir MI250 =~ build-only ]]
[2026-07-15T22:01:45.941Z] ++ GPU_FLAGS='--group-add video --device=/dev/kfd --device=/dev/dri'
[2026-07-15T22:01:45.941Z] ++++ dirname prepare_docker_env.sh
[2026-07-15T22:01:45.941Z] +++ cd .
[2026-07-15T22:01:45.941Z] +++ pwd
[2026-07-15T22:01:45.941Z] ++ script_dir=/home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3447
[2026-07-15T22:01:45.941Z] ++ docker run --name pytorch-ci-container -e 'EXTENSION_BUILD_GFXARCH=gfx90a;gfx908;gfx942' -e CI=1 -e TEST_CONFIG=default -e SHARD_NUMBER=1 -e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip -e PREBUILT=false -e TEST_CORE=true -e TEST_CORE=true -v /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3447:/host_workspace -v /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3447:/scripts --entrypoint bash -d --network=host --group-add video --device=/dev/kfd --device=/dev/dri --ipc=host --pid=host --shm-size 8G --cap-add=SYS_PTRACE --security-opt seccomp=unconfined rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942 -lc 'sleep infinity'
[2026-07-15T22:01:45.941Z] cab8d25a67ee0859a6110a3a2ed38b380be0e36e478325a0916064485c704955
[2026-07-15T22:01:45.941Z] ++ env -i python3 /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3447/pytorch/tools/stats/export_test_times.py
[2026-07-15T22:01:45.941Z] Exporting test times from test-infra
[2026-07-15T22:01:45.941Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-times.json to /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3447/pytorch/.additional_ci_files/test-times.json
[2026-07-15T22:01:45.941Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-class-times.json to /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3447/pytorch/.additional_ci_files/test-class-times.json
[2026-07-15T22:01:45.941Z] ++ python3 /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3447/pytorch/tools/stats/export_test_times.py
[2026-07-15T22:01:45.941Z] Exporting test times from test-infra
[2026-07-15T22:01:45.941Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-times.json to /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3447/pytorch/.additional_ci_files/test-times.json
[2026-07-15T22:01:45.941Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-class-times.json to /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3447/pytorch/.additional_ci_files/test-class-times.json
[2026-07-15T22:01:45.941Z] ++ [[ -d /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3447/pytorch/.additional_ci_files ]]
[2026-07-15T22:01:45.941Z] ++ sed -i -E 's/"\S+rocm\S+"/""/g' /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3447/pytorch/.additional_ci_files/test-times.json
[2026-07-15T22:01:45.941Z] ++ [[ false != \t\r\u\e ]]
[2026-07-15T22:01:45.941Z] ++ docker exec -u root pytorch-ci-container bash -c 'rm -rf /var/lib/jenkins/pytorch'
[2026-07-15T22:01:45.941Z] ++ docker cp /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3447/pytorch pytorch-ci-container:/var/lib/jenkins/pytorch
[2026-07-15T22:01:50.072Z] ++ docker exec -u root pytorch-ci-container bash -c 'chown root:root -R /var/lib/jenkins/pytorch'
[2026-07-15T22:01:50.072Z] ++ [[ 1 -eq 1 ]]
[2026-07-15T22:01:50.072Z] ++ docker cp /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3447/rocm_composable_kernel-7.1.1.dev0+g4266f8-py3-none-any.whl /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3447/torch-2.14.0a0+git579ba57-cp312-cp312-linux_x86_64.whl pytorch-ci-container:/var/lib/jenkins/pytorch
[2026-07-15T22:01:50.072Z] docker: 'docker cp' requires 2 arguments
[2026-07-15T22:01:50.072Z]
[2026-07-15T22:01:50.072Z] Usage: docker cp [OPTIONS] CONTAINER:SRC_PATH DEST_PATH|-
[2026-07-15T22:01:50.072Z] docker cp [OPTIONS] SRC_PATH|- CONTAINER:DEST_PATH
[2026-07-15T22:01:50.072Z]
[2026-07-15T22:01:50.072Z] Run 'docker cp --help' for more information
Tests / Test PyTorch / Test PyTorch / Run pytorch_test_1 / Error signal
Error in error step, with arguments pytorch_test_1 failed.
pytorch_test_1 failed
Tests / Test PyTorch / Test PyTorch / Run pytorch_test_2 / Shell Script
Error in sh step, with arguments ./test_pytorch_test.sh.
script returned exit code 1
Build log
[2026-07-15T22:07:25.088Z] + ./test_pytorch_test.sh
[2026-07-15T22:07:25.088Z] + [[ -z '' ]]
[2026-07-15T22:07:25.088Z] + DEFAULT_EXECUTION_OPTIONS='-e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip --copy-whls'
[2026-07-15T22:07:25.088Z] + source prepare_docker_env.sh -e 'EXTENSION_BUILD_GFXARCH=gfx90a;gfx908;gfx942' -e CI=1 -e TEST_CONFIG=default -e SHARD_NUMBER=2 -e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip --copy-whls
[2026-07-15T22:07:25.088Z] ++ set -ex
[2026-07-15T22:07:25.088Z] ++ set -o pipefail
[2026-07-15T22:07:25.088Z] ++ copy_whls=0
[2026-07-15T22:07:25.088Z] ++ envvars=()
[2026-07-15T22:07:25.088Z] ++ [[ 13 -gt 0 ]]
[2026-07-15T22:07:25.088Z] ++ key=-e
[2026-07-15T22:07:25.088Z] ++ case $key in
[2026-07-15T22:07:25.088Z] ++ envvars+=("-e $2")
[2026-07-15T22:07:25.088Z] ++ shift
[2026-07-15T22:07:25.088Z] ++ shift
[2026-07-15T22:07:25.088Z] ++ [[ 11 -gt 0 ]]
[2026-07-15T22:07:25.088Z] ++ key=-e
[2026-07-15T22:07:25.088Z] ++ case $key in
[2026-07-15T22:07:25.088Z] ++ envvars+=("-e $2")
[2026-07-15T22:07:25.088Z] ++ shift
[2026-07-15T22:07:25.088Z] ++ shift
[2026-07-15T22:07:25.088Z] ++ [[ 9 -gt 0 ]]
[2026-07-15T22:07:25.088Z] ++ key=-e
[2026-07-15T22:07:25.088Z] ++ case $key in
[2026-07-15T22:07:25.088Z] ++ envvars+=("-e $2")
[2026-07-15T22:07:25.088Z] ++ shift
[2026-07-15T22:07:25.088Z] ++ shift
[2026-07-15T22:07:25.088Z] ++ [[ 7 -gt 0 ]]
[2026-07-15T22:07:25.088Z] ++ key=-e
[2026-07-15T22:07:25.088Z] ++ case $key in
[2026-07-15T22:07:25.088Z] ++ envvars+=("-e $2")
[2026-07-15T22:07:25.088Z] ++ shift
[2026-07-15T22:07:25.088Z] ++ shift
[2026-07-15T22:07:25.088Z] ++ [[ 5 -gt 0 ]]
[2026-07-15T22:07:25.088Z] ++ key=-e
[2026-07-15T22:07:25.088Z] ++ case $key in
[2026-07-15T22:07:25.088Z] ++ envvars+=("-e $2")
[2026-07-15T22:07:25.088Z] ++ shift
[2026-07-15T22:07:25.088Z] ++ shift
[2026-07-15T22:07:25.088Z] ++ [[ 3 -gt 0 ]]
[2026-07-15T22:07:25.088Z] ++ key=-e
[2026-07-15T22:07:25.088Z] ++ case $key in
[2026-07-15T22:07:25.088Z] ++ envvars+=("-e $2")
[2026-07-15T22:07:25.088Z] ++ shift
[2026-07-15T22:07:25.088Z] ++ shift
[2026-07-15T22:07:25.088Z] ++ [[ 1 -gt 0 ]]
[2026-07-15T22:07:25.088Z] ++ key=--copy-whls
[2026-07-15T22:07:25.088Z] ++ case $key in
[2026-07-15T22:07:25.088Z] ++ copy_whls=1
[2026-07-15T22:07:25.088Z] ++ shift
[2026-07-15T22:07:25.088Z] ++ [[ 0 -gt 0 ]]
[2026-07-15T22:07:25.088Z] ++ envvars+=("-e PREBUILT=${PREBUILT:-false}")
[2026-07-15T22:07:25.088Z] ++ envvars+=("-e TEST_CORE=${TEST_CORE:-false}")
[2026-07-15T22:07:25.088Z] ++ [[ false == \t\r\u\e ]]
[2026-07-15T22:07:25.088Z] ++ [[ true == \t\r\u\e ]]
[2026-07-15T22:07:25.088Z] ++ envvars+=("-e TEST_CORE=${TEST_CORE}")
[2026-07-15T22:07:25.088Z] ++ docker ps -a
[2026-07-15T22:07:25.089Z] CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
[2026-07-15T22:07:25.089Z] cab8d25a67ee rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942 "bash -lc 'sleep inf…" 5 minutes ago Up 5 minutes pytorch-ci-container
[2026-07-15T22:07:25.089Z] 33af8c9e058f dashboard-dashboard-ui "/docker-entrypoint.…" 3 weeks ago Up 40 minutes 80/tcp, 0.0.0.0:3000->3000/tcp, [::]:3000->3000/tcp rocagents-dashboard-ui
[2026-07-15T22:07:25.089Z] 7c6be83a2db1 dashboard-dashboard-api "uvicorn main:app --…" 3 weeks ago Up 40 minutes (healthy) 0.0.0.0:8080->8080/tcp, [::]:8080->8080/tcp rocagents-dashboard-api
[2026-07-15T22:07:25.089Z] 780dc8fe5a7b rocm/eigen-test:ubuntu20.04-rocm5.3.0-eigen "/scripts/run_tests_…" 6 weeks ago Exited (1) 6 weeks ago infallible_chatelet
[2026-07-15T22:07:25.089Z] 2d7786cfd066 deployment-rocagents "python src/multi_re…" 2 months ago Up 40 minutes (healthy) rocagents-orchestrator
[2026-07-15T22:07:25.089Z] ++ docker rm -f pytorch-ci-container
[2026-07-15T22:07:25.607Z] ++ docker pull rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942
[2026-07-15T22:07:26.122Z] pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942: Pulling from rocm/pytorch-ci-private
[2026-07-15T22:07:26.122Z] Digest: sha256:b836aa0e221ac85a2184ec24a67b9d5089f3caa45d182cf521dee7d38112aa81
[2026-07-15T22:07:26.122Z] Status: Image is up to date for rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942
[2026-07-15T22:07:26.122Z] docker.io/rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942
[2026-07-15T22:07:26.122Z] ++ GPU_FLAGS=
[2026-07-15T22:07:26.122Z] ++ echo 'NODE_LABELS: multi_gpu rocm dlmodels gpu_8 gfx90a ixt-hq-ubb4-31 mlir MI250 multi_gpu rocm dlmodels gpu_8 gfx90a ixt-hq-ubb4-31 mlir MI250'
[2026-07-15T22:07:26.122Z] NODE_LABELS: multi_gpu rocm dlmodels gpu_8 gfx90a ixt-hq-ubb4-31 mlir MI250 multi_gpu rocm dlmodels gpu_8 gfx90a ixt-hq-ubb4-31 mlir MI250
[2026-07-15T22:07:26.122Z] ++ [[ ! multi_gpu rocm dlmodels gpu_8 gfx90a ixt-hq-ubb4-31 mlir MI250 =~ build-only ]]
[2026-07-15T22:07:26.122Z] ++ GPU_FLAGS='--group-add video --device=/dev/kfd --device=/dev/dri'
[2026-07-15T22:07:26.122Z] ++++ dirname prepare_docker_env.sh
[2026-07-15T22:07:26.122Z] +++ cd .
[2026-07-15T22:07:26.122Z] +++ pwd
[2026-07-15T22:07:26.122Z] ++ script_dir=/home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3447
[2026-07-15T22:07:26.122Z] ++ docker run --name pytorch-ci-container -e 'EXTENSION_BUILD_GFXARCH=gfx90a;gfx908;gfx942' -e CI=1 -e TEST_CONFIG=default -e SHARD_NUMBER=2 -e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip -e PREBUILT=false -e TEST_CORE=true -e TEST_CORE=true -v /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3447:/host_workspace -v /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3447:/scripts --entrypoint bash -d --network=host --group-add video --device=/dev/kfd --device=/dev/dri --ipc=host --pid=host --shm-size 8G --cap-add=SYS_PTRACE --security-opt seccomp=unconfined rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942 -lc 'sleep infinity'
[2026-07-15T22:07:26.122Z] 0158d7fc8ecb44b1ba58755106ccbfe9d8b4c3efd18d967727647828fbf21626
[2026-07-15T22:07:26.122Z] ++ env -i python3 /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3447/pytorch/tools/stats/export_test_times.py
[2026-07-15T22:07:27.214Z] Exporting test times from test-infra
[2026-07-15T22:07:27.214Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-times.json to /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3447/pytorch/.additional_ci_files/test-times.json
[2026-07-15T22:07:27.214Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-class-times.json to /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3447/pytorch/.additional_ci_files/test-class-times.json
[2026-07-15T22:07:27.214Z] ++ python3 /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3447/pytorch/tools/stats/export_test_times.py
[2026-07-15T22:07:27.214Z] Exporting test times from test-infra
[2026-07-15T22:07:27.214Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-times.json to /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3447/pytorch/.additional_ci_files/test-times.json
[2026-07-15T22:07:27.214Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-class-times.json to /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3447/pytorch/.additional_ci_files/test-class-times.json
[2026-07-15T22:07:27.214Z] ++ [[ -d /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3447/pytorch/.additional_ci_files ]]
[2026-07-15T22:07:27.214Z] ++ sed -i -E 's/"\S+rocm\S+"/""/g' /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3447/pytorch/.additional_ci_files/test-times.json
[2026-07-15T22:07:27.214Z] ++ [[ false != \t\r\u\e ]]
[2026-07-15T22:07:27.214Z] ++ docker exec -u root pytorch-ci-container bash -c 'rm -rf /var/lib/jenkins/pytorch'
[2026-07-15T22:07:27.214Z] ++ docker cp /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3447/pytorch pytorch-ci-container:/var/lib/jenkins/pytorch
[2026-07-15T22:07:31.424Z] ++ docker exec -u root pytorch-ci-container bash -c 'chown root:root -R /var/lib/jenkins/pytorch'
[2026-07-15T22:07:31.945Z] ++ [[ 1 -eq 1 ]]
[2026-07-15T22:07:31.945Z] ++ docker cp /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3447/rocm_composable_kernel-7.1.1.dev0+g4266f8-py3-none-any.whl /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3447/torch-2.14.0a0+git579ba57-cp312-cp312-linux_x86_64.whl pytorch-ci-container:/var/lib/jenkins/pytorch
[2026-07-15T22:07:31.945Z] docker: 'docker cp' requires 2 arguments
[2026-07-15T22:07:31.945Z]
[2026-07-15T22:07:31.945Z] Usage: docker cp [OPTIONS] CONTAINER:SRC_PATH DEST_PATH|-
[2026-07-15T22:07:31.945Z] docker cp [OPTIONS] SRC_PATH|- CONTAINER:DEST_PATH
[2026-07-15T22:07:31.945Z]
[2026-07-15T22:07:31.945Z] Run 'docker cp --help' for more information
Tests / Test PyTorch / Test PyTorch / Run pytorch_test_2 / Error signal
Error in error step, with arguments pytorch_test_2 failed.
pytorch_test_2 failed
Tests / Test Distributed / Test Distributed / Run pytorch_distributed_1 / Shell Script
Error in sh step, with arguments ./test_pytorch_test_distributed.sh.
script returned exit code 1
Build log
[2026-07-15T21:51:23.234Z] + ./test_pytorch_test_distributed.sh
[2026-07-15T21:51:23.234Z] + [[ -z '' ]]
[2026-07-15T21:51:23.234Z] + DEFAULT_EXECUTION_OPTIONS='-e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip --copy-whls'
[2026-07-15T21:51:23.234Z] + source prepare_docker_env.sh -e 'EXTENSION_BUILD_GFXARCH=gfx90a;gfx908;gfx942' -e CI=1 -e TEST_CONFIG=distributed -e SHARD_NUMBER=1 -e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip --copy-whls
[2026-07-15T21:51:23.234Z] ++ set -ex
[2026-07-15T21:51:23.234Z] ++ set -o pipefail
[2026-07-15T21:51:23.234Z] ++ copy_whls=0
[2026-07-15T21:51:23.234Z] ++ envvars=()
[2026-07-15T21:51:23.234Z] ++ [[ 13 -gt 0 ]]
[2026-07-15T21:51:23.234Z] ++ key=-e
[2026-07-15T21:51:23.234Z] ++ case $key in
[2026-07-15T21:51:23.234Z] ++ envvars+=("-e $2")
[2026-07-15T21:51:23.234Z] ++ shift
[2026-07-15T21:51:23.234Z] ++ shift
[2026-07-15T21:51:23.234Z] ++ [[ 11 -gt 0 ]]
[2026-07-15T21:51:23.234Z] ++ key=-e
[2026-07-15T21:51:23.234Z] ++ case $key in
[2026-07-15T21:51:23.234Z] ++ envvars+=("-e $2")
[2026-07-15T21:51:23.234Z] ++ shift
[2026-07-15T21:51:23.234Z] ++ shift
[2026-07-15T21:51:23.234Z] ++ [[ 9 -gt 0 ]]
[2026-07-15T21:51:23.234Z] ++ key=-e
[2026-07-15T21:51:23.234Z] ++ case $key in
[2026-07-15T21:51:23.234Z] ++ envvars+=("-e $2")
[2026-07-15T21:51:23.234Z] ++ shift
[2026-07-15T21:51:23.234Z] ++ shift
[2026-07-15T21:51:23.234Z] ++ [[ 7 -gt 0 ]]
[2026-07-15T21:51:23.234Z] ++ key=-e
[2026-07-15T21:51:23.234Z] ++ case $key in
[2026-07-15T21:51:23.234Z] ++ envvars+=("-e $2")
[2026-07-15T21:51:23.234Z] ++ shift
[2026-07-15T21:51:23.234Z] ++ shift
[2026-07-15T21:51:23.234Z] ++ [[ 5 -gt 0 ]]
[2026-07-15T21:51:23.234Z] ++ key=-e
[2026-07-15T21:51:23.234Z] ++ case $key in
[2026-07-15T21:51:23.234Z] ++ envvars+=("-e $2")
[2026-07-15T21:51:23.234Z] ++ shift
[2026-07-15T21:51:23.234Z] ++ shift
[2026-07-15T21:51:23.234Z] ++ [[ 3 -gt 0 ]]
[2026-07-15T21:51:23.234Z] ++ key=-e
[2026-07-15T21:51:23.234Z] ++ case $key in
[2026-07-15T21:51:23.234Z] ++ envvars+=("-e $2")
[2026-07-15T21:51:23.234Z] ++ shift
[2026-07-15T21:51:23.234Z] ++ shift
[2026-07-15T21:51:23.234Z] ++ [[ 1 -gt 0 ]]
[2026-07-15T21:51:23.234Z] ++ key=--copy-whls
[2026-07-15T21:51:23.234Z] ++ case $key in
[2026-07-15T21:51:23.234Z] ++ copy_whls=1
[2026-07-15T21:51:23.234Z] ++ shift
[2026-07-15T21:51:23.234Z] ++ [[ 0 -gt 0 ]]
[2026-07-15T21:51:23.234Z] ++ envvars+=("-e PREBUILT=${PREBUILT:-false}")
[2026-07-15T21:51:23.234Z] ++ envvars+=("-e TEST_CORE=${TEST_CORE:-false}")
[2026-07-15T21:51:23.234Z] ++ [[ false == \t\r\u\e ]]
[2026-07-15T21:51:23.234Z] ++ [[ true == \t\r\u\e ]]
[2026-07-15T21:51:23.234Z] ++ envvars+=("-e TEST_CORE=${TEST_CORE}")
[2026-07-15T21:51:23.234Z] ++ docker ps -a
[2026-07-15T21:51:23.234Z] CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
[2026-07-15T21:51:23.234Z] da028e45d685 rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-27d7487d6141807234c885bc3b76f82d055522fd-gfx908_gfx90a_gfx942 "bash -lc 'sleep inf…" About an hour ago Up About an hour pytorch-ci-container
[2026-07-15T21:51:23.234Z] ++ docker rm -f pytorch-ci-container
[2026-07-15T21:51:24.308Z] ++ docker pull rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942
[2026-07-15T21:51:24.830Z] pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942: Pulling from rocm/pytorch-ci-private
[2026-07-15T21:51:24.830Z] Digest: sha256:b836aa0e221ac85a2184ec24a67b9d5089f3caa45d182cf521dee7d38112aa81
[2026-07-15T21:51:24.830Z] Status: Image is up to date for rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942
[2026-07-15T21:51:24.830Z] docker.io/rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942
[2026-07-15T21:51:24.830Z] ++ GPU_FLAGS=
[2026-07-15T21:51:24.830Z] ++ echo 'NODE_LABELS: multi_gpu rocm pytorch-nightly selftest ixt-hq-15 gfx908 multi_gpu rocm pytorch-nightly selftest ixt-hq-15 gfx908'
[2026-07-15T21:51:24.830Z] NODE_LABELS: multi_gpu rocm pytorch-nightly selftest ixt-hq-15 gfx908 multi_gpu rocm pytorch-nightly selftest ixt-hq-15 gfx908
[2026-07-15T21:51:24.830Z] ++ [[ ! multi_gpu rocm pytorch-nightly selftest ixt-hq-15 gfx908 =~ build-only ]]
[2026-07-15T21:51:24.830Z] ++ GPU_FLAGS='--group-add video --device=/dev/kfd --device=/dev/dri'
[2026-07-15T21:51:24.830Z] ++++ dirname prepare_docker_env.sh
[2026-07-15T21:51:24.830Z] +++ cd .
[2026-07-15T21:51:24.830Z] +++ pwd
[2026-07-15T21:51:24.830Z] ++ script_dir=/home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3447
[2026-07-15T21:51:24.830Z] ++ docker run --name pytorch-ci-container -e 'EXTENSION_BUILD_GFXARCH=gfx90a;gfx908;gfx942' -e CI=1 -e TEST_CONFIG=distributed -e SHARD_NUMBER=1 -e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip -e PREBUILT=false -e TEST_CORE=true -e TEST_CORE=true -v /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3447:/host_workspace -v /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3447:/scripts --entrypoint bash -d --network=host --group-add video --device=/dev/kfd --device=/dev/dri --ipc=host --pid=host --shm-size 8G --cap-add=SYS_PTRACE --security-opt seccomp=unconfined rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942 -lc 'sleep infinity'
[2026-07-15T21:51:24.830Z] 52e9c3fc927e31cf2d9dc420c0c330a8442a838f9c74efa1c2147f57d597f0e7
[2026-07-15T21:51:25.356Z] ++ env -i python3 /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3447/pytorch/tools/stats/export_test_times.py
[2026-07-15T21:51:26.428Z] Exporting test times from test-infra
[2026-07-15T21:51:26.428Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-times.json to /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3447/pytorch/.additional_ci_files/test-times.json
[2026-07-15T21:51:26.428Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-class-times.json to /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3447/pytorch/.additional_ci_files/test-class-times.json
[2026-07-15T21:51:26.428Z] ++ python3 /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3447/pytorch/tools/stats/export_test_times.py
[2026-07-15T21:51:26.428Z] Exporting test times from test-infra
[2026-07-15T21:51:26.428Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-times.json to /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3447/pytorch/.additional_ci_files/test-times.json
[2026-07-15T21:51:26.428Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-class-times.json to /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3447/pytorch/.additional_ci_files/test-class-times.json
[2026-07-15T21:51:26.428Z] ++ [[ -d /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3447/pytorch/.additional_ci_files ]]
[2026-07-15T21:51:26.428Z] ++ sed -i -E 's/"\S+rocm\S+"/""/g' /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3447/pytorch/.additional_ci_files/test-times.json
[2026-07-15T21:51:26.428Z] ++ [[ false != \t\r\u\e ]]
[2026-07-15T21:51:26.428Z] ++ docker exec -u root pytorch-ci-container bash -c 'rm -rf /var/lib/jenkins/pytorch'
[2026-07-15T21:51:26.428Z] ++ docker cp /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3447/pytorch pytorch-ci-container:/var/lib/jenkins/pytorch
[2026-07-15T21:51:33.999Z] ++ docker exec -u root pytorch-ci-container bash -c 'chown root:root -R /var/lib/jenkins/pytorch'
[2026-07-15T21:51:33.999Z] ++ [[ 1 -eq 1 ]]
[2026-07-15T21:51:33.999Z] ++ docker cp /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3447/rocm_composable_kernel-7.1.1.dev0+g4266f8-py3-none-any.whl /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3447/torch-2.14.0a0+git579ba57-cp312-cp312-linux_x86_64.whl pytorch-ci-container:/var/lib/jenkins/pytorch
[2026-07-15T21:51:33.999Z] "docker cp" requires exactly 2 arguments.
[2026-07-15T21:51:33.999Z] See 'docker cp --help'.
[2026-07-15T21:51:33.999Z]
[2026-07-15T21:51:33.999Z] Usage: docker cp [OPTIONS] CONTAINER:SRC_PATH DEST_PATH|-
[2026-07-15T21:51:33.999Z] docker cp [OPTIONS] SRC_PATH|- CONTAINER:DEST_PATH
[2026-07-15T21:51:33.999Z]
[2026-07-15T21:51:33.999Z] Copy files/folders between a container and the local filesystem
Tests / Test Distributed / Test Distributed / Run pytorch_distributed_1 / Error signal
Error in error step, with arguments pytorch_distributed_1 failed.
pytorch_distributed_1 failed
Tests / Test Distributed / Test Distributed / Run pytorch_distributed_2 / Shell Script
Error in sh step, with arguments ./test_pytorch_test_distributed.sh.
script returned exit code 1
Build log
[2026-07-15T21:59:13.518Z] + ./test_pytorch_test_distributed.sh
[2026-07-15T21:59:13.518Z] + [[ -z '' ]]
[2026-07-15T21:59:13.518Z] + DEFAULT_EXECUTION_OPTIONS='-e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip --copy-whls'
[2026-07-15T21:59:13.518Z] + source prepare_docker_env.sh -e 'EXTENSION_BUILD_GFXARCH=gfx90a;gfx908;gfx942' -e CI=1 -e TEST_CONFIG=distributed -e SHARD_NUMBER=2 -e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip --copy-whls
[2026-07-15T21:59:13.518Z] ++ set -ex
[2026-07-15T21:59:13.518Z] ++ set -o pipefail
[2026-07-15T21:59:13.518Z] ++ copy_whls=0
[2026-07-15T21:59:13.518Z] ++ envvars=()
[2026-07-15T21:59:13.518Z] ++ [[ 13 -gt 0 ]]
[2026-07-15T21:59:13.518Z] ++ key=-e
[2026-07-15T21:59:13.518Z] ++ case $key in
[2026-07-15T21:59:13.518Z] ++ envvars+=("-e $2")
[2026-07-15T21:59:13.518Z] ++ shift
[2026-07-15T21:59:13.518Z] ++ shift
[2026-07-15T21:59:13.518Z] ++ [[ 11 -gt 0 ]]
[2026-07-15T21:59:13.518Z] ++ key=-e
[2026-07-15T21:59:13.518Z] ++ case $key in
[2026-07-15T21:59:13.518Z] ++ envvars+=("-e $2")
[2026-07-15T21:59:13.518Z] ++ shift
[2026-07-15T21:59:13.518Z] ++ shift
[2026-07-15T21:59:13.518Z] ++ [[ 9 -gt 0 ]]
[2026-07-15T21:59:13.518Z] ++ key=-e
[2026-07-15T21:59:13.518Z] ++ case $key in
[2026-07-15T21:59:13.518Z] ++ envvars+=("-e $2")
[2026-07-15T21:59:13.518Z] ++ shift
[2026-07-15T21:59:13.518Z] ++ shift
[2026-07-15T21:59:13.518Z] ++ [[ 7 -gt 0 ]]
[2026-07-15T21:59:13.518Z] ++ key=-e
[2026-07-15T21:59:13.518Z] ++ case $key in
[2026-07-15T21:59:13.518Z] ++ envvars+=("-e $2")
[2026-07-15T21:59:13.518Z] ++ shift
[2026-07-15T21:59:13.518Z] ++ shift
[2026-07-15T21:59:13.518Z] ++ [[ 5 -gt 0 ]]
[2026-07-15T21:59:13.518Z] ++ key=-e
[2026-07-15T21:59:13.518Z] ++ case $key in
[2026-07-15T21:59:13.518Z] ++ envvars+=("-e $2")
[2026-07-15T21:59:13.518Z] ++ shift
[2026-07-15T21:59:13.518Z] ++ shift
[2026-07-15T21:59:13.518Z] ++ [[ 3 -gt 0 ]]
[2026-07-15T21:59:13.518Z] ++ key=-e
[2026-07-15T21:59:13.518Z] ++ case $key in
[2026-07-15T21:59:13.518Z] ++ envvars+=("-e $2")
[2026-07-15T21:59:13.518Z] ++ shift
[2026-07-15T21:59:13.518Z] ++ shift
[2026-07-15T21:59:13.518Z] ++ [[ 1 -gt 0 ]]
[2026-07-15T21:59:13.518Z] ++ key=--copy-whls
[2026-07-15T21:59:13.518Z] ++ case $key in
[2026-07-15T21:59:13.518Z] ++ copy_whls=1
[2026-07-15T21:59:13.518Z] ++ shift
[2026-07-15T21:59:13.518Z] ++ [[ 0 -gt 0 ]]
[2026-07-15T21:59:13.518Z] ++ envvars+=("-e PREBUILT=${PREBUILT:-false}")
[2026-07-15T21:59:13.518Z] ++ envvars+=("-e TEST_CORE=${TEST_CORE:-false}")
[2026-07-15T21:59:13.518Z] ++ [[ false == \t\r\u\e ]]
[2026-07-15T21:59:13.518Z] ++ [[ true == \t\r\u\e ]]
[2026-07-15T21:59:13.518Z] ++ envvars+=("-e TEST_CORE=${TEST_CORE}")
[2026-07-15T21:59:13.518Z] ++ docker ps -a
[2026-07-15T21:59:13.518Z] CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
[2026-07-15T21:59:13.518Z] 52e9c3fc927e rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942 "bash -lc 'sleep inf…" 7 minutes ago Up 7 minutes pytorch-ci-container
[2026-07-15T21:59:13.518Z] ++ docker rm -f pytorch-ci-container
[2026-07-15T21:59:14.617Z] ++ docker pull rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942
[2026-07-15T21:59:15.155Z] pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942: Pulling from rocm/pytorch-ci-private
[2026-07-15T21:59:15.687Z] Digest: sha256:b836aa0e221ac85a2184ec24a67b9d5089f3caa45d182cf521dee7d38112aa81
[2026-07-15T21:59:15.687Z] Status: Image is up to date for rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942
[2026-07-15T21:59:15.687Z] docker.io/rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942
[2026-07-15T21:59:15.687Z] ++ GPU_FLAGS=
[2026-07-15T21:59:15.687Z] ++ echo 'NODE_LABELS: multi_gpu rocm pytorch-nightly selftest ixt-hq-15 gfx908 multi_gpu rocm pytorch-nightly selftest ixt-hq-15 gfx908'
[2026-07-15T21:59:15.687Z] NODE_LABELS: multi_gpu rocm pytorch-nightly selftest ixt-hq-15 gfx908 multi_gpu rocm pytorch-nightly selftest ixt-hq-15 gfx908
[2026-07-15T21:59:15.687Z] ++ [[ ! multi_gpu rocm pytorch-nightly selftest ixt-hq-15 gfx908 =~ build-only ]]
[2026-07-15T21:59:15.687Z] ++ GPU_FLAGS='--group-add video --device=/dev/kfd --device=/dev/dri'
[2026-07-15T21:59:15.687Z] ++++ dirname prepare_docker_env.sh
[2026-07-15T21:59:15.687Z] +++ cd .
[2026-07-15T21:59:15.687Z] +++ pwd
[2026-07-15T21:59:15.687Z] ++ script_dir=/home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3447
[2026-07-15T21:59:15.687Z] ++ docker run --name pytorch-ci-container -e 'EXTENSION_BUILD_GFXARCH=gfx90a;gfx908;gfx942' -e CI=1 -e TEST_CONFIG=distributed -e SHARD_NUMBER=2 -e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip -e PREBUILT=false -e TEST_CORE=true -e TEST_CORE=true -v /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3447:/host_workspace -v /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3447:/scripts --entrypoint bash -d --network=host --group-add video --device=/dev/kfd --device=/dev/dri --ipc=host --pid=host --shm-size 8G --cap-add=SYS_PTRACE --security-opt seccomp=unconfined rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942 -lc 'sleep infinity'
[2026-07-15T21:59:15.687Z] dd7e7732bbbbe0fc1769d912502ce7bb4b5081dd6cf3ab642f988f775a57eaa7
[2026-07-15T21:59:15.687Z] ++ env -i python3 /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3447/pytorch/tools/stats/export_test_times.py
[2026-07-15T21:59:16.801Z] Exporting test times from test-infra
[2026-07-15T21:59:16.801Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-times.json to /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3447/pytorch/.additional_ci_files/test-times.json
[2026-07-15T21:59:16.801Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-class-times.json to /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3447/pytorch/.additional_ci_files/test-class-times.json
[2026-07-15T21:59:16.801Z] ++ python3 /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3447/pytorch/tools/stats/export_test_times.py
[2026-07-15T21:59:16.801Z] Exporting test times from test-infra
[2026-07-15T21:59:16.801Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-times.json to /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3447/pytorch/.additional_ci_files/test-times.json
[2026-07-15T21:59:16.801Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-class-times.json to /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3447/pytorch/.additional_ci_files/test-class-times.json
[2026-07-15T21:59:16.801Z] ++ [[ -d /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3447/pytorch/.additional_ci_files ]]
[2026-07-15T21:59:16.801Z] ++ sed -i -E 's/"\S+rocm\S+"/""/g' /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3447/pytorch/.additional_ci_files/test-times.json
[2026-07-15T21:59:16.801Z] ++ [[ false != \t\r\u\e ]]
[2026-07-15T21:59:16.801Z] ++ docker exec -u root pytorch-ci-container bash -c 'rm -rf /var/lib/jenkins/pytorch'
[2026-07-15T21:59:16.801Z] ++ docker cp /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3447/pytorch pytorch-ci-container:/var/lib/jenkins/pytorch
[2026-07-15T21:59:24.520Z] ++ docker exec -u root pytorch-ci-container bash -c 'chown root:root -R /var/lib/jenkins/pytorch'
[2026-07-15T21:59:24.520Z] ++ [[ 1 -eq 1 ]]
[2026-07-15T21:59:24.520Z] ++ docker cp /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3447/rocm_composable_kernel-7.1.1.dev0+g4266f8-py3-none-any.whl /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3447/torch-2.14.0a0+git579ba57-cp312-cp312-linux_x86_64.whl pytorch-ci-container:/var/lib/jenkins/pytorch
[2026-07-15T21:59:24.520Z] "docker cp" requires exactly 2 arguments.
[2026-07-15T21:59:24.520Z] See 'docker cp --help'.
[2026-07-15T21:59:24.520Z]
[2026-07-15T21:59:24.520Z] Usage: docker cp [OPTIONS] CONTAINER:SRC_PATH DEST_PATH|-
[2026-07-15T21:59:24.520Z] docker cp [OPTIONS] SRC_PATH|- CONTAINER:DEST_PATH
[2026-07-15T21:59:24.520Z]
[2026-07-15T21:59:24.520Z] Copy files/folders between a container and the local filesystem
Tests / Test Distributed / Test Distributed / Run pytorch_distributed_2 / Error signal
Error in error step, with arguments pytorch_distributed_2 failed.
pytorch_distributed_2 failed
Tests / Test Inductor / Test Inductor / Run pytorch_inductor_1 / Shell Script
Error in sh step, with arguments ./test_pytorch_inductor.sh.
script returned exit code 1
Build log
[2026-07-15T21:52:17.553Z] + ./test_pytorch_inductor.sh
[2026-07-15T21:52:17.553Z] + [[ -z '' ]]
[2026-07-15T21:52:17.553Z] + DEFAULT_EXECUTION_OPTIONS='-e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip --copy-whls'
[2026-07-15T21:52:17.553Z] + source prepare_docker_env.sh -e 'EXTENSION_BUILD_GFXARCH=gfx90a;gfx908;gfx942' -e CI=1 -e TEST_CONFIG=inductor -e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip --copy-whls
[2026-07-15T21:52:17.553Z] ++ set -ex
[2026-07-15T21:52:17.553Z] ++ set -o pipefail
[2026-07-15T21:52:17.553Z] ++ copy_whls=0
[2026-07-15T21:52:17.553Z] ++ envvars=()
[2026-07-15T21:52:17.553Z] ++ [[ 11 -gt 0 ]]
[2026-07-15T21:52:17.553Z] ++ key=-e
[2026-07-15T21:52:17.553Z] ++ case $key in
[2026-07-15T21:52:17.553Z] ++ envvars+=("-e $2")
[2026-07-15T21:52:17.553Z] ++ shift
[2026-07-15T21:52:17.553Z] ++ shift
[2026-07-15T21:52:17.553Z] ++ [[ 9 -gt 0 ]]
[2026-07-15T21:52:17.553Z] ++ key=-e
[2026-07-15T21:52:17.553Z] ++ case $key in
[2026-07-15T21:52:17.553Z] ++ envvars+=("-e $2")
[2026-07-15T21:52:17.553Z] ++ shift
[2026-07-15T21:52:17.553Z] ++ shift
[2026-07-15T21:52:17.553Z] ++ [[ 7 -gt 0 ]]
[2026-07-15T21:52:17.553Z] ++ key=-e
[2026-07-15T21:52:17.553Z] ++ case $key in
[2026-07-15T21:52:17.553Z] ++ envvars+=("-e $2")
[2026-07-15T21:52:17.553Z] ++ shift
[2026-07-15T21:52:17.553Z] ++ shift
[2026-07-15T21:52:17.553Z] ++ [[ 5 -gt 0 ]]
[2026-07-15T21:52:17.553Z] ++ key=-e
[2026-07-15T21:52:17.553Z] ++ case $key in
[2026-07-15T21:52:17.553Z] ++ envvars+=("-e $2")
[2026-07-15T21:52:17.553Z] ++ shift
[2026-07-15T21:52:17.553Z] ++ shift
[2026-07-15T21:52:17.553Z] ++ [[ 3 -gt 0 ]]
[2026-07-15T21:52:17.553Z] ++ key=-e
[2026-07-15T21:52:17.553Z] ++ case $key in
[2026-07-15T21:52:17.553Z] ++ envvars+=("-e $2")
[2026-07-15T21:52:17.553Z] ++ shift
[2026-07-15T21:52:17.553Z] ++ shift
[2026-07-15T21:52:17.553Z] ++ [[ 1 -gt 0 ]]
[2026-07-15T21:52:17.553Z] ++ key=--copy-whls
[2026-07-15T21:52:17.553Z] ++ case $key in
[2026-07-15T21:52:17.553Z] ++ copy_whls=1
[2026-07-15T21:52:17.553Z] ++ shift
[2026-07-15T21:52:17.553Z] ++ [[ 0 -gt 0 ]]
[2026-07-15T21:52:17.553Z] ++ envvars+=("-e PREBUILT=${PREBUILT:-false}")
[2026-07-15T21:52:17.553Z] ++ envvars+=("-e TEST_CORE=${TEST_CORE:-false}")
[2026-07-15T21:52:17.553Z] ++ [[ false == \t\r\u\e ]]
[2026-07-15T21:52:17.553Z] ++ [[ true == \t\r\u\e ]]
[2026-07-15T21:52:17.553Z] ++ envvars+=("-e TEST_CORE=${TEST_CORE}")
[2026-07-15T21:52:17.553Z] ++ docker ps -a
[2026-07-15T21:52:17.553Z] CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
[2026-07-15T21:52:17.553Z] ++ docker rm -f pytorch-ci-container
[2026-07-15T21:52:17.553Z] ++ docker pull rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942
[2026-07-15T21:52:18.868Z] pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942: Pulling from rocm/pytorch-ci-private
[2026-07-15T21:52:25.283Z] 99e6462d03e0: Pulling fs layer
[2026-07-15T21:52:25.283Z] 3d56742d201d: Pulling fs layer
[2026-07-15T21:52:25.283Z] ed874654c321: Pulling fs layer
[2026-07-15T21:52:25.283Z] 78cb638d6686: Pulling fs layer
[2026-07-15T21:52:25.283Z] d9a8b776bea7: Pulling fs layer
[2026-07-15T21:52:25.283Z] 3d56742d201d: Pulling fs layer
[2026-07-15T21:52:25.283Z] f3058198fba5: Pulling fs layer
[2026-07-15T21:52:25.283Z] bf3e4cf00cd6: Pulling fs layer
[2026-07-15T21:52:25.283Z] f391428cfe0f: Pulling fs layer
[2026-07-15T21:52:25.283Z] 8d78d529f04d: Pulling fs layer
[2026-07-15T21:52:25.283Z] eab367f1ca39: Pulling fs layer
[2026-07-15T21:52:25.283Z] cb259a83ac3d: Pulling fs layer
[2026-07-15T21:52:25.283Z] d7df6379f9cc: Pulling fs layer
[2026-07-15T21:52:25.283Z] 906ac4de0a2b: Pulling fs layer
[2026-07-15T21:52:25.283Z] c608e00b1282: Pulling fs layer
[2026-07-15T21:52:25.283Z] 95d704643610: Pulling fs layer
[2026-07-15T21:52:25.283Z] 34e29023481f: Pulling fs layer
[2026-07-15T21:52:25.283Z] 4c8a34755929: Pulling fs layer
[2026-07-15T21:52:25.283Z] 5f8dd10c9c78: Pulling fs layer
[2026-07-15T21:52:25.283Z] 423e26ddb517: Pulling fs layer
[2026-07-15T21:52:25.283Z] 65f281ce2e70: Pulling fs layer
[2026-07-15T21:52:25.283Z] 284e8cbb8a05: Pulling fs layer
[2026-07-15T21:52:25.283Z] 9548298aa5f4: Pulling fs layer
[2026-07-15T21:52:25.283Z] 8d998c79d309: Pulling fs layer
[2026-07-15T21:52:25.283Z] 3d56742d201d: Pulling fs layer
[2026-07-15T21:52:25.283Z] 86cc05da26d1: Pulling fs layer
[2026-07-15T21:52:25.283Z] ed86e5a5d9b1: Pulling fs layer
[2026-07-15T21:52:25.283Z] b09b7d04a8d9: Pulling fs layer
[2026-07-15T21:52:25.283Z] ba1fabf878bd: Pulling fs layer
[2026-07-15T21:52:25.283Z] 4f4fb700ef54: Pulling fs layer
[2026-07-15T21:52:25.283Z] 78a3c5370db4: Pulling fs layer
[2026-07-15T21:52:25.283Z] 4f4fb700ef54: Pulling fs layer
[2026-07-15T21:52:25.283Z] db0af168a1a4: Pulling fs layer
[2026-07-15T21:52:25.283Z] 747272797eac: Pulling fs layer
[2026-07-15T21:52:25.283Z] 4106a853273f: Pulling fs layer
[2026-07-15T21:52:25.283Z] fef65a673dc9: Pulling fs layer
[2026-07-15T21:52:25.283Z] 0e6d9b5f5e8d: Pulling fs layer
[2026-07-15T21:52:25.283Z] 0fcc3b23bd99: Pulling fs layer
[2026-07-15T21:52:25.283Z] 42239a12ed73: Pulling fs layer
[2026-07-15T21:52:25.283Z] c0664a7365cb: Pulling fs layer
[2026-07-15T21:52:25.283Z] f0ef294e4f9b: Pulling fs layer
[2026-07-15T21:52:25.283Z] 1057fba0c7df: Pulling fs layer
[2026-07-15T21:52:25.283Z] 241cbb1c2f26: Pulling fs layer
[2026-07-15T21:52:25.283Z] 3d56742d201d: Pulling fs layer
[2026-07-15T21:52:25.283Z] 3d56742d201d: Pulling fs layer
[2026-07-15T21:52:25.283Z] 4cfd21f31c8b: Pulling fs layer
[2026-07-15T21:52:25.283Z] 5edf52847c35: Pulling fs layer
[2026-07-15T21:52:25.283Z] 27fcad4d96a6: Pulling fs layer
[2026-07-15T21:52:25.283Z] b96eaed0e751: Pulling fs layer
[2026-07-15T21:52:25.283Z] 9ae773ccc4b2: Pulling fs layer
[2026-07-15T21:52:25.283Z] dedc784bccba: Pulling fs layer
[2026-07-15T21:52:25.283Z] f790d498bebb: Pulling fs layer
[2026-07-15T21:52:25.283Z] 8b3d007242f8: Pulling fs layer
[2026-07-15T21:52:25.283Z] f5afebc61536: Pulling fs layer
[2026-07-15T21:52:25.283Z] 0a0ffadd8b4f: Pulling fs layer
[2026-07-15T21:52:25.283Z] cccff27d82fb: Pulling fs layer
[2026-07-15T21:52:25.283Z] 07b45639fc6f: Pulling fs layer
[2026-07-15T21:52:25.283Z] 69145e0460f6: Download complete
[2026-07-15T21:52:25.283Z] 5f8dd10c9c78: Download complete
[2026-07-15T21:52:25.283Z] 27fcad4d96a6: Download complete
[2026-07-15T21:52:25.283Z] 3d56742d201d: Download complete
[2026-07-15T21:52:25.283Z] 78a3c5370db4: Download complete
[2026-07-15T21:52:25.283Z] fef65a673dc9: Download complete
[2026-07-15T21:52:25.283Z] 423e26ddb517: Download complete
[2026-07-15T21:52:25.283Z] 906ac4de0a2b: Download complete
[2026-07-15T21:52:25.283Z] 9548298aa5f4: Download complete
[2026-07-15T21:52:25.283Z] f391428cfe0f: Download complete
[2026-07-15T21:52:25.283Z] bf3e4cf00cd6: Download complete
[2026-07-15T21:52:25.283Z] 8b3d007242f8: Download complete
[2026-07-15T21:52:25.283Z] 8d998c79d309: Download complete
[2026-07-15T21:52:25.283Z] 86cc05da26d1: Download complete
[2026-07-15T21:52:25.283Z] f0ef294e4f9b: Download complete
[2026-07-15T21:52:25.283Z] 9ae773ccc4b2: Download complete
[2026-07-15T21:52:25.283Z] eab367f1ca39: Download complete
[2026-07-15T21:52:25.283Z] 0fcc3b23bd99: Download complete
[2026-07-15T21:52:25.283Z] 4cfd21f31c8b: Download complete
[2026-07-15T21:52:25.283Z] 99e6462d03e0: Download complete
[2026-07-15T21:52:25.283Z] ed874654c321: Download complete
[2026-07-15T21:52:25.283Z] c0664a7365cb: Download complete
[2026-07-15T21:52:25.283Z] 241cbb1c2f26: Download complete
[2026-07-15T21:52:25.283Z] b96eaed0e751: Download complete
[2026-07-15T21:52:25.283Z] 0a0ffadd8b4f: Download complete
[2026-07-15T21:52:25.283Z] 07b45639fc6f: Download complete
[2026-07-15T21:52:25.283Z] f5afebc61536: Download complete
[2026-07-15T21:52:25.283Z] d7df6379f9cc: Download complete
[2026-07-15T21:52:25.283Z] c608e00b1282: Download complete
[2026-07-15T21:52:25.283Z] b09b7d04a8d9: Download complete
[2026-07-15T21:52:25.283Z] db0af168a1a4: Download complete
[2026-07-15T21:52:25.283Z] 747272797eac: Download complete
[2026-07-15T21:52:25.283Z] f790d498bebb: Download complete
[2026-07-15T21:52:25.283Z] 34e29023481f: Download complete
[2026-07-15T21:52:25.283Z] f3058198fba5: Download complete
[2026-07-15T21:52:25.283Z] 8d78d529f04d: Download complete
[2026-07-15T21:52:25.283Z] 284e8cbb8a05: Download complete
[2026-07-15T21:52:25.283Z] ed86e5a5d9b1: Download complete
[2026-07-15T21:52:25.283Z] 42239a12ed73: Download complete
[2026-07-15T21:52:25.283Z] 65f281ce2e70: Download complete
[2026-07-15T21:52:25.283Z] 4f4fb700ef54: Download complete
[2026-07-15T21:52:25.671Z] 0e6d9b5f5e8d: Download complete
[2026-07-15T21:52:25.671Z] 4106a853273f: Download complete
[2026-07-15T21:52:26.055Z] ba1fabf878bd: Download complete
[2026-07-15T21:52:26.055Z] cb259a83ac3d: Download complete
[2026-07-15T21:52:26.055Z] 4c8a34755929: Download complete
[2026-07-15T21:52:26.440Z] 95d704643610: Download complete
[2026-07-15T21:52:26.825Z] 99e6462d03e0: Pull complete
[2026-07-15T21:52:26.825Z] cb259a83ac3d: Pull complete
[2026-07-15T21:52:28.142Z] cccff27d82fb: Download complete
[2026-07-15T21:52:32.314Z] 1057fba0c7df: Download complete
[2026-07-15T21:52:38.717Z] 34e29023481f: Pull complete
[2026-07-15T21:52:38.717Z] 8d78d529f04d: Pull complete
[2026-07-15T21:52:38.717Z] 65f281ce2e70: Pull complete
[2026-07-15T21:52:38.717Z] 0e6d9b5f5e8d: Pull complete
[2026-07-15T21:52:38.717Z] 1057fba0c7df: Pull complete
[2026-07-15T21:52:38.717Z] 906ac4de0a2b: Pull complete
[2026-07-15T21:52:38.717Z] d7df6379f9cc: Pull complete
[2026-07-15T21:52:38.717Z] c608e00b1282: Pull complete
[2026-07-15T21:52:43.911Z] 78cb638d6686: Download complete
[2026-07-15T21:53:12.603Z] 5edf52847c35: Download complete
[2026-07-15T21:54:01.878Z] 9548298aa5f4: Pull complete
[2026-07-15T21:54:01.878Z] 86cc05da26d1: Pull complete
[2026-07-15T21:54:01.878Z] ed874654c321: Pull complete
[2026-07-15T21:54:01.878Z] db0af168a1a4: Pull complete
[2026-07-15T21:54:01.878Z] 747272797eac: Pull complete
[2026-07-15T21:54:01.878Z] 5edf52847c35: Pull complete
[2026-07-15T21:54:01.878Z] 8b3d007242f8: Pull complete
[2026-07-15T21:54:01.878Z] 0a0ffadd8b4f: Pull complete
[2026-07-15T21:54:21.787Z] d9a8b776bea7: Download complete
[2026-07-15T21:55:10.862Z] dedc784bccba: Download complete
[2026-07-15T21:56:50.269Z] dedc784bccba: Pull complete
[2026-07-15T21:56:50.269Z] 27fcad4d96a6: Pull complete
[2026-07-15T21:56:50.269Z] bf3e4cf00cd6: Pull complete
[2026-07-15T21:56:50.269Z] 9ae773ccc4b2: Pull complete
[2026-07-15T21:56:50.269Z] f5afebc61536: Pull complete
[2026-07-15T21:56:50.269Z] f3058198fba5: Pull complete
[2026-07-15T21:56:50.269Z] 78cb638d6686: Pull complete
[2026-07-15T21:56:50.269Z] c0664a7365cb: Pull complete
[2026-07-15T21:56:50.269Z] b96eaed0e751: Pull complete
[2026-07-15T21:56:50.269Z] 07b45639fc6f: Pull complete
[2026-07-15T21:56:50.269Z] f790d498bebb: Pull complete
[2026-07-15T21:56:50.269Z] cccff27d82fb: Pull complete
[2026-07-15T21:56:50.269Z] 423e26ddb517: Pull complete
[2026-07-15T21:56:50.269Z] 4106a853273f: Pull complete
[2026-07-15T21:56:50.269Z] 95d704643610: Pull complete
[2026-07-15T21:56:50.269Z] 5f8dd10c9c78: Pull complete
[2026-07-15T21:56:50.269Z] 3d56742d201d: Pull complete
[2026-07-15T21:56:50.269Z] 4cfd21f31c8b: Pull complete
[2026-07-15T21:56:50.269Z] b09b7d04a8d9: Pull complete
[2026-07-15T21:56:50.269Z] 78a3c5370db4: Pull complete
[2026-07-15T21:56:50.269Z] f0ef294e4f9b: Pull complete
[2026-07-15T21:56:50.269Z] 0fcc3b23bd99: Pull complete
[2026-07-15T21:56:50.269Z] 241cbb1c2f26: Pull complete
[2026-07-15T21:56:50.269Z] 284e8cbb8a05: Pull complete
[2026-07-15T21:56:50.269Z] 42239a12ed73: Pull complete
[2026-07-15T21:57:49.149Z] f391428cfe0f: Pull complete
[2026-07-15T21:57:49.149Z] eab367f1ca39: Pull complete
[2026-07-15T21:57:49.149Z] d9a8b776bea7: Pull complete
[2026-07-15T21:57:49.149Z] fef65a673dc9: Pull complete
Output truncated.
Details
- Kill older PR Builds (1.5 sec)
- Initialize (1 hr 21 min)
- Download CI scripts (36 sec)
- Checkout Pytorch (3 min 14 sec)
- Check base Docker image existence (5.8 sec)
- Pull Docker Image (4 min 57 sec)
- Build PyTorch (1 hr 11 min)
- Tests (19 min)
- Test PyTorch (8 ms)
- Test Distributed (7 ms)
- Test Inductor (7 ms)
- Test PyTorch Slow (7 ms)
- Test PyTorch Slow (7.9 sec)
- Microbenchmark (15 sec)
- Microbenchmark (7.9 sec)
- Post Build (1.7 sec)
- Declarative: Post Actions (5 sec)
Loading