Skip to content

[release/2.13] [ROCm] derive the hipcub compatible cccl version from 'HIPCUB_CCCL_VERSION'#3442

Open
jithunnair-amd wants to merge 1 commit into
ROCm:release/2.13from
jithunnair-amd:2.13-cherry-pick-pr-188072
Open

[release/2.13] [ROCm] derive the hipcub compatible cccl version from 'HIPCUB_CCCL_VERSION'#3442
jithunnair-amd wants to merge 1 commit into
ROCm:release/2.13from
jithunnair-amd:2.13-cherry-pick-pr-188072

[ROCm] derive the hipcub compatible cccl version from 'HIPCUB_CCCL_VE…

757b6cc
Select commit
Loading
Failed to load commit list.
ROCm Repo Management API / Jenkins failed Jul 17, 2026 in 2h 10m 48s

Tests/Test Inductor/Run pytorch_inductor_1: error in 'error' step

Tests / Test PyTorch / Test PyTorch / Run pytorch_test_1 / Shell Script

Error in sh step, with arguments ./test_pytorch_test.sh.

script returned exit code 1
Build log 
[2026-07-17T07:44:33.297Z] + ./test_pytorch_test.sh
[2026-07-17T07:44:33.297Z] + [[ -z '' ]]
[2026-07-17T07:44:33.297Z] + DEFAULT_EXECUTION_OPTIONS='-e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts         -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip         --copy-whls'
[2026-07-17T07:44:33.297Z] + source prepare_docker_env.sh -e 'EXTENSION_BUILD_GFXARCH=gfx90a;gfx908;gfx942' -e CI=1 -e TEST_CONFIG=default -e SHARD_NUMBER=1 -e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip --copy-whls
[2026-07-17T07:44:33.297Z] ++ set -ex
[2026-07-17T07:44:33.297Z] ++ set -o pipefail
[2026-07-17T07:44:33.297Z] ++ copy_whls=0
[2026-07-17T07:44:33.297Z] ++ envvars=()
[2026-07-17T07:44:33.297Z] ++ [[ 13 -gt 0 ]]
[2026-07-17T07:44:33.297Z] ++ key=-e
[2026-07-17T07:44:33.297Z] ++ case $key in
[2026-07-17T07:44:33.297Z] ++ envvars+=("-e $2")
[2026-07-17T07:44:33.297Z] ++ shift
[2026-07-17T07:44:33.297Z] ++ shift
[2026-07-17T07:44:33.297Z] ++ [[ 11 -gt 0 ]]
[2026-07-17T07:44:33.297Z] ++ key=-e
[2026-07-17T07:44:33.297Z] ++ case $key in
[2026-07-17T07:44:33.297Z] ++ envvars+=("-e $2")
[2026-07-17T07:44:33.297Z] ++ shift
[2026-07-17T07:44:33.297Z] ++ shift
[2026-07-17T07:44:33.297Z] ++ [[ 9 -gt 0 ]]
[2026-07-17T07:44:33.297Z] ++ key=-e
[2026-07-17T07:44:33.297Z] ++ case $key in
[2026-07-17T07:44:33.297Z] ++ envvars+=("-e $2")
[2026-07-17T07:44:33.297Z] ++ shift
[2026-07-17T07:44:33.297Z] ++ shift
[2026-07-17T07:44:33.297Z] ++ [[ 7 -gt 0 ]]
[2026-07-17T07:44:33.297Z] ++ key=-e
[2026-07-17T07:44:33.297Z] ++ case $key in
[2026-07-17T07:44:33.297Z] ++ envvars+=("-e $2")
[2026-07-17T07:44:33.297Z] ++ shift
[2026-07-17T07:44:33.297Z] ++ shift
[2026-07-17T07:44:33.297Z] ++ [[ 5 -gt 0 ]]
[2026-07-17T07:44:33.297Z] ++ key=-e
[2026-07-17T07:44:33.297Z] ++ case $key in
[2026-07-17T07:44:33.297Z] ++ envvars+=("-e $2")
[2026-07-17T07:44:33.297Z] ++ shift
[2026-07-17T07:44:33.297Z] ++ shift
[2026-07-17T07:44:33.297Z] ++ [[ 3 -gt 0 ]]
[2026-07-17T07:44:33.297Z] ++ key=-e
[2026-07-17T07:44:33.297Z] ++ case $key in
[2026-07-17T07:44:33.297Z] ++ envvars+=("-e $2")
[2026-07-17T07:44:33.297Z] ++ shift
[2026-07-17T07:44:33.297Z] ++ shift
[2026-07-17T07:44:33.297Z] ++ [[ 1 -gt 0 ]]
[2026-07-17T07:44:33.297Z] ++ key=--copy-whls
[2026-07-17T07:44:33.297Z] ++ case $key in
[2026-07-17T07:44:33.297Z] ++ copy_whls=1
[2026-07-17T07:44:33.297Z] ++ shift
[2026-07-17T07:44:33.297Z] ++ [[ 0 -gt 0 ]]
[2026-07-17T07:44:33.297Z] ++ envvars+=("-e PREBUILT=${PREBUILT:-false}")
[2026-07-17T07:44:33.297Z] ++ envvars+=("-e TEST_CORE=${TEST_CORE:-false}")
[2026-07-17T07:44:33.297Z] ++ [[ false == \t\r\u\e ]]
[2026-07-17T07:44:33.297Z] ++ [[ true == \t\r\u\e ]]
[2026-07-17T07:44:33.297Z] ++ envvars+=("-e TEST_CORE=${TEST_CORE}")
[2026-07-17T07:44:33.297Z] ++ docker ps -a
[2026-07-17T07:44:33.297Z] CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES
[2026-07-17T07:44:33.297Z] ++ docker rm -f pytorch-ci-container
[2026-07-17T07:44:33.297Z] ++ docker pull rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-27d7487d6141807234c885bc3b76f82d055522fd-gfx908_gfx90a_gfx942
[2026-07-17T07:44:50.575Z] pytorch-linux-noble-rocm7.2.4-py3.12-27d7487d6141807234c885bc3b76f82d055522fd-gfx908_gfx90a_gfx942: Pulling from rocm/pytorch-ci-private
[2026-07-17T07:44:54.842Z] ca2678b20700: Pulling fs layer
[2026-07-17T07:44:54.842Z] 2ad1733fbd4f: Pulling fs layer
[2026-07-17T07:44:54.842Z] 4d307752764b: Pulling fs layer
[2026-07-17T07:44:54.842Z] 1f32f350bea4: Pulling fs layer
[2026-07-17T07:44:54.842Z] 906ac4de0a2b: Pulling fs layer
[2026-07-17T07:44:54.842Z] 376b4789221e: Pulling fs layer
[2026-07-17T07:44:54.842Z] 5846e8e82d69: Pulling fs layer
[2026-07-17T07:44:54.842Z] 974f46a80f1e: Pulling fs layer
[2026-07-17T07:44:54.842Z] a661a3e0a52e: Pulling fs layer
[2026-07-17T07:44:54.842Z] 626c31f89101: Pulling fs layer
[2026-07-17T07:44:54.842Z] 9492bd48d097: Pulling fs layer
[2026-07-17T07:44:54.842Z] 4f4fb700ef54: Pulling fs layer
[2026-07-17T07:44:54.842Z] 2b5553ca587e: Pulling fs layer
[2026-07-17T07:44:54.842Z] d49e4624054e: Pulling fs layer
[2026-07-17T07:44:54.842Z] f006a76c61e8: Pulling fs layer
[2026-07-17T07:44:54.842Z] aeea1a04eb86: Pulling fs layer
[2026-07-17T07:44:54.842Z] 42a99e52f938: Pulling fs layer
[2026-07-17T07:44:54.842Z] c0664a7365cb: Pulling fs layer
[2026-07-17T07:44:54.842Z] d9289c72d3d7: Pulling fs layer
[2026-07-17T07:44:54.842Z] aeea1a04eb86: Pulling fs layer
[2026-07-17T07:44:54.842Z] 3b2e63278265: Pulling fs layer
[2026-07-17T07:44:54.842Z] 78a3c5370db4: Pulling fs layer
[2026-07-17T07:44:54.842Z] aeea1a04eb86: Pulling fs layer
[2026-07-17T07:44:54.842Z] f8f114c00167: Pulling fs layer
[2026-07-17T07:44:54.842Z] 2e147acf21bf: Pulling fs layer
[2026-07-17T07:44:54.842Z] 1ed9f8c32de0: Pulling fs layer
[2026-07-17T07:44:54.842Z] db0af168a1a4: Pulling fs layer
[2026-07-17T07:44:54.842Z] 582c5c9e769f: Pulling fs layer
[2026-07-17T07:44:54.842Z] a48fe49c074d: Pulling fs layer
[2026-07-17T07:44:54.842Z] 86cc05da26d1: Pulling fs layer
[2026-07-17T07:44:54.842Z] aeea1a04eb86: Pulling fs layer
[2026-07-17T07:44:54.842Z] 5d4725036ac1: Pulling fs layer
[2026-07-17T07:44:54.842Z] 27fcad4d96a6: Pulling fs layer
[2026-07-17T07:44:54.842Z] 381079e824de: Pulling fs layer
[2026-07-17T07:44:54.842Z] bf3e4cf00cd6: Pulling fs layer
[2026-07-17T07:44:54.842Z] 5ab16f3f3b07: Pulling fs layer
[2026-07-17T07:44:54.842Z] 9d96e6a6c786: Pulling fs layer
[2026-07-17T07:44:54.842Z] ad8ac94589aa: Pulling fs layer
[2026-07-17T07:44:54.842Z] c6cfbbbf995c: Pulling fs layer
[2026-07-17T07:44:54.842Z] 9c4ca0035d95: Pulling fs layer
[2026-07-17T07:44:54.842Z] 93747a912780: Pulling fs layer
[2026-07-17T07:44:54.842Z] f3058198fba5: Pulling fs layer
[2026-07-17T07:44:54.842Z] b96eaed0e751: Pulling fs layer
[2026-07-17T07:44:54.842Z] 9110ef432697: Pulling fs layer
[2026-07-17T07:44:54.842Z] 4f4fb700ef54: Pulling fs layer
[2026-07-17T07:44:54.842Z] 90d28eda335d: Pulling fs layer
[2026-07-17T07:44:54.842Z] 5f8dd10c9c78: Pulling fs layer
[2026-07-17T07:44:54.842Z] c459752d4f0e: Pulling fs layer
[2026-07-17T07:44:54.842Z] aa60dad79264: Pulling fs layer
[2026-07-17T07:44:54.842Z] eeed50dfa35d: Pulling fs layer
[2026-07-17T07:44:54.842Z] 11547fb624d2: Pulling fs layer
[2026-07-17T07:44:54.842Z] fef65a673dc9: Pulling fs layer
[2026-07-17T07:44:54.842Z] 535a46026bba: Pulling fs layer
[2026-07-17T07:44:54.842Z] c9dec199a57f: Pulling fs layer
[2026-07-17T07:44:54.842Z] f391428cfe0f: Pulling fs layer
[2026-07-17T07:44:54.842Z] f96aeaebbb33: Pulling fs layer
[2026-07-17T07:44:54.842Z] aeea1a04eb86: Pulling fs layer
[2026-07-17T07:44:55.261Z] 032a39a51a7b: Download complete
[2026-07-17T07:44:55.679Z] 2ad1733fbd4f: Download complete
[2026-07-17T07:44:55.679Z] 906ac4de0a2b: Download complete
[2026-07-17T07:44:55.679Z] 4f4fb700ef54: Download complete
[2026-07-17T07:44:55.679Z] 2b5553ca587e: Download complete
[2026-07-17T07:44:55.679Z] a661a3e0a52e: Download complete
[2026-07-17T07:44:55.679Z] 1f32f350bea4: Download complete
[2026-07-17T07:44:55.679Z] f006a76c61e8: Download complete
[2026-07-17T07:44:55.679Z] 9c4ca0035d95: Download complete
[2026-07-17T07:44:55.679Z] aeea1a04eb86: Download complete
[2026-07-17T07:44:55.679Z] 626c31f89101: Download complete
[2026-07-17T07:44:55.679Z] 974f46a80f1e: Download complete
[2026-07-17T07:44:55.679Z] c0664a7365cb: Download complete
[2026-07-17T07:44:55.679Z] d49e4624054e: Download complete
[2026-07-17T07:44:55.679Z] f96aeaebbb33: Download complete
[2026-07-17T07:44:55.679Z] 42a99e52f938: Download complete
[2026-07-17T07:44:55.679Z] d9289c72d3d7: Download complete
[2026-07-17T07:44:55.679Z] f8f114c00167: Download complete
[2026-07-17T07:44:55.679Z] 86cc05da26d1: Download complete
[2026-07-17T07:44:55.679Z] c6cfbbbf995c: Download complete
[2026-07-17T07:44:55.679Z] 5f8dd10c9c78: Download complete
[2026-07-17T07:44:55.679Z] 2e147acf21bf: Download complete
[2026-07-17T07:44:55.679Z] 1ed9f8c32de0: Download complete
[2026-07-17T07:44:55.679Z] bf3e4cf00cd6: Download complete
[2026-07-17T07:44:55.679Z] 5846e8e82d69: Download complete
[2026-07-17T07:44:55.679Z] f3058198fba5: Download complete
[2026-07-17T07:44:55.679Z] 535a46026bba: Download complete
[2026-07-17T07:44:55.679Z] 5d4725036ac1: Download complete
[2026-07-17T07:44:55.679Z] 9d96e6a6c786: Download complete
[2026-07-17T07:44:55.679Z] 78a3c5370db4: Download complete
[2026-07-17T07:44:55.679Z] db0af168a1a4: Download complete
[2026-07-17T07:44:55.679Z] 90d28eda335d: Download complete
[2026-07-17T07:44:55.679Z] 9110ef432697: Download complete
[2026-07-17T07:44:55.679Z] 381079e824de: Download complete
[2026-07-17T07:44:55.679Z] ad8ac94589aa: Download complete
[2026-07-17T07:44:55.679Z] 9492bd48d097: Download complete
[2026-07-17T07:44:55.679Z] 27fcad4d96a6: Download complete
[2026-07-17T07:44:55.679Z] 11547fb624d2: Download complete
[2026-07-17T07:44:56.066Z] f391428cfe0f: Download complete
[2026-07-17T07:44:56.066Z] a48fe49c074d: Download complete
[2026-07-17T07:44:56.066Z] aa60dad79264: Download complete
[2026-07-17T07:44:56.066Z] b96eaed0e751: Download complete
[2026-07-17T07:44:56.066Z] fef65a673dc9: Download complete
[2026-07-17T07:44:56.066Z] ca2678b20700: Download complete
[2026-07-17T07:44:56.457Z] c9dec199a57f: Download complete
[2026-07-17T07:44:56.457Z] 582c5c9e769f: Download complete
[2026-07-17T07:44:56.840Z] 3b2e63278265: Download complete
[2026-07-17T07:44:56.840Z] 1f32f350bea4: Pull complete
[2026-07-17T07:44:57.228Z] ca2678b20700: Pull complete
[2026-07-17T07:44:59.159Z] c459752d4f0e: Download complete
[2026-07-17T07:45:02.533Z] 4d307752764b: Download complete
[2026-07-17T07:45:10.571Z] 2ad1733fbd4f: Pull complete
[2026-07-17T07:45:10.571Z] 906ac4de0a2b: Pull complete
[2026-07-17T07:45:10.571Z] 974f46a80f1e: Pull complete
[2026-07-17T07:45:10.571Z] c6cfbbbf995c: Pull complete
[2026-07-17T07:45:10.571Z] 5846e8e82d69: Pull complete
[2026-07-17T07:45:10.571Z] 4d307752764b: Pull complete
[2026-07-17T07:45:10.571Z] f006a76c61e8: Pull complete
[2026-07-17T07:45:10.571Z] d9289c72d3d7: Pull complete
[2026-07-17T07:45:13.187Z] 376b4789221e: Download complete
[2026-07-17T07:45:42.203Z] 5ab16f3f3b07: Download complete
[2026-07-17T07:46:23.571Z] f8f114c00167: Pull complete
[2026-07-17T07:46:23.571Z] 86cc05da26d1: Pull complete
[2026-07-17T07:46:23.571Z] 2e147acf21bf: Pull complete
[2026-07-17T07:46:23.571Z] db0af168a1a4: Pull complete
[2026-07-17T07:46:23.571Z] 9110ef432697: Pull complete
[2026-07-17T07:46:23.571Z] 381079e824de: Pull complete
[2026-07-17T07:46:23.571Z] a48fe49c074d: Pull complete
[2026-07-17T07:46:23.571Z] 5ab16f3f3b07: Pull complete
[2026-07-17T07:46:57.965Z] 93747a912780: Download complete
[2026-07-17T07:47:47.154Z] eeed50dfa35d: Download complete
[2026-07-17T07:49:26.507Z] 27fcad4d96a6: Pull complete
[2026-07-17T07:49:26.507Z] eeed50dfa35d: Pull complete
[2026-07-17T07:49:26.507Z] 1ed9f8c32de0: Pull complete
[2026-07-17T07:49:26.507Z] bf3e4cf00cd6: Pull complete
[2026-07-17T07:49:26.507Z] 376b4789221e: Pull complete
[2026-07-17T07:49:26.507Z] f3058198fba5: Pull complete
[2026-07-17T07:49:26.507Z] 535a46026bba: Pull complete
[2026-07-17T07:49:26.886Z] 2b5553ca587e: Pull complete
[2026-07-17T07:49:26.886Z] 626c31f89101: Pull complete
[2026-07-17T07:49:26.886Z] c0664a7365cb: Pull complete
[2026-07-17T07:49:26.886Z] b96eaed0e751: Pull complete
[2026-07-17T07:49:26.886Z] c459752d4f0e: Pull complete
[2026-07-17T07:49:27.266Z] 4f4fb700ef54: Pull complete
[2026-07-17T07:49:27.266Z] a661a3e0a52e: Pull complete
[2026-07-17T07:49:27.266Z] aeea1a04eb86: Pull complete
[2026-07-17T07:49:27.266Z] d49e4624054e: Pull complete
[2026-07-17T07:49:27.266Z] 42a99e52f938: Pull complete
[2026-07-17T07:49:27.266Z] 5f8dd10c9c78: Pull complete
[2026-07-17T07:49:27.266Z] 5d4725036ac1: Pull complete
[2026-07-17T07:49:27.266Z] 78a3c5370db4: Pull complete
[2026-07-17T07:49:27.266Z] 90d28eda335d: Pull complete
[2026-07-17T07:49:27.266Z] 9492bd48d097: Pull complete
[2026-07-17T07:49:27.266Z] aa60dad79264: Pull complete
[2026-07-17T07:49:27.266Z] 3b2e63278265: Pull complete
[2026-07-17T07:49:27.647Z] 9d96e6a6c786: Pull complete
[2026-07-17T07:49:27.647Z] ad8ac94589aa: Pull complete
[2026-07-17T07:50:38.183Z] 9c4ca0035d95: Pull complete
[2026-07-17T07:50:38.183Z] f391428cfe0f: Pull complete
[2026-07-17T07:50:38.183Z] 93747a912780: Pull complete
[2026-07-17T07:50:38.183Z] f96aeaebbb33: Pull complete
[2026-07-17T07:50:38.183Z] 11547fb624d2: Pull complete
[2026-07-17T07:50:38.183Z] fef65a673dc9: Pull complete
[2026-07-17T07:50:38.183Z] c9dec199a57f: Pull complete
[2026-07-17T07:50:38.183Z] 582c5c9e769f: Pull complete
[2026-07-17T07:50:38.183Z] Digest: sha256:667c3601e4a4a05ec0db1d4d16d410447a47e0b598e0b2d24623546afcd677fb
[2026-07-17T07:50:38.183Z] Status: Downloaded newer image for rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-27d7487d6141807234c885bc3b76f82d055522fd-gfx908_gfx90a_gfx942
[2026-07-17T07:50:38.183Z] docker.io/rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-27d7487d6141807234c885bc3b76f82d055522fd-gfx908_gfx90a_gfx942
[2026-07-17T07:50:38.183Z] ++ GPU_FLAGS=
[2026-07-17T07:50:38.183Z] ++ echo 'NODE_LABELS: k8s-mi250-1-t0g51 gfx90a mlir k8s-mi250-1-t0g51 gfx90a mlir'
[2026-07-17T07:50:38.183Z] NODE_LABELS: k8s-mi250-1-t0g51 gfx90a mlir k8s-mi250-1-t0g51 gfx90a mlir
[2026-07-17T07:50:38.183Z] ++ [[ ! k8s-mi250-1-t0g51 gfx90a mlir =~ build-only ]]
[2026-07-17T07:50:38.183Z] ++ GPU_FLAGS='--group-add video --device=/dev/kfd --device=/dev/dri'
[2026-07-17T07:50:38.183Z] ++++ dirname prepare_docker_env.sh
[2026-07-17T07:50:38.183Z] +++ cd .
[2026-07-17T07:50:38.183Z] +++ pwd
[2026-07-17T07:50:38.183Z] ++ script_dir=/home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3442
[2026-07-17T07:50:38.183Z] ++ docker run --name pytorch-ci-container -e 'EXTENSION_BUILD_GFXARCH=gfx90a;gfx908;gfx942' -e CI=1 -e TEST_CONFIG=default -e SHARD_NUMBER=1 -e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip -e PREBUILT=false -e TEST_CORE=true -e TEST_CORE=true -v /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3442:/host_workspace -v /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3442:/scripts --entrypoint bash -d --network=host --group-add video --device=/dev/kfd --device=/dev/dri --ipc=host --pid=host --shm-size 8G --cap-add=SYS_PTRACE --security-opt seccomp=unconfined rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-27d7487d6141807234c885bc3b76f82d055522fd-gfx908_gfx90a_gfx942 -lc 'sleep infinity'
[2026-07-17T07:50:38.183Z] 8cecd49c405144d95e377cbeea71fe142b085ee582099f87f99fb253831a465b
[2026-07-17T07:50:38.183Z] ++ env -i python3 /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3442/pytorch/tools/stats/export_test_times.py
[2026-07-17T07:50:39.140Z] Exporting test times from test-infra
[2026-07-17T07:50:39.140Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-times.json to /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3442/pytorch/.additional_ci_files/test-times.json
[2026-07-17T07:50:39.140Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-class-times.json to /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3442/pytorch/.additional_ci_files/test-class-times.json
[2026-07-17T07:50:39.140Z] ++ python3 /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3442/pytorch/tools/stats/export_test_times.py
[2026-07-17T07:50:39.140Z] Exporting test times from test-infra
[2026-07-17T07:50:39.140Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-times.json to /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3442/pytorch/.additional_ci_files/test-times.json
[2026-07-17T07:50:39.140Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-class-times.json to /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3442/pytorch/.additional_ci_files/test-class-times.json
[2026-07-17T07:50:39.140Z] ++ [[ -d /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3442/pytorch/.additional_ci_files ]]
[2026-07-17T07:50:39.140Z] ++ sed -i -E 's/"\S+rocm\S+"/""/g' /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3442/pytorch/.additional_ci_files/test-times.json
[2026-07-17T07:50:39.140Z] ++ [[ false != \t\r\u\e ]]
[2026-07-17T07:50:39.140Z] ++ docker exec -u root pytorch-ci-container bash -c 'rm -rf /var/lib/jenkins/pytorch'
[2026-07-17T07:50:39.140Z] ++ docker cp /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3442/pytorch pytorch-ci-container:/var/lib/jenkins/pytorch
[2026-07-17T07:50:43.714Z] ++ docker exec -u root pytorch-ci-container bash -c 'chown root:root -R /var/lib/jenkins/pytorch'
[2026-07-17T07:50:43.714Z] ++ [[ 1 -eq 1 ]]
[2026-07-17T07:50:43.714Z] ++ docker cp /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3442/rocm_composable_kernel-7.1.1.dev0+g4266f8-py3-none-any.whl /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3442/torch-2.13.0a0+git757b6cc-cp312-cp312-linux_x86_64.whl pytorch-ci-container:/var/lib/jenkins/pytorch
[2026-07-17T07:50:43.714Z] docker: 'docker cp' requires 2 arguments
[2026-07-17T07:50:43.714Z] 
[2026-07-17T07:50:43.714Z] Usage:  docker cp [OPTIONS] CONTAINER:SRC_PATH DEST_PATH|-
[2026-07-17T07:50:43.714Z] 	docker cp [OPTIONS] SRC_PATH|- CONTAINER:DEST_PATH
[2026-07-17T07:50:43.714Z] 
[2026-07-17T07:50:43.714Z] Run 'docker cp --help' for more information

Tests / Test PyTorch / Test PyTorch / Run pytorch_test_1 / Error signal

Error in error step, with arguments pytorch_test_1 failed.

pytorch_test_1 failed

Tests / Test PyTorch / Test PyTorch / Run pytorch_test_2 / Shell Script

Error in sh step, with arguments ./test_pytorch_test.sh.

script returned exit code 1
Build log 
[2026-07-17T08:00:37.153Z] + ./test_pytorch_test.sh
[2026-07-17T08:00:37.153Z] + [[ -z '' ]]
[2026-07-17T08:00:37.153Z] + DEFAULT_EXECUTION_OPTIONS='-e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts         -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip         --copy-whls'
[2026-07-17T08:00:37.153Z] + source prepare_docker_env.sh -e 'EXTENSION_BUILD_GFXARCH=gfx90a;gfx908;gfx942' -e CI=1 -e TEST_CONFIG=default -e SHARD_NUMBER=2 -e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip --copy-whls
[2026-07-17T08:00:37.153Z] ++ set -ex
[2026-07-17T08:00:37.153Z] ++ set -o pipefail
[2026-07-17T08:00:37.153Z] ++ copy_whls=0
[2026-07-17T08:00:37.153Z] ++ envvars=()
[2026-07-17T08:00:37.153Z] ++ [[ 13 -gt 0 ]]
[2026-07-17T08:00:37.153Z] ++ key=-e
[2026-07-17T08:00:37.153Z] ++ case $key in
[2026-07-17T08:00:37.153Z] ++ envvars+=("-e $2")
[2026-07-17T08:00:37.153Z] ++ shift
[2026-07-17T08:00:37.153Z] ++ shift
[2026-07-17T08:00:37.153Z] ++ [[ 11 -gt 0 ]]
[2026-07-17T08:00:37.153Z] ++ key=-e
[2026-07-17T08:00:37.153Z] ++ case $key in
[2026-07-17T08:00:37.153Z] ++ envvars+=("-e $2")
[2026-07-17T08:00:37.153Z] ++ shift
[2026-07-17T08:00:37.153Z] ++ shift
[2026-07-17T08:00:37.153Z] ++ [[ 9 -gt 0 ]]
[2026-07-17T08:00:37.153Z] ++ key=-e
[2026-07-17T08:00:37.153Z] ++ case $key in
[2026-07-17T08:00:37.153Z] ++ envvars+=("-e $2")
[2026-07-17T08:00:37.153Z] ++ shift
[2026-07-17T08:00:37.153Z] ++ shift
[2026-07-17T08:00:37.153Z] ++ [[ 7 -gt 0 ]]
[2026-07-17T08:00:37.153Z] ++ key=-e
[2026-07-17T08:00:37.153Z] ++ case $key in
[2026-07-17T08:00:37.153Z] ++ envvars+=("-e $2")
[2026-07-17T08:00:37.153Z] ++ shift
[2026-07-17T08:00:37.153Z] ++ shift
[2026-07-17T08:00:37.153Z] ++ [[ 5 -gt 0 ]]
[2026-07-17T08:00:37.153Z] ++ key=-e
[2026-07-17T08:00:37.153Z] ++ case $key in
[2026-07-17T08:00:37.153Z] ++ envvars+=("-e $2")
[2026-07-17T08:00:37.153Z] ++ shift
[2026-07-17T08:00:37.153Z] ++ shift
[2026-07-17T08:00:37.153Z] ++ [[ 3 -gt 0 ]]
[2026-07-17T08:00:37.153Z] ++ key=-e
[2026-07-17T08:00:37.153Z] ++ case $key in
[2026-07-17T08:00:37.153Z] ++ envvars+=("-e $2")
[2026-07-17T08:00:37.153Z] ++ shift
[2026-07-17T08:00:37.153Z] ++ shift
[2026-07-17T08:00:37.153Z] ++ [[ 1 -gt 0 ]]
[2026-07-17T08:00:37.153Z] ++ key=--copy-whls
[2026-07-17T08:00:37.153Z] ++ case $key in
[2026-07-17T08:00:37.153Z] ++ copy_whls=1
[2026-07-17T08:00:37.153Z] ++ shift
[2026-07-17T08:00:37.153Z] ++ [[ 0 -gt 0 ]]
[2026-07-17T08:00:37.153Z] ++ envvars+=("-e PREBUILT=${PREBUILT:-false}")
[2026-07-17T08:00:37.153Z] ++ envvars+=("-e TEST_CORE=${TEST_CORE:-false}")
[2026-07-17T08:00:37.153Z] ++ [[ false == \t\r\u\e ]]
[2026-07-17T08:00:37.153Z] ++ [[ true == \t\r\u\e ]]
[2026-07-17T08:00:37.153Z] ++ envvars+=("-e TEST_CORE=${TEST_CORE}")
[2026-07-17T08:00:37.153Z] ++ docker ps -a
[2026-07-17T08:00:37.153Z] CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES
[2026-07-17T08:00:37.153Z] ++ docker rm -f pytorch-ci-container
[2026-07-17T08:00:37.153Z] ++ docker pull rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-27d7487d6141807234c885bc3b76f82d055522fd-gfx908_gfx90a_gfx942
[2026-07-17T08:00:38.149Z] pytorch-linux-noble-rocm7.2.4-py3.12-27d7487d6141807234c885bc3b76f82d055522fd-gfx908_gfx90a_gfx942: Pulling from rocm/pytorch-ci-private
[2026-07-17T08:00:39.157Z] 4d307752764b: Pulling fs layer
[2026-07-17T08:00:39.157Z] 1f32f350bea4: Pulling fs layer
[2026-07-17T08:00:39.157Z] 974f46a80f1e: Pulling fs layer
[2026-07-17T08:00:39.157Z] c6cfbbbf995c: Pulling fs layer
[2026-07-17T08:00:39.157Z] ca2678b20700: Pulling fs layer
[2026-07-17T08:00:39.157Z] 582c5c9e769f: Pulling fs layer
[2026-07-17T08:00:39.157Z] 5846e8e82d69: Pulling fs layer
[2026-07-17T08:00:39.157Z] b96eaed0e751: Pulling fs layer
[2026-07-17T08:00:39.157Z] 2b5553ca587e: Pulling fs layer
[2026-07-17T08:00:39.157Z] 1ed9f8c32de0: Pulling fs layer
[2026-07-17T08:00:39.157Z] 3b2e63278265: Pulling fs layer
[2026-07-17T08:00:39.157Z] 376b4789221e: Pulling fs layer
[2026-07-17T08:00:39.157Z] 2ad1733fbd4f: Pulling fs layer
[2026-07-17T08:00:39.157Z] f8f114c00167: Pulling fs layer
[2026-07-17T08:00:39.157Z] aeea1a04eb86: Pulling fs layer
[2026-07-17T08:00:39.157Z] a661a3e0a52e: Pulling fs layer
[2026-07-17T08:00:39.157Z] 381079e824de: Pulling fs layer
[2026-07-17T08:00:39.157Z] f3058198fba5: Pulling fs layer
[2026-07-17T08:00:39.157Z] 9d96e6a6c786: Pulling fs layer
[2026-07-17T08:00:39.157Z] 27fcad4d96a6: Pulling fs layer
[2026-07-17T08:00:39.157Z] 5ab16f3f3b07: Pulling fs layer
[2026-07-17T08:00:39.157Z] 626c31f89101: Pulling fs layer
[2026-07-17T08:00:39.157Z] 9492bd48d097: Pulling fs layer
[2026-07-17T08:00:39.157Z] 90d28eda335d: Pulling fs layer
[2026-07-17T08:00:39.157Z] 93747a912780: Pulling fs layer
[2026-07-17T08:00:39.157Z] 42a99e52f938: Pulling fs layer
[2026-07-17T08:00:39.157Z] 535a46026bba: Pulling fs layer
[2026-07-17T08:00:39.157Z] aeea1a04eb86: Pulling fs layer
[2026-07-17T08:00:39.157Z] bf3e4cf00cd6: Pulling fs layer
[2026-07-17T08:00:39.157Z] 4f4fb700ef54: Pulling fs layer
[2026-07-17T08:00:39.157Z] aeea1a04eb86: Pulling fs layer
[2026-07-17T08:00:39.157Z] 9c4ca0035d95: Pulling fs layer
[2026-07-17T08:00:39.157Z] 78a3c5370db4: Pulling fs layer
[2026-07-17T08:00:39.157Z] c0664a7365cb: Pulling fs layer
[2026-07-17T08:00:39.157Z] 2e147acf21bf: Pulling fs layer
[2026-07-17T08:00:39.157Z] 5d4725036ac1: Pulling fs layer
[2026-07-17T08:00:39.157Z] f006a76c61e8: Pulling fs layer
[2026-07-17T08:00:39.157Z] eeed50dfa35d: Pulling fs layer
[2026-07-17T08:00:39.157Z] 5f8dd10c9c78: Pulling fs layer
[2026-07-17T08:00:39.157Z] aa60dad79264: Pulling fs layer
[2026-07-17T08:00:39.157Z] 9110ef432697: Pulling fs layer
[2026-07-17T08:00:39.157Z] ad8ac94589aa: Pulling fs layer
[2026-07-17T08:00:39.157Z] a48fe49c074d: Pulling fs layer
[2026-07-17T08:00:39.157Z] 906ac4de0a2b: Pulling fs layer
[2026-07-17T08:00:39.157Z] aeea1a04eb86: Pulling fs layer
[2026-07-17T08:00:39.157Z] f391428cfe0f: Pulling fs layer
[2026-07-17T08:00:39.157Z] d9289c72d3d7: Pulling fs layer
[2026-07-17T08:00:39.157Z] fef65a673dc9: Pulling fs layer
[2026-07-17T08:00:39.157Z] 4f4fb700ef54: Pulling fs layer
[2026-07-17T08:00:39.157Z] c459752d4f0e: Pulling fs layer
[2026-07-17T08:00:39.157Z] d49e4624054e: Pulling fs layer
[2026-07-17T08:00:39.157Z] aeea1a04eb86: Pulling fs layer
[2026-07-17T08:00:39.157Z] f96aeaebbb33: Pulling fs layer
[2026-07-17T08:00:39.157Z] db0af168a1a4: Pulling fs layer
[2026-07-17T08:00:39.157Z] c9dec199a57f: Pulling fs layer
[2026-07-17T08:00:39.157Z] 11547fb624d2: Pulling fs layer
[2026-07-17T08:00:39.157Z] 86cc05da26d1: Pulling fs layer
[2026-07-17T08:00:39.157Z] 032a39a51a7b: Download complete
[2026-07-17T08:00:39.157Z] 1f32f350bea4: Download complete
[2026-07-17T08:00:39.157Z] 27fcad4d96a6: Download complete
[2026-07-17T08:00:39.157Z] 974f46a80f1e: Download complete
[2026-07-17T08:00:39.157Z] 90d28eda335d: Download complete
[2026-07-17T08:00:39.157Z] c6cfbbbf995c: Download complete
[2026-07-17T08:00:39.626Z] b96eaed0e751: Download complete
[2026-07-17T08:00:39.626Z] 1ed9f8c32de0: Download complete
[2026-07-17T08:00:39.626Z] f3058198fba5: Download complete
[2026-07-17T08:00:39.626Z] 5f8dd10c9c78: Download complete
[2026-07-17T08:00:39.626Z] a661a3e0a52e: Download complete
[2026-07-17T08:00:39.626Z] a48fe49c074d: Download complete
[2026-07-17T08:00:39.626Z] 2b5553ca587e: Download complete
[2026-07-17T08:00:39.626Z] d49e4624054e: Download complete
[2026-07-17T08:00:39.626Z] 2e147acf21bf: Download complete
[2026-07-17T08:00:39.626Z] 2ad1733fbd4f: Download complete
[2026-07-17T08:00:39.626Z] 5d4725036ac1: Download complete
[2026-07-17T08:00:39.626Z] 906ac4de0a2b: Download complete
[2026-07-17T08:00:39.626Z] 535a46026bba: Download complete
[2026-07-17T08:00:39.626Z] 11547fb624d2: Download complete
[2026-07-17T08:00:39.626Z] 9c4ca0035d95: Download complete
[2026-07-17T08:00:39.626Z] ad8ac94589aa: Download complete
[2026-07-17T08:00:39.626Z] 5846e8e82d69: Download complete
[2026-07-17T08:00:39.626Z] 9d96e6a6c786: Download complete
[2026-07-17T08:00:39.626Z] 626c31f89101: Download complete
[2026-07-17T08:00:39.626Z] bf3e4cf00cd6: Download complete
[2026-07-17T08:00:39.626Z] fef65a673dc9: Download complete
[2026-07-17T08:00:39.626Z] f96aeaebbb33: Download complete
[2026-07-17T08:00:39.626Z] aeea1a04eb86: Download complete
[2026-07-17T08:00:39.626Z] c0664a7365cb: Download complete
[2026-07-17T08:00:39.626Z] 9110ef432697: Download complete
[2026-07-17T08:00:39.626Z] f006a76c61e8: Download complete
[2026-07-17T08:00:39.626Z] 42a99e52f938: Download complete
[2026-07-17T08:00:39.626Z] db0af168a1a4: Download complete
[2026-07-17T08:00:39.626Z] 381079e824de: Download complete
[2026-07-17T08:00:39.626Z] d9289c72d3d7: Download complete
[2026-07-17T08:00:39.626Z] 4f4fb700ef54: Download complete
[2026-07-17T08:00:39.626Z] f8f114c00167: Download complete
[2026-07-17T08:00:39.626Z] 86cc05da26d1: Download complete
[2026-07-17T08:00:39.626Z] f391428cfe0f: Download complete
[2026-07-17T08:00:39.626Z] 78a3c5370db4: Download complete
[2026-07-17T08:00:39.626Z] aa60dad79264: Download complete
[2026-07-17T08:00:40.095Z] 9492bd48d097: Download complete
[2026-07-17T08:00:40.095Z] 582c5c9e769f: Download complete
[2026-07-17T08:00:40.095Z] ca2678b20700: Download complete
[2026-07-17T08:00:40.095Z] c9dec199a57f: Download complete
[2026-07-17T08:00:40.570Z] 3b2e63278265: Download complete
[2026-07-17T08:00:41.051Z] ca2678b20700: Pull complete
[2026-07-17T08:00:41.051Z] 1f32f350bea4: Pull complete
[2026-07-17T08:00:42.557Z] c459752d4f0e: Download complete
[2026-07-17T08:00:46.307Z] 4d307752764b: Download complete
[2026-07-17T08:00:53.088Z] 4d307752764b: Pull complete
[2026-07-17T08:00:53.088Z] 974f46a80f1e: Pull complete
[2026-07-17T08:00:53.088Z] c6cfbbbf995c: Pull complete
[2026-07-17T08:00:53.088Z] 2ad1733fbd4f: Pull complete
[2026-07-17T08:00:53.088Z] 906ac4de0a2b: Pull complete
[2026-07-17T08:00:53.088Z] 5846e8e82d69: Pull complete
[2026-07-17T08:00:53.088Z] f006a76c61e8: Pull complete
[2026-07-17T08:00:53.088Z] d9289c72d3d7: Pull complete
[2026-07-17T08:00:57.617Z] 376b4789221e: Download complete
[2026-07-17T08:01:22.724Z] 5ab16f3f3b07: Download complete
[2026-07-17T08:02:04.729Z] 5ab16f3f3b07: Pull complete
[2026-07-17T08:02:04.729Z] a48fe49c074d: Pull complete
[2026-07-17T08:02:04.729Z] 2e147acf21bf: Pull complete
[2026-07-17T08:02:04.729Z] aeea1a04eb86: Pull complete
[2026-07-17T08:02:04.729Z] 9110ef432697: Pull complete
[2026-07-17T08:02:04.729Z] db0af168a1a4: Pull complete
[2026-07-17T08:02:04.729Z] 381079e824de: Pull complete
[2026-07-17T08:02:04.729Z] f8f114c00167: Pull complete
[2026-07-17T08:02:04.729Z] 86cc05da26d1: Pull complete
[2026-07-17T08:02:39.643Z] 93747a912780: Download complete
[2026-07-17T08:03:21.063Z] eeed50dfa35d: Download complete
[2026-07-17T08:05:00.679Z] eeed50dfa35d: Pull complete
[2026-07-17T08:05:00.679Z] 27fcad4d96a6: Pull complete
[2026-07-17T08:05:00.679Z] 1ed9f8c32de0: Pull complete
[2026-07-17T08:05:00.679Z] bf3e4cf00cd6: Pull complete
[2026-07-17T08:05:01.495Z] f3058198fba5: Pull complete
[2026-07-17T08:05:01.495Z] 535a46026bba: Pull complete
[2026-07-17T08:05:01.495Z] 376b4789221e: Pull complete
[2026-07-17T08:05:03.657Z] b96eaed0e751: Pull complete
[2026-07-17T08:05:03.657Z] 2b5553ca587e: Pull complete
[2026-07-17T08:05:03.657Z] 626c31f89101: Pull complete
[2026-07-17T08:05:03.657Z] c0664a7365cb: Pull complete
[2026-07-17T08:05:03.657Z] c459752d4f0e: Pull complete
[2026-07-17T08:05:03.657Z] 3b2e63278265: Pull complete
[2026-07-17T08:05:04.061Z] 90d28eda335d: Pull complete
[2026-07-17T08:05:04.061Z] 5f8dd10c9c78: Pull complete
[2026-07-17T08:05:04.061Z] a661a3e0a52e: Pull complete
[2026-07-17T08:05:04.061Z] d49e4624054e: Pull complete
[2026-07-17T08:05:04.061Z] 5d4725036ac1: Pull complete
[2026-07-17T08:05:04.061Z] ad8ac94589aa: Pull complete
[2026-07-17T08:05:04.061Z] 9d96e6a6c786: Pull complete
[2026-07-17T08:05:04.061Z] 42a99e52f938: Pull complete
[2026-07-17T08:05:04.061Z] 78a3c5370db4: Pull complete
[2026-07-17T08:05:04.061Z] aa60dad79264: Pull complete
[2026-07-17T08:05:04.061Z] 9492bd48d097: Pull complete
[2026-07-17T08:06:14.880Z] 9c4ca0035d95: Pull complete
[2026-07-17T08:06:14.880Z] f391428cfe0f: Pull complete
[2026-07-17T08:06:14.880Z] 93747a912780: Pull complete
[2026-07-17T08:06:14.880Z] c9dec199a57f: Pull complete
[2026-07-17T08:06:14.880Z] f96aeaebbb33: Pull complete
[2026-07-17T08:06:14.880Z] fef65a673dc9: Pull complete
[2026-07-17T08:06:14.880Z] 4f4fb700ef54: Pull complete
[2026-07-17T08:06:14.880Z] 11547fb624d2: Pull complete
[2026-07-17T08:06:14.880Z] 582c5c9e769f: Pull complete
[2026-07-17T08:06:14.880Z] Digest: sha256:667c3601e4a4a05ec0db1d4d16d410447a47e0b598e0b2d24623546afcd677fb
[2026-07-17T08:06:14.880Z] Status: Downloaded newer image for rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-27d7487d6141807234c885bc3b76f82d055522fd-gfx908_gfx90a_gfx942
[2026-07-17T08:06:14.880Z] docker.io/rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-27d7487d6141807234c885bc3b76f82d055522fd-gfx908_gfx90a_gfx942
[2026-07-17T08:06:14.880Z] ++ GPU_FLAGS=
[2026-07-17T08:06:14.880Z] ++ echo 'NODE_LABELS: gfx90a mlir k8s-mi250-1-cwhsb gfx90a mlir k8s-mi250-1-cwhsb'
[2026-07-17T08:06:14.880Z] NODE_LABELS: gfx90a mlir k8s-mi250-1-cwhsb gfx90a mlir k8s-mi250-1-cwhsb
[2026-07-17T08:06:14.880Z] ++ [[ ! gfx90a mlir k8s-mi250-1-cwhsb =~ build-only ]]
[2026-07-17T08:06:14.880Z] ++ GPU_FLAGS='--group-add video --device=/dev/kfd --device=/dev/dri'
[2026-07-17T08:06:14.880Z] ++++ dirname prepare_docker_env.sh
[2026-07-17T08:06:14.880Z] +++ cd .
[2026-07-17T08:06:14.880Z] +++ pwd
[2026-07-17T08:06:14.880Z] ++ script_dir=/home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3442
[2026-07-17T08:06:14.880Z] ++ docker run --name pytorch-ci-container -e 'EXTENSION_BUILD_GFXARCH=gfx90a;gfx908;gfx942' -e CI=1 -e TEST_CONFIG=default -e SHARD_NUMBER=2 -e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip -e PREBUILT=false -e TEST_CORE=true -e TEST_CORE=true -v /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3442:/host_workspace -v /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3442:/scripts --entrypoint bash -d --network=host --group-add video --device=/dev/kfd --device=/dev/dri --ipc=host --pid=host --shm-size 8G --cap-add=SYS_PTRACE --security-opt seccomp=unconfined rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-27d7487d6141807234c885bc3b76f82d055522fd-gfx908_gfx90a_gfx942 -lc 'sleep infinity'
[2026-07-17T08:06:14.880Z] 1e2beafbbb8d463d9711539fc9818d7102fec592f786ed979d0745c72dece91f
[2026-07-17T08:06:14.881Z] ++ env -i python3 /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3442/pytorch/tools/stats/export_test_times.py
[2026-07-17T08:06:14.881Z] Exporting test times from test-infra
[2026-07-17T08:06:14.881Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-times.json to /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3442/pytorch/.additional_ci_files/test-times.json
[2026-07-17T08:06:14.881Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-class-times.json to /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3442/pytorch/.additional_ci_files/test-class-times.json
[2026-07-17T08:06:14.881Z] ++ python3 /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3442/pytorch/tools/stats/export_test_times.py
[2026-07-17T08:06:14.881Z] Exporting test times from test-infra
[2026-07-17T08:06:14.881Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-times.json to /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3442/pytorch/.additional_ci_files/test-times.json
[2026-07-17T08:06:14.881Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-class-times.json to /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3442/pytorch/.additional_ci_files/test-class-times.json
[2026-07-17T08:06:14.881Z] ++ [[ -d /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3442/pytorch/.additional_ci_files ]]
[2026-07-17T08:06:14.881Z] ++ sed -i -E 's/"\S+rocm\S+"/""/g' /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3442/pytorch/.additional_ci_files/test-times.json
[2026-07-17T08:06:14.881Z] ++ [[ false != \t\r\u\e ]]
[2026-07-17T08:06:14.881Z] ++ docker exec -u root pytorch-ci-container bash -c 'rm -rf /var/lib/jenkins/pytorch'
[2026-07-17T08:06:14.881Z] ++ docker cp /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3442/pytorch pytorch-ci-container:/var/lib/jenkins/pytorch
[2026-07-17T08:06:18.473Z] ++ docker exec -u root pytorch-ci-container bash -c 'chown root:root -R /var/lib/jenkins/pytorch'
[2026-07-17T08:06:18.473Z] ++ [[ 1 -eq 1 ]]
[2026-07-17T08:06:18.473Z] ++ docker cp /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3442/rocm_composable_kernel-7.1.1.dev0+g4266f8-py3-none-any.whl /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3442/torch-2.13.0a0+git757b6cc-cp312-cp312-linux_x86_64.whl pytorch-ci-container:/var/lib/jenkins/pytorch
[2026-07-17T08:06:18.473Z] docker: 'docker cp' requires 2 arguments
[2026-07-17T08:06:18.474Z] 
[2026-07-17T08:06:18.474Z] Usage:  docker cp [OPTIONS] CONTAINER:SRC_PATH DEST_PATH|-
[2026-07-17T08:06:18.474Z] 	docker cp [OPTIONS] SRC_PATH|- CONTAINER:DEST_PATH
[2026-07-17T08:06:18.474Z] 
[2026-07-17T08:06:18.474Z] Run 'docker cp --help' for more information

Tests / Test PyTorch / Test PyTorch / Run pytorch_test_2 / Error signal

Error in error step, with arguments pytorch_test_2 failed.

pytorch_test_2 failed

Tests / Test Distributed / Test Distributed / Run pytorch_distributed_1 / Shell Script

Error in sh step, with arguments ./test_pytorch_test_distributed.sh.

script returned exit code 1
Build log 
[2026-07-17T07:42:37.965Z] + ./test_pytorch_test_distributed.sh
[2026-07-17T07:42:37.965Z] + [[ -z '' ]]
[2026-07-17T07:42:37.965Z] + DEFAULT_EXECUTION_OPTIONS='-e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts         -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip         --copy-whls'
[2026-07-17T07:42:37.965Z] + source prepare_docker_env.sh -e 'EXTENSION_BUILD_GFXARCH=gfx90a;gfx908;gfx942' -e CI=1 -e TEST_CONFIG=distributed -e SHARD_NUMBER=1 -e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip --copy-whls
[2026-07-17T07:42:37.965Z] ++ set -ex
[2026-07-17T07:42:37.965Z] ++ set -o pipefail
[2026-07-17T07:42:37.965Z] ++ copy_whls=0
[2026-07-17T07:42:37.965Z] ++ envvars=()
[2026-07-17T07:42:37.965Z] ++ [[ 13 -gt 0 ]]
[2026-07-17T07:42:37.965Z] ++ key=-e
[2026-07-17T07:42:37.965Z] ++ case $key in
[2026-07-17T07:42:37.965Z] ++ envvars+=("-e $2")
[2026-07-17T07:42:37.965Z] ++ shift
[2026-07-17T07:42:37.965Z] ++ shift
[2026-07-17T07:42:37.965Z] ++ [[ 11 -gt 0 ]]
[2026-07-17T07:42:37.965Z] ++ key=-e
[2026-07-17T07:42:37.965Z] ++ case $key in
[2026-07-17T07:42:37.965Z] ++ envvars+=("-e $2")
[2026-07-17T07:42:37.965Z] ++ shift
[2026-07-17T07:42:37.965Z] ++ shift
[2026-07-17T07:42:37.965Z] ++ [[ 9 -gt 0 ]]
[2026-07-17T07:42:37.965Z] ++ key=-e
[2026-07-17T07:42:37.965Z] ++ case $key in
[2026-07-17T07:42:37.965Z] ++ envvars+=("-e $2")
[2026-07-17T07:42:37.965Z] ++ shift
[2026-07-17T07:42:37.965Z] ++ shift
[2026-07-17T07:42:37.965Z] ++ [[ 7 -gt 0 ]]
[2026-07-17T07:42:37.965Z] ++ key=-e
[2026-07-17T07:42:37.965Z] ++ case $key in
[2026-07-17T07:42:37.965Z] ++ envvars+=("-e $2")
[2026-07-17T07:42:37.965Z] ++ shift
[2026-07-17T07:42:37.965Z] ++ shift
[2026-07-17T07:42:37.965Z] ++ [[ 5 -gt 0 ]]
[2026-07-17T07:42:37.965Z] ++ key=-e
[2026-07-17T07:42:37.965Z] ++ case $key in
[2026-07-17T07:42:37.965Z] ++ envvars+=("-e $2")
[2026-07-17T07:42:37.965Z] ++ shift
[2026-07-17T07:42:37.965Z] ++ shift
[2026-07-17T07:42:37.965Z] ++ [[ 3 -gt 0 ]]
[2026-07-17T07:42:37.965Z] ++ key=-e
[2026-07-17T07:42:37.965Z] ++ case $key in
[2026-07-17T07:42:37.965Z] ++ envvars+=("-e $2")
[2026-07-17T07:42:37.965Z] ++ shift
[2026-07-17T07:42:37.965Z] ++ shift
[2026-07-17T07:42:37.965Z] ++ [[ 1 -gt 0 ]]
[2026-07-17T07:42:37.965Z] ++ key=--copy-whls
[2026-07-17T07:42:37.965Z] ++ case $key in
[2026-07-17T07:42:37.965Z] ++ copy_whls=1
[2026-07-17T07:42:37.965Z] ++ shift
[2026-07-17T07:42:37.965Z] ++ [[ 0 -gt 0 ]]
[2026-07-17T07:42:37.965Z] ++ envvars+=("-e PREBUILT=${PREBUILT:-false}")
[2026-07-17T07:42:37.965Z] ++ envvars+=("-e TEST_CORE=${TEST_CORE:-false}")
[2026-07-17T07:42:37.965Z] ++ [[ false == \t\r\u\e ]]
[2026-07-17T07:42:37.965Z] ++ [[ true == \t\r\u\e ]]
[2026-07-17T07:42:37.965Z] ++ envvars+=("-e TEST_CORE=${TEST_CORE}")
[2026-07-17T07:42:37.965Z] ++ docker ps -a
[2026-07-17T07:42:37.965Z] CONTAINER ID   IMAGE                                                                                                                        COMMAND                  CREATED       STATUS       PORTS     NAMES
[2026-07-17T07:42:37.965Z] 37c880b284c4   rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-1b618e3c916dd1605bea8b2bec1c54e0e08d0eb3-gfx908_gfx90a_gfx942   "bash -lc 'sleep inf…"   4 hours ago   Up 4 hours             pytorch-ci-container
[2026-07-17T07:42:37.965Z] ++ docker rm -f pytorch-ci-container
[2026-07-17T07:42:43.784Z] ++ docker pull rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-27d7487d6141807234c885bc3b76f82d055522fd-gfx908_gfx90a_gfx942
[2026-07-17T07:42:43.784Z] pytorch-linux-noble-rocm7.2.4-py3.12-27d7487d6141807234c885bc3b76f82d055522fd-gfx908_gfx90a_gfx942: Pulling from rocm/pytorch-ci-private
[2026-07-17T07:42:43.784Z] Digest: sha256:667c3601e4a4a05ec0db1d4d16d410447a47e0b598e0b2d24623546afcd677fb
[2026-07-17T07:42:43.784Z] Status: Image is up to date for rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-27d7487d6141807234c885bc3b76f82d055522fd-gfx908_gfx90a_gfx942
[2026-07-17T07:42:43.784Z] docker.io/rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-27d7487d6141807234c885bc3b76f82d055522fd-gfx908_gfx90a_gfx942
[2026-07-17T07:42:43.784Z] ++ GPU_FLAGS=
[2026-07-17T07:42:43.784Z] ++ echo 'NODE_LABELS: multi_gpu rocm pytorch-nightly selftest ixt-hq-15 gfx908 multi_gpu rocm pytorch-nightly selftest ixt-hq-15 gfx908'
[2026-07-17T07:42:43.784Z] NODE_LABELS: multi_gpu rocm pytorch-nightly selftest ixt-hq-15 gfx908 multi_gpu rocm pytorch-nightly selftest ixt-hq-15 gfx908
[2026-07-17T07:42:43.784Z] ++ [[ ! multi_gpu rocm pytorch-nightly selftest ixt-hq-15 gfx908 =~ build-only ]]
[2026-07-17T07:42:43.784Z] ++ GPU_FLAGS='--group-add video --device=/dev/kfd --device=/dev/dri'
[2026-07-17T07:42:43.784Z] ++++ dirname prepare_docker_env.sh
[2026-07-17T07:42:43.784Z] +++ cd .
[2026-07-17T07:42:43.784Z] +++ pwd
[2026-07-17T07:42:43.784Z] ++ script_dir=/home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3442
[2026-07-17T07:42:43.784Z] ++ docker run --name pytorch-ci-container -e 'EXTENSION_BUILD_GFXARCH=gfx90a;gfx908;gfx942' -e CI=1 -e TEST_CONFIG=distributed -e SHARD_NUMBER=1 -e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip -e PREBUILT=false -e TEST_CORE=true -e TEST_CORE=true -v /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3442:/host_workspace -v /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3442:/scripts --entrypoint bash -d --network=host --group-add video --device=/dev/kfd --device=/dev/dri --ipc=host --pid=host --shm-size 8G --cap-add=SYS_PTRACE --security-opt seccomp=unconfined rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-27d7487d6141807234c885bc3b76f82d055522fd-gfx908_gfx90a_gfx942 -lc 'sleep infinity'
[2026-07-17T07:42:44.374Z] 54c2dc388f17bad0d438502fe171f817acc2b43d2149f66bd2e462914d0bf6c1
[2026-07-17T07:42:44.374Z] ++ env -i python3 /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3442/pytorch/tools/stats/export_test_times.py
[2026-07-17T07:42:45.538Z] Exporting test times from test-infra
[2026-07-17T07:42:45.538Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-times.json to /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3442/pytorch/.additional_ci_files/test-times.json
[2026-07-17T07:42:45.538Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-class-times.json to /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3442/pytorch/.additional_ci_files/test-class-times.json
[2026-07-17T07:42:45.538Z] ++ python3 /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3442/pytorch/tools/stats/export_test_times.py
[2026-07-17T07:42:45.538Z] Exporting test times from test-infra
[2026-07-17T07:42:45.538Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-times.json to /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3442/pytorch/.additional_ci_files/test-times.json
[2026-07-17T07:42:45.538Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-class-times.json to /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3442/pytorch/.additional_ci_files/test-class-times.json
[2026-07-17T07:42:46.119Z] ++ [[ -d /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3442/pytorch/.additional_ci_files ]]
[2026-07-17T07:42:46.119Z] ++ sed -i -E 's/"\S+rocm\S+"/""/g' /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3442/pytorch/.additional_ci_files/test-times.json
[2026-07-17T07:42:46.119Z] ++ [[ false != \t\r\u\e ]]
[2026-07-17T07:42:46.119Z] ++ docker exec -u root pytorch-ci-container bash -c 'rm -rf /var/lib/jenkins/pytorch'
[2026-07-17T07:42:46.119Z] ++ docker cp /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3442/pytorch pytorch-ci-container:/var/lib/jenkins/pytorch
[2026-07-17T07:42:52.832Z] ++ docker exec -u root pytorch-ci-container bash -c 'chown root:root -R /var/lib/jenkins/pytorch'
[2026-07-17T07:42:52.832Z] ++ [[ 1 -eq 1 ]]
[2026-07-17T07:42:52.832Z] ++ docker cp /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3442/rocm_composable_kernel-7.1.1.dev0+g4266f8-py3-none-any.whl /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3442/torch-2.13.0a0+git757b6cc-cp312-cp312-linux_x86_64.whl pytorch-ci-container:/var/lib/jenkins/pytorch
[2026-07-17T07:42:52.832Z] "docker cp" requires exactly 2 arguments.
[2026-07-17T07:42:52.832Z] See 'docker cp --help'.
[2026-07-17T07:42:52.832Z] 
[2026-07-17T07:42:52.832Z] Usage:  docker cp [OPTIONS] CONTAINER:SRC_PATH DEST_PATH|-
[2026-07-17T07:42:52.832Z] 	docker cp [OPTIONS] SRC_PATH|- CONTAINER:DEST_PATH
[2026-07-17T07:42:52.832Z] 
[2026-07-17T07:42:52.832Z] Copy files/folders between a container and the local filesystem

Tests / Test Distributed / Test Distributed / Run pytorch_distributed_1 / Error signal

Error in error step, with arguments pytorch_distributed_1 failed.

pytorch_distributed_1 failed

Tests / Test Distributed / Test Distributed / Run pytorch_distributed_2 / Shell Script

Error in sh step, with arguments ./test_pytorch_test_distributed.sh.

script returned exit code 1
Build log 
[2026-07-17T07:47:30.121Z] + ./test_pytorch_test_distributed.sh
[2026-07-17T07:47:30.121Z] + [[ -z '' ]]
[2026-07-17T07:47:30.121Z] + DEFAULT_EXECUTION_OPTIONS='-e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts         -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip         --copy-whls'
[2026-07-17T07:47:30.121Z] + source prepare_docker_env.sh -e 'EXTENSION_BUILD_GFXARCH=gfx90a;gfx908;gfx942' -e CI=1 -e TEST_CONFIG=distributed -e SHARD_NUMBER=2 -e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip --copy-whls
[2026-07-17T07:47:30.121Z] ++ set -ex
[2026-07-17T07:47:30.121Z] ++ set -o pipefail
[2026-07-17T07:47:30.121Z] ++ copy_whls=0
[2026-07-17T07:47:30.121Z] ++ envvars=()
[2026-07-17T07:47:30.121Z] ++ [[ 13 -gt 0 ]]
[2026-07-17T07:47:30.121Z] ++ key=-e
[2026-07-17T07:47:30.121Z] ++ case $key in
[2026-07-17T07:47:30.121Z] ++ envvars+=("-e $2")
[2026-07-17T07:47:30.121Z] ++ shift
[2026-07-17T07:47:30.121Z] ++ shift
[2026-07-17T07:47:30.121Z] ++ [[ 11 -gt 0 ]]
[2026-07-17T07:47:30.121Z] ++ key=-e
[2026-07-17T07:47:30.121Z] ++ case $key in
[2026-07-17T07:47:30.121Z] ++ envvars+=("-e $2")
[2026-07-17T07:47:30.121Z] ++ shift
[2026-07-17T07:47:30.121Z] ++ shift
[2026-07-17T07:47:30.121Z] ++ [[ 9 -gt 0 ]]
[2026-07-17T07:47:30.121Z] ++ key=-e
[2026-07-17T07:47:30.121Z] ++ case $key in
[2026-07-17T07:47:30.121Z] ++ envvars+=("-e $2")
[2026-07-17T07:47:30.121Z] ++ shift
[2026-07-17T07:47:30.121Z] ++ shift
[2026-07-17T07:47:30.121Z] ++ [[ 7 -gt 0 ]]
[2026-07-17T07:47:30.121Z] ++ key=-e
[2026-07-17T07:47:30.121Z] ++ case $key in
[2026-07-17T07:47:30.121Z] ++ envvars+=("-e $2")
[2026-07-17T07:47:30.121Z] ++ shift
[2026-07-17T07:47:30.121Z] ++ shift
[2026-07-17T07:47:30.121Z] ++ [[ 5 -gt 0 ]]
[2026-07-17T07:47:30.121Z] ++ key=-e
[2026-07-17T07:47:30.121Z] ++ case $key in
[2026-07-17T07:47:30.121Z] ++ envvars+=("-e $2")
[2026-07-17T07:47:30.121Z] ++ shift
[2026-07-17T07:47:30.121Z] ++ shift
[2026-07-17T07:47:30.121Z] ++ [[ 3 -gt 0 ]]
[2026-07-17T07:47:30.121Z] ++ key=-e
[2026-07-17T07:47:30.121Z] ++ case $key in
[2026-07-17T07:47:30.121Z] ++ envvars+=("-e $2")
[2026-07-17T07:47:30.121Z] ++ shift
[2026-07-17T07:47:30.121Z] ++ shift
[2026-07-17T07:47:30.121Z] ++ [[ 1 -gt 0 ]]
[2026-07-17T07:47:30.121Z] ++ key=--copy-whls
[2026-07-17T07:47:30.121Z] ++ case $key in
[2026-07-17T07:47:30.121Z] ++ copy_whls=1
[2026-07-17T07:47:30.121Z] ++ shift
[2026-07-17T07:47:30.121Z] ++ [[ 0 -gt 0 ]]
[2026-07-17T07:47:30.121Z] ++ envvars+=("-e PREBUILT=${PREBUILT:-false}")
[2026-07-17T07:47:30.121Z] ++ envvars+=("-e TEST_CORE=${TEST_CORE:-false}")
[2026-07-17T07:47:30.121Z] ++ [[ false == \t\r\u\e ]]
[2026-07-17T07:47:30.121Z] ++ [[ true == \t\r\u\e ]]
[2026-07-17T07:47:30.121Z] ++ envvars+=("-e TEST_CORE=${TEST_CORE}")
[2026-07-17T07:47:30.121Z] ++ docker ps -a
[2026-07-17T07:47:30.121Z] CONTAINER ID   IMAGE                                                                                                                        COMMAND                  CREATED         STATUS         PORTS     NAMES
[2026-07-17T07:47:30.121Z] 54c2dc388f17   rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-27d7487d6141807234c885bc3b76f82d055522fd-gfx908_gfx90a_gfx942   "bash -lc 'sleep inf…"   4 minutes ago   Up 4 minutes             pytorch-ci-container
[2026-07-17T07:47:30.121Z] ++ docker rm -f pytorch-ci-container
[2026-07-17T07:47:31.206Z] ++ docker pull rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-27d7487d6141807234c885bc3b76f82d055522fd-gfx908_gfx90a_gfx942
[2026-07-17T07:47:31.722Z] pytorch-linux-noble-rocm7.2.4-py3.12-27d7487d6141807234c885bc3b76f82d055522fd-gfx908_gfx90a_gfx942: Pulling from rocm/pytorch-ci-private
[2026-07-17T07:47:31.722Z] Digest: sha256:667c3601e4a4a05ec0db1d4d16d410447a47e0b598e0b2d24623546afcd677fb
[2026-07-17T07:47:31.722Z] Status: Image is up to date for rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-27d7487d6141807234c885bc3b76f82d055522fd-gfx908_gfx90a_gfx942
[2026-07-17T07:47:31.722Z] docker.io/rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-27d7487d6141807234c885bc3b76f82d055522fd-gfx908_gfx90a_gfx942
[2026-07-17T07:47:31.722Z] ++ GPU_FLAGS=
[2026-07-17T07:47:31.722Z] ++ echo 'NODE_LABELS: multi_gpu rocm pytorch-nightly selftest ixt-hq-15 gfx908 multi_gpu rocm pytorch-nightly selftest ixt-hq-15 gfx908'
[2026-07-17T07:47:31.722Z] NODE_LABELS: multi_gpu rocm pytorch-nightly selftest ixt-hq-15 gfx908 multi_gpu rocm pytorch-nightly selftest ixt-hq-15 gfx908
[2026-07-17T07:47:31.722Z] ++ [[ ! multi_gpu rocm pytorch-nightly selftest ixt-hq-15 gfx908 =~ build-only ]]
[2026-07-17T07:47:31.722Z] ++ GPU_FLAGS='--group-add video --device=/dev/kfd --device=/dev/dri'
[2026-07-17T07:47:31.722Z] ++++ dirname prepare_docker_env.sh
[2026-07-17T07:47:31.722Z] +++ cd .
[2026-07-17T07:47:31.722Z] +++ pwd
[2026-07-17T07:47:31.722Z] ++ script_dir=/home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3442
[2026-07-17T07:47:31.722Z] ++ docker run --name pytorch-ci-container -e 'EXTENSION_BUILD_GFXARCH=gfx90a;gfx908;gfx942' -e CI=1 -e TEST_CONFIG=distributed -e SHARD_NUMBER=2 -e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip -e PREBUILT=false -e TEST_CORE=true -e TEST_CORE=true -v /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3442:/host_workspace -v /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3442:/scripts --entrypoint bash -d --network=host --group-add video --device=/dev/kfd --device=/dev/dri --ipc=host --pid=host --shm-size 8G --cap-add=SYS_PTRACE --security-opt seccomp=unconfined rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-27d7487d6141807234c885bc3b76f82d055522fd-gfx908_gfx90a_gfx942 -lc 'sleep infinity'
[2026-07-17T07:47:31.722Z] cc0c630b7dce62e9606fd89e504de9700e4a5c0cfd763a452c8616b27098dc70
[2026-07-17T07:47:32.264Z] ++ env -i python3 /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3442/pytorch/tools/stats/export_test_times.py
[2026-07-17T07:47:32.790Z] Exporting test times from test-infra
[2026-07-17T07:47:32.790Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-times.json to /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3442/pytorch/.additional_ci_files/test-times.json
[2026-07-17T07:47:32.790Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-class-times.json to /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3442/pytorch/.additional_ci_files/test-class-times.json
[2026-07-17T07:47:32.790Z] ++ python3 /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3442/pytorch/tools/stats/export_test_times.py
[2026-07-17T07:47:33.311Z] Exporting test times from test-infra
[2026-07-17T07:47:33.311Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-times.json to /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3442/pytorch/.additional_ci_files/test-times.json
[2026-07-17T07:47:33.311Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-class-times.json to /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3442/pytorch/.additional_ci_files/test-class-times.json
[2026-07-17T07:47:33.311Z] ++ [[ -d /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3442/pytorch/.additional_ci_files ]]
[2026-07-17T07:47:33.311Z] ++ sed -i -E 's/"\S+rocm\S+"/""/g' /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3442/pytorch/.additional_ci_files/test-times.json
[2026-07-17T07:47:33.311Z] ++ [[ false != \t\r\u\e ]]
[2026-07-17T07:47:33.311Z] ++ docker exec -u root pytorch-ci-container bash -c 'rm -rf /var/lib/jenkins/pytorch'
[2026-07-17T07:47:33.311Z] ++ docker cp /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3442/pytorch pytorch-ci-container:/var/lib/jenkins/pytorch
[2026-07-17T07:47:39.560Z] ++ docker exec -u root pytorch-ci-container bash -c 'chown root:root -R /var/lib/jenkins/pytorch'
[2026-07-17T07:47:40.072Z] ++ [[ 1 -eq 1 ]]
[2026-07-17T07:47:40.072Z] ++ docker cp /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3442/rocm_composable_kernel-7.1.1.dev0+g4266f8-py3-none-any.whl /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3442/torch-2.13.0a0+git757b6cc-cp312-cp312-linux_x86_64.whl pytorch-ci-container:/var/lib/jenkins/pytorch
[2026-07-17T07:47:40.072Z] "docker cp" requires exactly 2 arguments.
[2026-07-17T07:47:40.072Z] See 'docker cp --help'.
[2026-07-17T07:47:40.072Z] 
[2026-07-17T07:47:40.072Z] Usage:  docker cp [OPTIONS] CONTAINER:SRC_PATH DEST_PATH|-
[2026-07-17T07:47:40.072Z] 	docker cp [OPTIONS] SRC_PATH|- CONTAINER:DEST_PATH
[2026-07-17T07:47:40.072Z] 
[2026-07-17T07:47:40.072Z] Copy files/folders between a container and the local filesystem

Tests / Test Distributed / Test Distributed / Run pytorch_distributed_2 / Error signal

Error in error step, with arguments pytorch_distributed_2 failed.

pytorch_distributed_2 failed

Tests / Test Inductor / Test Inductor / Run pytorch_inductor_1 / Shell Script

Error in sh step, with arguments ./test_pytorch_inductor.sh.

script returned exit code 1
Build log 
[2026-07-17T07:54:21.566Z] + ./test_pytorch_inductor.sh
[2026-07-17T07:54:21.566Z] + [[ -z '' ]]
[2026-07-17T07:54:21.566Z] + DEFAULT_EXECUTION_OPTIONS='-e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts         -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip         --copy-whls'
[2026-07-17T07:54:21.566Z] + source prepare_docker_env.sh -e 'EXTENSION_BUILD_GFXARCH=gfx90a;gfx908;gfx942' -e CI=1 -e TEST_CONFIG=inductor -e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip --copy-whls
[2026-07-17T07:54:21.566Z] ++ set -ex
[2026-07-17T07:54:21.566Z] ++ set -o pipefail
[2026-07-17T07:54:21.566Z] ++ copy_whls=0
[2026-07-17T07:54:21.566Z] ++ envvars=()
[2026-07-17T07:54:21.566Z] ++ [[ 11 -gt 0 ]]
[2026-07-17T07:54:21.566Z] ++ key=-e
[2026-07-17T07:54:21.566Z] ++ case $key in
[2026-07-17T07:54:21.566Z] ++ envvars+=("-e $2")
[2026-07-17T07:54:21.566Z] ++ shift
[2026-07-17T07:54:21.566Z] ++ shift
[2026-07-17T07:54:21.566Z] ++ [[ 9 -gt 0 ]]
[2026-07-17T07:54:21.566Z] ++ key=-e
[2026-07-17T07:54:21.566Z] ++ case $key in
[2026-07-17T07:54:21.566Z] ++ envvars+=("-e $2")
[2026-07-17T07:54:21.566Z] ++ shift
[2026-07-17T07:54:21.566Z] ++ shift
[2026-07-17T07:54:21.566Z] ++ [[ 7 -gt 0 ]]
[2026-07-17T07:54:21.566Z] ++ key=-e
[2026-07-17T07:54:21.566Z] ++ case $key in
[2026-07-17T07:54:21.566Z] ++ envvars+=("-e $2")
[2026-07-17T07:54:21.566Z] ++ shift
[2026-07-17T07:54:21.566Z] ++ shift
[2026-07-17T07:54:21.566Z] ++ [[ 5 -gt 0 ]]
[2026-07-17T07:54:21.566Z] ++ key=-e
[2026-07-17T07:54:21.566Z] ++ case $key in
[2026-07-17T07:54:21.566Z] ++ envvars+=("-e $2")
[2026-07-17T07:54:21.566Z] ++ shift
[2026-07-17T07:54:21.566Z] ++ shift
[2026-07-17T07:54:21.566Z] ++ [[ 3 -gt 0 ]]
[2026-07-17T07:54:21.566Z] ++ key=-e
[2026-07-17T07:54:21.566Z] ++ case $key in
[2026-07-17T07:54:21.566Z] ++ envvars+=("-e $2")
[2026-07-17T07:54:21.566Z] ++ shift
[2026-07-17T07:54:21.566Z] ++ shift
[2026-07-17T07:54:21.566Z] ++ [[ 1 -gt 0 ]]
[2026-07-17T07:54:21.566Z] ++ key=--copy-whls
[2026-07-17T07:54:21.566Z] ++ case $key in
[2026-07-17T07:54:21.566Z] ++ copy_whls=1
[2026-07-17T07:54:21.566Z] ++ shift
[2026-07-17T07:54:21.566Z] ++ [[ 0 -gt 0 ]]
[2026-07-17T07:54:21.566Z] ++ envvars+=("-e PREBUILT=${PREBUILT:-false}")
[2026-07-17T07:54:21.566Z] ++ envvars+=("-e TEST_CORE=${TEST_CORE:-false}")
[2026-07-17T07:54:21.566Z] ++ [[ false == \t\r\u\e ]]
[2026-07-17T07:54:21.566Z] ++ [[ true == \t\r\u\e ]]
[2026-07-17T07:54:21.566Z] ++ envvars+=("-e TEST_CORE=${TEST_CORE}")
[2026-07-17T07:54:21.566Z] ++ docker ps -a
[2026-07-17T07:54:21.566Z] CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES
[2026-07-17T07:54:21.566Z] ++ docker rm -f pytorch-ci-container
[2026-07-17T07:54:21.566Z] ++ docker pull rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-27d7487d6141807234c885bc3b76f82d055522fd-gfx908_gfx90a_gfx942
[2026-07-17T07:54:22.868Z] pytorch-linux-noble-rocm7.2.4-py3.12-27d7487d6141807234c885bc3b76f82d055522fd-gfx908_gfx90a_gfx942: Pulling from rocm/pytorch-ci-private
[2026-07-17T07:54:23.249Z] 4d307752764b: Pulling fs layer
[2026-07-17T07:54:23.249Z] ca2678b20700: Pulling fs layer
[2026-07-17T07:54:23.249Z] 582c5c9e769f: Pulling fs layer
[2026-07-17T07:54:23.249Z] 9110ef432697: Pulling fs layer
[2026-07-17T07:54:23.249Z] d9289c72d3d7: Pulling fs layer
[2026-07-17T07:54:23.249Z] a48fe49c074d: Pulling fs layer
[2026-07-17T07:54:23.249Z] 1f32f350bea4: Pulling fs layer
[2026-07-17T07:54:23.249Z] bf3e4cf00cd6: Pulling fs layer
[2026-07-17T07:54:23.249Z] c6cfbbbf995c: Pulling fs layer
[2026-07-17T07:54:23.249Z] 5ab16f3f3b07: Pulling fs layer
[2026-07-17T07:54:23.249Z] 2ad1733fbd4f: Pulling fs layer
[2026-07-17T07:54:23.249Z] 86cc05da26d1: Pulling fs layer
[2026-07-17T07:54:23.249Z] 376b4789221e: Pulling fs layer
[2026-07-17T07:54:23.249Z] 5846e8e82d69: Pulling fs layer
[2026-07-17T07:54:23.249Z] db0af168a1a4: Pulling fs layer
[2026-07-17T07:54:23.249Z] 906ac4de0a2b: Pulling fs layer
[2026-07-17T07:54:23.249Z] 974f46a80f1e: Pulling fs layer
[2026-07-17T07:54:23.249Z] aeea1a04eb86: Pulling fs layer
[2026-07-17T07:54:23.249Z] 9c4ca0035d95: Pulling fs layer
[2026-07-17T07:54:23.249Z] f3058198fba5: Pulling fs layer
[2026-07-17T07:54:23.249Z] 2e147acf21bf: Pulling fs layer
[2026-07-17T07:54:23.249Z] 1ed9f8c32de0: Pulling fs layer
[2026-07-17T07:54:23.249Z] c0664a7365cb: Pulling fs layer
[2026-07-17T07:54:23.249Z] f391428cfe0f: Pulling fs layer
[2026-07-17T07:54:23.249Z] aeea1a04eb86: Pulling fs layer
[2026-07-17T07:54:23.249Z] ad8ac94589aa: Pulling fs layer
[2026-07-17T07:54:23.249Z] 27fcad4d96a6: Pulling fs layer
[2026-07-17T07:54:23.249Z] 2b5553ca587e: Pulling fs layer
[2026-07-17T07:54:23.249Z] 3b2e63278265: Pulling fs layer
[2026-07-17T07:54:23.249Z] 93747a912780: Pulling fs layer
[2026-07-17T07:54:23.249Z] f8f114c00167: Pulling fs layer
[2026-07-17T07:54:23.249Z] 9d96e6a6c786: Pulling fs layer
[2026-07-17T07:54:23.249Z] aeea1a04eb86: Pulling fs layer
[2026-07-17T07:54:23.249Z] a661a3e0a52e: Pulling fs layer
[2026-07-17T07:54:23.249Z] fef65a673dc9: Pulling fs layer
[2026-07-17T07:54:23.249Z] 90d28eda335d: Pulling fs layer
[2026-07-17T07:54:23.249Z] 626c31f89101: Pulling fs layer
[2026-07-17T07:54:23.249Z] aa60dad79264: Pulling fs layer
[2026-07-17T07:54:23.249Z] 5f8dd10c9c78: Pulling fs layer
[2026-07-17T07:54:23.249Z] c459752d4f0e: Pulling fs layer
[2026-07-17T07:54:23.249Z] d49e4624054e: Pulling fs layer
[2026-07-17T07:54:23.249Z] f96aeaebbb33: Pulling fs layer
[2026-07-17T07:54:23.249Z] 4f4fb700ef54: Pulling fs layer
[2026-07-17T07:54:23.249Z] aeea1a04eb86: Pulling fs layer
[2026-07-17T07:54:23.249Z] 9492bd48d097: Pulling fs layer
[2026-07-17T07:54:23.249Z] 5d4725036ac1: Pulling fs layer
[2026-07-17T07:54:23.249Z] aeea1a04eb86: Pulling fs layer
[2026-07-17T07:54:23.249Z] 11547fb624d2: Pulling fs layer
[2026-07-17T07:54:23.249Z] eeed50dfa35d: Pulling fs layer
[2026-07-17T07:54:23.249Z] 78a3c5370db4: Pulling fs layer
[2026-07-17T07:54:23.249Z] c9dec199a57f: Pulling fs layer
[2026-07-17T07:54:23.249Z] 381079e824de: Pulling fs layer
[2026-07-17T07:54:23.249Z] f006a76c61e8: Pulling fs layer
[2026-07-17T07:54:23.249Z] 42a99e52f938: Pulling fs layer
[2026-07-17T07:54:23.249Z] 535a46026bba: Pulling fs layer
[2026-07-17T07:54:23.249Z] b96eaed0e751: Pulling fs layer
[2026-07-17T07:54:23.249Z] 4f4fb700ef54: Pulling fs layer
[2026-07-17T07:54:23.663Z] 032a39a51a7b: Download complete
[2026-07-17T07:54:23.663Z] 1f32f350bea4: Download complete
[2026-07-17T07:54:23.663Z] 2ad1733fbd4f: Download complete
[2026-07-17T07:54:23.663Z] c6cfbbbf995c: Download complete
[2026-07-17T07:54:23.663Z] 86cc05da26d1: Download complete
[2026-07-17T07:54:23.663Z] 9110ef432697: Download complete
[2026-07-17T07:54:23.663Z] d9289c72d3d7: Download complete
[2026-07-17T07:54:23.663Z] 2e147acf21bf: Download complete
[2026-07-17T07:54:23.663Z] bf3e4cf00cd6: Download complete
[2026-07-17T07:54:24.047Z] 4f4fb700ef54: Download complete
[2026-07-17T07:54:24.047Z] 9d96e6a6c786: Download complete
[2026-07-17T07:54:24.047Z] 5d4725036ac1: Download complete
[2026-07-17T07:54:24.047Z] 27fcad4d96a6: Download complete
[2026-07-17T07:54:24.047Z] 1ed9f8c32de0: Download complete
[2026-07-17T07:54:24.047Z] ad8ac94589aa: Download complete
[2026-07-17T07:54:24.047Z] db0af168a1a4: Download complete
[2026-07-17T07:54:24.047Z] 906ac4de0a2b: Download complete
[2026-07-17T07:54:24.047Z] f3058198fba5: Download complete
[2026-07-17T07:54:24.047Z] a48fe49c074d: Download complete
[2026-07-17T07:54:24.047Z] aeea1a04eb86: Download complete
[2026-07-17T07:54:24.047Z] 974f46a80f1e: Download complete
[2026-07-17T07:54:24.047Z] c0664a7365cb: Download complete
[2026-07-17T07:54:24.047Z] 9c4ca0035d95: Download complete
[2026-07-17T07:54:24.047Z] b96eaed0e751: Download complete
[2026-07-17T07:54:24.047Z] 90d28eda335d: Download complete
[2026-07-17T07:54:24.047Z] f96aeaebbb33: Download complete
[2026-07-17T07:54:24.047Z] 381079e824de: Download complete
[2026-07-17T07:54:24.047Z] 5846e8e82d69: Download complete
[2026-07-17T07:54:24.047Z] a661a3e0a52e: Download complete
[2026-07-17T07:54:24.047Z] aa60dad79264: Download complete
[2026-07-17T07:54:24.047Z] 535a46026bba: Download complete
[2026-07-17T07:54:24.047Z] 11547fb624d2: Download complete
[2026-07-17T07:54:24.047Z] fef65a673dc9: Download complete
[2026-07-17T07:54:24.047Z] d49e4624054e: Download complete
[2026-07-17T07:54:24.047Z] f006a76c61e8: Download complete
[2026-07-17T07:54:24.047Z] 5f8dd10c9c78: Download complete
[2026-07-17T07:54:24.047Z] f8f114c00167: Download complete
[2026-07-17T07:54:24.047Z] 42a99e52f938: Download complete
[2026-07-17T07:54:24.047Z] 626c31f89101: Download complete
[2026-07-17T07:54:24.047Z] f391428cfe0f: Download complete
[2026-07-17T07:54:24.047Z] 78a3c5370db4: Download complete
[2026-07-17T07:54:24.047Z] 2b5553ca587e: Download complete
[2026-07-17T07:54:24.436Z] 9492bd48d097: Download complete
[2026-07-17T07:54:24.436Z] c9dec199a57f: Download complete
[2026-07-17T07:54:24.436Z] ca2678b20700: Download complete
[2026-07-17T07:54:24.824Z] 582c5c9e769f: Download complete
[2026-07-17T07:54:25.224Z] 3b2e63278265: Download complete
[2026-07-17T07:54:25.224Z] 1f32f350bea4: Pull complete
[2026-07-17T07:54:25.224Z] ca2678b20700: Pull complete
[2026-07-17T07:54:27.097Z] c459752d4f0e: Download complete
[2026-07-17T07:54:30.362Z] 4d307752764b: Download complete
[2026-07-17T07:54:38.105Z] 2ad1733fbd4f: Pull complete
[2026-07-17T07:54:38.105Z] c6cfbbbf995c: Pull complete
[2026-07-17T07:54:38.105Z] 906ac4de0a2b: Pull complete
[2026-07-17T07:54:38.105Z] 974f46a80f1e: Pull complete
[2026-07-17T07:54:38.105Z] 5846e8e82d69: Pull complete
[2026-07-17T07:54:38.105Z] f006a76c61e8: Pull complete
[2026-07-17T07:54:38.105Z] 4d307752764b: Pull complete
[2026-07-17T07:54:38.105Z] d9289c72d3d7: Pull complete
[2026-07-17T07:54:38.105Z] aeea1a04eb86: Pull complete
[2026-07-17T07:54:42.220Z] 376b4789221e: Download complete
[2026-07-17T07:55:10.834Z] 5ab16f3f3b07: Download complete
[2026-07-17T07:55:51.929Z] 86cc05da26d1: Pull complete
[2026-07-17T07:55:51.930Z] 9110ef432697: Pull complete
[2026-07-17T07:55:51.930Z] 2e147acf21bf: Pull complete
[2026-07-17T07:55:51.930Z] db0af168a1a4: Pull complete
[2026-07-17T07:55:51.930Z] a48fe49c074d: Pull complete
[2026-07-17T07:55:51.930Z] 381079e824de: Pull complete
[2026-07-17T07:55:51.930Z] f8f114c00167: Pull complete
[2026-07-17T07:55:51.930Z] 5ab16f3f3b07: Pull complete
[2026-07-17T07:56:20.508Z] 93747a912780: Download complete
[2026-07-17T07:57:09.784Z] eeed50dfa35d: Download complete
[2026-07-17T07:58:49.805Z] eeed50dfa35d: Pull complete
[2026-07-17T07:58:49.805Z] bf3e4cf00cd6: Pull complete
Output truncated.

Details

  • Kill older PR Builds (1.6 sec)
  • Initialize (1 hr 42 min)
    • Download CI scripts (1 min 5 sec)
    • Checkout Pytorch (3 min 24 sec)
    • Check base Docker image existence (20 sec)
    • Pull Docker Image (5 min 14 sec)
    • Build PyTorch (1 hr 30 min)
  • Tests (28 min)
    • Test PyTorch (11 ms)
      • Test PyTorch (28 min)
        • Run pytorch_test_1 (12 min)
          Error: script returned exit code 1 - logs
          Error: pytorch_test_1 failed - logs
        • Run pytorch_test_2 (15 min)
          Error: script returned exit code 1 - logs
          Error: pytorch_test_2 failed - logs
    • Test Distributed (7 ms)
      • Test Distributed (9 min 31 sec)
        • Run pytorch_distributed_1 (4 min 30 sec)
          Error: script returned exit code 1 - logs
          Error: pytorch_distributed_1 failed - logs
        • Run pytorch_distributed_2 (4 min 46 sec)
          Error: script returned exit code 1 - logs
          Error: pytorch_distributed_2 failed - logs
    • Test Inductor (9 ms)
      • Test Inductor (22 min)
        • Run pytorch_inductor_1 (21 min)
          Error: script returned exit code 1 - logs
          Error: pytorch_inductor_1 failed - logs
    • Test PyTorch Slow (8 ms)
      • Test PyTorch Slow (12 sec)
    • Microbenchmark (25 sec)
      • Microbenchmark (12 sec)
  • Post Build (1.8 sec)
  • Declarative: Post Actions (3.8 sec)
View more details on ROCm Repo Management API