Skip to content

CI: benchmark pip cache reuse #24064

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
vyasr wants to merge 2 commits into from
Closed

CI: benchmark pip cache reuse #24064

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
3884a71
ci: establish pip cache baseline
vyasr Sep 9, 2026
c2ba6e7
ci: prototype shared pip cache within jobs
vyasr Sep 9, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/workflows/pr.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
# Full-CI baseline for the pip-cache experiment.
name: pr
on:
push:
Expand Down
2 changes: 1 addition & 1 deletion ci/build_wheel_common.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@

source rapids-configure-sccache
source rapids-datetime-string
source rapids-init-pip
source ci/rapids-init-pip-prototype.sh

export SCCACHE_S3_USE_PREPROCESSOR_CACHE_MODE=true

Expand Down
2 changes: 1 addition & 1 deletion ci/cudf_pandas_scripts/pandas-tests/merge-nightly.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@

set -euo pipefail

source rapids-init-pip
source ci/rapids-init-pip-prototype.sh
# shellcheck source=ci/cudf_pandas_scripts/pandas-tests/shard-results.sh
source ci/cudf_pandas_scripts/pandas-tests/shard-results.sh

Expand Down
2 changes: 1 addition & 1 deletion ci/cudf_pandas_scripts/pandas-tests/run.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@

set -euo pipefail

source rapids-init-pip
source ci/rapids-init-pip-prototype.sh

EXITCODE=0
trap "EXITCODE=1" ERR
Expand Down
2 changes: 1 addition & 1 deletion ci/cudf_pandas_scripts/pandas-tests/summary.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@
# every fallible command is guarded explicitly and the script always exits 0.
set -uo pipefail

source rapids-init-pip
source ci/rapids-init-pip-prototype.sh
# shellcheck source=ci/cudf_pandas_scripts/pandas-tests/shard-results.sh
source ci/cudf_pandas_scripts/pandas-tests/shard-results.sh

Expand Down
2 changes: 1 addition & 1 deletion ci/cudf_pandas_scripts/run_tests.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@

set -eoxu pipefail

source rapids-init-pip
source ci/rapids-init-pip-prototype.sh

RAPIDS_TESTS_DIR=${RAPIDS_TESTS_DIR:-"${PWD}/test-results"}
RAPIDS_COVERAGE_DIR=${RAPIDS_COVERAGE_DIR:-"${PWD}/coverage-results"}
Expand Down
21 changes: 21 additions & 0 deletions ci/rapids-init-pip-prototype.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
#!/bin/bash
# SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

# Use the unreleased rapids-init-pip change under evaluation. Keeping the tool in
# the workspace makes it available to every pip invocation in this CI job, while
# deliberately not persisting it across jobs.
RAPIDS_INIT_PIP_SHA=a1aa4d1a80fcbf09da0f0b8eebf0e41ab42c0f48
RAPIDS_INIT_PIP_DIR="${GITHUB_WORKSPACE:-${PWD}}/.cache/gha-tools/${RAPIDS_INIT_PIP_SHA}"
RAPIDS_INIT_PIP_PATH="${RAPIDS_INIT_PIP_DIR}/rapids-init-pip"

if [[ ! -f "${RAPIDS_INIT_PIP_PATH}" ]]; then
mkdir -p "${RAPIDS_INIT_PIP_DIR}"
curl --fail --location --silent --show-error \
"https://raw.githubusercontent.com/rapidsai/gha-tools/${RAPIDS_INIT_PIP_SHA}/tools/rapids-init-pip" \
--output "${RAPIDS_INIT_PIP_PATH}"
fi

# shellcheck disable=SC1090
source "${RAPIDS_INIT_PIP_PATH}"
python -m pip cache dir
2 changes: 1 addition & 1 deletion ci/test_cudf_polars_polars_tests.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@

set -euo pipefail

source rapids-init-pip
source ci/rapids-init-pip-prototype.sh

rapids-logger "Download wheels"

Expand Down
2 changes: 1 addition & 1 deletion ci/test_narwhals.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ set -euo pipefail
CI_DIR="$(dirname "$(realpath "${BASH_SOURCE[0]}")")"
cd "${CI_DIR}/../"

source rapids-init-pip
source ci/rapids-init-pip-prototype.sh

# Common setup steps shared by Python test jobs
source ./ci/test_python_common.sh test_python_narwhals
Expand Down
2 changes: 1 addition & 1 deletion ci/test_wheel_cudf.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@

set -euo pipefail

source rapids-init-pip
source ci/rapids-init-pip-prototype.sh

RAPIDS_PY_CUDA_SUFFIX="$(rapids-wheel-ctk-name-gen "${RAPIDS_CUDA_VERSION}")"
RUN_PYLIBCUDF_TESTS="${RUN_PYLIBCUDF_TESTS:-true}"
Expand Down
2 changes: 1 addition & 1 deletion ci/test_wheel_cudf_polars.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@

set -euo pipefail

source rapids-init-pip
source ci/rapids-init-pip-prototype.sh

rapids-logger "Download wheels"

Expand Down
2 changes: 1 addition & 1 deletion ci/test_wheel_dask_cudf.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@

set -euo pipefail

source rapids-init-pip
source ci/rapids-init-pip-prototype.sh

RAPIDS_PY_CUDA_SUFFIX="$(rapids-wheel-ctk-name-gen "${RAPIDS_CUDA_VERSION}")"
DASK_CUDF_WHEELHOUSE=$(rapids-download-from-github "$(rapids-artifact-name wheel_python dask-cudf cudf --pure --arch any --cuda "$RAPIDS_CUDA_VERSION")")
Expand Down
Loading