Improve build-te for build-jax step in CI - #2245
Conversation
|
/ok to test 3448da0 |
|
/ok to test 3448da0 |
|
/ok to test 6285bfc |
|
/ok to test cb1317d |
| @@ -0,0 +1,160 @@ | |||
| #!/bin/bash | |||
There was a problem hiding this comment.
There was a problem hiding this comment.
0.17.0 is released, can this all be dropped?
There was a problem hiding this comment.
Not yet, I tried to get this script in a better shape though.
The reason is that sccache 0.17.0 was released before the CUDA 13.3 fix, so there is no prebuilt sccache binary that contains the CUDA 13.3 fix yet
| # Install ccache if not present (needs >= 4.1 for Redis remote storage support) | ||
| if ! command -v ccache &> /dev/null; then | ||
| apt-get update && apt-get install -y --no-install-recommends ccache | ||
| NVTE_CCACHE_BIN="${NVTE_CCACHE_BIN:-ccache}" |
There was a problem hiding this comment.
If we want to use sscache, then we need the install script
| export NVTE_USE_CCACHE=1 | ||
| ccache --zero-stats | ||
| export NVTE_CCACHE_BIN | ||
| case "${CACHE_PROGRAM}" in |
There was a problem hiding this comment.
Here we're initializing the compiler-cache backend.
For sccache, we explicitly start the daemon, disable its idle shutdown for the
duration of the long CUDA build (line 216)
| @@ -0,0 +1,44 @@ | |||
| #!/usr/bin/env bash | |||
There was a problem hiding this comment.
build-te-with-sccache.sh is a thin BuildKit-specific wrapper around
build-te.sh. It keeps Docker secret handling separate from the normal
Transformer Engine build logic.
When ENABLE_NVTE_SCCACHE=1, the wrapper:
- Verifies that the temporary AWS and S3 BuildKit secret files are present.
- Reads them without command tracing and exports them only to the current
process tree. - Configures sccache to use TLS and an architecture-specific S3 prefix.
- Adds
--ccacheto thebuild-te.shinvocation.
olupton
left a comment
olupton
left a comment
There was a problem hiding this comment.
I don't see a CI run from the latest diff? It seems like a lot of code and branches. Do we really need to support both ccache and sccache?
| needs: build-base | ||
| runs-on: ${{ inputs.ARCHITECTURE == 'amd64' && 'linux-amd64-cpu32m' || 'linux-arm64-cpu32m' }} | ||
| env: | ||
| CACHE_AWS_REGION: ${{ vars.AWS_REGION || secrets.AWS_REGION }} |
There was a problem hiding this comment.
Old test, going to remove this bit
| id: build-jax | ||
| uses: ./.github/actions/build-container | ||
| env: | ||
| SCCACHE_BUCKET: ${{ secrets.BAZEL_REMOTE_BUCKET }} |
There was a problem hiding this comment.
Why 4 entries in mealkit-secret-envs and 1 here?
There was a problem hiding this comment.
The env creates and env variable for build-jax, and it does not need the AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN, as the OIDC action that we have is already automatically providing these variables.
The mealkit-secret-envs is an explicit list of which existing runner environment variables we have that may cross into the Docker build. Since then we have Docker running, we do nee dto pass thes evariable somehow,and this is the only way I found.
| @@ -0,0 +1,160 @@ | |||
| #!/bin/bash | |||
There was a problem hiding this comment.
0.17.0 is released, can this all be dropped?
|
/ok to test 95ab9f9 |
2 participants
No description provided.