pytorch主干nightly集成验证 - #141
Open
kerer-ai wants to merge 43 commits into
Open
Conversation
…h upstream - Add builder Dockerfiles (manylinux2_28, x86_64/aarch64) for torch_npu wheel builds - Add test Dockerfiles (ubuntu:22.04 + Miniforge3 conda py_3.10, aligned with upstream CUDA 13.0 image) - Switch test images from system Python to conda + named env pattern - Align requirements-test.txt with upstream requirements-ci.txt (py3.10/jammy profile) - Switch to PyTorch nightly index for latest daily builds - Add docker_build.sh with explicit case-statement tag mapping - Add .ci/pytorch/ build scripts (common.sh, build_pytorch.sh, build_torch_npu.sh, build.sh, integration_verify.sh) - Update _build.yml with real build commands replacing simulated placeholders - Add build-docker-images.yml workflow: PR/push triggers build+push all 8 images, aarch64 on ARM native runners Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…function GHA expressions do not support 'split'. matrix-prep now outputs a JSON array directly, consumed by fromJSON() in the build matrix. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…6_64 - Remove builder Dockerfiles and requirements-builder.txt - Remove test/Dockerfile.x86_64 - Simplify docker_build.sh to only aarch64 A2/A3 - Simplify build-docker-images.yml to aarch64-only, ubuntu-24.04-arm runner - Update _build.yml default image tag to aarch64 - Update README Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Docker image no longer installs PyTorch (built separately at CI runtime) - Move torchvision, torch_geometric, torch-scatter to requirements-post.txt - requirements-post.txt installed after PyTorch + torch_npu are built - Add post-build dependency step to build.sh and _build.yml - Dockerfile now only installs base requirements (no torch needed) Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Replace generic DOCKER_REGISTRY push with quay.io/kerer/pytorch using docker/login-action@v3. Upload each built image path as an artifact, and add a summary job that prints image names and docker pull commands in a Markdown table via GITHUB_STEP_SUMMARY. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Add test-build-trigger.yml (PR trigger on master) and _build.yml (reusable workflow) that checkout upstream PyTorch main HEAD, build from source, then checkout downstream torch_npu master and verify both packages import successfully. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
CLA Signature Passkerer-ai, thanks for your pull request. All authors of the commits have signed the CLA. 👍 |
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
CLA Signature Passkerer-ai, thanks for your pull request. All authors of the commits have signed the CLA. 👍 |
- Add ccache to accelerate C/C++ compilation across runs - Add pip cache via actions/cache@v4 for faster dependency installs - Add BUILD_WITHOUT_SHA=1 env for more cacheable builds - Capture build logs to /tmp and upload as artifacts for debugging - Add proper exit code handling in build steps - Enhance build summary with ccache statistics Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
CLA Signature Passkerer-ai, thanks for your pull request. All authors of the commits have signed the CLA. 👍 |
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
CLA Signature Passkerer-ai, thanks for your pull request. All authors of the commits have signed the CLA. 👍 |
…erfile - fetch-depth: 0 prevents shallow clone conflicts with recursive submodules - Install ccache in Dockerfile so future images have it pre-installed Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
CLA Signature Passkerer-ai, thanks for your pull request. All authors of the commits have signed the CLA. 👍 |
Setting CC="ccache gcc" causes ccache to be invoked directly when CMake builds .S assembly files (qnnpack confu loses the "gcc" part), making ccache reject -D preprocessor flags as invalid options. CMAKE_C_COMPILER_LAUNCHER tells CMake to prefix the compiler natively, which correctly handles C, C++, and assembly compilation. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
CLA Signature Passkerer-ai, thanks for your pull request. All authors of the commits have signed the CLA. 👍 |
When the source-built PyTorch is newer than what torch_npu master expects, the code generator adds TORCH_FEATURE_VERSION guards to C shim headers that differ from checked-in files, causing a RuntimeError. Run codegen with --update_aoti_c_shim before ci/build.sh to sync headers first. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
CLA Signature Passkerer-ai, thanks for your pull request. All authors of the commits have signed the CLA. 👍 |
When the op-plugin submodule lacks the config directory for the source-built PyTorch version (e.g. no v2r13 for 2.13.0a0), fall back to the latest available config and symlink it so ci/build.sh can also resolve the expected path. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
CLA Signature Passkerer-ai, thanks for your pull request. All authors of the commits have signed the CLA. 👍 |
The manual gen_backend_stubs invocation required constructing version- specific config paths (v2r13 etc.) which break when the op-plugin submodule structure changes. Instead, patch generate_code.sh in-place with sed to add the --update_aoti_c_shim flag, then let ci/build.sh run the full codegen pipeline which handles all version mapping correctly. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
CLA Signature Passkerer-ai, thanks for your pull request. All authors of the commits have signed the CLA. 👍 |
Don't touch source code in CI. Let the build fail naturally with full error output for diagnosis. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
CLA Signature Passkerer-ai, thanks for your pull request. All authors of the commits have signed the CLA. 👍 |
Allow parallel runs during testing phase. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
CLA Signature Passkerer-ai, thanks for your pull request. All authors of the commits have signed the CLA. 👍 |
pip wheel + pip install puts torch in site-packages, but Python in the pytorch/ source dir imports the local torch/ directory (which contains source-only _C folder, not compiled .so). This fails with: "Failed to load PyTorch C extensions" Fix: cd /tmp before import, then cd back. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
CLA Signature Passkerer-ai, thanks for your pull request. All authors of the commits have signed the CLA. 👍 |
Cache restore (ccache 5.5GB) took ~7min via network, while full recompile takes <5min. Cache save/upload also cost another ~8min post-job. Remove: pip cache, ccache cache, ccache install, ccache configuration in both build steps, ccache cleanup, and ccache summary output. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
CLA Signature Passkerer-ai, thanks for your pull request. All authors of the commits have signed the CLA. 👍 |
Include the dynamically generated AOTI C shim header in the build artifact upload for debugging AOTInductor compatibility issues. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
CLA Signature Passkerer-ai, thanks for your pull request. All authors of the commits have signed the CLA. 👍 |
Now that c_shim_npu.h is kept in sync with the repo, the CI runs in validation mode — it regenerates the header and fails if it differs from the checked-in version, instead of silently overwriting it. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
CLA Signature Passkerer-ai, thanks for your pull request. All authors of the commits have signed the CLA. 👍 |
Update the checked-in AOTI C shim header to match the version generated by the current PyTorch nightly's torchgen fallback ops list. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
CLA Signature Passkerer-ai, thanks for your pull request. All authors of the commits have signed the CLA. 👍 |
Sync with latest upstream Ascend/pytorch master.
CLA Signature Passkerer-ai, thanks for your pull request. All authors of the commits have signed the CLA. 👍 |
CLA Signature Passkerer-ai, thanks for your pull request. All authors of the commits have signed the CLA. 👍 |
…lity PyTorch PR #173895 hard-removed the deprecated named tensor feature, including Dimname type, NamedTensorUtils, and namedinference APIs. This caused torch_npu codegen to fail with "KeyError: Dimname" when building against PyTorch 2.13.0a0. Remove all Dimname/NamedTensor references across 15 files: - npu_native_functions.yaml: drop .names operator overloads - AutoCastOps.cpp: drop dim_Dimname/dim_DimnameList kernel registrations - C++ source files: remove propagate_names/namedinference calls - Headers/templates: remove #include <ATen/NamedTensorUtils.h> and using Dimname Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
CLA Signature Passkerer-ai, thanks for your pull request. All authors of the commits have signed the CLA. 👍 |
CLA Signature Passkerer-ai, thanks for your pull request. All authors of the commits have signed the CLA. 👍 |
CLA Signature Passkerer-ai, thanks for your pull request. All authors of the commits have signed the CLA. 👍 |
…ences Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
CLA Signature Passkerer-ai, thanks for your pull request. All authors of the commits have signed the CLA. 👍 |
…cker images - Align requirements-test.txt with upstream PyTorch requirements-ci.txt: - pillow: 12.2.0 -> 12.3.0 - pip: 26.0.1 -> 26.1.2 - aiohttp: 3.13.4 -> 3.14.3 - typing-extensions: 4.15.0 -> 4.12.2 - librosa: 0.10.2 -> >=0.6.2 (match upstream py3.10 unbounded) - transformers: 4.40.0 -> 4.36.2 (match upstream install_onnx.sh) - Remove unused deps not referenced in torch-npu, upstream pytorch, or nightly CI: beartype, attrs, decorator, importlib_metadata, zstandard - Add requirements-test-annotated.md with detailed dependency documentation - Add timestamp suffix (YYYYMMDDHHMM) to docker image tags in docker_build.sh - Update build-docker-images.yml to use docker_build.sh for timestamped tags
CLA Signature Passkerer-ai, thanks for your pull request. All authors of the commits have signed the CLA. 👍 |
docker_build.sh passes the full name (with timestamp) as --tag without
a colon, so Docker treats it as repository name with tag 'latest'.
The previous detection logic used --format '{{.Repository}}:{{.Tag}}'
and grepped for [0-9]{12}$ which failed because the output ended with
':latest' instead of the timestamp.
Fix by matching on repository name only, and use the timestamped repo
name as the remote tag.
CLA Signature Passkerer-ai, thanks for your pull request. All authors of the commits have signed the CLA. 👍 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.