Add golden store publish scripts + nightly CI wiring - #703
Draft
yuhuchua-qti wants to merge 8 commits into
Draft
Conversation
…blishing PR2 of the QNN EP UT migration stack. Adds: - resolve_tool_versions.sh: single source of truth for QAIRT + ORT runtime version strings (file-only detection, no env override). Dual-mode (sourceable + executable). Shared by the manifest writer and the future accuracy gate reader. - publish_goldens.sh: thin orchestrator that regenerates goldens, selects only accuracy-PASSING op groups, packages them with a version-stamped manifest.json, and uploads to Artifactory (default dry-run). - pytest suite (17 tests) covering version resolution precedence, sdk.yaml parsing variants, undeterminable exit codes, and edge cases. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Follows the PR1 rename of --update-goldens to --generate-goldens in run_snapshot_accuracy.sh. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Move the pass-group selection logic from an inline bash heredoc in publish_goldens.sh into filter_accuracy_pass_groups.py. This makes the classification logic independently testable and reusable by the future accuracy gate (PR4). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds the CI half of PR2 (merged PR2+PR3): a scheduled job that runs publish_goldens.sh to refresh the Artifactory golden store nightly. - qualcomm-internal-golden-publish.yml: reusable workflow (workflow_call + workflow_dispatch) that builds a coverage-instrumented binary, then runs publish_goldens.sh. Inputs: filter, repo_subpath, skip_regen, dry_run. dry_run defaults true so a manual dispatch never clobbers the shared store by accident; the scheduled producer opts into the real upload explicitly. - qualcomm-internal-release-nightly.yml: adds a parallel golden-publish job on the 9am cron, passing dry_run=false. - build_and_test.py: adds coverage_build_linux_x86_64 (coverage build only, no HTML report). publish_goldens.sh does its own golden regen + accuracy verify, so the job needs the instrumented binary but not the report, avoiding a wasted second test run. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Golden stores live under qnn-ep-test-store/, grouped by subject: ut-snapshot/ (this suite) and, later, modelzoo/. Retention keys off the path segment, not the top folder: qnn-ep-test-store/<subject>/latest/ -> never pruned (gates read it) qnn-ep-test-store/<subject>/archive/ -> pruned after 10d - expired_golden_artifacts.json: AQL matching qnn-ep-test-store/*/archive/* created before 10d. latest/ and any permanent data (no archive/ segment) are never matched, so retention can't strand a store without a golden. - qualcomm-internal-clean-artifacts.yml: new "Delete old golden archives" step on the existing nightly cleanup cron. - publish_goldens.sh / golden-publish.yml: repo_subpath default moves from qnn-ut-goldens to qnn-ep-test-store/ut-snapshot. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
PR1 renamed accuracy suites from QnnUnit_Accuracy_<Group>Test to QnnUnit_<Group>_Accuracy[_<Variant>]Test. The old regex no longer matched any suite, so publish_goldens.sh would always find zero passing groups. Update the regex and its two doc comments to match the new naming; verified against a synthetic gtest JSON report.
yuhuchua-qti
force-pushed
the
dev/yuhuchua/qnn-ut-golden-publish
branch
from
August 12, 2026 07:03
04feff2 to
455fe84
Compare
Temporary wrapper: on push to the PR2 branch, run golden-publish in dry-run (nothing touches Artifactory) and upload the packaged zip as a downloadable artifact. Passes dry_run + filter explicitly because a push event has no inputs context. DELETE THIS COMMIT BEFORE MERGE. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The wrapper called golden-publish.yml (which requests checks: write) with default token permissions, causing a startup_failure. Match the reusable workflow's permissions. DELETE WITH THE OTHER TEMP COMMIT BEFORE MERGE. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Stacked on PR1 (
migrate-clip-ut, #399). Delivers the producer side of the QNN EP unit-test golden pipeline: scripts that regenerate goldens, keep only accuracy-passing op groups, version-stamp them, and publish to Artifactory — plus the nightly CI job that runs it and the retention policy that prunes old archives.Pure additive. No production op-builder / snapshot / accuracy source is touched. The consumer (per-PR accuracy-routing gate) is a separate later PR.
What's in it
Version resolver — single source of truth
qcom/scripts/linux/resolve_tool_versions.sh— source-able and executable. Resolves QAIRT version (fromsdk.yamlacrossQAIRT_SDK_ROOT/QNN_SDK_ROOT/SNPE_ROOT) and ORT runtime version (from$ORT_PREBUILT_ROOT/VERSION_NUMBER, then/VERSION). Deliberately does not read repo-rootVERSION_NUMBER(that's the plugin/wheel version, not the ORT runtime). Undeterminable → exit 3 (graceful), so callers treat it as a safe signal. Writer and future gate stay consistent by sharing this code, never a process.Publisher — thin orchestrator
qcom/scripts/linux/publish_goldens.sh— regen goldens viarun_snapshot_accuracy.sh --generate-goldens→ read the accuracy report → package only groups whose everyQnnUnit_Accuracy_<Group>Testcase PASSED → write a version-stampedmanifest.json→ upload a write-oncearchive/<date>-<sha>/and the mutablelatest/pointer. Default is dry-run;--publishis required to actually upload (it overwrites the sharedlatest/pointer — never implicit).qcom/scripts/linux/filter_accuracy_pass_groups.py— extracted pass-group selection (a group passes iff all its accuracy cases COMPLETED with no failures).Nightly CI wiring
.github/workflows/qualcomm-internal-golden-publish.yml— reusable workflow (workflow_call+workflow_dispatch). Builds the coverage-instrumented binary, then runspublish_goldens.sh.dry_rundefaults true so manual dispatch never clobbers the shared store; the scheduled producer opts into the real upload explicitly..github/workflows/qualcomm-internal-release-nightly.yml— adds agolden-publishjob (9am UTC cron) passingdry_run: false.qcom/build_and_test.py— newcoverage_build_linux_x86_64task (coverage build only, no HTML report — snapshot/accuracy tiers only compile under--enable-coverage).Golden store layout + retention
qnn-ep-test-store/ut-snapshot/{archive,latest}/. Theqnn-ep-test-store/parent is shared with future siblings (e.g. modelzoo goldens).qcom/scripts/artifactory/aql/expired_golden_artifacts.json— AQL that matchesqnn-ep-test-store/*/archive/*older than 10d. Retention keys off thearchive/segment only, solatest/pointers (and any permanent data outsidearchive/) are never matched — retention can never leave a store without a current golden..github/workflows/qualcomm-internal-clean-artifacts.yml— new "Delete old golden archives" step (runs on the existing daily cleanup cron, 1h before the publish cron).Test plan
pytest qcom/scripts/linux/tests— 17 tests pass (resolver priority order, sdk.yaml variants, undeterminable → exit 3).publish_goldens.shdry-run end-to-end against a real coverage build-dir: regen → parse accuracy report → stage passing groups → buildgoldens.zip+manifest.json→ prints the twojf rt uploadcommands without executing.manifest.jsonqairt_version/ort_versionmatchresolve_tool_versions.sh both;git_shamatchesgit rev-parse --short=10 HEAD.🤖 Generated with Claude Code