Skip to content

Add golden store publish scripts + nightly CI wiring - #703

Draft
yuhuchua-qti wants to merge 8 commits into
dev/yuhuchua/migrate-clip-utfrom
dev/yuhuchua/qnn-ut-golden-publish
Draft

Add golden store publish scripts + nightly CI wiring#703
yuhuchua-qti wants to merge 8 commits into
dev/yuhuchua/migrate-clip-utfrom
dev/yuhuchua/qnn-ut-golden-publish

Conversation

@yuhuchua-qti

Copy link
Copy Markdown
Collaborator

Summary

Stacked on PR1 (migrate-clip-ut, #399). Delivers the producer side of the QNN EP unit-test golden pipeline: scripts that regenerate goldens, keep only accuracy-passing op groups, version-stamp them, and publish to Artifactory — plus the nightly CI job that runs it and the retention policy that prunes old archives.

Pure additive. No production op-builder / snapshot / accuracy source is touched. The consumer (per-PR accuracy-routing gate) is a separate later PR.

What's in it

Version resolver — single source of truth

  • qcom/scripts/linux/resolve_tool_versions.sh — source-able and executable. Resolves QAIRT version (from sdk.yaml across QAIRT_SDK_ROOT/QNN_SDK_ROOT/SNPE_ROOT) and ORT runtime version (from $ORT_PREBUILT_ROOT/VERSION_NUMBER, then /VERSION). Deliberately does not read repo-root VERSION_NUMBER (that's the plugin/wheel version, not the ORT runtime). Undeterminable → exit 3 (graceful), so callers treat it as a safe signal. Writer and future gate stay consistent by sharing this code, never a process.

Publisher — thin orchestrator

  • qcom/scripts/linux/publish_goldens.sh — regen goldens via run_snapshot_accuracy.sh --generate-goldens → read the accuracy report → package only groups whose every QnnUnit_Accuracy_<Group>Test case PASSED → write a version-stamped manifest.json → upload a write-once archive/<date>-<sha>/ and the mutable latest/ pointer. Default is dry-run; --publish is required to actually upload (it overwrites the shared latest/ pointer — never implicit).
  • qcom/scripts/linux/filter_accuracy_pass_groups.py — extracted pass-group selection (a group passes iff all its accuracy cases COMPLETED with no failures).

Nightly CI wiring

  • .github/workflows/qualcomm-internal-golden-publish.yml — reusable workflow (workflow_call + workflow_dispatch). Builds the coverage-instrumented binary, then runs publish_goldens.sh. dry_run defaults true so manual dispatch never clobbers the shared store; the scheduled producer opts into the real upload explicitly.
  • .github/workflows/qualcomm-internal-release-nightly.yml — adds a golden-publish job (9am UTC cron) passing dry_run: false.
  • qcom/build_and_test.py — new coverage_build_linux_x86_64 task (coverage build only, no HTML report — snapshot/accuracy tiers only compile under --enable-coverage).

Golden store layout + retention

  • Store lives under qnn-ep-test-store/ut-snapshot/{archive,latest}/. The qnn-ep-test-store/ parent is shared with future siblings (e.g. modelzoo goldens).
  • qcom/scripts/artifactory/aql/expired_golden_artifacts.json — AQL that matches qnn-ep-test-store/*/archive/* older than 10d. Retention keys off the archive/ segment only, so latest/ pointers (and any permanent data outside archive/) are never matched — retention can never leave a store without a current golden.
  • .github/workflows/qualcomm-internal-clean-artifacts.yml — new "Delete old golden archives" step (runs on the existing daily cleanup cron, 1h before the publish cron).

Test plan

  • pytest qcom/scripts/linux/tests — 17 tests pass (resolver priority order, sdk.yaml variants, undeterminable → exit 3).
  • publish_goldens.sh dry-run end-to-end against a real coverage build-dir: regen → parse accuracy report → stage passing groups → build goldens.zip + manifest.json → prints the two jf rt upload commands without executing.
  • manifest.json qairt_version/ort_version match resolve_tool_versions.sh both; git_sha matches git rev-parse --short=10 HEAD.
  • Empty-pass guard: zero passing groups → die, no zip/upload.
  • Live Artifactory upload verified via nightly (post-merge).

🤖 Generated with Claude Code

yuhuchua-qti and others added 6 commits August 12, 2026 14:45
…blishing

PR2 of the QNN EP UT migration stack. Adds:
- resolve_tool_versions.sh: single source of truth for QAIRT + ORT runtime
  version strings (file-only detection, no env override). Dual-mode
  (sourceable + executable). Shared by the manifest writer and the future
  accuracy gate reader.
- publish_goldens.sh: thin orchestrator that regenerates goldens, selects
  only accuracy-PASSING op groups, packages them with a version-stamped
  manifest.json, and uploads to Artifactory (default dry-run).
- pytest suite (17 tests) covering version resolution precedence, sdk.yaml
  parsing variants, undeterminable exit codes, and edge cases.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Follows the PR1 rename of --update-goldens to --generate-goldens in
run_snapshot_accuracy.sh.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Move the pass-group selection logic from an inline bash heredoc in
publish_goldens.sh into filter_accuracy_pass_groups.py. This makes
the classification logic independently testable and reusable by the
future accuracy gate (PR4).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds the CI half of PR2 (merged PR2+PR3): a scheduled job that runs
publish_goldens.sh to refresh the Artifactory golden store nightly.

- qualcomm-internal-golden-publish.yml: reusable workflow (workflow_call +
  workflow_dispatch) that builds a coverage-instrumented binary, then runs
  publish_goldens.sh. Inputs: filter, repo_subpath, skip_regen, dry_run.
  dry_run defaults true so a manual dispatch never clobbers the shared store
  by accident; the scheduled producer opts into the real upload explicitly.
- qualcomm-internal-release-nightly.yml: adds a parallel golden-publish job
  on the 9am cron, passing dry_run=false.
- build_and_test.py: adds coverage_build_linux_x86_64 (coverage build only,
  no HTML report). publish_goldens.sh does its own golden regen + accuracy
  verify, so the job needs the instrumented binary but not the report,
  avoiding a wasted second test run.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Golden stores live under qnn-ep-test-store/, grouped by subject:
ut-snapshot/ (this suite) and, later, modelzoo/. Retention keys off the
path segment, not the top folder:

  qnn-ep-test-store/<subject>/latest/   -> never pruned (gates read it)
  qnn-ep-test-store/<subject>/archive/  -> pruned after 10d

- expired_golden_artifacts.json: AQL matching qnn-ep-test-store/*/archive/*
  created before 10d. latest/ and any permanent data (no archive/ segment)
  are never matched, so retention can't strand a store without a golden.
- qualcomm-internal-clean-artifacts.yml: new "Delete old golden archives"
  step on the existing nightly cleanup cron.
- publish_goldens.sh / golden-publish.yml: repo_subpath default moves from
  qnn-ut-goldens to qnn-ep-test-store/ut-snapshot.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
PR1 renamed accuracy suites from QnnUnit_Accuracy_<Group>Test to
QnnUnit_<Group>_Accuracy[_<Variant>]Test. The old regex no longer
matched any suite, so publish_goldens.sh would always find zero
passing groups. Update the regex and its two doc comments to match
the new naming; verified against a synthetic gtest JSON report.
@yuhuchua-qti
yuhuchua-qti force-pushed the dev/yuhuchua/qnn-ut-golden-publish branch from 04feff2 to 455fe84 Compare August 12, 2026 07:03
yuhuchua-qti and others added 2 commits August 12, 2026 16:33
Temporary wrapper: on push to the PR2 branch, run golden-publish in
dry-run (nothing touches Artifactory) and upload the packaged zip as a
downloadable artifact. Passes dry_run + filter explicitly because a push
event has no inputs context. DELETE THIS COMMIT BEFORE MERGE.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The wrapper called golden-publish.yml (which requests checks: write) with
default token permissions, causing a startup_failure. Match the reusable
workflow's permissions. DELETE WITH THE OTHER TEMP COMMIT BEFORE MERGE.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant