Repository navigation
Ship prebuilt static-library tarballs as release assets - #2444
Merged
Merged
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #2444 +/- ##
=======================================
Coverage 85.36% 85.37%
=======================================
Files 126 126
Lines 22862 22868 +6
=======================================
+ Hits 19516 19523 +7
+ Misses 3346 3345 -1 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
ronaldtse
force-pushed
the
prebuilt-release-assets
branch
12 times, most recently
from
July 29, 2026 05:52
5deb062 to
38d8424
Compare
4 tasks
ronaldtse
force-pushed
the
prebuilt-release-assets
branch
from
August 1, 2026 08:24
3a8c919 to
62283f2
Compare
ni4
approved these changes
Aug 4, 2026
ni4
left a comment
Contributor
There was a problem hiding this comment.
LGTM as merge conflicts/CI fixed!
Contributor
Author
|
Merging with 1 approval (ni4). CI/release infrastructure — builds and ships prebuilt static-library tarballs as release assets. Thank you @ni4 for the review! |
Adds per-target static-library tarballs of librnp + its full dependency stack (sexpp, Botan or OpenSSL, json-c, zlib, bzip2) as additional GitHub release assets, so downstream consumers that cannot reach a system package manager (Cargo vendored mode, embedded cross-compile targets, Windows app vendors, CI pinning a specific librnp version) can link against librnp without compiling the full C++ + Botan stack from source. Closes #2443. == What ships 14 tarballs per release (7 target triples x 2 backends). Each tarball contains librnp.a plus all transitive static dependencies, the public FFI headers, CMake config, pkg-config files, and a MANIFEST.txt with link instructions. A .sha256 sidecar and optional GPG .asc detached signature accompany each tarball. Targets: x86_64/aarch64 Linux glibc, x86_64/aarch64 Linux musl (via alpine container), x86_64/aarch64 macOS, x86_64 Windows MSVC. == Architecture Data-driven: backend and target properties live in config files (ci/backends/<name>.env, ci/targets/<triple>.env) that are the single source of truth for both the build scripts and the validator. Adding a backend or target = adding one config file. The build is split across two scripts: * ci/build_prebuilt.sh -- Linux glibc, Linux musl, macOS. Downloads pinned dep versions, builds them as static libraries, builds librnp against them, stages into a tarball. * ci/build_prebuilt.ps1 -- Windows MSVC via vcpkg's x64-windows-static triplet. ci/test_prebuilt.sh validates every tarball against docs/specs/tarball-layout.adoc: naming, layout, required headers and archives, pkg-config path integrity, MANIFEST.txt consistency, and an end-to-end compile+link+run that calls rnp_ffi_create(). == Workflow design The prebuild matrix lives in .github/workflows/prebuilt.yml, a reusable workflow with three triggers: * pull_request -- path-filtered to prebuilt scripts; uploads tarballs as workflow artifacts for reviewer download. * workflow_call -- called from release.yml after the GitHub release exists; attaches tarballs to the release via softprops/action-gh-release. * workflow_dispatch -- manual run against an existing tag. release.yml calls prebuilt.yml via workflow_call with the release tag. The prebuild job depends on the release job so the source tarball is created first. == Specs * docs/specs/tarball-layout.adoc -- the contract every tarball must satisfy. test_prebuilt.sh mechanically asserts it. * docs/specs/target-matrix.adoc -- tier definitions, support window, add/deprecate policy. * docs/specs/backends.adoc -- backend requirements, config-file contract, add/deprecate policy. * docs/packaging.adoc -- consumer-facing docs with asset naming, tarball layout, linking instructions, and reproducibility notes. == Other changes * ci/botan3-modules, ci/botan3-pqc-modules: renamed curve25519 to x25519 (Botan 3.0 rename) and added per-curve pcurves_* modules that rnp's CMake feature detection requires on Botan 3.7+. * ci/lib/sign.sh: shared GPG signing helper. * ci/prebuilt-versions.env: pinned dep versions (single source of truth for the source-based build path). * ci/check_reproducibility.sh: informational diff tool (wired as a PR-only CI job that always exits 0).
- _typos.toml: allow "Sur" (macOS Big Sur proper noun) - ci/build_prebuilt.sh: use find instead of ls for backend/target listing (shellcheck SC2012) - cmake/Modules/FindOpenSSLFeatures.cmake: link Crypt32/Ws2_32/ Advapi32/User32 into the findopensslfeatures probe when building against static OpenSSL on MSVC. Without these, libcrypto.lib's references to CertOpenStore / socket BIOs / etc. leave 38 unresolved externals and the probe fails to link. - ci/build_prebuilt.ps1: drop the CMAKE_EXE_LINKER_FLAGS workaround that didn't actually propagate into the sub-CMake build; the fix above replaces it.
The ludeeus/action-shellcheck action runs with SHELLCHECK_OPTS=-x
(--external-sources), which makes shellcheck try to follow every
source statement. The runtime-computed paths ($BACKEND_CONFIG,
$TARGET_CONFIG, ${SCRIPT_DIR}/backends/${BACKEND}.env, etc.) can't
be resolved at lint time, producing SC1090/SC1091 notes that the
action treats as failures.
Replace the # shellcheck source=... directives (which asked shellcheck
to follow a constant path it couldn't find with explicit
disable=SC1090 (non-constant source) or disable=SC1091 (unresolvable
SCRIPT_DIR-relative path) so the lint is honest about what it can
and can't statically verify.
EOF
)
MSVC doesn't ship <getopt.h> or <dirent.h>; rnp's src/common and src/rnp need them (via the uniwin.h / getoptwin.h shims). The existing windows-native.yml CI already installs both ports via vcpkg — do the same in the prebuilt Windows build so CMake can resolve GETOPT_INCLUDE_DIR and DIRENT_INCLUDE_DIR instead of aborting at configure time with "variables set to NOTFOUND".
The MSYS2 tar that Git for Windows puts first in PATH fails with "Cannot connect to D: resolve failed" when given a Windows-style output path like D:\a\rnp\rnp\artifacts\.... It expects POSIX paths (/d/a/rnp/...) and the cygpath round-trip is unreliable. Use C:\Windows\System32\tar.exe (Windows native, preinstalled on Windows 10 1803+ / Server 2019+, available on windows-latest runners). It handles Windows paths directly and produces the same .tar.gz the Linux/macOS build does.
RNP_ARTIFACT_DIR on Windows resolves to a Windows-style path like D:\a\rnp\rnp/artifacts inside the bash validation step. The MSYS2 tar that test_prebuilt.sh invokes cannot open that path (it expects POSIX /d/a/rnp/rnp/artifacts and fails with Cannot connect to D: resolve failed). Convert at the workflow boundary with cygpath -u so the script sees a path it can open.
vcpkg's x64-windows-static triplet ships libraries under different names than the conventional ones used on Unix: zlib zlib.lib -> zs.lib bzip2 libbz2.lib -> bz2.lib json-c libjson-c -> json-c.lib (no leading lib) botan libbotan-3 -> botan.lib (no -3 suffix on Windows) Update BACKEND_WINDOWS_LINK_LIBS and BACKEND_WINDOWS_LIB_NAMES in both backend configs so the validation step finds the archives that vcpkg actually produces.
test_prebuilt.sh always read BACKEND_LIB_NAMES (the Unix naming) even on Windows, so the validation kept failing on lib/z.lib even after aligning BACKEND_WINDOWS_LIB_NAMES with vcpkg naming. Pick the right list based on TARGET_OS so the structural check matches the actual vcpkg x64-windows-static output.
ec.cpp and exdsa_ecdhkem.cpp call Botan::EC_Group::from_name() but only include botan/ecdh.h, which forward-declares EC_Group via ecc_key.h without the full class definition. On Linux and macOS the source-built Botan transitively pulls in ec_group.h through some header chain, but vcpkg's x64-windows-static botan port does not, causing ec.cpp to fail with 'use of undefined type Botan::EC_Group'. Include the header explicitly in both files for the class they actually use, instead of relying on transitive includes that vary between Botan builds.
vcpkg's x64-windows-static triplet compiles every dep (botan, json-c, zlib, bzip2, etc.) with /MT (static MSVC runtime). The rnp build was defaulting to /MD (dynamic runtime), which caused LNK2038 mismatches when linking the example and CLI binaries against botan-3.lib (Botan's allocator touches runtime internals and the linker flags the discrepancy). OpenSSL did not surface the issue because its API does not directly reference runtime-internal types. Pass CMAKE_MSVC_RUNTIME_LIBRARY=MultiThreaded so the entire bundle is consistently /MT, matching what consumers of the static bundle are expected to use anyway.
actions/checkout v4, actions/cache v4, actions/upload-artifact v4, and softprops/action-gh-release v2 all ship with Node.js 20, which GitHub Actions is deprecating. Upgrade to the latest stable majors that bundle Node.js 24: actions/checkout v4 -> v7 actions/cache v4 -> v6 actions/upload-artifact v4 -> v7 softprops/action-gh-release v2 -> v3 ilammy/msvc-dev-cmd@v1 is already on the v1 major line (v1.13.0) and does not have a Node 20 problem, so it stays.
GitHub Actions is deprecating Node.js 20 on runners. Several actions used across our workflows still ship with the Node.js 20 runtime, producing deprecation warnings on every run. Upgrade to the latest stable major for each: actions/checkout v4 -> v7 actions/cache v4 -> v6 actions/upload-artifact v4 -> v7 actions/download-artifact v4 -> v8 peter-evans/repository-dispatch v2 -> v4 DoozyX/clang-format-lint-action v0.18.2 -> v0.20 Actions already on their latest major tag (codeql-action v3, msys2/setup-msys2 v2, install-nix-action v31, coverity-scan v1) and actions that track @master (typos, shellcheck, oss-fuzz) are left unchanged.
Supports the CI dep-fan-out orchestration (time-machine.yml and
centos-and-fedora.yml already use this needs: + upload/download-
artifact pattern for their build → test fan-out).
Three invocation modes:
build_prebuilt.sh deps <target> [workdir]
Phase 1: build zlib + bzip2 + json-c into \$WORK/prefix and
stop. The CI job uploads that prefix as an artifact so both
per-backend rnp jobs reuse it.
build_prebuilt.sh rnp <backend> <version> <target> [workdir]
Phase 2: expects deps already in \$WORK/prefix. Builds the
backend (botan/openssl), then rnp, then packages.
build_prebuilt.sh <backend> <version> <target> [workdir]
Backward-compat single-pass mode for local testing.
Backend config sourcing is skipped in deps mode since no backend-
specific code is built.
Restructure prebuilt.yml to share common dep builds across backends,
mirroring the pattern already used in centos-and-fedora.yml
(package-source → package → rpm-tests) and time-machine.yml
(build → test).
Three orchestrated phases:
config (1 job)
Single source of truth for the target list. Emits two matrices
via jq: unix_targets (6, for the deps phase) and rnp_matrix
(14, the full target × backend cartesian product). Adding a
target = one line in the targets JSON; both downstream phases
pick it up automatically.
build-deps (6 jobs: Unix targets only)
Builds zlib + bzip2 + json-c into \$WORK/prefix and uploads it
as an artifact (deps-<triple>). Dep cache keyed by target only
(not target+backend) so both backends' rnp jobs benefit.
build-rnp (14 jobs: all targets × backends)
Unix targets download the deps artifact and run
build_prebuilt.sh rnp (deps already in \$prefix — no rebuild).
Windows runs the monolithic PowerShell pass (vcpkg deps are not
cleanly splittable via artifacts).
Net: common deps built 6 times (once per Unix target) instead of 12
(once per Unix target × backend). macOS savings at 10x runner cost.
Reproducibility check moved to release-only (it was informational
anyway and ran on every PR push).
alpine+ARM64 runners refuse to execute JS actions (actions/cache, actions/upload-artifact, actions/download-artifact). The deps fan-out therefore can't share its output across the alpine+ARM64 matrix entries, which failed both the build-deps upload step and the build-rnp download step. Three coordinated changes keep the fan-out working on every other target while degrading gracefully on alpine+ARM64: * build-deps: skip the actions/upload-artifact step for alpine+ARM64 (the actions/cache step already had the same conditional). The job now succeeds on that target, with deps built into the per-runner prefix but not shared. * build-rnp: skip the actions/download-artifact step for alpine+ARM64, matching the build-deps upload skip. * build_prebuilt.sh rnp mode: when \$PREFIX/lib/libz.a is missing, build the common deps inline before the backend + rnp build. This is the same work the single-pass mode does, so the alpine+ARM64 path costs exactly what the pre-fan-out baseline did -- and the fan-out still saves the duplicated build on every other target.
…ount
In container jobs (alpine), \${{ github.workspace }} resolves to the
host path /home/runner/work/rnp/rnp, which is NOT visible from inside
the container -- the container mounts the workspace at /__w/rnp/rnp
instead. The build_prebuilt.sh script was creating directories at
the host path inside the container's ephemeral filesystem, so the
files never landed on the mount point and the upload-artifact step
(reading from the host side) reported 'No files were found'.
Fix: inside bash steps, use \$GITHUB_WORKSPACE (correctly set to
/__w/rnp/rnp in containers, identical to \${{ github.workspace }} in
non-container jobs). The JS actions (cache, upload-artifact,
download-artifact) continue using \${{ github.workspace }} because
they read from the host filesystem, and the container mount makes
the two paths equivalent.
Also move RNP_ARTIFACT_DIR under work/ so the tarballs and deps
share the same parent and the mount mapping is consistent.
GitHub has deprecated macos-13 (Intel) runners -- the deps job for x86_64-apple-darwin sat queued for 20+ hours waiting for a runner that will never come, blocking the entire build-rnp phase. Remove the Intel macOS target entirely: - matrix entry in prebuilt.yml (6 targets now, was 7) - ci/targets/x86_64-apple-darwin.env (orphaned) - Intel rows in docs/specs/target-matrix.adoc and docs/packaging.adoc - tarball count updated 14 -> 12 (6 targets x 2 backends) Intel macOS consumers can use the aarch64-apple-darwin tarball via Rosetta, or build from source. A future universal-binary target (x86_64+aarch64 cross-compiled on macos-14) can be added separately.
rnp uses Botan's C FFI API (botan_block_cipher_*, botan_rng_*, botan_nist_kw_*, etc.) throughout its crypto layer. vcpkg's default botan port omits the ffi module, so the static botan.lib was missing all the botan_* symbols, producing 17 unresolved externals when linking the rnp examples and CLI tools. Install botan[ffi] to enable the module. vcpkg applies the feature to the same x64-windows-static triplet we already use.
…bug)
vcpkg's botan port for the x64-windows-static triplet produces a
.lib whose headers still declare the C FFI symbols
(botan_block_cipher_*, botan_rng_*, botan_nist_kw_*, etc.) with
__declspec(dllimport). When rnp links statically against that
.lib, the MSVC linker prepends __imp_ to every botan_* reference
and fails with 17 unresolved externals (LNK2019 / LNK1120).
The vcpkg botan port has no 'ffi' feature to toggle either
('botan has no feature named ffi'). Fixing this properly needs
either an upstream vcpkg port fix or a from-source Botan build on
Windows -- both are follow-up work. The Windows + openssl tarball
covers the primary Windows use case in the meantime.
Revert the invalid botan[ffi] attempt in build_prebuilt.ps1, filter
the (x86_64-pc-windows-msvc, botan) entry out of the rnp_matrix in
the config job, and document the gap in packaging.adoc. Matrix now
ships 11 tarballs per release (was 12).
ronaldtse
force-pushed
the
prebuilt-release-assets
branch
from
August 4, 2026 15:42
dfa9e67 to
0f28c88
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds prebuilt static-library tarballs to GitHub releases, addressing #2443.
Each tarball bundles
librnp.aplus a curated static build of everytransitive dependency (sexpp, Botan or OpenSSL, json-c, zlib, bzip2), so
downstream consumers that cannot reach a system package manager (Cargo,
embedded cross-compile targets, Windows app vendors) can link against
librnp without compiling the full C++ + Botan stack from source.
The build runs as a new
prebuildmatrix job in.github/workflows/release.ymlthat depends on the existingreleasejob, so the source tarball and GitHub release are created first; the
prebuilt tarballs are then uploaded as additional assets to the same
release.
Targets (v1)
[cols="2,1", options="header"]
|===
| Target | Backend(s)
|
x86_64-unknown-linux-gnu| botan, openssl|
aarch64-unknown-linux-gnu| botan, openssl|
x86_64-apple-darwin| botan, openssl|
aarch64-apple-darwin| botan, openssl|===
8 tarballs per release. Windows MSVC (
x86_64-pc-windows-msvc) andLinux musl (
x86_64-unknown-linux-musl,aarch64-unknown-linux-musl)are tracked as follow-up — the script's
detect_targetalreadyrecognizes them, but the dependency build steps need platform-specific
work that's out of scope for this PR.
What's in each tarball
What changed
ci/build_prebuilt.sh(new)The static-build orchestrator. Downloads pinned versions of each
dependency, builds them as static libraries into a temporary prefix,
builds librnp against that prefix, and stages the resulting headers and
archives into the tarball layout shown above. Outputs:
rnp-v<version>-<target>-<backend>.tar.gzrnp-v<version>-<target>-<backend>.sha256Dependency versions are pinned in the script for reproducibility:
ci/botan3-modules,ci/botan3-pqc-modulesRenamed
curve25519tox25519. Botan 3 renamed this module; theexisting files were broken for any Botan 3.x build using them.
ci/botan-modules(used bycodeql.ymlagainst Botan 2.17.3) isunchanged.
.github/workflows/release.ymlAdded
prebuildmatrix job with 8 entries (4 targets x 2 backends).Each job:
ci/build_prebuilt.shrnp-*.tar.gz+rnp-*.sha256to the release viagh release upload --clobberThe job depends on
releaseso it only fires after the GitHub releaseexists.
fail-fast: falseensures one target failing doesn't kill theothers.
docs/packaging.adocNew "Prebuilt static libraries (release assets)" section documenting
asset naming, available targets, tarball layout, and consumption
patterns (CMake + raw compiler flags). Cross-references the existing
sections on
find_packageand pkg-config.Validation
ci/build_prebuilt.shend-to-end locally on macOS arm64with the botan backend: builds zlib, bzip2, json-c, Botan 3.10.0
(minimized build with PQC modules), and rnp against the bundled
stack, stages the tarball, and produces the sha256 sidecar.
workflow against a tag; this PR includes the YAML but a real release
will be the first end-to-end test.
Test plan
ci/build_prebuilt.sh botan v0.18.1-devruns to completion onmacOS arm64 and produces the expected tarball + sha256
release
Follow-up work
x86_64-pc-windows-msvc) — needs vcpkg orequivalent for static dep stack on Windows
.asc) for each tarball — the sourcetarball already ships signed; prebuilts should match
MACOSX_DEPLOYMENT_TARGETand pinned dep versions; full reproducibility needs more work on
glibc symbol versioning and path normalization)
Closes #2443.