Skip to content

Ship prebuilt static-library tarballs as release assets - #2444

Merged
ronaldtse merged 20 commits into
mainfrom
prebuilt-release-assets
Aug 4, 2026
Merged

ronaldtse merged 20 commits into
mainfrom
prebuilt-release-assets

Conversation

@ronaldtse

Copy link
Copy Markdown
Contributor

Summary

Adds prebuilt static-library tarballs to GitHub releases, addressing #2443.
Each tarball bundles librnp.a plus a curated static build of every
transitive dependency (sexpp, Botan or OpenSSL, json-c, zlib, bzip2), so
downstream consumers that cannot reach a system package manager (Cargo,
embedded cross-compile targets, Windows app vendors) can link against
librnp without compiling the full C++ + Botan stack from source.

The build runs as a new prebuild matrix job in
.github/workflows/release.yml that depends on the existing release
job, so the source tarball and GitHub release are created first; the
prebuilt tarballs are then uploaded as additional assets to the same
release.

Targets (v1)

[cols="2,1", options="header"]
|===
| Target | Backend(s)
| x86_64-unknown-linux-gnu | botan, openssl
| aarch64-unknown-linux-gnu | botan, openssl
| x86_64-apple-darwin | botan, openssl
| aarch64-apple-darwin | botan, openssl
|===

8 tarballs per release. Windows MSVC (x86_64-pc-windows-msvc) and
Linux musl (x86_64-unknown-linux-musl, aarch64-unknown-linux-musl)
are tracked as follow-up — the script's detect_target already
recognizes them, but the dependency build steps need platform-specific
work that's out of scope for this PR.

What's in each tarball

rnp-v<version>-<target>-<backend>.tar.gz
├── include/
│   ├── rnp/         # public FFI headers (rnp.h, rnp_err.h, rnp_export.h, rnp_ver.h)
│   ├── sexpp/       # sexpp headers
│   ├── botan-3/     # (botan) or openssl/ (openssl)
│   ├── json-c/
│   ├── bzlib.h
│   └── zlib.h
├── lib/
│   ├── librnp.a
│   ├── libsexpp.a
│   ├── libbotan-3.a  # (botan) or libcrypto.a + libssl.a (openssl)
│   ├── libjson-c.a
│   ├── libz.a
│   ├── libbz2.a
│   ├── cmake/rnp/   # find_package(rnp)
│   └── pkgconfig/
└── MANIFEST.txt     # summary, dependency versions, link flags

What changed

ci/build_prebuilt.sh (new)

The static-build orchestrator. Downloads pinned versions of each
dependency, builds them as static libraries into a temporary prefix,
builds librnp against that prefix, and stages the resulting headers and
archives into the tarball layout shown above. Outputs:

  • rnp-v<version>-<target>-<backend>.tar.gz
  • rnp-v<version>-<target>-<backend>.sha256

Dependency versions are pinned in the script for reproducibility:

  • zlib 1.3.1
  • bzip2 1.0.8
  • json-c 0.19
  • Botan 3.10.0 (botan backend)
  • OpenSSL 3.6.0 (openssl backend)

ci/botan3-modules, ci/botan3-pqc-modules

Renamed curve25519 to x25519. Botan 3 renamed this module; the
existing files were broken for any Botan 3.x build using them.
ci/botan-modules (used by codeql.yml against Botan 2.17.3) is
unchanged.

.github/workflows/release.yml

Added prebuild matrix job with 8 entries (4 targets x 2 backends).
Each job:

  1. Checks out the source at the release tag
  2. Installs build deps (Linux/macOS handled separately)
  3. Runs ci/build_prebuilt.sh
  4. Uploads rnp-*.tar.gz + rnp-*.sha256 to the release via
    gh release upload --clobber

The job depends on release so it only fires after the GitHub release
exists. fail-fast: false ensures one target failing doesn't kill the
others.

docs/packaging.adoc

New "Prebuilt static libraries (release assets)" section documenting
asset naming, available targets, tarball layout, and consumption
patterns (CMake + raw compiler flags). Cross-references the existing
sections on find_package and pkg-config.

Validation

  • Validated ci/build_prebuilt.sh end-to-end locally on macOS arm64
    with the botan backend: builds zlib, bzip2, json-c, Botan 3.10.0
    (minimized build with PQC modules), and rnp against the bundled
    stack, stages the tarball, and produces the sha256 sidecar.
  • The 8-entry matrix itself can only be exercised by running the
    workflow against a tag; this PR includes the YAML but a real release
    will be the first end-to-end test.

Test plan

  • ci/build_prebuilt.sh botan v0.18.1-dev runs to completion on
    macOS arm64 and produces the expected tarball + sha256
  • CI green on this branch (workflow syntax check at minimum)
  • First tag-triggered run produces all 8 tarballs attached to the
    release

Follow-up work

  • Windows MSVC target (x86_64-pc-windows-msvc) — needs vcpkg or
    equivalent for static dep stack on Windows
  • Linux musl targets — needs alpine container or musl cross-toolchain
  • GPG detached signatures (.asc) for each tarball — the source
    tarball already ships signed; prebuilts should match
  • Reproducible builds (currently best-effort via MACOSX_DEPLOYMENT_TARGET
    and pinned dep versions; full reproducibility needs more work on
    glibc symbol versioning and path normalization)

Closes #2443.

@codecov

codecov Bot commented Jul 28, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 85.37%. Comparing base (ffa3ad7) to head (0f28c88).
⚠️ Report is 29 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #2444   +/-   ##
=======================================
  Coverage   85.36%   85.37%           
=======================================
  Files         126      126           
  Lines       22862    22868    +6     
=======================================
+ Hits        19516    19523    +7     
+ Misses       3346     3345    -1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@ronaldtse
ronaldtse force-pushed the prebuilt-release-assets branch 12 times, most recently from 5deb062 to 38d8424 Compare July 29, 2026 05:52
@ronaldtse
ronaldtse requested review from antonsviridenko and ni4 July 31, 2026 16:19
@ronaldtse
ronaldtse force-pushed the prebuilt-release-assets branch from 3a8c919 to 62283f2 Compare August 1, 2026 08:24

@ni4 ni4 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM as merge conflicts/CI fixed!

@ronaldtse

Copy link
Copy Markdown
Contributor Author

Merging with 1 approval (ni4). CI/release infrastructure — builds and ships prebuilt static-library tarballs as release assets. Thank you @ni4 for the review!

Adds per-target static-library tarballs of librnp + its full
dependency stack (sexpp, Botan or OpenSSL, json-c, zlib, bzip2) as
additional GitHub release assets, so downstream consumers that
cannot reach a system package manager (Cargo vendored mode,
embedded cross-compile targets, Windows app vendors, CI pinning a
specific librnp version) can link against librnp without compiling
the full C++ + Botan stack from source.

Closes #2443.

== What ships

14 tarballs per release (7 target triples x 2 backends). Each
tarball contains librnp.a plus all transitive static dependencies,
the public FFI headers, CMake config, pkg-config files, and a
MANIFEST.txt with link instructions. A .sha256 sidecar and optional
GPG .asc detached signature accompany each tarball.

Targets: x86_64/aarch64 Linux glibc, x86_64/aarch64 Linux musl
(via alpine container), x86_64/aarch64 macOS, x86_64 Windows MSVC.

== Architecture

Data-driven: backend and target properties live in config files
(ci/backends/<name>.env, ci/targets/<triple>.env) that are the
single source of truth for both the build scripts and the
validator. Adding a backend or target = adding one config file.

The build is split across two scripts:
  * ci/build_prebuilt.sh   -- Linux glibc, Linux musl, macOS.
    Downloads pinned dep versions, builds them as static libraries,
    builds librnp against them, stages into a tarball.
  * ci/build_prebuilt.ps1  -- Windows MSVC via vcpkg's
    x64-windows-static triplet.

ci/test_prebuilt.sh validates every tarball against
docs/specs/tarball-layout.adoc: naming, layout, required headers
and archives, pkg-config path integrity, MANIFEST.txt consistency,
and an end-to-end compile+link+run that calls rnp_ffi_create().

== Workflow design

The prebuild matrix lives in .github/workflows/prebuilt.yml, a
reusable workflow with three triggers:
  * pull_request  -- path-filtered to prebuilt scripts; uploads
    tarballs as workflow artifacts for reviewer download.
  * workflow_call -- called from release.yml after the GitHub
    release exists; attaches tarballs to the release via
    softprops/action-gh-release.
  * workflow_dispatch -- manual run against an existing tag.

release.yml calls prebuilt.yml via workflow_call with the release
tag. The prebuild job depends on the release job so the source
tarball is created first.

== Specs

  * docs/specs/tarball-layout.adoc    -- the contract every
    tarball must satisfy. test_prebuilt.sh mechanically asserts it.
  * docs/specs/target-matrix.adoc     -- tier definitions, support
    window, add/deprecate policy.
  * docs/specs/backends.adoc          -- backend requirements,
    config-file contract, add/deprecate policy.
  * docs/packaging.adoc               -- consumer-facing docs with
    asset naming, tarball layout, linking instructions, and
    reproducibility notes.

== Other changes

  * ci/botan3-modules, ci/botan3-pqc-modules: renamed curve25519
    to x25519 (Botan 3.0 rename) and added per-curve pcurves_*
    modules that rnp's CMake feature detection requires on
    Botan 3.7+.
  * ci/lib/sign.sh: shared GPG signing helper.
  * ci/prebuilt-versions.env: pinned dep versions (single source
    of truth for the source-based build path).
  * ci/check_reproducibility.sh: informational diff tool (wired as
    a PR-only CI job that always exits 0).
- _typos.toml: allow "Sur" (macOS Big Sur proper noun)
- ci/build_prebuilt.sh: use find instead of ls for backend/target
  listing (shellcheck SC2012)
- cmake/Modules/FindOpenSSLFeatures.cmake: link Crypt32/Ws2_32/
  Advapi32/User32 into the findopensslfeatures probe when building
  against static OpenSSL on MSVC. Without these, libcrypto.lib's
  references to CertOpenStore / socket BIOs / etc. leave 38
  unresolved externals and the probe fails to link.
- ci/build_prebuilt.ps1: drop the CMAKE_EXE_LINKER_FLAGS workaround
  that didn't actually propagate into the sub-CMake build; the fix
  above replaces it.
The ludeeus/action-shellcheck action runs with SHELLCHECK_OPTS=-x
(--external-sources), which makes shellcheck try to follow every
source statement. The runtime-computed paths ($BACKEND_CONFIG,
$TARGET_CONFIG, ${SCRIPT_DIR}/backends/${BACKEND}.env, etc.) can't
be resolved at lint time, producing SC1090/SC1091 notes that the
action treats as failures.

Replace the # shellcheck source=... directives (which asked shellcheck
to follow a constant path it couldn't find with explicit
disable=SC1090 (non-constant source) or disable=SC1091 (unresolvable
SCRIPT_DIR-relative path) so the lint is honest about what it can
and can't statically verify.
EOF
)
MSVC doesn't ship <getopt.h> or <dirent.h>; rnp's src/common and
src/rnp need them (via the uniwin.h / getoptwin.h shims). The
existing windows-native.yml CI already installs both ports via
vcpkg — do the same in the prebuilt Windows build so CMake can
resolve GETOPT_INCLUDE_DIR and DIRENT_INCLUDE_DIR instead of
aborting at configure time with "variables set to NOTFOUND".
The MSYS2 tar that Git for Windows puts first in PATH fails with
"Cannot connect to D: resolve failed" when given a Windows-style
output path like D:\a\rnp\rnp\artifacts\.... It expects POSIX paths
(/d/a/rnp/...) and the cygpath round-trip is unreliable.

Use C:\Windows\System32\tar.exe (Windows native, preinstalled on
Windows 10 1803+ / Server 2019+, available on windows-latest runners).
It handles Windows paths directly and produces the same .tar.gz the
Linux/macOS build does.
RNP_ARTIFACT_DIR on Windows resolves to a Windows-style path
like D:\a\rnp\rnp/artifacts inside the bash validation step.
The MSYS2 tar that test_prebuilt.sh invokes cannot open that path
(it expects POSIX /d/a/rnp/rnp/artifacts and fails with Cannot
connect to D: resolve failed). Convert at the workflow boundary
with cygpath -u so the script sees a path it can open.
vcpkg's x64-windows-static triplet ships libraries under different
names than the conventional ones used on Unix:

  zlib   zlib.lib    ->  zs.lib
  bzip2  libbz2.lib  ->  bz2.lib
  json-c libjson-c   ->  json-c.lib  (no leading lib)
  botan  libbotan-3  ->  botan.lib   (no -3 suffix on Windows)

Update BACKEND_WINDOWS_LINK_LIBS and BACKEND_WINDOWS_LIB_NAMES in
both backend configs so the validation step finds the archives that
vcpkg actually produces.
test_prebuilt.sh always read BACKEND_LIB_NAMES (the Unix naming) even
on Windows, so the validation kept failing on lib/z.lib even after
aligning BACKEND_WINDOWS_LIB_NAMES with vcpkg naming. Pick the right
list based on TARGET_OS so the structural check matches the actual
vcpkg x64-windows-static output.
ec.cpp and exdsa_ecdhkem.cpp call Botan::EC_Group::from_name() but
only include botan/ecdh.h, which forward-declares EC_Group via
ecc_key.h without the full class definition. On Linux and macOS
the source-built Botan transitively pulls in ec_group.h through
some header chain, but vcpkg's x64-windows-static botan port does
not, causing ec.cpp to fail with 'use of undefined type Botan::EC_Group'.

Include the header explicitly in both files for the class they
actually use, instead of relying on transitive includes that vary
between Botan builds.
vcpkg's x64-windows-static triplet compiles every dep (botan,
json-c, zlib, bzip2, etc.) with /MT (static MSVC runtime). The rnp
build was defaulting to /MD (dynamic runtime), which caused LNK2038
mismatches when linking the example and CLI binaries against
botan-3.lib (Botan's allocator touches runtime internals and the
linker flags the discrepancy). OpenSSL did not surface the issue
because its API does not directly reference runtime-internal types.

Pass CMAKE_MSVC_RUNTIME_LIBRARY=MultiThreaded so the entire bundle
is consistently /MT, matching what consumers of the static bundle
are expected to use anyway.
actions/checkout v4, actions/cache v4, actions/upload-artifact v4,
and softprops/action-gh-release v2 all ship with Node.js 20, which
GitHub Actions is deprecating. Upgrade to the latest stable majors
that bundle Node.js 24:

  actions/checkout          v4 -> v7
  actions/cache             v4 -> v6
  actions/upload-artifact   v4 -> v7
  softprops/action-gh-release v2 -> v3

ilammy/msvc-dev-cmd@v1 is already on the v1 major line (v1.13.0)
and does not have a Node 20 problem, so it stays.
GitHub Actions is deprecating Node.js 20 on runners. Several actions
used across our workflows still ship with the Node.js 20 runtime,
producing deprecation warnings on every run. Upgrade to the latest
stable major for each:

  actions/checkout                      v4  -> v7
  actions/cache                         v4  -> v6
  actions/upload-artifact               v4  -> v7
  actions/download-artifact             v4  -> v8
  peter-evans/repository-dispatch       v2  -> v4
  DoozyX/clang-format-lint-action       v0.18.2 -> v0.20

Actions already on their latest major tag (codeql-action v3,
msys2/setup-msys2 v2, install-nix-action v31, coverity-scan v1)
and actions that track @master (typos, shellcheck, oss-fuzz) are
left unchanged.
Supports the CI dep-fan-out orchestration (time-machine.yml and
centos-and-fedora.yml already use this needs: + upload/download-
artifact pattern for their build → test fan-out).

Three invocation modes:

  build_prebuilt.sh deps <target> [workdir]
      Phase 1: build zlib + bzip2 + json-c into \$WORK/prefix and
      stop. The CI job uploads that prefix as an artifact so both
      per-backend rnp jobs reuse it.

  build_prebuilt.sh rnp <backend> <version> <target> [workdir]
      Phase 2: expects deps already in \$WORK/prefix. Builds the
      backend (botan/openssl), then rnp, then packages.

  build_prebuilt.sh <backend> <version> <target> [workdir]
      Backward-compat single-pass mode for local testing.

Backend config sourcing is skipped in deps mode since no backend-
specific code is built.
Restructure prebuilt.yml to share common dep builds across backends,
mirroring the pattern already used in centos-and-fedora.yml
(package-source → package → rpm-tests) and time-machine.yml
(build → test).

Three orchestrated phases:

  config (1 job)
    Single source of truth for the target list. Emits two matrices
    via jq: unix_targets (6, for the deps phase) and rnp_matrix
    (14, the full target × backend cartesian product). Adding a
    target = one line in the targets JSON; both downstream phases
    pick it up automatically.

  build-deps (6 jobs: Unix targets only)
    Builds zlib + bzip2 + json-c into \$WORK/prefix and uploads it
    as an artifact (deps-<triple>). Dep cache keyed by target only
    (not target+backend) so both backends' rnp jobs benefit.

  build-rnp (14 jobs: all targets × backends)
    Unix targets download the deps artifact and run
    build_prebuilt.sh rnp (deps already in \$prefix — no rebuild).
    Windows runs the monolithic PowerShell pass (vcpkg deps are not
    cleanly splittable via artifacts).

Net: common deps built 6 times (once per Unix target) instead of 12
(once per Unix target × backend). macOS savings at 10x runner cost.
Reproducibility check moved to release-only (it was informational
anyway and ran on every PR push).
alpine+ARM64 runners refuse to execute JS actions (actions/cache,
actions/upload-artifact, actions/download-artifact). The deps
fan-out therefore can't share its output across the alpine+ARM64
matrix entries, which failed both the build-deps upload step and
the build-rnp download step.

Three coordinated changes keep the fan-out working on every other
target while degrading gracefully on alpine+ARM64:

* build-deps: skip the actions/upload-artifact step for
  alpine+ARM64 (the actions/cache step already had the same
  conditional). The job now succeeds on that target, with deps
  built into the per-runner prefix but not shared.

* build-rnp: skip the actions/download-artifact step for
  alpine+ARM64, matching the build-deps upload skip.

* build_prebuilt.sh rnp mode: when \$PREFIX/lib/libz.a is
  missing, build the common deps inline before the backend + rnp
  build. This is the same work the single-pass mode does, so the
  alpine+ARM64 path costs exactly what the pre-fan-out baseline
  did -- and the fan-out still saves the duplicated build on every
  other target.
…ount

In container jobs (alpine), \${{ github.workspace }} resolves to the
host path /home/runner/work/rnp/rnp, which is NOT visible from inside
the container -- the container mounts the workspace at /__w/rnp/rnp
instead. The build_prebuilt.sh script was creating directories at
the host path inside the container's ephemeral filesystem, so the
files never landed on the mount point and the upload-artifact step
(reading from the host side) reported 'No files were found'.

Fix: inside bash steps, use \$GITHUB_WORKSPACE (correctly set to
/__w/rnp/rnp in containers, identical to \${{ github.workspace }} in
non-container jobs). The JS actions (cache, upload-artifact,
download-artifact) continue using \${{ github.workspace }} because
they read from the host filesystem, and the container mount makes
the two paths equivalent.

Also move RNP_ARTIFACT_DIR under work/ so the tarballs and deps
share the same parent and the mount mapping is consistent.
GitHub has deprecated macos-13 (Intel) runners -- the deps job for
x86_64-apple-darwin sat queued for 20+ hours waiting for a runner
that will never come, blocking the entire build-rnp phase.

Remove the Intel macOS target entirely:
  - matrix entry in prebuilt.yml (6 targets now, was 7)
  - ci/targets/x86_64-apple-darwin.env (orphaned)
  - Intel rows in docs/specs/target-matrix.adoc and docs/packaging.adoc
  - tarball count updated 14 -> 12 (6 targets x 2 backends)

Intel macOS consumers can use the aarch64-apple-darwin tarball via
Rosetta, or build from source. A future universal-binary target
(x86_64+aarch64 cross-compiled on macos-14) can be added separately.
rnp uses Botan's C FFI API (botan_block_cipher_*, botan_rng_*,
botan_nist_kw_*, etc.) throughout its crypto layer. vcpkg's default
botan port omits the ffi module, so the static botan.lib was missing
all the botan_* symbols, producing 17 unresolved externals when
linking the rnp examples and CLI tools.

Install botan[ffi] to enable the module. vcpkg applies the feature
to the same x64-windows-static triplet we already use.
…bug)

vcpkg's botan port for the x64-windows-static triplet produces a
.lib whose headers still declare the C FFI symbols
(botan_block_cipher_*, botan_rng_*, botan_nist_kw_*, etc.) with
__declspec(dllimport). When rnp links statically against that
.lib, the MSVC linker prepends __imp_ to every botan_* reference
and fails with 17 unresolved externals (LNK2019 / LNK1120).

The vcpkg botan port has no 'ffi' feature to toggle either
('botan has no feature named ffi'). Fixing this properly needs
either an upstream vcpkg port fix or a from-source Botan build on
Windows -- both are follow-up work. The Windows + openssl tarball
covers the primary Windows use case in the meantime.

Revert the invalid botan[ffi] attempt in build_prebuilt.ps1, filter
the (x86_64-pc-windows-msvc, botan) entry out of the rnp_matrix in
the config job, and document the gap in packaging.adoc. Matrix now
ships 11 tarballs per release (was 12).
@ronaldtse
ronaldtse force-pushed the prebuilt-release-assets branch from dfa9e67 to 0f28c88 Compare August 4, 2026 15:42
@ronaldtse
ronaldtse merged commit b6112ce into main Aug 4, 2026
71 of 113 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Ship prebuilt static libraries as GitHub release assets

2 participants