Flavored (HEAD) vendored build segfaults in signing on macOS/ARM64 — pre-existing, not the pinning sync

[ronaldtse]

Summary

The vendored HEAD flavor (--features vendored,pqc or crypto-refresh) segfaults on macOS (ARM64, Apple clang 15) when exercising the signing path. The release flavor (0.18.1 tarball) is unaffected locally (full suite green), and the flavored suite passes on ubuntu CI (pqc + crypto-refresh job) — this appears to be macOS-specific and, per the investigation below, pre-dates the pinning sync (#); it was simply never exercised on macOS before, because until now flavored builds were only ever run by ubuntu CI.

Repro

cargo build --features vendored,pqc,crypto-refresh --example <probe>
# probe: Context::new + RSA-2048 keygen (ok) + Signer::new(.., Mode::Detached).add_signer(..).build_to_memory()

• keygen: ok
• encrypt + decrypt roundtrip: ok
• sign (Detached / Inline / Cleartext): SIGSEGV
• verify of a valid external detached signature: aborts with uncaught rnp::rnp_exception

Crash

EXC_BAD_ACCESS (code=1, address=0x400000000000018)
frame #0: rnp::Hash_Botan::add(void const*, unsigned long) + 20

Hash_Botan holds a garbage/destroyed botan_hash_t; only one frame unwinds — consistent with memory corruption. The hash object is used inside rnp_op_sign_execute while hashing the literal-data payload.

Not a regression of the pinning sync

Control experiment on the same machine: a worktree of main before the sync (floating clone + json-c linked + old prepare flow) crashes identically. The crash reproduces against pins ee97ed760cf (2026-08-28 — the exact commit whose ubuntu CI run was green) and 470695b98ab (current tip), so it is not tied to the recent upstream window (json-c → nlohmann/json etc.) either.

Suspects

• The flavored Botan is built with the PQC module set (ml_kem,ml_dsa,slh_dsa_sha2,slh_dsa_shake) via BOTAN_CONFIGURE_ENABLE_MODULES; we have prior history of module-filtered Botan builds being sensitive to the toolchain on ARM64 macOS (the old CLAUDE.md note about miscompiled calc_sig_words under Homebrew LLVM at -O3). Current builds use Apple clang 15 (/usr/bin/g++ shim), which is not the previously-blamed compiler — but a similar codegen interaction with the minimized module set is plausible.
• Alternatively an upstream main issue that only manifests on macOS (dynload/lockfile/entropy paths differ) — the ubuntu CI wouldn't see it.

Next steps

• Reproduce with a Debug build of librnp (vendored, flavored) for a full backtrace.
• Try the flavored Botan with --module-policy= unchanged / full module set to isolate the module filter.
• If it is the module set on Apple clang, pin the flavored Botan to a full build on macOS (build-time cost only).