ci: per-crate feature matrix check (cargo hack --each-feature) to close the workspace-unification blind spot #240

Problem

CI's feature coverage has a structural blind spot: both existing checks run workspace-scoped, so cargo unifies features across all members and every crate's feature set is validated only in combination — never in the configurations a downstream consumer or a cargo publish would actually build.

$ cargo check --workspace --all-features        # 0 errors on the commit that introduced #233
$ cargo check --workspace --no-default-features # 0 errors on the commit that introduced #236

Two shipped bugs prove the class (writeups: #233, #236, and the verification matrix in #233 (comment)):

• bug: arc-consensus-types arbitrary feature does not build standalone #233 — arc-consensus-types/arbitrary didn't build standalone; masked because arc-consensus-db and arc-node-consensus dev-depend on the feature and other members' flags supplied the missing deps under unification. Fixed in fix(types): make the arbitrary feature self-contained #231.
• bug: arc-consensus-types does not build with --no-default-features; signer-local gates a mandatory dependency #236 — arc-consensus-types --no-default-features didn't build at all; the signer-local feature gated a dependency four modules import unconditionally. Fixed in fix(types): make malachitebft-signing-ed25519 a required dependency #237.

Both directions of the same defect: a feature that only builds because of workspace unification, and a crate that only builds because of its default features. Since crates/types inherits publish from the workspace, per-crate feature buildability is publishing correctness, not hygiene.

Proposed job

feature-matrix:
  runs-on: ubuntu-latest
  steps:
    - uses: actions/checkout@v4
    - uses: dtolnay/rust-toolchain@stable   # pin to rust-toolchain.toml (1.91.1)
    - uses: taiki-e/install-action@cargo-hack
    - uses: Swatinem/rust-cache@v2
    - run: cargo hack check --each-feature --workspace --exclude arc-signer
    - run: cargo hack check --each-feature -p arc-signer --at-least-one-of local,remote

cargo-hack decomposes --workspace into per-package runs, which is the entire mechanism — adding flags to the existing workspace-scoped jobs catches nothing (verified in the comment linked above).

Known constraints (measured, not speculative)

1. arc-signer fails --no-default-features by design — its compile_error!("At least one signing provider feature must be enabled") guard is correct code (genuine either/or between local/remote providers), but cargo-hack counts it as a failure. Hence the split invocation above; --at-least-one-of local,remote keeps every other combination of its 5 features covered instead of exempting the crate entirely.
2. Cost: full matrix is 64 configurations, ~69 min cold-cache local (@mehmetkr-31's measurement). Only 8 of 25 crates declare features; if the warm-cache CI number is still too slow for per-PR, the fallback is scoping to feature-declaring crates or running the matrix on merge-queue/nightly rather than every push. Suggest measuring one warm run before deciding.
3. Baseline: goes green only after fix(types): make the arbitrary feature self-contained #231 and fix(types): make malachitebft-signing-ed25519 a required dependency #237 both land — the job's first run doubles as regression proof for both fixes.

Acceptance

• Job red on main prior to fix(types): make the arbitrary feature self-contained #231/fix(types): make malachitebft-signing-ed25519 a required dependency #237, green after both merge.
• Reverting either fix turns the job red (the two bugs are the test fixture).

@mehmetkr-31 offered to write the workflow PR (thanks for the matrix run and the arc-signer catch) — filing this per the division of labor agreed on #233.