Skip to content

fix(wallet-toolbox): send and return BEEF as bytes so negotiated binary JSON carries it - #774

Merged
ty-everett merged 12 commits into
bsv-blockchain:mainfrom
shruggr:fix/storage-binary-beef
Oct 10, 2026
Merged

ty-everett merged 12 commits into
bsv-blockchain:mainfrom
shruggr:fix/storage-binary-beef

Conversation

@shruggr

@shruggr shruggr commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

Program and scope

Compact BEEF transport uses the existing negotiated binary JSON codec for signer/internalization bytes, create-action/expiry request BEEF, Knex response BEEF/source transactions, and schema-declared row bytes. Older peers retain numeric-array JSON and identical transaction/proof content. Authentication, request/response ceilings and arbitrary numeric-array values are preserved.

Integrated qualified #775 and current main. Exact reviewed head: 9c28e5037a71d959d76fd00a0655489f0d803d54. The main integration changed ancestry only: the candidate tree remained identical.

Impact

  • Public package bytes affected: Node, browser/client and mobile wallet packages, included in the unpublished 2.14.7 candidate. No publication claimed.
  • Security/compatibility review: negotiation remains explicit; only schema-declared valid large byte arrays are tagged; legacy encoding, authentication, validation, payload ceilings and source object ownership remain intact.
  • Changelog, package pages, release-note/migration ledger and generated API documentation updated. No public API, persisted schema or legacy peer migration.

Verification

  • Node 24.18.0 / pnpm 10.33.2; frozen lifecycle-disabled installation and allowlisted tooling.
  • pnpm health:check, pnpm lint, pnpm format:check, pnpm build, pnpm typecheck, pnpm audit:security, pnpm docs:build: passed; zero high/critical advisories.
  • Focused signer/client/server/row/list tests passed: 66 tests including 150 KB BEEF requests, negotiated/legacy response limits, byte equality, no source mutation, invalid-byte preservation and unrelated numeric arrays.
  • Complete Node wallet coverage: 292 suites / 3,171 tests passed, four workers, unchanged governed exclusions and timeouts. Client 22 and mobile 46 coverage tests passed.
  • Initial unbounded local attempts exposed a Node/V8 garbage-collector SIGSEGV and an authenticated large-record sync failure. The isolated sync regression and complete four-worker rerun passed. No assertion, test, timeout or gate was relaxed.
  • All other workspace tests and conformance passed with root/wallet runners explicitly excluded from the dependent command. Existing governed skips/todo remain unchanged.
  • Repository patch coverage: 100.00% (41/41) against the unchanged 90% requirement.
  • All three pack:check contracts passed, including exports/declarations/source maps/publint/strict type resolution and clean consumers.
  • Real browser: Vite 2,228,550 raw / 533,683 gzip / 407,031 Brotli bytes, 132 modules; esbuild 1,737,475 / 483,032 / 379,329 bytes. Governed composition/budgets passed.
  • Metro/Hermes passed: Metro 2,235,853 / 593,746 / 454,134 bytes; Hermes 4,658,005 / 1,946,634 / 1,515,317 bytes.
  • Governed wallet property suite passed; deterministic documentation generation and git diff --check passed. Generated test reports excluded.
  • Complete incremental diff self-reviewed for correctness, security, compatibility, public API, artifacts, dependencies, documentation and operations.
  • Hosted exact-head coverage, platform, consumers, conformance, mutation/aggregate gates all terminal and successful. No new skip, suppression, finding baseline or coverage reduction.

Hosted CI 38095324610, conformance 38095324644 and CodeQL 38095324668 passed on the exact head. Aggregate and merge gates, Sonar zero-findings, Socket and patch coverage passed; expected scope skips were validated by the merge gate.

Security and dependencies

  • No new dependency, override, dismissal, suppression or workflow-permission change.
  • Negative/boundary tests preserve legacy peers, invalid bytes, payload ceilings and schema field ownership.
  • Exact-head CodeQL has no new alert; open merge-ref alerts: 0.
  • Exact-head Sonar has zero new findings and zero unreviewed hotspots.

Release and operations

  • Patch behavior included in the held 2.14.7 wallet candidate with release notes and no-migration guidance.
  • No workstation npm publication or repository settings change.
  • Upgrade both clients and active storage to obtain request and response savings. Existing payload ceilings still apply. The protected npm OIDC workflow will publish the final qualified wallet candidate after all four scoped David PRs merge; source merge alone is not publication.

Completion evidence

  • Final-head required checks and review conversations complete; 0 review threads.
  • Remote main merge verified: dc2f727, merged 2026-10-10T23:41:38Z; remote tree exactly matches the qualified head.

Original implementation explanation

With binary JSON negotiated, the storage client and server still sent their largest byte fields as JSON number arrays. The codec only encodes Uint8Array values, and these fields were number[].

A request BEEF over 100,000 bytes also exceeds the 100,000-item array limit in @bsv/auth-express-middleware and fails with HTTP 400 ERR_AUTH_MALFORMED.

Changes

Requests, when the client has binaryRequests on and the server advertises binary:

  • createAction, prepareNoSendExpiry and activateNoSendExpiry send inputBEEF as bytes.
  • internalizeAction sends tx as bytes.

Responses, when the client sends X-BSV-Binary-Encoding:

  • listOutputs returns BEEF as bytes.
  • createAction returns sourceTransaction as bytes.
  • findOutputsAuth (lockingScript) and findProvenTxReqs (rawTx, inputBEEF) rows are sent as bytes. The client converts them back to number[].

Peers that have not negotiated binary get number arrays as before.

StorageKnex now produces the listOutputs BEEF and sourceTransaction as Uint8Array in process too. The declared types already allow it, and Wallet.listOutputs re-serializes as before.

Not changed, because existing peers do not support bytes there: competingBeef in processAction / internalizeAction results, byte fields in error payloads, and merklePath in updateProvenTxReqWithNewProvenTx.

Tests

  • New tests cover each changed field: it is sent or returned as bytes and tagged $bsvBinary on the wire.
  • Over authenticated HTTP, a 150 KB BEEF sent as number[] returns the 400; sent as bytes it passes authentication.
  • Local results: 290 suites passed, 2 skipped; 3127 tests passed, 7 skipped. pnpm lint, pnpm health:check, and the package typecheck and format:check pass.

shruggr and others added 5 commits October 3, 2026 14:24
…ry JSON carries it

binaryRequests encodes only Uint8Array values, but the toolbox built its
largest storage payloads as number[]: StorageClientBase.createAction set
inputBEEF from pruned.toBinary(), and the signer's internalizeAction passed
ab.toBinaryAtomic(txid). They travelled as JSON number arrays even with
binary negotiated, and a BEEF over ~100 KB exceeded the auth middleware's
100,000-value request limit.

Use the SDK's toUint8Array() / toUint8ArrayAtomic() for createAction and
no-send-expiry inputBEEF and internalizeAction AtomicBEEF, and return
listOutputs BEEF, createAction sourceTransaction and competingBeef as
Uint8Array from Knex storage. BEEF and AtomicBEEF already allow Uint8Array,
and the server's byte validation accepts it. Legacy peers that have not
negotiated binary still receive number arrays.
BRC-177 Prefunding step 2 requires the anchor to be large enough both to
fund the protected action and to be reclaimed, net of a reclaim fee. The
anchor was sized for the protected action only, so value-neutral actions
(e.g. a 1-sat token transfer) were refused because the anchor could not
pay for its own reclaim.

Size the anchor as the larger of the protected action's need and the
reclaim floor (reclaim fee plus minimum reclaim output). The protected
action carries no change, so the funding planner now runs with a zero
change cap and pays any surplus as fee. generateChangeSdk accepts
maxChangeOutputs: 0 to mean "create no change".

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R53QzSzGv4LkfmxBd4URiD
Replace the "maxChangeOutputs: 0 means no change" overload with a
dedicated GenerateChangeSdkParams.surplusToFee flag. When set, the
planner creates no change output and pays any surplus beyond the
required fee as fee. maxChangeOutputs keeps its original rule (minimum
1, default 8); the BRC-177 protected branch goes back to a cap of 1 and
sets surplusToFee. The protected funding plan is the only setter.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R53QzSzGv4LkfmxBd4URiD
Compute the change-output cap as before, then zero it when surplusToFee
is set. No behaviour change (SonarCloud typescript:S3358).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R53QzSzGv4LkfmxBd4URiD
… JSON

StorageServer converts the byte columns of findOutputsAuth and
findProvenTxReqs results when the client negotiated binary JSON, as it
already does for getSyncChunk. Clients convert them back to number[] in
validateEntities.

Restore competingBeef to number[] in createAction's WERR_REVIEW_ACTIONS.
Thrown errors are serialized with stringifyBRC100, so it never reached
the binary codec.
@shruggr
shruggr marked this pull request as ready for review October 3, 2026 22:49
shruggr added a commit to yours-org/yours-wallet that referenced this pull request Oct 4, 2026
Binary BEEF over negotiated binary JSON (bsv-blockchain/ts-stack#774),
BRC-169 handle payments.
@ty-everett
ty-everett marked this pull request as draft October 10, 2026 23:30
@sonarqubecloud

Copy link
Copy Markdown

@codecov

codecov Bot commented Oct 10, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@ty-everett
ty-everett marked this pull request as ready for review October 10, 2026 23:41
@ty-everett
ty-everett merged commit dc2f727 into bsv-blockchain:main Oct 10, 2026
44 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants