Reuse advice commitments for sparse witness deltas

[ValarDragon]

Summary

• reuse the first circuit's prepared advice commitments when a later circuit
  has a cheaper sparse witness delta;
• make the density gate exact, then use a small prepared-recoder work check to
  reject observed digit/window regressions; and
• preserve the direct path for ineligible or rejected proofs.

For advice polynomial a_i and blind r_i, the routed path uses

Com(a_i, r_i) = Com(a_0, r_0) + Com(a_i - a_0, r_i - r_0).

The original advice polynomials and blinds remain available for openings.
Proof format, transcript order, RNG consumption, and verification are
unchanged.

Routing

Routing is limited to prepared multicore, no-orbits, k = 11 proofs with at
least two circuits and no more workers than the prepared backend's measured
cap.

For each later circuit, the exact full-row density pass requires:

• no advice column to gain nonzero coefficients;
• at least 256 fewer aggregate nonzero coefficients; and
• savings strictly greater than one eighth of the direct nonzeros.

Every column must then pass an independent eight-row prepared-recoder work
check. Pasta compares main-window span and digit visits, plus residual-tail
span and visits, componentwise. The density decision is exact. The eight-row
work check remains a performance heuristic: it can reject a profitable delta
or miss unobserved digit work, but it cannot change the commitment result.
Unsupported and exceptional cases fail closed to the direct path.

On the real post-NU6.3 two-spend fixture, all ten later-action advice columns
route. Exact direct/delta nonzero counts fall from 15,865 to 13,654, saving
2,211 entries (13.94%), and no column gains nonzeros. In the eight-row work
sample, main-plus-tail digit visits fall from 1,069 to 891 (16.65%). The padded
two-action fixture saves only 3.69% and declines the exact density gate; all
three later circuits in the four-action fixture decline too.

End-to-end Ironwood measurements

Negative is faster. A single diagnostic executable selected unmodified main
(M), this PR (S), or the stacked scheduling candidate (O). Each
invocation proved the real post-NU6.3 two-spend fixture with a prepared k = 11 key. Criterion used Flat sampling, ten samples, a two-second warmup, and a
15-second measurement. Four balanced, interleaved macroblocks were retained
per host.

Host	Workers	Contrast	Invocation means	Effect	95% whole-block bootstrap
Linux x86_64	8	this PR/main (S/M)	142.376/143.019 ms	-0.449%	[-0.561%, -0.338%]
Apple M4	10	this PR/main (S/M)	67.980/68.227 ms	-0.365%	[-0.753%, +0.170%]
Linux x86_64	8	stacked overlap/this PR (O/S)	141.456/142.651 ms	-0.837%	[-1.322%, -0.385%]
Apple M4	10	stacked overlap/this PR (O/S)	67.738/68.020 ms	-0.414%	[-0.541%, -0.314%]

All four Linux S/M blocks favored this PR. Three of four M4 blocks favored
it; the M4 point estimate is favorable but dispersed, and its conservative
Student-t interval also crosses zero. All four O/S blocks favored the stacked
candidate on both hosts. The stacked result is reported here to show the
measured phase interaction; it is not part of this PR.

Those four-block measurements were frozen at main 42f5dd4e. #403 is
target-equivalent for that experiment, but #405 changes runtime field doubling
on Apple AArch64 and may change the phase balance. The five #328 commits were
therefore restacked on exact current main 4d14a1d1 as 0dd71aba; their
range-diff and patch IDs are unchanged, with no conflicts.

The exact-current same-binary confirmation retained four Linux blocks and two
M4 blocks. Linux was extended without rebuilding because its initial S/M
pair was mixed:

Host	Workers	Contrast	Invocation means	Effect	95% whole-block bootstrap	Blocks
Linux x86_64	8	this PR/main (S/M)	141.006833/141.492766 ms	-0.344038%	[-0.733611%, -0.025098%]	+0.098234%, -0.394182%, -0.927426%, -0.149885%
Apple M4	10	this PR/main (S/M)	67.736104/67.987879 ms	-0.370645%	[-0.500048%, -0.241074%]	-0.500048%, -0.241074%
Linux x86_64	8	stacked overlap/this PR (O/S)	139.787874/140.860498 ms	-0.761552%	[-1.082534%, -0.449238%]	-0.663125%, -1.221944%, -0.318462%, -0.840539%
Apple M4	10	stacked overlap/this PR (O/S)	67.430197/67.668560 ms	-0.352367%	[-0.399280%, -0.305433%]	-0.305433%, -0.399280%

Exact-current topology is main 4d14a1d1, this PR 0dd71aba, stacked v2
2a06abe3, and diagnostic 32ef6b23. The source bundle SHA-256 is
3e2e97484837b4213d0e2b185173fa9dfbf3db8145db84f23bc74b9d364d1f23.
Exact-current result archives are
ebd168c5208494d5eb00cb3e8d0529d45480e86f448816fda0fd7f9eeb527860
and 06200616bfabe664f1d9e761132e9521e2ac0310f3497ffd03f8fb682d378326
for the Linux initial run and extension, and
1018ec9dbdcab55bbfd2e87b7b731d1125b8908cbf0496e9e318b392a4a8b5aa
on M4. The no-rebuild extension runner and combined analyzer are
e04f59fa12b16ac5eff72715192f81dc0d07f9984c4ef8e1a590c5e36fce72f7
and 4177be3ce918b15d1254962ee81cb9c7a910bf2d04a357704c49808a43874de9.
Proof, route, binary, source, worker-width, and contamination gates all passed.
Linux retained and excluded landscape-sysinfo- and fwupd-contaminated
extension attempts. M4 retained and excluded two launchd-contaminated
attempts.

The same binary generated, verified, and byte-compared a 7,264-byte proof in
every mode, SHA-256
04d18c448a3fc2497daca547a90ef9a5b6d0bb8a1391e39f55529e1f95a51dc8.
Preflight route markers confirmed the real fixture routed, while padded
two-action and four-action fixtures declined. Accepted blocks had stable
binary/source hashes and no in-run competing process, swap, steal, or thermal
warning; rejected attempts were retained and replaced.

The frozen same-binary source bundle is
ae596cdaa133cab426d93f6fc3082a7ab291888178751381d3cb75131e9bb51c.
Result archives are retained for both hosts.

Route-negative cost

The exact density scan is not free. A separate two-block padded two-action
screen measured +0.140% on Linux (blocks -0.129%/+0.409%, noisy) and +0.173%
on M4 (blocks +0.209%/+0.138%). These proofs decline before the work check and
otherwise retain the original schedule. The target real-witness saving above
is larger.

Alternatives measured

• Parallelizing the ten-column work check (P) was rejected: it regressed M4
  by +0.510% in both retained blocks and was only -0.166% on Linux. The entire
  serial check is about 8.9 microseconds, so even deleting it caps the direct
  full-proof gain near 0.013%; the observed larger movement is scheduling
  noise or contention.

• The original stacked scheduler (v1), which placed more planning work in
  the overlap, regressed Reuse advice commitments for sparse witness deltas #328 by about +0.30% on M4 and +0.34% on Linux.

• A v3 scheduler that moved the work check before the fork but overlapped
  delta materialization regressed Reuse advice commitments for sparse witness deltas #328 by +0.774% on M4 and +0.952% on Linux.

• The surviving stacked v2 deliberately leaves the serial work check inside
  the commitment arm. That stagger lets transforms start before delta
  materialization and MSM pressure arrive. Its remaining route-negative risk
  is the case where the exact count passes but the work check rejects after
  transforms have started. A forced first-/last-column rejection gate favored
  the stacked schedule:

  • Linux X0/D0: -0.712%, two-block bootstrap
    [-1.090%, -0.333%];
  • Linux X9/D9: -0.455%, [-0.586%, -0.324%];
  • M4 X0/D0: -0.230%, [-0.521%, +0.061%]; and
  • M4 X9/D9: -0.487%, [-0.592%, -0.382%].

  D preserves Reuse advice commitments for sparse witness deltas #328's nested direct fallback; X starts transforms before
  the forced rejection and then uses global direct commitments. Both order
  strata favored X in all four contrasts.

Correctness and compatibility

• Seeded prepared and unprepared proofs are byte-identical and verify.
• One-, four-, and configured-width tests cover one, two, and four circuits.
• Tests cover accepted, mixed, direct-only, insufficient-savings,
  magnitude-inversion, high-window, sample-evasion, and exact-density-evasion
  profiles.
• Malformed shapes, identity/duplicate prepared bases, unsupported prepared
  comparisons, and exceptional inputs fail closed.
• Regression tests preserve instance/transcript failure side effects before
  synthesis and synthesis failure before prover RNG use.
• Default, no-default, multicore, and orbits configurations pass.

On the exact-current stacked head, the full Halo2 suites pass with 238 default,
191 no-default, and 199 batch,orbits tests. The focused advice-delta tests pass
5/5, the Pasta multicore scalar-work tests pass 2/2, and the Orchard library
suite passes 170 tests with 3 ignored. Formatting and git diff --check also
pass.

Risks and cost

• This path is variable-time in witness values and can expose relative witness
  sparsity and similarity through proving latency. The affected public proving
  APIs document that contract.
• The thresholds target the real Ironwood k = 11 two-spend workload. Other
  eligible prepared circuits can route when they satisfy the same gates.
• Eligible multi-circuit proofs that decline at the density gate pay the exact
  count scan. The padded screen above measures this overhead.
• A routed later circuit materializes one scalar delta per advice cell:
  approximately 640 KiB for ten 2,048-entry columns of 32-byte scalars, plus
  vector metadata.

API surface

• Adds the feature-gated public default trait method
  PreparedZeroCheck::scalar_work_is_at_most_vartime. This must cross the
  existing Halo2/Pasta crate boundary through
  Arc<dyn PreparedZeroCheck<_>>; the changelog records its public re-export.
• Adds the crate-private
  Params::prepared_lagrange_commitments_active(&self, usize) -> bool.
• Widens the private Codebook::recode_pair helper to pub(super) so the
  prepared backend can compare exact recoder work.
• Expands security documentation on the existing public proving APIs.

No other signature or visibility changes are introduced.

Current head 0dd71aba is clean and mergeable. All 65 GitHub checks are
green.

[ValarDragon]

I'm really excited for this, once we figure out how to expose it for voting as well, will be a big win.

Were fine with the timing variance leakage here. In practical terms, this is likely the largest timing variation we are introducing into proving. (E.g. a 1% proving time variation dependent on if your spending notes created from a single change split)

[v12-auditor]

Note

Complete: Audit complete. V12 did not find any issues that need review.

Open the full results here.

Analyzed three files, diff c0060c7...8466214.

[ValarDragon]

damn just caught it was benchmarking the 2-action tx using the same g_d^{old}, which is applicable in some cases, e.g. change splitting, but also wrong in many others

[ValarDragon]

Rebased conflict-free onto current main 42f5dd4e and force-pushed as
6f400097. The #328 patch ID is identical. Current main now includes #399's
substantial permutation-product work reduction (plus #402's documentation-only
accounting correction), so a fresh full-prover planner matrix is in progress.

This refresh also replaces the old 256-row density sample with exact full-row
direct/delta nonzero counts. The count pass now rejects a constructed profile
that could satisfy the old deterministic sample while being worse on the
unseen rows. The eight-row prepared-recoder comparison remains a documented
performance heuristic; it is no longer presented as an exact density guard.

On the real post-NU6.3 two-spend fixture, all ten later-action advice columns
route. Exact nonzero counts fall from 15,865 to 13,654 (-13.94%), while the
observed eight-row digit visits fall from 1,069 to 891 (-16.65%). The padded
fixture saves only 3.69% and declines the gate; all three later circuits in the
four-action fixture decline too.

The most recent completed same-binary full-prover results use the earlier
f21c359e baseline:

• Linux x86_64, 8 workers, two balanced blocks: -0.566%, 142.431 versus
  143.242 ms, 95% whole-block bootstrap [-0.659%, -0.472%]. Both blocks and
  both order strata favor this PR.
• Apple M4, 10 workers, four balanced blocks: -0.395%, 68.214 versus
  68.484 ms, whole-block bootstrap [-0.668%, -0.114%]. Three of four
  individual blocks, both order strata, and both run halves favor this PR.
  The conservative four-block t interval [-0.952%, +0.166%] reflects the
  small sample and dispersion.

A two-block route-negative padded screen quantifies the exact count-scan cost
when no later circuit routes:

• Linux, 8 workers: +0.140%, 143.782 versus 143.581 ms; blocks -0.129% and
  +0.409% (noisy).
• Apple M4, 10 workers: +0.173%, 68.645 versus 68.526 ms; blocks +0.209% and
  +0.138%.

Thus the target real witness win is larger than the observed route-negative
overhead, but the scan is not free. After a rejected plan, the original direct
commitment/transform schedule is retained. The four-action route-negative
markers pass, but that profile has not been timed on this exact stack.

All modes generated, verified, and byte-compared the same 7,264-byte proof.
The local final-stack validation includes 5/5 focused advice-delta tests,
Halo2 default/no-default/batch,orbits library suites (230/187/195 tests),
the Orchard library suite (169 passed, 3 ignored), formatting, diff, and
changelog checks. The same 5/5 focused tests pass after the conflict-free
42f5dd4e rebase. All 65 checks were green on the preceding head; fresh CI is
running after this rebase.

API-surface audit:

• one feature-gated public default trait method,
  PreparedZeroCheck::scalar_work_is_at_most_vartime;
• one pub(crate) method,
  Params::prepared_lagrange_commitments_active;
• one helper widened from private to pub(super), Codebook::recode_pair;
• documentation-only changes to existing public proving APIs.

There are no other signature or visibility changes. The public trait method is
needed across the Halo2/Pasta crate boundary through the existing
Arc<dyn PreparedZeroCheck<_>>; the changelog records the public re-export.

[ValarDragon]

Rebased onto current main 4d14a1d1 as 0dd71aba; the five-commit
range-diff is all = and individual and combined patch IDs are unchanged.
#403 is target-equivalent for the frozen 42f5dd4e measurements. #405 changes
runtime field doubling on Apple AArch64 and could change the phase balance, so
we ran an exact-current same-binary confirmation rather than assuming the old
denominator transfers. Linux was extended to four blocks without rebuilding
after its initial S/M pair was mixed; M4 retained two blocks.

Host	Contrast	Current means	Current result	95% whole-block bootstrap	Blocks
Linux x86_64, 8 workers	S/M	141.006833/141.492766 ms	-0.344038%	[-0.733611%, -0.025098%]	+0.098234%, -0.394182%, -0.927426%, -0.149885%
Apple M4, 10 workers	S/M	67.736104/67.987879 ms	-0.370645%	[-0.500048%, -0.241074%]	-0.500048%, -0.241074%
Linux x86_64, 8 workers	O/S	139.787874/140.860498 ms	-0.761552%	[-1.082534%, -0.449238%]	-0.663125%, -1.221944%, -0.318462%, -0.840539%
Apple M4, 10 workers	O/S	67.430197/67.668560 ms	-0.352367%	[-0.399280%, -0.305433%]	-0.305433%, -0.399280%

Exact-current topology is 4d14a1d1 -> 0dd71aba -> 2a06abe3 ->
32ef6b23; bundle SHA-256 3e2e97484837b4213d0e2b185173fa9dfbf3db8145db84f23bc74b9d364d1f23.
Exact-current result archives are ebd168c5208494d5eb00cb3e8d0529d45480e86f448816fda0fd7f9eeb527860
and 06200616bfabe664f1d9e761132e9521e2ac0310f3497ffd03f8fb682d378326
for Linux, and 1018ec9dbdcab55bbfd2e87b7b731d1125b8908cbf0496e9e318b392a4a8b5aa
for M4. Combined Linux analyzer SHA-256 is
4177be3ce918b15d1254962ee81cb9c7a910bf2d04a357704c49808a43874de9.
All integrity gates passed. The extension retained and excluded
landscape-sysinfo and fwupd attempts; M4 retained and excluded two launchd
attempts.

The prior 42f5dd4e same-binary, real post-NU6.3 two-spend result has four
balanced blocks per host:

• Linux x86_64, 8 workers: -0.449%, 142.376 versus 143.019 ms, 95%
  whole-block bootstrap [-0.561%, -0.338%]; all four blocks favor Reuse advice commitments for sparse witness deltas #328.
• Apple M4, 10 workers: -0.365%, 67.980 versus 68.227 ms, bootstrap
  [-0.753%, +0.170%]; three of four blocks favor Reuse advice commitments for sparse witness deltas #328, so this row is
  favorable but noisy.

The exact density route remains positive for all ten later-action advice
columns in the real fixture (15,865 to 13,654 nonzeros, -13.94%). The padded
two-action and four-action fixtures decline. The separate padded two-action
screen measured the count-scan cost at +0.140% on Linux (noisy) and +0.173% on
M4.

The same binary produced and verified byte-identical 7,264-byte proofs in all
modes (SHA-256 04d18c448a3fc2497daca547a90ef9a5b6d0bb8a1391e39f55529e1f95a51dc8).
Route markers, source/binary hashes, host-width checks, and contamination gates
passed; rejected attempts were retained and replaced.

We also rejected parallelizing the small ten-column work gate: P/S was
+0.510% on M4 and -0.166% on Linux, while the entire serial gate is only about
8.9 microseconds (a ~0.013% full-proof ceiling). This avoids adding complexity
for a result dominated by scheduling noise.

API-surface audit is unchanged:

• one feature-gated public default trait method,
  PreparedZeroCheck::scalar_work_is_at_most_vartime;
• one pub(crate) method,
  Params::prepared_lagrange_commitments_active;
• one helper widened from private to pub(super), Codebook::recode_pair;
• documentation-only edits to existing public proving APIs.

There are no other signature or visibility changes.

Current head 0dd71aba is clean and mergeable, with all 65 GitHub checks
green.

Stacked scheduling status: in the frozen four-block experiment, the surviving
v2 candidate improves #328 by -0.837% on Linux and -0.414% on M4. Earlier
v1, parallel-work, and v3 schedules were rejected. The final forced
work-reject screen also favored v2:

• Linux X0/D0: -0.712%, two-block bootstrap
  [-1.090%, -0.333%];
• Linux X9/D9: -0.455%, [-0.586%, -0.324%];
• M4 X0/D0: -0.230%, [-0.521%, +0.061%]; and
• M4 X9/D9: -0.487%, [-0.592%, -0.382%].

Both order strata favored v2 in every contrast. All 32 accepted timing legs
were clean; Linux swap/steal counters were unchanged, and M4 reported zero
swap and no thermal or performance warnings.