fix(header-chain): preserve auxiliary capacity and diversity

[evan-forbes]

Motivation

The dual-stack continuous-sync canary repeatedly received
AuxiliaryLimitExceeded while VCT repair was pending. Header-chain admission
evaluated new deliveries against the pre-retention engine state. It could reject
a transition whose retention pass would evict an old delivery and keep the
settled state within the aggregate limit.

The retained run does not expose auxiliary-delivery or retained-node counts. It
therefore cannot show whether the failed transition would have evicted a
delivery. The shared error also does not identify whether the event-local
per-header limit, the event-local aggregate limit, the retained per-header
limit, or the retained aggregate limit caused the refusal.

Version 1.3 allows 16 retained auxiliary deliveries per header and 65,536
retained auxiliary deliveries across the graph. Transport retries can store the
same payload under new delivery IDs. An attacker can also send several distinct
payloads from one peer. Both cases can consume capacity without adding an
independent repair candidate.

This change raises the per-header limit to 32 distinct semantic payloads. It
keeps the 65,536-row aggregate limit. It also removes the pre-retention
false-rejection case. It does not bound the VCT repair episode or claim to fix
every cause of that livelock.

Solution

• Keep aggregate and per-header event bounds before projection.
• Enforce retained per-header and aggregate auxiliary limits after finality and
  retention produce the settled projection.
• Enforce the same limits during independent plan verification.
• Identify a semantic payload by its header hash and tree-aux fields.
• Exclude transport ownership, supplier, body-size hint, and delivery ID from
  the semantic identity.
• Retain one row for each semantic payload.
• Retain at most one rooted semantic payload from one supplier for one header.
• Raise the retained per-header semantic-payload limit from 16 to 32.

The event-local per-header bound prevents one aggregate-sized event from forcing
quadratic vector insertion work for one header. The post-retention bound allows
retention to replace old deliveries without a false capacity rejection.

The first retained row preserves provenance for a semantic payload. Header sync
continues to track which peers announced the header. A repeated delivery from a
new peer does not need another durable payload row.

The planner may represent a transition above the aggregate limit while it
calculates retention. The engine never commits that intermediate projection.
The final atomic transition must remain within both retained limits.

Alternatives

• Raise both limits without deduplication. This option delays saturation but
  lets transport retries and supplier-controlled duplicates consume the new
  capacity.
• Retain one delivery per peer. This option protects peer diversity but lets
  Sybil identities consume the per-header limit with identical payloads.
• Retain only unique payloads. This option protects semantic diversity but lets
  one supplier consume every slot with distinct payloads.
• Replace an existing payload when a new delivery arrives. This option lets an
  attacker displace useful retained input. The engine instead keeps the first
  retained row for each semantic payload.
• Run retention as a separate transition before admission. This option adds a
  durable transition and creates a race between retention and later admission.
• Refuse the delivery, run retention, and retry. This option requires reliable
  retry coordination and can repeat the same failure.
• Predict retention during admission. This option duplicates retention policy
  and can disagree with the settlement calculation.
• Reserve replacement capacity. This option requires explicit slot ownership
  and a larger protocol change.
• Keep a fixed-size replacement structure for each target. This option provides
  stronger containment but belongs with the state-owned repair protocol.

The selected policy combines semantic deduplication, one rooted slot per
supplier, and a 32-payload per-header limit. The policy preserves useful
payload diversity without adding a new durable supplier-set schema.

Testing

• cargo test -p zakura-header-chain --lib
• cargo test -p zakura-state --lib
• cargo clippy -p zakura-header-chain --all-targets -- -D warnings
• cargo fmt --all -- --check
• ./scripts/changelog.py check-pr --base origin/main --head HEAD --pr 845

The replacement regression starts at a one-row aggregate auxiliary cap. The
transition adds one delivery while retention evicts the losing branch delivery.
The test verifies that the planner admits the transition and that the final
delta replaces the old delivery without exceeding the cap.

The saturation regression starts at the same cap without an eviction. It
verifies that the planner still returns AuxiliaryLimitExceeded. The
event-local test covers the per-header work bound. The invariant tests corrupt
candidates above the aggregate and per-header retained limits. Production and
exhaustive verification reject those candidates.

The semantic-diversity regression admits payload A and ignores the same payload
from another supplier. It then admits payload B and ignores another rooted
payload from B's supplier. The fingerprint regression proves that transport
metadata cannot make a duplicate payload distinct. Independent verification
rejects duplicate semantic payloads and multiple rooted payloads from one
supplier.

Follow-up Work

• Redesign continuous-sync stall detection in fix(continuous-sync): require evidence before halting stalled nodes #846.
• Add the state-owned repair episode and rejected-input protocol in fix(vct)!: require authoritative repair progress #847. A
  completed local failure can still repeat until that protocol lands.
• Add separate capacity metrics for each auxiliary limit before using another
  production run to attribute a refusal to a specific saturation mode.

[ValarDragon]

I assume we have a later plan to make this a precise constant and if you send the incorrect number you get banned?

[ValarDragon]

Ai finding which I imagine is fixed in follow-on's

1. High — Existing duplicate payloads survive upgrades and can preserve the livelock

The new invariant only checks semantic uniqueness for newly inserted delivery IDs (auxiliary.rs L121-L126). It never checks all retained rows pairwise.

That matters because pre-PR databases may legitimately contain multiple delivery IDs for an identical payload—the exact transport-retry behavior this PR addresses. Startup recovery
still checks only:

• delivery-ID uniqueness,
• foreign-key consistency,
• raw row-count limits.

It does not enforce semantic uniqueness or the one-rooted-payload-per-supplier rule (authoritative_rows.rs L62-L87).

I confirmed diagnostically that hydration accepts two different delivery IDs with the same semantic fingerprint.

Consequences:

• old retry duplicates continue consuming aggregate capacity;
• duplicates on protected/selected headers may not be removed by retention;
• an already saturated node can continue returning AuxiliaryLimitExceeded after upgrading;
• the projected state violates the newly documented LC-AUX-06 rule.

Recommendation: add an explicit durable migration or recovery repair that deterministically compacts existing semantic duplicates, then verify semantic/source uniqueness over the
entire final delivery set. Add an upgrade test starting from a v1.3 database containing duplicates.

2. Medium — Delete-plus-Put can bypass aggregate invariant accounting

The verifier calculates:

old count - all deleted IDs + only previously absent Put IDs

(auxiliary.rs L50-L70)

It does not reject the same delivery ID appearing in both a Delete and a Put.

For an existing ID, such a plan subtracts the deletion but does not add the replacement. Because auxiliary changes are applied in order, this sequence can finish above the aggregate
limit:

Delete(existing A)
Put(existing A)
Put(new B)

At a full aggregate limit, the verifier counts this as unchanged, while the committed state gains B. Repeating the pattern can hide multiple new rows.

PlanCandidate is sealed inside the crate, so there is no direct peer-controlled construction path today. Nevertheless, this defeats the independent verifier’s safety purpose if the
planner produces a conflicting delta now or later.

Recommendation: reject duplicate deletes and any delivery ID present in both Put and Delete, or calculate the final keyed delivery set by replaying changes in order. Test both Put/
Delete orderings at the exact aggregate cap.

[ValarDragon]

I did review this before, should've merged. I think this addresses the bug, and we just have a few more that are likely already fixed in follow-on's!

Nice job, and hope to tighten this with message expectation work!