multi: reorg-aware chain observation (chainsource + backends)

[ellemouton]

Summary

PR 1 of the six-PR basic-v1 reorg-safety stack tracked by
lightninglabs/lumos#454.

This PR is the chain-observation layer. It makes confirmation and spend
observation reversible end to end:

Observed → Reorged → Observed → Done

For txconfirm:

TxConfirmed → TxReorged → TxConfirmed → TxFinalized

The first positive observation is no longer terminal. Higher layers can keep a
fact provisional, react when it leaves the best chain, and observe it again
after reconfirmation.

What is included

• chainsource: multi-shot confirmation/spend actors, reorg and done
  events, ordered delivery, and height-based finality synthesis for transports
  that do not provide native Done. Transport boundaries explicitly defer a
  buffered Done until the positive confirmation/spend identity has crossed
  first.
• txconfirm: reversible confirmation notifications, a distinct terminal
  TxFinalized state, and correct terminal-only replay for late subscribers.
• LND/lndclient backend: preserve reorg-capable observation instead of
  terminating after the first positive event. Today this transport supplies a
  payload-less reorg ping, so chainsource synthesizes finality from block
  height.
• lwwallet/Esplora: detect same-height, shorter, deeper, and mid-catch-up
  replacements, then forward disconnected/connected blocks in order.
• Neutrino: forward confirmation/spend reorg and done notifications.
• Wallet subscriber compatibility: adapt the boarding-sweep txconfirm
  subscriber to the new four-event lifecycle. Without this, TxReorged and
  TxFinalized fall through the old mapper as failure-shaped notifications.
  Broader boarding spend-watch recovery belongs to Client PR 2.
• Harness and systests: real chainsource and txconfirm reorg round trips.

Native LND transport completion

This PR remains compatible with today's released lnd/lndclient by synthesizing
terminal Done at the chainsource boundary. The proper native transport path
is being completed in parallel:

• lightningnetwork/lnd#10943
  carries confirmation re-org depth and explicit confirmation/spend Done on
  the ChainNotifier gRPC streams.
• lightninglabs/lndclient#283
  exposes those native fields/events through lndclient.

After both land and Wavelength bumps the dependencies, the LND backend can use
native depth/Done directly while height synthesis remains a compatibility
fallback for older transports.

Deliberate boundary

This PR does not implement:

• batch/VTXO canonicality or lineage storage;
• registration-before-exposure;
• client/server admission gates;
• conditional restore;
• unroll lineage gating;
• one-confirmation VTXO exposure; or
• the final configured policy-finality boundary.

FinalityDepth is configurable observation policy. This PR retains the
interim default of 6 because no one-confirmation lineage is exposed by it.
Client PR 2 makes the shared basic-v1 terminal boundary configurable so a
deployment can choose how deep a reorg remains recoverable.

This is necessary observation plumbing; it does not, by itself, claim
end-to-end reorg safety through the configured policy depth.

Basic-v1 stack

1. this PR — reorg-aware chain observation and required subscriber
   compatibility.
2. client — configurable-depth boarding/unroll recovery foundation and a
   disabled capability.
3. client — durable complete-lineage batch canonicality, fail-closed
   restart reconciliation, and objective conditional restore.
4. client — registration-before-exposure and every one-confirmation
   lineage admission/pre-side-effect gate.
5. server — configurable provisional safety, reversible operations, and
   complete evidence/admission/pre-side-effect gates.
6. server itests — prove running and offline recovery through a configured
   test depth, sealing at that policy boundary, and the final S1-S5/R2/F9
   contract across both sides.

Generic automatic replay, durable lineage waiters, per-operation usability
depths, and detection after the configured safety horizon remain follow-ups.

Validation

At 233abdd6:

• go test ./lwwallet ./txconfirm -count=1
• go test -race ./lwwallet -run TestTipPollerReorgDuringMultiBlockAdvance -count=10
• go test -race ./lwwallet -run TestTipPollerDeeperForwardReorg -count=100
• go test -race ./lwwallet -count=10
• go test -race ./txconfirm -run TestLateFinalizedSubscriberRetrySkipsConfirmed -count=10
• go test -race ./chainsource ./chainbackends ./btcwbackend -count=5
• focused positive-before-Done regressions across those three packages,
  -count=20
• make lint-changed-local — 0 issues

The prior unit-race failure exposed nondeterminism in the deeper-forward
TipPoller test harness: its background poll could observe the deliberately
mutated stub chain between the test's two intended snapshots. That test now
drives the two observations manually and passed 100 focused race runs plus ten
full-package race runs. Fresh full CI is running for this rewritten head. A
human approving review remains the final merge gate.

[gemini-code-assist]

Warning

Gemini encountered an error creating the review. You can try again by commenting /gemini review.

[ellemouton]

@claude review this

[claude]

Claude finished @ellemouton's task in 1m 40s —— View job

---

Review: reorg-aware chain observation

I reviewed the ~9.4k-line diff (4a1f9b1^...HEAD) across all five subsystems. Overall this is careful, well-tested work: the inclusive finality-depth math (epoch.Height - confirmHeight + 1, firing at H + FinalityDepth - 1) is off-by-one-correct on both the epoch and on-arm paths, the FSM reorg lifecycle (Confirmed → Reorged → re-Confirmed → Done) has no stuck-state I could find, the arming context lifetimes match the intent of the recent commits, and the actor paths avoid unbounded batching. The findings below are the exceptions, most severe first.

Findings came from a deep multi-pass read; I verified the top two against the source directly. A couple are flagged as "needs confirmation" where I couldn't read pinned external deps.

---

🔴 High

1. boarding_sweep_actor: a genuine post-reorg failure is silently suppressed (accounting divergence). wallet/boarding_sweep_actor.go:1186-1212 (Reorged arm) is a pure no-op, but the first TxConfirmed already drove the store to a terminal-success status: the Confirmed arm calls reconcileSweepInputsOnConfirm (:1182), which via MarkBoardingSweepInputSpent flips the sweep row to confirmed/external_resolved (db/boarding_sweep_store.go:406-425) on a provisional confirmation. The TxFailed guard (:1237-1252) then ignores any failure whenever isTerminalSuccessSweepStatus(rec.Status). Scenario: our sweep confirms → store=confirmed + ledger deposit emitted → block reorged out → Reorged no-ops → a competing spend of the boarding output confirms on the new chain → txconfirm emits TxFailed → guard sees confirmed and suppresses it. End state: store + ledger report a confirmed sweep, but the funds left via another tx, with only a WARN. A confirmation should not be treated as irreversible until Finalized is observed. Fix this →

---

🟠 Medium

2. chainsource: finality synthesis is permanently disabled if the block-epoch stream closes after arming. In the epoch case (conf_actor.go:434-438, spend_actor.go:404-408), a closed channel does blockEpochs = nil; continue but never clears a.blockReg. The re-arm guard (conf_actor.go:354, spend_actor.go:325) requires a.blockReg == nil, so once the epoch stream closes it can never re-arm. This is reachable: the lndclient block-epoch forwarder closes epochChan on any stream hiccup (chainbackends/lnd.go:526-531, independent of the still-live conf stream), and since lndclient never writes the backend Done, the watch is stranded in provisional until daemon restart — exactly what the arm-retry logic was meant to prevent. (I confirmed blockReg is never reset.) Fix this →

3. chainbackends: gRPC conf/spend stream errors are logged but never propagated. The errChan reader goroutines (lndclient_adapters.go:206-219 conf, :364-377 spend) emit a WarnS and exit without calling cancel(). lndclient does not close confChan/spendChan on Recv() error, so forwardOrderedReorg keeps blocking and the downstream orderedConfirmed channel is never closed — meaning chainsource's failConfirmation (conf_actor.go:293-307, built precisely to fail the watch on a closed Confirmed) never fires. If the stream dies before the first event, finality is never armed and the watch hangs indistinguishably from a slow confirmation (the 15s timeout only bounds the initial Register* call). Contrast the block-epoch forwarder (lnd.go:526-531), which correctly returns on errChan. Fix this →

4. txconfirm: the per-tick retry of a timed-out initial TxConfirmed is unreachable (regressed at-least-once delivery). retryTerminalNotifications is only called under if isTerminalTxState(state) (actor.go:1064, :1276), and this PR made Confirmed non-terminal (isTerminalTxState returns true only for Finalized/Failed, :1233-1241). So the *trackedTxStateConfirmed case (:1892) and retryConfirmedRedelivery are now dead from the tick path — yet five comments (e.g. :1899, :1916-1925) still document them as running "every actor tick". Pre-PR the gate was state == Confirmed || state == Failed. Effect: if a legacy retry-tracked subscriber's mailbox is saturated >terminalNotifyTimeout at confirmation, pendingConfirmed stays true but is never retried; on the lndclient backend (no Done) that TxConfirmed is permanently lost. (Verified the gating myself.) Fix this →

5. lwwallet: a transient Esplora error fires a spurious Reorged on the spend path. processReorgEvent calls reorgSpendReg for every active spend registration on each reorg (including ones whose block wasn't disconnected); those hit the fallback checkSingleSpend (chain_backend.go:1478-1490), which collapses transient HTTP error, "not spent", and parse failure all to (nil, {}). With current == nil the guard at :1046 falls through and fires Reorged. The conf path deliberately avoids this via the tri-state confirmedBlockHash (:1390-1438, "firing Reorged here would strand the consumer on a false alarm"); the spend path lacks the equivalent guard. Self-heals next tick but emits a false reorg to the SpendActor. Fix this →

6. boarding_sweep_actor: pendingSweeps entries + chainsource spend sub-actors leak; the Finalized arm is a no-op. The only successful-path cleanup (cancelSweepSpendWatches + delete(pendingSweeps)) lives in the err == nil branch of handleSweepSpendNotification (:1114-1151). But reconcileSweepInputsOnConfirm on TxConfirmed (:1182) marks the input rows spent first, so the real per-input spend events then hit the status guard, get ErrNoRows, and return early at :1101 before cleanup. Whenever TxConfirmed wins the race against the spend notifications (both fire on the same block), the in-memory maps and registered SpendActors leak. The Finalized arm (:1214-1226) — the correct terminal signal — should do this cleanup but only logs.

7. btcwbackend / lwwallet: Done may be swallowed if lnd closes the channel instead of sending (needs confirmation). The forwarders (btcwbackend/chain_backend.go:466-479, :579-592) treat event.Done as value-carrying: case _, ok := <-event.Done: if !ok { event.Done = nil; continue }, forwarding Done only on ok == true. If lnd's chainntnfs closes ConfirmationEvent.Done/SpendEvent.Done at reorg-safety depth rather than sending, ok == false and Done is never forwarded. The unit test uses a value-send stub (chain_backend_reorg_test.go:136) so it wouldn't catch this. Worth verifying against the pinned lnd revision whether Done is sent-on or closed.

---

🟡 Low / informational

• chainsource: lastSeq starts at 0, so a buffered Reorged (seq ≥ 1) can be applied before its preceding first Confirmed is dequeued (conf_actor.go:399-422), emitting a ConfReorgedEvent with confirmHeight == 0/lastEvent == nil for a txid the consumer never saw confirmed. End state stays correct; worth gating deliverConfReorged on a prior positive event or documenting the guarantee.
• lwwallet: the TipPoller poll/one-shot Esplora calls use context.Background(), so Stop() can't cancel an in-flight HTTP request — shutdown blocks on the HTTP client's own timeout (tip_poller.go, chain_backend.go one-shots). EsploraChainService threads runCtx correctly; align these.
• lwwallet: EsploraChainService.processReorgEvent (esplora_chain.go:787-793) transiently zeroes the cached tip Hash/Timestamp (sets only Height), so a concurrent BlockStamp() mid-reorg returns a zero hash at a nonzero height — an invalid stamp a future caller could trust.
• chainbackends: the "priority draining" comment (lndclient_adapters.go:229-232, mirrored for spend) contradicts the actual unbiased 3-way select (and the function's own doc). Ordering holds via sequential blocking sends + highest-seq-wins, not priority — recommend deleting the misleading claim.
• chainbackends: RegisterBlocks forwarder (lnd.go:588-630) still captures the request-scoped ctx for its whole lifetime, the pattern this PR fixed for conf/spend (notifyCtx from context.Background()). Benign today (sole caller passes the actor root ctx) but inconsistent and latent.
• boarding_sweep_actor: the Reorged arm comment claims it "leaves spend watches and pending state armed" for a re-reconcile, but in the common case the first TxConfirmed already cleared/orphaned that state, so the promised re-run doesn't happen — misleading invariant.

---

Not verified here (live outside these files / external deps): ledger idempotency on re-confirm is asserted by comment only; the lnd Done send-vs-close semantics for #7. Everything else was traced in-repo.
· reorg-observe

[ellemouton]

@claude please review this PR one more time for merge readiness.

[claude]

Claude finished @ellemouton's task in 2m 3s —— View job

---

Merge-readiness re-review

• Confirm branch state vs. prior review
• Re-verify the boarding-sweep accounting + leak findings
• Re-verify chainsource re-arm + chainbackends error-propagation findings
• Re-verify the txconfirm retry-path regression
• Give a go / no-go recommendation

Branch state: no commits have landed since my previous review — HEAD is still 3412852. I re-read the code directly rather than trusting the earlier pass, and all seven findings are still present verbatim. I re-confirmed the four that matter most for a merge decision:

#	Location	Status (re-verified)
1	wallet/boarding_sweep_actor.go:1186 / :1237	✅ still present — Reorged arm is a WarnS-only no-op; TxFailed is suppressed whenever isTerminalSuccessSweepStatus(rec.Status), and the provisional first TxConfirmed already drove that status via reconcileSweepInputsOnConfirm (:1182).
2	chainsource/conf_actor.go:434 / spend_actor.go:404	✅ still present — epoch-close does blockEpochs = nil; continue but never clears a.blockReg; the re-arm guard at :354 requires a.blockReg == nil, so synthesis can never re-arm.
4	txconfirm/actor.go:1276 / :1892	✅ still present — retryTerminalNotifications runs only under isTerminalTxState (Finalized/Failed only, :1233), so the *trackedTxStateConfirmed retry (:1892) is dead from the tick path while :1919 still documents it as running "every actor tick".
6	wallet/boarding_sweep_actor.go:1214	✅ still present — the Finalized arm is log-only; the sole pendingSweeps/spend-sub-actor cleanup lives on the resolved branch of the spend path (:1147), which reconcileSweepInputsOnConfirm can front-run.

Findings #3, #5, #7 and the Low items are likewise unchanged.

---

Recommendation: not yet — a few in-scope correctness items should land first

This is careful, well-tested work and the observation core is sound. But three of the open findings are correctness regressions inside the exact plumbing this PR ships (the chain-observation layer), not in the deferred PR-2 scope, and one is a silent data-integrity suppression. I'd treat these as merge blockers:

Blockers (in-scope, correctness):

• multI: add initial repo scaffolding #2 — finality synthesis strands after an epoch-stream close. This is in the heart of the PR. On the lndclient backend (which never sends backend Done), a single block-epoch stream hiccup leaves the watch permanently provisional until daemon restart — defeating the arm-retry logic this PR introduces. Fix →
• multi: stage boarding actor client  #4 — timed-out initial TxConfirmed is never retried. Making Confirmed non-terminal silently killed the per-tick retryConfirmedRedelivery. On a backend without Done, a TxConfirmed lost to a saturated mailbox is now lost permanently — a regression in the at-least-once contract the code still claims in comments. Either re-run the redelivery for Confirmed entries each tick, or update the now-false comments and confirm the loss is acceptable. Fix →
• chainbackend+chainsource: add chainsource actor and backend impl #3 — gRPC conf/spend stream error never propagates. The errChan readers WarnS and exit without cancel(), so failConfirmation never fires and a stream that dies before the first event hangs indistinguishably from a slow confirmation. The block-epoch forwarder already gets this right (lnd.go:526); mirror it. Fix →

Strongly recommend before merge (data integrity, but touches PR-2 territory):

• lib: start adding lib helpers #1 — silent suppression of a genuine post-reorg failure. A confirmation is treated as irreversible before Finalized, so a competing spend confirming after a reorg is swallowed to a WARN with the store/ledger left reporting a sweep that no longer happened. I understand the boarding spend-watch recovery is explicitly deferred to Client PR 2 — but this PR is what makes Confirmed reversible and what re-shapes this subscriber, so shipping a state where a real divergence is silently masked is worth closing (or at minimum gating the suppression on Finalized and filing a tracked follow-up with a code pointer). Fix →

Can be follow-ups (leak / cosmetic / needs-confirmation):

• [test, ignore] harness: basic infrastructure test environment #6 (pendingSweeps + spend sub-actor leak — move cleanup into the Finalized arm), harness: basic infrastructure test environment #5 (spurious spend-path Reorged on transient Esplora error; self-heals next tick), multi: clean up unnecessary stuff and use instance logging in favor of global logger #7 (verify lnd Done send-vs-close semantics against the pinned revision), and the Low/informational items (stale comments in lndclient_adapters.go:229, RegisterBlocks request-ctx capture, TipPoller context.Background() in one-shots, BlockStamp() zero-hash window, lastSeq-starts-at-0). These don't block correctness of the merged result but [test, ignore] harness: basic infrastructure test environment #6 in particular will accumulate goroutines/registrations over a long-lived daemon and should not linger long.

---

Bottom line: the design and test coverage are in good shape, but #2, #3 and #4 are in-scope regressions/hangs in the observation layer itself and #1 is a silent divergence — I'd want at least those addressed (or #1 explicitly deferred with a tracked issue + code comment) before merge. Everything else is fair game for a follow-up. If you'd like, comment @claude fix #2 #3 #4 and I'll implement the three blocking fixes with tests on this branch.
· reorg-observe

[ellemouton]

@claude please re-review updated head b43d2b3 for merge readiness. The prior #1-#6 findings were addressed with regression tests; please verify the fixes and flag any remaining blockers.

[claude]

Claude finished @ellemouton's task in 3m 43s —— View job

---

Merge-readiness re-review — HEAD b43d2b3

• Inspect the fix commit b43d2b3
• Verify lib: start adding lib helpers #1 — boarding-sweep post-reorg failure suppression
• Verify multI: add initial repo scaffolding #2 — chainsource finality re-arm after epoch close
• Verify chainbackend+chainsource: add chainsource actor and backend impl #3 — chainbackends gRPC stream error propagation
• Verify multi: stage boarding actor client  #4 — txconfirm confirmed-redelivery retry path
• Verify harness: basic infrastructure test environment #5 — lwwallet spurious spend-path Reorged
• Verify [test, ignore] harness: basic infrastructure test environment #6 — boarding-sweep pendingSweeps/sub-actor leak
• Flag remaining blockers + go/no-go

One new commit landed since my last pass — b43d2b3 "multi: Fix reorg observation lifecycle gaps" (+609/−126 across the six subsystems). I re-read each fix against source and confirmed each ships with a regression test.

#	Fix (verified in b43d2b3)	Regression test	Status
2	clearFinalityRegistration() now nils a.blockReg on epoch-stream close (direct close and the drainToLatestEpoch close path), then re-arms when confirmHeight != 0. The re-arm guard blockReg == nil && !arming now passes. (conf_actor.go:436-522, spend_actor.go:405-489)	TestConfActorRearmsFinalityAfterEpochStreamClose, TestSpendActorRearms…	✅
3	Both errChan reader goroutines call cancel() on stream error (lndclient_adapters.go:216, :374). That cancels ctx, forwardOrderedReorg returns, its deferred close(orderedConfirmed) fires → failConfirmation fires downstream.	TestLndClientConfStreamErrorClosesRegistration, …SpendStream…	✅
4	handleBlockObserved now retries Confirmed entries every tick via an explicit if state == TxStateConfirmed { retryLifecycleNotifications; continue } (actor.go:1282-1286); helper renamed and comments corrected. retryConfirmedRedelivery is live again.	TestLifecycleDeliveryRetriesAfterTellFailure (now drives an emitBlock tick, not emitConfDone)	✅
5	checkSingleSpend returns a third error value; reorgSpendReg and deliverSpendIfNew bail on error instead of collapsing transient HTTP/parse failure into (nil, {}) → no spurious Reorged. (chain_backend.go:1055-1065, :1491-1545)	TestChainBackendSpendReorgSkipsTransientStatusFailure	✅
1	The provisional first TxConfirmed no longer commits the store: the Confirmed arm is now log-only, and handleSweepSpendNotification defers the sweep's own-tx input spend (SpendingTxid == pending.txid) until finality (:1093-1112). Store/ledger transitions moved to reconcileSweepInputsOnFinalized + emitSweepConfirmedLedger in the Finalized arm. The TxFailed guard can now only suppress once the record is genuinely terminal-success (finalized, or externally resolved by a real competing spend) — a real post-reorg divergence is no longer masked.	TestSweepTxNotificationConfirmedRemainsProvisional, …FinalizedEmitsLedger, …FinalizedExternalDestSkipsCreated	✅
6	The Finalized arm now performs the sole cleanup — delete(a.pendingSweeps, txid) + cancelSweepSpendWatches(pending) (which also clears pendingSweepInputs and unregisters the SpendActors). Because own-tx spends are deferred (fix #1), they no longer front-run and orphan this state.	TestSweepTxNotificationFinalizedCommitsAndCleansUp	✅

No dangling references to the removed reconcileSweepInputsOnConfirm, and the Finalized arm ordering is correct (reconcile reads pending.inputs before cancelSweepSpendWatches clears them). The misleading "priority draining" comment (a prior Low item) was also corrected.

Note: local test execution is blocked in this sandbox (no exec permission for go test/make unit), so I verified the fixes and their tests by source inspection rather than by running them here. The fixes each carry a targeted regression test and CI runs the full suite.

---

Recommendation: go — no remaining in-scope blockers

All four items I flagged as merge blockers last round (#2, #3, #4, and the #1 data-integrity suppression) are resolved, and the two leak/false-alarm items (#5, #6) are fixed too. The fixes are minimal, match the intended lifecycle (Confirmed provisional → Finalized commits), and don't introduce new stuck states that I could find.

Follow-up, non-blocking (unchanged from before):

• multi: clean up unnecessary stuff and use instance logging in favor of global logger #7 — verify lnd's chainntnfs Done is sent-on vs closed at reorg-safety depth against the pinned revision; the btcwbackend forwarders (chain_backend.go:466, :579) treat it as value-carrying, and the unit stub uses a value-send. Worth a quick confirmation but outside this diff's control.
• The remaining Low/informational items from the first review (RegisterBlocks request-ctx capture in lnd.go, TipPoller context.Background() in one-shots blocking Stop(), the BlockStamp() zero-hash window in esplora_chain.go:787, and lastSeq-starts-at-0 in conf_actor.go) are all self-healing or benign-today and fine as follow-ups.

This is good to merge from a correctness standpoint; human review remains the final gate.
· reorg-observe

[ellemouton]

@claude please re-review the latest head. An independent whole-substrate pass found two additional cases now folded into their owning commits: TipPoller verifies PrevBlock continuity at every height of a multi-block catch-up (including a reorg landing mid-walk), and a late subscriber attaching to an already-finalized tx no longer receives a spurious TxConfirmed on delivery retry. Please focus on those fixes and their regression tests, and flag any remaining merge blocker.

[claude]

Claude finished @ellemouton's task in 2m 54s —— View job

---

Re-review: latest head — two folded fixes + merge-readiness

• Confirm branch/head state
• Verify TipPoller PrevBlock continuity across multi-block catch-up (+ mid-walk reorg)
• Verify late-finalized-subscriber no spurious TxConfirmed on retry
• Check regression tests for both
• Flag any remaining merge blocker

I re-read the checked-out reorg-observe working tree directly. The two new cases described are folded into their owning commits and both are correct with targeted regression tests. All six prior blockers remain resolved.

Note: go test/make unit are not executable in this sandbox, so I verified by source inspection; CI runs the full suite.

---

1. TipPoller PrevBlock continuity at every catch-up height ✅

advance() (lwwallet/tip_poller.go:699-804) now fetches the raw header at every height in the oldHeight+1..newHeight walk and checks rawHdr.PrevBlock != previousHash (:769), advancing previousHash = hash only after a clean broadcast (:802). Checking only the first height would have missed a reorg landing after an earlier block in the same walk was already broadcast — the comment at :748-750 calls this out precisely.

The mid-walk case is handled correctly: on mismatch it re-resolves the live tip at newHeight and calls handleReorg(newHeight, tipHash) (:787). Because the earlier iterations already ran broadcastTipBlock (advancing t.tipHeight and recording hashes), handleReorg sees cachedTipHeight at the last-broadcast height and builds Disconnected over forkHeight+1..cachedTipHeight (:1007-1011) — so the blocks broadcast during this very walk are the ones disconnected. No out-of-order or leaked-forward event.

Regression test TestTipPollerReorgDuringMultiBlockAdvance (tip_poller_reorg_test.go:282) swaps the chain precisely on the /block-height/112 request mid-walk and asserts ForkHeight==110, Disconnected==[orphaned111], and Connected==[111,112] in ascending order — exactly the mid-walk boundary. Good coverage; note the whole-cycle abort on a failed raw-header fetch (:754-767) preserves the "never emit out-of-order" guarantee.

2. Late-finalized subscriber: no spurious TxConfirmed on retry ✅

The fix is in attachExistingSubscriber (txconfirm/actor.go:1375-1390): the *trackedTxStateFinalized case sets subscriber.pendingConfirmed = false unconditionally before attempting notifyOneFinalized, and only re-attaches on failure. This is the crux — a late attacher never observed the tx confirm live, and TxFinalized already carries the authoritative ConfirmHeight, so it is not owed an initial TxConfirmed.

The consequence is correct on both retry seams:

• The per-tick retry routes a Finalized entry to notifyFinalized (:1909-1910), not retryConfirmedRedelivery. With pendingConfirmed == false, the if subscriber.pendingConfirmed guard in notifyFinalized (:2215) is skipped, so only TxFinalized is (re)delivered.
• retryConfirmedRedelivery itself early-continues on !pendingConfirmed (:1936), so it is a no-op for this subscriber regardless.

The distinction from a live subscriber is preserved: one that rode Confirmed→Finalized with a still-pending initial delivery keeps pendingConfirmed == true and correctly gets TxConfirmed before TxFinalized (:2214-2233), honoring the at-least-once contract. Only the late-attach path is special-cased.

Regression test TestLateFinalizedSubscriberRetrySkipsConfirmed (actor_test.go:1036) attaches with pendingConfirmed: true, forces the first delivery to fail (retained, attemptsCount==1), then drives notifyFinalized and asserts the sole message is TxFinalized with no trailing notification (:1069-1074). Directly exercises the regression.

---

Prior blockers — still resolved at this head

Re-confirmed each survived the fold into its owning commit:

#	Check	Status
1	reconcileSweepInputsOnConfirm fully removed (no refs); commit is now log-only, store/ledger transitions live in reconcileSweepInputsOnFinalized + emitSweepConfirmedLedger on the Finalized arm (boarding_sweep_actor.go:1237-1238)	✅
2	clearFinalityRegistration() nils a.blockReg (conf_actor.go:523, spend_actor.go:484) and is called on both epoch-close paths so synthesis can re-arm	✅
3	conf + spend errChan readers call cancel() on stream error (lndclient_adapters.go:216, :374)	✅
4	if state == TxStateConfirmed { retryLifecycleNotifications } on the block-tick path (actor.go:1283-1284) — Confirmed redelivery live again	✅
5	checkSingleSpend returns an explicit error (chain_backend.go:1497) so a transient Esplora failure no longer collapses to a spurious Reorged	✅
6	Finalized arm performs the sole cleanup, correctly ordered — reconcile reads pending inputs (:1237) before cancelSweepSpendWatches clears them (:1243)	✅

Recommendation: go — no remaining in-scope merge blocker

Both folded fixes are minimal, match the intended lifecycle, and carry regression tests that exercise the exact failure they close. I found no new stuck states or ordering regressions.

Non-blocking follow-ups (unchanged): #7 (verify lnd chainntnfs Done is sent-on vs closed against the pinned revision — the btcwbackend forwarders treat it as value-carrying) and the Low/informational items (RegisterBlocks request-ctx capture, TipPoller context.Background() one-shots blocking Stop(), the BlockStamp() zero-hash window, lastSeq-starts-at-0). All self-healing or benign-today.

Human review remains the final merge gate.
· reorg-observe

[ellemouton]

The prior unit-race failure in TestTipPollerDeeperForwardReorg was a test-harness race, not a production TipPoller failure. The background poller could observe the deliberately mutated stub chain between the test's two intended snapshots, splitting the expected six-block reorg into two valid observations.

I made only that test use a one-hour background interval and explicitly call poll() after each atomic stub-chain mutation. This preserves the scenario while making the snapshot under assertion deterministic. Local verification: 100 focused race runs and 10 full lwwallet race runs passed. The fix is squashed into the original lwwallet commit; fresh CI is now running on 292394eb.

[ellemouton]

@claude please perform one final whole-PR review of head 233abdd6, with special attention to the new positive-before-Done ordering invariant.

Independent tracing found that chainntnfs exposes Confirmed/Spend and Done on separate size-1 buffered channels. For an already-deep observation, both may be ready and Go can select Done first even though the notifier emitted the positive event first. That could previously terminate a forwarder/watch without ever delivering the confirmation block metadata or spender identity.

The rewritten head now uses one small chainsource.PositiveDoneOrder state machine at the LND, Neutrino, ConfActor, and SpendActor boundaries. Done is retained until the matching positive observation crosses; an intervening reorg clears a merely pending terminal signal. Regression tests force Done first for confirmation and spend at all three layers. Focused tests passed 20x under -race, the three full packages passed 5x under -race, and make lint-changed-local reports 0 issues. Please check the whole PR rather than only this delta and report any merge blocker.

[claude]

Claude finished @ellemouton's task in 3m 31s —— View job

---

Final whole-PR review — head 233abdd6

• Confirm head state (233abdd6, 6 commits)
• Review the PositiveDoneOrder state machine
• Verify positive-before-Done at ConfActor / SpendActor
• Verify positive-before-Done at LND + Neutrino forwarders
• Re-confirm prior blockers (lib: start adding lib helpers #1–[test, ignore] harness: basic infrastructure test environment #6) still resolved
• Whole-PR sweep for new merge blockers

I re-read the checked-out tree at 233abdd6 directly, with the requested focus on the new ordering invariant, then swept the rest of the diff.

Note: go test/make are not executable in this sandbox, so I verified by source inspection; the focused/full -race runs and make lint-changed-local reported in the PR body are the execution evidence, and CI runs the full suite.

---

PositiveDoneOrder — the ordering invariant is correct ✅

The core insight is right: chainntnfs (and the lndclient bridge) expose the positive event and Done on independent size-1 buffered channels, so a select over both can pick Done first for an already-deep observation — terminating a watch/forwarder with no block metadata or spender identity. The PositiveDoneOrder machine (chainsource/lifecycle_order.go) closes this cleanly:

• ObserveDone() returns true only if a positive already crossed; otherwise it latches pendingDone and the caller retains the terminal signal.
• ObservePositive() returns the latched pendingDone, so the deferred Done is released immediately after the positive is delivered.
• ObserveReorg() clears both positive and pendingDone, so a Done selected before any positive cannot survive an intervening reorg.

Applied consistently and symmetrically at all four boundaries I was asked to check:

Boundary	Positive→Done release	Retain-on-early-Done	Reorg clears pending
LND conf (chainbackends/lnd.go:412/430/437)	✅	✅ (event.Done = nil)	✅
LND spend (lnd.go:543/561/568)	✅	✅	✅
Neutrino conf (btcwbackend/chain_backend.go:635/655/663)	✅	✅	✅
Neutrino spend (chain_backend.go:759/779/787)	✅	✅	✅
ConfActor (chainsource/conf_actor.go:347/418/426, helpers :513-551)	✅	✅ (registration.Done = nil)	✅
SpendActor (spend_actor.go:319/399/406)	✅	✅	✅

I checked the two failure shapes that matter and neither hangs: the deep-observation case (Done+positive both buffered) delivers positive→Done in order and terminates; disabling the drained one-shot (Done = nil) after latching also correctly avoids a spin if the upstream closes rather than sends Done (the open question from finding #7). The actor-level doneOrder is belt-and-suspenders over the forwarder's, and the lastSeq highest-seq-wins gate composes correctly — a stale (discarded) positive does not call ObservePositive, so it can't prematurely release a latched Done.

Prior blockers #1–#6 — all still resolved at this head ✅

Re-verified each survived the squash to 6 commits:

#	Check at 233abdd6	Status
1	reconcileSweepInputsOnConfirm gone; Confirmed arm is log-only (boarding_sweep_actor.go:1192); store/ledger commit + TxFailed-suppression gate moved to the Finalized path so a genuine post-reorg failure is no longer masked (:1237, :1255)	✅
2	clearFinalityRegistration() nils blockReg (conf_actor.go:561, spend_actor.go:497) and runs on both epoch-close paths → synthesis re-arms	✅
3	conf+spend errChan readers cancel() on stream error → failConfirmation/failSpend fire	✅
4	Confirmed redelivery live on the block-tick path (non-terminal Confirmed)	✅
5	checkSingleSpend returns an explicit error → transient Esplora failure no longer collapses to a spurious Reorged	✅
6	Sole pendingSweeps/sub-actor cleanup on the Finalized arm, correctly ordered (reconcile reads pending inputs at :1237 before cancelSweepSpendWatches at :1243); own-tx spends deferred until finality (:1103) so they can't front-run it	✅

---

Recommendation: go — no remaining in-scope merge blocker

The positive-before-Done invariant is implemented correctly and uniformly across the LND, Neutrino, ConfActor, and SpendActor boundaries, with regression tests forcing Done-first at each layer. All six prior blockers remain fixed. I found no new stuck states, ordering regressions, or hangs in the whole-PR sweep.

Non-blocking follow-ups (unchanged + one new low observation):

• New (low / out-of-scope): once a Done is latched (registration.Done = nil / event.Done = nil) and then an intervening reorg clears it via ObserveReorg(), the one-shot Done channel stays disabled — a native-Done backend (Neutrino) that re-emits Done after the re-confirmation would not be re-observed. This is only reachable when Done was buffered before the first positive and a reorg then unwinds a confirmation already at reorg-safety depth — i.e. a reorg deeper than the safety horizon, which is exactly the deferred out-of-scope territory. lndclient/lwwallet are unaffected (finality comes from height synthesis, a separate arm). Worth a one-line comment noting the latched channel is intentionally not re-enabled, but not a blocker.
• multi: clean up unnecessary stuff and use instance logging in favor of global logger #7 — verify lnd chainntnfs Done is sent-on vs closed at reorg-safety depth against the pinned revision (forwarders now tolerate a close via the Done = nil guard, so this is lower-risk than in the first review, but still worth confirming).
• The remaining Low/informational items (RegisterBlocks request-ctx capture in lnd.go, TipPoller context.Background() one-shots blocking Stop(), the BlockStamp() zero-hash window in esplora_chain.go, lastSeq-starts-at-0) are all self-healing or benign-today.

This is good to merge from a correctness standpoint; human review remains the final gate.
· reorg-observe

[github-actions]

Successfully created backport PR for v0.1.x-branch:

• [v0.1.x-branch] Backport #895: multi: reorg-aware chain observation (chainsource + backends) #981