fix(blockassembly): stop re-marking the whole mempool on every reorg

CHANGELOG.md

@@ -106,6 +106,47 @@ Upgrading from v0.16.0 changes nothing for them; upgrading from an earlier relea
 
 ### Breaking — behaviour changes on upgrade
 
+- **A reorg no longer re-marks block assembly's whole mempool.** Block assembly marks
+  only the transactions it moved back and that are not mined on the new chain. Block
+  validation now repairs a block it marked mined that a competing chain later overtakes,
+  on its setMined worker, in slices of `blockvalidation_tipRepairSliceSize` transactions
+  (new setting, default 250000); each slice can delay the next block's mined status by
+  one slice. **The pruner deletes nothing while a repair is unfinished.** Block validation
+  records the overtaken blocks it has not finished repairing under the blockchain state
+  key `OvertakenTipRepairHold`, before it sets the mined status of the block that
+  overtook them, and the pruner skips UTXO deletion (phase 2) and blob deletion while any
+  is listed; parent preservation (phase 1) still runs, and deletion resumes on its own
+  once the repair finishes. A subtree of an overtaken block that cannot be read is tried
+  again on every block and holds pruning until it reads or the block leaves the
+  1000-block repair window. A subtree or block that is gone from its store can never be
+  read: a gone subtree is given up at once, and a gone block after 10 attempts, both
+  counted in `teranode_blockvalidation_tip_repair_abandoned_total`. Their transactions
+  may stay marked mined while off the chain, and the pruner can then delete parents they
+  need. `teranode_pruner_overtaken_tip_repair_held_blocks` and
+  `teranode_blockvalidation_tip_repair_subtrees_set_aside` show a hold; a hold that stays
+  above zero means a subtree file is corrupt or a store keeps failing, and disk use grows
+  until it is resolved. A repair that keeps failing no longer stops healthy blocks: their
+  mined status does not wait for it. The hold survives a restart, and the blocks it lists
+  are repaired after it. **A single block validation instance must write mined status
+  for the chain**: the hold is one record, so two instances can release each other's
+  hold. A block overtaken while block validation is down and more than 10 blocks below
+  the tip when it starts is not repaired. Upgrade the pruner no later than block
+  validation: a newer block validation with an older pruner writes the hold but nothing
+  reads it. Rolling block validation back after it wrote a non-empty hold leaves the
+  record in place, and a newer pruner keeps holding until block validation is upgraded
+  again or the `OvertakenTipRepairHold` state row is deleted. No action needed for a
+  single-instance deployment. (#2065)
+- **A missing blockchain state key is a not-found error.** `GetState` on the SQL
+  blockchain store returned the raw `sql.ErrNoRows` for a key never written; it now
+  returns an `ERR_NOT_FOUND` error wrapping it, whose code survives gRPC. Callers that
+  match `sql.ErrNoRows`, with `errors.Is` or on its text, still match, locally and over
+  gRPC; new callers should match `errors.ErrNotFound`. (#2065)
+- **`GetLastNBlocks` honours `fromHeight` when orphans are included.** The SQL
+  blockchain store ignored `fromHeight` with `includeOrphans=true` and always returned
+  the highest blocks. It now returns only blocks at or below `fromHeight`, as documented.
+  This changes the asset API's block list endpoints when a client passes both
+  `includeOrphans` and `fromHeight`: they now page down instead of repeating the top of
+  the chain. (#2065)
 - **A short catch-up from `RUNNING` hands its blocks' unseen transactions to block
   assembly, and catch-up no longer fetches a chain with less work than the best tip.**
   Every catch-up moves the FSM to `CATCHINGBLOCKS`, and subtree validation used to stop
@@ -857,21 +898,21 @@ Upgrading from v0.16.0 changes nothing for them; upgrading from an earlier relea
 - **Conflict-check failures in subtree validation are no longer reported as
   context cancellations.** Two follow-ups to the entry above, both in subtree
   validation's conflict check:
-  - On the fresh path (`CheckBlockSubtrees` on a new subtree, `CheckSubtreeFromBlock`,
-    live peer subtrees), a store fault, the `utxostore_conflictingChildrenMaxNodes`
-    walk limit, a frozen output, an input out of range and the fail-fast
-    missing-tx threshold are no longer wrapped as a context cancellation. Each is
-    still retried and never charged to a peer. What changes is how they show up:
-    on the live path `subtreeMessageHandler` logs them at error level ("error
-    processing kafka message") where it logged a "Context canceled, skipping"
-    warning, and a catch-up run that ends on the walk limit, a frozen output or an
-    out-of-range input records `local_processing_fault` instead of
-    `local_context_cancelled` as its last error type. A missing record, a
-    cancelled check, and a record the store could not decode keep the old
-    classification.
-  - On the fresh path a proven conflict is rejected with `TX_INVALID` even when
-    the check's context is cancelled while it runs, as the cached path and
-    blessing a missing transaction already did. Before, it was retried.
+    - On the fresh path (`CheckBlockSubtrees` on a new subtree, `CheckSubtreeFromBlock`,
+      live peer subtrees), a store fault, the `utxostore_conflictingChildrenMaxNodes`
+      walk limit, a frozen output, an input out of range and the fail-fast
+      missing-tx threshold are no longer wrapped as a context cancellation. Each is
+      still retried and never charged to a peer. What changes is how they show up:
+      on the live path `subtreeMessageHandler` logs them at error level ("error
+      processing kafka message") where it logged a "Context canceled, skipping"
+      warning, and a catch-up run that ends on the walk limit, a frozen output or an
+      out-of-range input records `local_processing_fault` instead of
+      `local_context_cancelled` as its last error type. A missing record, a
+      cancelled check, and a record the store could not decode keep the old
+      classification.
+    - On the fresh path a proven conflict is rejected with `TX_INVALID` even when
+      the check's context is cancelled while it runs, as the cached path and
+      blessing a missing transaction already did. Before, it was retried.
 
   Nothing to configure. (bsv-blockchain/teranode#2066)
 

cmd/rewindblockchain/rewindblockchain/phase3_finalize.go

@@ -59,8 +59,8 @@ func (e *env) resetBlockAssemblerState(ctx context.Context, pf *preflightResult)
 func (e *env) readBlockPersisterHeight(ctx context.Context) (uint32, bool, error) {
 	existing, err := e.blockchainStore.GetState(ctx, blockPersisterHeightKey)
 	if err != nil {
-		// SQL blockchain store returns sql.ErrNoRows directly for missing keys;
-		// future implementations may wrap into errors.ErrNotFound.
+		// The SQL blockchain store returns errors.ErrNotFound wrapping sql.ErrNoRows
+		// for a missing key; either match is enough.
 		if errors.Is(err, sql.ErrNoRows) || errors.Is(err, errors.ErrNotFound) {
 			e.logger.Debugf("state[%s] not present: %v", blockPersisterHeightKey, err)
 			return 0, false, nil

docs/references/settings/services/blockvalidation_settings.md

@@ -69,6 +69,7 @@ loader (0, false and false), so for these keys the two agree. The
 | MaxBlocksBehindBlockAssembly | int | 20 | blockvalidation_maxBlocksBehindBlockAssembly | **CRITICAL** - Max blocks behind block assembly |
 | PeriodicProcessingInterval | time.Duration | 1m | blockvalidation_periodic_processing_interval | Periodic processing interval |
 | RecentBlockIDsLimit | uint64 | 50000 | blockvalidation_recentBlockIDsLimit | **CRITICAL** - Fast-path double-spend checking window |
+| TipRepairSliceSize | int | 250000 | blockvalidation_tipRepairSliceSize | Transactions per overtaken-tip repair step on the setMined worker; bounds how long a step delays the next block's mined status. The pruner deletes nothing until the repair has finished |
 | MaxParallelForks | int | 4 | blockvalidation_max_parallel_forks | Maximum parallel fork processing |
 | MaxTrackedForks | int | 1000 | blockvalidation_max_tracked_forks | Maximum total forks tracked |
 | NearForkThreshold | int | 0 | blockvalidation_near_fork_threshold | Near fork detection (0=coinbase maturity/2) |

docs/references/stores/utxo_reference.md

@@ -256,7 +256,9 @@ type Store interface {
     // MarkTransactionsOnLongestChain marks transactions as being on the longest chain or not.
     // When onLongestChain is true, the unminedSince field is unset (transaction is mined).
     // When onLongestChain is false, the unminedSince field is set to the current block height.
-    MarkTransactionsOnLongestChain(ctx context.Context, txHashes []chainhash.Hash, onLongestChain bool) error
+    // WithSkipMissing() makes a record missing from the store a logged skip instead of an
+    // error for onLongestChain=false.
+    MarkTransactionsOnLongestChain(ctx context.Context, txHashes []chainhash.Hash, onLongestChain bool, opts ...MarkLongestChainOption) error
 
     // SetBlockHeight updates the current block height in the store.
     //

docs/topics/stores/utxo.md

@@ -632,8 +632,8 @@ The current writers, re-derived directly from the code (not from the settings/de
 | Service | File(s) | Writes | Why |
 |---|---|---|---|
 | **Validator** | `services/validator/Validator.go` | `SpendAndCreate` (creates the tx's outputs, spends its inputs), `SetLocked` | Core transaction validation: on accepting a new transaction it atomically creates its UTXOs and spends the ones it consumes. `SetLocked` completes a two-phase commit on the tx's own hash: when the tx is queued for block assembly its outputs are created locked (`WithLocked(true)`), and unlocked only once it has been added successfully (`twoPhaseCommitTransaction`) — a crash in between leaves them unspendable rather than spendable. |
-| **Block Validation** | `services/blockvalidation/BlockValidation.go`, `services/blockvalidation/quick_validate.go` | `SpendAndCreate` (`WithCreateOnly`/`WithSpendOnly`, quick-validate replay), `SetMinedMulti` (steady state, via `model.UpdateTxMinedStatus` in `setTxMined`), `utxo.SetMinedMultiChunked` (→ `SetMinedMulti`, quick-validate replay), `SetLocked` | Validates incoming blocks: creates/spends UTXOs for transactions not already known, marks the block's transactions as mined (records `BlockID`/`SubtreeIdx`), then — unless `quickValidateSkipsUtxoLock` applies to this block — unlocks the block's own subtree transactions at the end of quick-validate (`unlockSubtreeTransactions`). |
-| **Block Assembly** | `services/blockassembly/BlockAssembler.go`, `services/blockassembly/subtreeprocessor/SubtreeProcessor.go` | `SpendAndCreate` (`WithCreateOnly` — creates the coinbase UTXOs, `processCoinbaseUtxos`), `Delete` (coinbase and its child spends on reorg, `removeCoinbaseUtxos` — Block Assembly is the only service that calls `Delete` directly), `MarkTransactionsOnLongestChain`, `SetLocked` (unlocks unmined transactions reloaded at startup; unlocks conflict-locked parents when healing a stale conflict intent), `utxo.ProcessConflicting` / `utxo.ReverseProcessConflicting` (→ `SetConflicting`, `Unspend`, `SpendAndCreate`, `BeginConflictIntent`/`CompleteConflictIntent`) | Owns chain-tip bookkeeping during block building and reorgs: creates the coinbase UTXO and deletes it (with its child spends) on reorg, flags which transactions sit on the currently-longest chain, unlocks unmined transactions reloaded at startup and conflict-locked parents when healing a stale intent, and drives the conflicting-transaction state machine (marking/unmarking double-spends and reversing that on reorg). |
+| **Block Validation** | `services/blockvalidation/BlockValidation.go`, `services/blockvalidation/quick_validate.go`, `services/blockvalidation/overtaken_tip_repair.go` | `SpendAndCreate` (`WithCreateOnly`/`WithSpendOnly`, quick-validate replay), `SetMinedMulti` (steady state, via `model.UpdateTxMinedStatus` in `setTxMined`), `utxo.SetMinedMultiChunked` (→ `SetMinedMulti`, quick-validate replay), `SetLocked`, `MarkTransactionsOnLongestChain` (overtaken-tip repair, `overtaken_tip_repair.go`) | Validates incoming blocks: creates/spends UTXOs for transactions not already known, marks the block's transactions as mined (records `BlockID`/`SubtreeIdx`), re-marks the transactions of a block it marked mined that has since left the longest chain, then — unless `quickValidateSkipsUtxoLock` applies to this block — unlocks the block's own subtree transactions at the end of quick-validate (`unlockSubtreeTransactions`). |
+| **Block Assembly** | `services/blockassembly/BlockAssembler.go`, `services/blockassembly/subtreeprocessor/SubtreeProcessor.go` | `SpendAndCreate` (`WithCreateOnly` — creates the coinbase UTXOs, `processCoinbaseUtxos`), `Delete` (coinbase and its child spends on reorg, `removeCoinbaseUtxos` — Block Assembly is the only service that calls `Delete` directly), `MarkTransactionsOnLongestChain`, `SetLocked` (unlocks unmined transactions reloaded at startup; unlocks conflict-locked parents when healing a stale conflict intent), `utxo.ProcessConflicting` / `utxo.ReverseProcessConflicting` (→ `SetConflicting`, `Unspend`, `SpendAndCreate`, `BeginConflictIntent`/`CompleteConflictIntent`) | Owns chain-tip bookkeeping during block building and reorgs: creates the coinbase UTXO and deletes it (with its child spends) on reorg, flags which transactions sit on the currently-longest chain (on a reorg, only the transactions it moved, not its whole mempool), unlocks unmined transactions reloaded at startup and conflict-locked parents when healing a stale intent, and drives the conflicting-transaction state machine (marking/unmarking double-spends and reversing that on reorg). |
 | **Legacy (p2p sync)** | `services/legacy/netsync/handle_block.go` | `SpendAndCreate`, `utxo.SetMinedMultiChunked` (→ `SetMinedMulti`) | The legacy Bitcoin-protocol block-sync path validates and applies blocks received from legacy peers the same way Block Validation does for the native protocol — same create/spend/set-mined writes, different ingestion path. |
 | **Alert System** | `services/alert/node.go` | `FreezeUTXOs`, `UnFreezeUTXOs`, `ReAssignUTXO` | Executes alert-network directives (court orders / network alerts) to freeze, unfreeze, or reassign specific UTXOs. |
 | **RPC** | `services/rpc/handlers.go` | `FreezeUTXOs`, `UnFreezeUTXOs`, `ReAssignUTXO` | Exposes the same freeze/unfreeze/reassign operations as operator-facing RPC calls (e.g. for manual intervention), independent of the Alert System's automated path. |