diff --git a/CHANGELOG.md b/CHANGELOG.md index 15bdb9bc96..8ee315d202 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -844,14 +844,37 @@ Upgrading from v0.16.0 changes nothing for them; upgrading from an earlier relea invalid transaction is; the retryable error is returned only when nothing is proven. A block that used to retry indefinitely may now be rejected on the first attempt. This includes a walk over the `utxostore_conflictingChildrenMaxNodes` - budget that already reached a mined member: raising the setting no longer - changes that verdict. The fresh-path change also reaches `ValidateSubtreeInternal`, + budget that already reached a mined counter-conflicting transaction (a + conflicting transaction, a counter-spender or a counter-spender's descendant; + the conflicting transactions' own descendants are only checked for frozen + outputs): raising the setting no longer changes that verdict. The fresh-path change also reaches `ValidateSubtreeInternal`, so `CheckSubtreeFromBlock` and live peer subtrees return a proven conflict as `TX_INVALID` too. On the live path, `subtreeMessageHandler` now logs it at error level ("error processing kafka message") where it logged a "Context canceled, skipping" warning, so operators will see new error-level log lines for such subtrees. Nothing to configure. (bsv-blockchain/teranode#2052) +- **Conflict-check failures in subtree validation are no longer reported as + context cancellations.** Two follow-ups to the entry above, both in subtree + validation's conflict check: + - On the fresh path (`CheckBlockSubtrees` on a new subtree, `CheckSubtreeFromBlock`, + live peer subtrees), a store fault, the `utxostore_conflictingChildrenMaxNodes` + walk limit, a frozen output, an input out of range and the fail-fast + missing-tx threshold are no longer wrapped as a context cancellation. Each is + still retried and never charged to a peer. What changes is how they show up: + on the live path `subtreeMessageHandler` logs them at error level ("error + processing kafka message") where it logged a "Context canceled, skipping" + warning, and a catch-up run that ends on the walk limit, a frozen output or an + out-of-range input records `local_processing_fault` instead of + `local_context_cancelled` as its last error type. A missing record, a + cancelled check, and a record the store could not decode keep the old + classification. + - On the fresh path a proven conflict is rejected with `TX_INVALID` even when + the check's context is cancelled while it runs, as the cached path and + blessing a missing transaction already did. Before, it was retried. + + Nothing to configure. (bsv-blockchain/teranode#2066) + ### Not breaking, but worth knowing - **Peer subtrees announced during a catchup are validated afterwards instead of diff --git a/docs/references/settings/stores/utxo_settings.md b/docs/references/settings/stores/utxo_settings.md index 89df471cb8..31f1417339 100644 --- a/docs/references/settings/stores/utxo_settings.md +++ b/docs/references/settings/stores/utxo_settings.md @@ -44,7 +44,7 @@ | MaxMinedBatchSize | int | 1024 | utxostore_maxMinedBatchSize | Max mined transaction batch size | | BlockHeightRetentionAdjustment | int32 | 0 | utxostore_blockHeightRetentionAdjustment | **CRITICAL** - Retention adjustment | | DisableDAHCleaner | bool | false | utxostore_disableDAHCleaner | **CRITICAL** - DAH cleaner process control | -| ConflictingChildrenMaxNodes | int | 100000 | utxostore_conflictingChildrenMaxNodes | Fail-closed node budget per root of a conflicting-descendant walk (<=0 = unlimited). Subtree validation walks a block's conflicting transactions as one batch from N roots (the transactions and their counter-spenders) and allows N × this budget: the old aggregate bound of N separate walks. Worst case, the visited set reaches N × this budget entries, and a single runaway cone fails after N × this budget reads rather than this budget. A walk that stops on the budget still rejects the block with `TX_INVALID` if it already reached a transaction mined on the candidate chain; otherwise the block fails with `UTXO_WALK_LIMIT_EXCEEDED` and is retried | +| ConflictingChildrenMaxNodes | int | 100000 | utxostore_conflictingChildrenMaxNodes | Fail-closed node budget per root of a conflicting-descendant walk (<=0 = unlimited). Subtree validation walks a block's conflicting transactions as one batch from N roots (the transactions and their counter-spenders) and allows N × this budget: the old aggregate bound of N separate walks. Worst case, the visited set reaches N × this budget entries, and a single runaway cone fails after N × this budget reads rather than this budget. A walk that stops on the budget still rejects the block with `TX_INVALID` if it already reached a counter-conflicting transaction (a conflicting transaction, a counter-spender or a counter-spender's descendant) mined on the candidate chain; the conflicting transactions' own descendants are only checked for frozen outputs. Otherwise the block fails with `UTXO_WALK_LIMIT_EXCEEDED` and is retried | ### PostgreSQL Connection Pool (PostgresPool) diff --git a/services/blockvalidation/BlockValidation_test.go b/services/blockvalidation/BlockValidation_test.go index c7fa8461b8..9d257facaa 100644 --- a/services/blockvalidation/BlockValidation_test.go +++ b/services/blockvalidation/BlockValidation_test.go @@ -302,6 +302,13 @@ func (m *MockSubtreeValidationClient) CheckBlockSubtrees(ctx context.Context, bl // proper test isolation. Each configure func adjusts the settings the stores and // the subtree validation server are built with. func setup(t *testing.T, configure ...func(*settings.Settings)) (utxostore.Store, subtreevalidation.Interface, blockchain.ClientI, blob.Store, blob.Store, func()) { + return setupWithSubtreeValidationStore(t, nil, configure...) +} + +// setupWithSubtreeValidationStore is setup, with the subtree validation server +// reading the UTXO store through wrap, when wrap is not nil. The returned store +// is the unwrapped one. +func setupWithSubtreeValidationStore(t *testing.T, wrap func(utxostore.Store) utxostore.Store, configure ...func(*settings.Settings)) (utxostore.Store, subtreevalidation.Interface, blockchain.ClientI, blob.Store, blob.Store, func()) { // we only need the httpClient, utxoStore and validatorClient when blessing a transaction httpmock.ActivateNonDefault(util.HTTPClient()) httpmock.RegisterResponder( @@ -352,7 +359,12 @@ func setup(t *testing.T, configure ...func(*settings.Settings)) (utxostore.Store nilConsumer := &kafka.KafkaConsumerGroup{} - subtreeValidationServer, err := subtreevalidation.New(context.Background(), ulogger.TestLogger{}, tSettings, subtreeStore, txStore, utxoStore, validatorClient, blockchainClient, nilConsumer, nilConsumer, nil, nil) + subtreeValidationStore := utxostore.Store(utxoStore) + if wrap != nil { + subtreeValidationStore = wrap(utxoStore) + } + + subtreeValidationServer, err := subtreevalidation.New(context.Background(), ulogger.TestLogger{}, tSettings, subtreeStore, txStore, subtreeValidationStore, validatorClient, blockchainClient, nilConsumer, nilConsumer, nil, nil) if err != nil { panic(err) } diff --git a/services/blockvalidation/proven_conflict_classification_test.go b/services/blockvalidation/proven_conflict_classification_test.go index daf29824c6..03b992c954 100644 --- a/services/blockvalidation/proven_conflict_classification_test.go +++ b/services/blockvalidation/proven_conflict_classification_test.go @@ -3,6 +3,7 @@ package blockvalidation import ( "context" "net/url" + "slices" "testing" "time" @@ -16,7 +17,10 @@ import ( "github.com/bsv-blockchain/teranode/settings" blockchain_store "github.com/bsv-blockchain/teranode/stores/blockchain" utxostore "github.com/bsv-blockchain/teranode/stores/utxo" + "github.com/bsv-blockchain/teranode/stores/utxo/fields" + "github.com/bsv-blockchain/teranode/stores/utxo/meta" "github.com/bsv-blockchain/teranode/ulogger" + "github.com/bsv-blockchain/teranode/util" "github.com/bsv-blockchain/teranode/util/test" "github.com/jarcoal/httpmock" "github.com/stretchr/testify/require" @@ -26,11 +30,12 @@ import ( // one transaction the UTXO store holds as conflicting: it spends parent:0, whose // recorded spender is counter. type conflictBlockFixture struct { - utxoStore utxostore.Store - bv *BlockValidation - block *model.Block - parent *bt.Tx - counter *bt.Tx + utxoStore utxostore.Store + bv *BlockValidation + block *model.Block + parent *bt.Tx + counter *bt.Tx + conflicting *bt.Tx } // p2pkhTx builds a tx with one P2PKH output, spending the given outputs. salt @@ -62,10 +67,23 @@ func spendAndStore(t *testing.T, ctx context.Context, store utxostore.Store, tx require.NoError(t, err) } +// conflictFixtureOptions varies the fixture. +type conflictFixtureOptions struct { + // counterOpts are the counter-spender's create options. + counterOpts []utxostore.CreateOption + // configure adjusts the settings of the subtree validation server. + configure []func(*settings.Settings) + // wrap, when set, is how the subtree validation server reads the UTXO store. + wrap func(utxostore.Store) utxostore.Store + // spendVout is the parent output the conflicting tx spends. The parent has + // one output, so any other index is out of range. + spendVout uint32 +} + // newConflictBlockFixture builds the fixture on a fresh setup. The parent is // mined in genesis, the candidate block's parent, so a counter-spender stored // with mined is mined on the candidate chain. -func newConflictBlockFixture(t *testing.T, counterOpts []utxostore.CreateOption, configure ...func(*settings.Settings)) *conflictBlockFixture { +func newConflictBlockFixture(t *testing.T, opts conflictFixtureOptions) *conflictBlockFixture { t.Helper() initPrometheusMetrics() @@ -79,7 +97,7 @@ func newConflictBlockFixture(t *testing.T, counterOpts []utxostore.CreateOption, s.ChainCfgParams.DaaForkHeight = 0 } - utxoStore, subtreeValidationClient, _, txStore, subtreeStore, deferFunc := setup(t, append([]func(*settings.Settings){syntheticPoW}, configure...)...) + utxoStore, subtreeValidationClient, _, txStore, subtreeStore, deferFunc := setupWithSubtreeValidationStore(t, opts.wrap, append([]func(*settings.Settings){syntheticPoW}, opts.configure...)...) t.Cleanup(deferFunc) inGenesis := utxostore.WithMinedBlockInfo(utxostore.MinedBlockInfo{BlockID: 0, BlockHeight: 0}) @@ -89,9 +107,10 @@ func newConflictBlockFixture(t *testing.T, counterOpts []utxostore.CreateOption, require.NoError(t, err) counter := p2pkhTx(t, 7002, parent) - spendAndStore(t, ctx, utxoStore, counter, counterOpts...) + spendAndStore(t, ctx, utxoStore, counter, opts.counterOpts...) conflicting := p2pkhTx(t, 7003, parent) + conflicting.Inputs[0].PreviousTxOutIndex = opts.spendVout _, _, err = utxoStore.SpendAndCreate(ctx, conflicting, 0, utxostore.WithConflicting(true), utxostore.WithCreateOnly()) require.NoError(t, err) @@ -154,11 +173,12 @@ func newConflictBlockFixture(t *testing.T, counterOpts []utxostore.CreateOption, require.NoError(t, err) return &conflictBlockFixture{ - utxoStore: utxoStore, - bv: NewBlockValidation(ctx, ulogger.TestLogger{}, tSettings, blockchainClient, subtreeStore, txStore, utxoStore, nil, subtreeValidationClient), - block: block, - parent: parent, - counter: counter, + utxoStore: utxoStore, + bv: NewBlockValidation(ctx, ulogger.TestLogger{}, tSettings, blockchainClient, subtreeStore, txStore, utxoStore, nil, subtreeValidationClient), + block: block, + parent: parent, + counter: counter, + conflicting: conflicting, } } @@ -169,12 +189,12 @@ func newConflictBlockFixture(t *testing.T, counterOpts []utxostore.CreateOption, // took the verdict for this node's own cancellation: the Kafka path queued the // block for another retry (isRecoverableBlockMessageError) and catch-up scored it // local_context_cancelled, so the block was retried indefinitely. The other -// failures of that path keep the wrap, and stay retryable. +// failures of that path stay this node's own, wrapped or not. func TestValidateBlock_ProvenConflictIsNotACancellation(t *testing.T) { ctx := context.Background() t.Run("a mined counter-spender rejects the block", func(t *testing.T) { - f := newConflictBlockFixture(t, []utxostore.CreateOption{utxostore.WithMinedBlockInfo(utxostore.MinedBlockInfo{BlockID: 0, BlockHeight: 0})}) + f := newConflictBlockFixture(t, conflictFixtureOptions{counterOpts: []utxostore.CreateOption{utxostore.WithMinedBlockInfo(utxostore.MinedBlockInfo{BlockID: 0, BlockHeight: 0})}}) subtreeErr := f.bv.validateBlockSubtrees(ctx, f.block, "", "http://localhost:8000", false) require.ErrorIs(t, subtreeErr, errors.ErrTxInvalid) @@ -191,54 +211,155 @@ func TestValidateBlock_ProvenConflictIsNotACancellation(t *testing.T) { require.False(t, isLocalCatchupFault(err), "catch-up must not score it as this node's own fault") require.True(t, isUnboundTxInvalidVerdict(err), "catch-up scores it as a consensus rejection") - // catch-up's classifier, as a catch-up run ending on this error applies it - server, _, _, cleanup := setupTestCatchupServer(t) - defer cleanup() - - catchupCtx := &CatchupContext{ - blockUpTo: f.block, - baseURL: "http://localhost:8000", - peerID: "primary", - startTime: time.Now(), - } - require.NoError(t, server.acquireCatchupLock(catchupCtx)) - - server.releaseCatchupLock(catchupCtx, &err) - require.NotNil(t, server.previousCatchupAttempt) - require.Equal(t, "validation_failure", server.previousCatchupAttempt.ErrorType, "not local_context_cancelled") + require.Equal(t, "validation_failure", releasedCatchupErrorType(t, f.block, err), "not local_context_cancelled") }) - t.Run("a pruned parent is a retryable failure", func(t *testing.T) { - f := newConflictBlockFixture(t, nil) - require.NoError(t, f.utxoStore.Delete(ctx, f.parent.TxIDChainHash())) - - subtreeErr := f.bv.validateBlockSubtrees(ctx, f.block, "", "http://localhost:8000", false) - require.ErrorIs(t, subtreeErr, errors.ErrTxNotFound) - require.ErrorIs(t, subtreeErr, errors.ErrContextCanceled) - require.NotErrorIs(t, subtreeErr, errors.ErrTxInvalid) - require.True(t, isRecoverableBlockMessageError(subtreeErr)) - }) - - t.Run("a walk over its budget is a retryable failure", func(t *testing.T) { - // two roots (the conflicting tx and its counter-spender): a budget of 2, - // against a counter-spender cone of 3 - f := newConflictBlockFixture(t, nil, func(s *settings.Settings) { - s.UtxoStore.ConflictingChildrenMaxNodes = 1 + // Every other failure of the conflict check, and of the decorate batches + // before it, is this node's own: block validation retries the block and never + // charges the peer for it. Each class is checked as the Kafka path and + // catch-up classify it. A missing record and a TX_INVALID under another code + // keep the CONTEXT_CANCELED wrap, which is what keeps them local on catch-up + // (see settledError); the rest leave without it. + for _, tc := range []struct { + name string + // build arranges the failure on a fresh fixture + build func(t *testing.T) *conflictBlockFixture + code *errors.Error + // cancelWrap is whether the failure still carries CONTEXT_CANCELED + cancelWrap bool + // corrupt is whether ValidateBlock returns a corrupt-body verdict + corrupt bool + // catchupLabel is the error type releaseCatchupLock records + catchupLabel string + }{ + { + name: "a pruned parent", + build: func(t *testing.T) *conflictBlockFixture { + f := newConflictBlockFixture(t, conflictFixtureOptions{}) + require.NoError(t, f.utxoStore.Delete(ctx, f.parent.TxIDChainHash())) + + return f + }, + code: errors.ErrTxNotFound, + cancelWrap: true, + catchupLabel: "local_context_cancelled", + }, + { + name: "a walk over its budget", + build: func(t *testing.T) *conflictBlockFixture { + // two roots (the conflicting tx and its counter-spender): a budget + // of 2, against a counter-spender cone of 3 + f := newConflictBlockFixture(t, conflictFixtureOptions{configure: []func(*settings.Settings){func(s *settings.Settings) { + s.UtxoStore.ConflictingChildrenMaxNodes = 1 + }}}) + + prev := f.counter + for i := uint32(0); i < 3; i++ { + child := p2pkhTx(t, 7100+i, prev) + spendAndStore(t, ctx, f.utxoStore, child) + prev = child + } + + return f + }, + code: errors.ErrUtxoWalkLimitExceeded, + catchupLabel: "local_processing_fault", + }, + { + name: "a store fault in the conflict check", + build: func(t *testing.T) *conflictBlockFixture { + return newFaultyConflictBlockFixture(t, errors.NewStorageError("store down"), false) + }, + code: errors.ErrStorageError, + catchupLabel: "local_storage_fault", + }, + { + name: "a store fault in a decorate batch", + build: func(t *testing.T) *conflictBlockFixture { + return newFaultyConflictBlockFixture(t, errors.NewStorageError("store down"), true) + }, + code: errors.ErrStorageError, + catchupLabel: "local_storage_fault", + }, + { + // ctx is live: a store fault is not a cancellation for naming a deadline + name: "a store fault naming a deadline in the conflict check", + build: func(t *testing.T) *conflictBlockFixture { + return newFaultyConflictBlockFixture(t, errors.NewStorageError("store down", context.DeadlineExceeded), false) + }, + code: errors.ErrStorageError, + catchupLabel: "local_storage_fault", + }, + { + name: "a store fault naming a deadline in a decorate batch", + build: func(t *testing.T) *conflictBlockFixture { + return newFaultyConflictBlockFixture(t, errors.NewStorageError("store down", context.DeadlineExceeded), true) + }, + code: errors.ErrStorageError, + catchupLabel: "local_storage_fault", + }, + { + name: "a frozen output in a counter-spender's cone", + build: func(t *testing.T) *conflictBlockFixture { + f := newConflictBlockFixture(t, conflictFixtureOptions{}) + + utxoHash, err := util.UTXOHashFromOutput(f.counter.TxIDChainHash(), f.counter.Outputs[0], 0) + require.NoError(t, err) + require.NoError(t, f.utxoStore.FreezeUTXOs(ctx, []*utxostore.Spend{{TxID: f.counter.TxIDChainHash(), Vout: 0, UTXOHash: utxoHash}}, test.CreateBaseTestSettings(t))) + + return f + }, + code: errors.ErrProcessing, + catchupLabel: "local_processing_fault", + }, + { + name: "an input out of range of its parent's outputs", + build: func(t *testing.T) *conflictBlockFixture { + return newConflictBlockFixture(t, conflictFixtureOptions{spendVout: 5}) + }, + code: errors.ErrProcessing, + catchupLabel: "local_processing_fault", + }, + { + // A record that was read but does not decode, which the aerospike + // store returns as TX_INVALID (classifyRecordError). ValidateBlock's + // invalid-tx branch already returns a corrupt verdict for it, as + // before: its infrastructure check looks for context.Canceled, not + // the CONTEXT_CANCELED code. The wrap is what keeps the Kafka path + // retrying it and catch-up from reporting the peer malicious. + name: "a TX_INVALID under another code", + build: func(t *testing.T) *conflictBlockFixture { + return newFaultyConflictBlockFixture(t, errors.NewTxInvalidError("could not process tx inpoints"), false) + }, + code: errors.ErrTxInvalid, + cancelWrap: true, + corrupt: true, + catchupLabel: "local_context_cancelled", + }, + } { + t.Run(tc.name, func(t *testing.T) { + f := tc.build(t) + + subtreeErr := f.bv.validateBlockSubtrees(ctx, f.block, "", "http://localhost:8000", false) + require.ErrorIs(t, subtreeErr, tc.code) + require.Equal(t, tc.cancelWrap, errors.Is(subtreeErr, errors.ErrContextCanceled), "CONTEXT_CANCELED: %v", subtreeErr) + require.True(t, isRecoverableBlockMessageError(subtreeErr)) + + err := f.bv.ValidateBlock(ctx, f.block, "http://localhost:8000") + require.ErrorIs(t, err, tc.code) + require.Equal(t, tc.corrupt, errors.IsBlockCorrupt(err), "corrupt verdict: %v", err) + require.NotErrorIs(t, err, errors.ErrBlockInvalid) + + // processBlockFound passes a corrupt or invalid verdict on as it is + // and wraps anything else in a service error, which the Kafka path + // retries; this one is retryable before that wrap too + require.True(t, isRecoverableBlockMessageError(err), "the Kafka path retries the block") + require.True(t, isLocalCatchupFault(err), "catch-up does not charge or rotate away from the peer") + require.False(t, shouldReportConsensusMalicious(err)) + require.False(t, isUnboundTxInvalidVerdict(err)) + require.Equal(t, tc.catchupLabel, releasedCatchupErrorType(t, f.block, err)) }) - - prev := f.counter - for i := uint32(0); i < 3; i++ { - child := p2pkhTx(t, 7100+i, prev) - spendAndStore(t, ctx, f.utxoStore, child) - prev = child - } - - subtreeErr := f.bv.validateBlockSubtrees(ctx, f.block, "", "http://localhost:8000", false) - require.ErrorIs(t, subtreeErr, errors.ErrUtxoWalkLimitExceeded) - require.ErrorIs(t, subtreeErr, errors.ErrContextCanceled) - require.NotErrorIs(t, subtreeErr, errors.ErrTxInvalid) - require.True(t, isRecoverableBlockMessageError(subtreeErr)) - }) + } // The block path does not fail fast, so the missing-tx threshold cannot be // driven through ValidateBlock. This is the shape processTxMetaUsingStore @@ -247,11 +368,82 @@ func TestValidateBlock_ProvenConflictIsNotACancellation(t *testing.T) { t.Run("the missing-tx threshold is a retryable failure", func(t *testing.T) { thresholdErr := errors.NewProcessingError("[CheckBlockSubtreesRequest] Failed to process transactions in batch 1", errors.NewProcessingError("[processTransactionsInLevels] Failed to check txMeta store", - errors.NewContextCanceledError("[processTxMetaUsingStore]", - errors.NewThresholdExceededError("threshold exceeded for missing txs: 2 > 1")))) + errors.NewThresholdExceededError("threshold exceeded for missing txs: 2 > 1"))) subtreeErr := errors.UnwrapGRPC(errors.WrapGRPC(thresholdErr)) require.ErrorIs(t, subtreeErr, errors.ErrThresholdExceeded) require.True(t, isRecoverableBlockMessageError(subtreeErr)) + require.True(t, isLocalCatchupFault(subtreeErr)) + }) +} + +// releasedCatchupErrorType is the error type releaseCatchupLock records for a +// catch-up run toward block that ends on err. +func releasedCatchupErrorType(t *testing.T, block *model.Block, err error) string { + t.Helper() + + server, _, _, cleanup := setupTestCatchupServer(t) + defer cleanup() + + catchupCtx := &CatchupContext{ + blockUpTo: block, + baseURL: "http://localhost:8000", + peerID: "primary", + startTime: time.Now(), + } + require.NoError(t, server.acquireCatchupLock(catchupCtx)) + + server.releaseCatchupLock(catchupCtx, &err) + require.NotNil(t, server.previousCatchupAttempt) + + return server.previousCatchupAttempt.ErrorType +} + +// faultyCheckStore fails the fresh path's reads of one tx with err: its decorate +// when decorate is set, otherwise the conflict check's read of its inpoints. +type faultyCheckStore struct { + utxostore.Store + tx chainhash.Hash + err error + decorate bool +} + +func (f *faultyCheckStore) Get(ctx context.Context, hash *chainhash.Hash, fs ...fields.FieldName) (*meta.Data, error) { + if !f.decorate && hash.Equal(f.tx) && slices.Contains(fs, fields.TxInpoints) { + return nil, f.err + } + + return f.Store.Get(ctx, hash, fs...) +} + +func (f *faultyCheckStore) BatchDecorate(ctx context.Context, items []*utxostore.UnresolvedMetaData, fs ...fields.FieldName) error { + if f.decorate && slices.ContainsFunc(items, func(item *utxostore.UnresolvedMetaData) bool { return item.Hash.Equal(f.tx) }) { + return f.err + } + + return f.Store.BatchDecorate(ctx, items, fs...) +} + +// newFaultyConflictBlockFixture is the fixture with the conflicting tx's reads +// failing as faultyCheckStore fails them. Decorate goes through BatchDecorate, +// once. +func newFaultyConflictBlockFixture(t *testing.T, err error, decorate bool) *conflictBlockFixture { + t.Helper() + + faulty := &faultyCheckStore{err: err, decorate: decorate} + + f := newConflictBlockFixture(t, conflictFixtureOptions{ + wrap: func(store utxostore.Store) utxostore.Store { + faulty.Store = store + return faulty + }, + configure: []func(*settings.Settings){func(s *settings.Settings) { + s.SubtreeValidation.BatchMissingTransactions = true + s.BlockValidation.ProcessTxMetaUsingStoreRetries = 0 + }}, }) + + faulty.tx = *f.conflicting.TxIDChainHash() + + return f } diff --git a/services/subtreevalidation/check_counter_conflicting_batch.go b/services/subtreevalidation/check_counter_conflicting_batch.go index 515579159f..6008eb78ee 100644 --- a/services/subtreevalidation/check_counter_conflicting_batch.go +++ b/services/subtreevalidation/check_counter_conflicting_batch.go @@ -83,9 +83,10 @@ const counterConflictingBatchFanOut = 128 // distinct roots (transactions and counter-spenders): the aggregate bound of the // per-transaction walks, so any block whose every per-transaction walk fit still // passes, however many conflicting transactions it carries. Exceeding it stops -// the walk. If the members it reached include one mined on the candidate chain, -// the verdict is ErrTxInvalid on the first attempt, and raising the budget does -// not change that (see VERDICT). Otherwise it fails closed with +// the walk, in either pass. If the counter-conflicting members it reached +// include one mined on the candidate chain, the verdict is ErrTxInvalid on the +// first attempt, and raising the budget does not change that (see VERDICT). +// Otherwise it fails closed with // ERR_UTXO_WALK_LIMIT_EXCEEDED, retryably, and raising the budget resumes // progress. The price is that a single runaway cone is stopped only after that // many reads, not after maxNodes. @@ -195,6 +196,9 @@ func (u *Server) checkCounterConflictingOnCurrentChainBatch(ctx context.Context, fail(missingRecordsError("walked record", missing)) } + // A walk that stops, on its budget or on a store fault, still has the counter + // cone it reached mined-checked: every member of it descends from a + // counter-spender, so any mined one is proof. counterConflicting := make(map[chainhash.Hash]struct{}, len(txs)+len(counterSpenders)+len(counterCone)) for _, hashes := range [][]chainhash.Hash{txs, counterSpenders, counterCone} { for _, hash := range hashes { @@ -431,7 +435,8 @@ func (u *Server) getBatch(ctx context.Context, hashes []chainhash.Hash, f ...fie // A failed read does not stop the others, and both results can be set: a mined // member, and the error of a member that could not be read. The caller decides // which wins. Each is the one of the lowest member hash, so neither depends on -// read timing. +// read timing. Cancelling the other reads after a failure would lose a mined +// member not read yet, and with it a proven conflict. func (u *Server) firstMinedOnChain(ctx context.Context, members map[chainhash.Hash]struct{}, blockIds map[uint32]bool, missingPolicy missingRecordPolicy) (*chainhash.Hash, error) { var ( mu sync.Mutex diff --git a/services/subtreevalidation/check_counter_conflicting_precedence_test.go b/services/subtreevalidation/check_counter_conflicting_precedence_test.go index c5c06d9348..945e1c2748 100644 --- a/services/subtreevalidation/check_counter_conflicting_precedence_test.go +++ b/services/subtreevalidation/check_counter_conflicting_precedence_test.go @@ -20,23 +20,115 @@ import ( "github.com/stretchr/testify/require" ) -// faultyGetStore fails the batch's phase-1 read of one hash, the one asking for -// its inpoints, with a storage error. The cached path reads the tx's body first -// (checkStoredTxAtHeight), and that read is left alone, so the fault reaches the -// batch. +// faultyGetStore fails the batch's reads of one hash with a storage error: its +// phase-1 read, the one asking for its inpoints, and its read as a member of the +// walk. The cached path reads the tx's body first (checkStoredTxAtHeight), and +// that read is left alone, so the fault reaches the batch. type faultyGetStore struct { utxo.Store fail chainhash.Hash } func (f *faultyGetStore) Get(ctx context.Context, hash *chainhash.Hash, fs ...fields.FieldName) (*meta.Data, error) { - if hash.Equal(f.fail) && slices.Contains(fs, fields.TxInpoints) { + if hash.Equal(f.fail) && !slices.Contains(fs, fields.Tx) { return nil, errors.NewStorageError("store down") } return f.Store.Get(ctx, hash, fs...) } +// buildCone stores a chain of n spenders below root, each spending output 0 of +// the one before, and returns it top first. The first is created with firstOpts. +func buildCone(t *testing.T, ctx context.Context, store utxo.Store, root *bt.Tx, n int, firstOpts ...utxo.CreateOption) []chainhash.Hash { + t.Helper() + + cone := make([]chainhash.Hash, 0, n) + + prev := root + for i := 0; i < n; i++ { + child := spendingChild(t, prev, 0) + + if i == 0 { + spendAndCreate(t, ctx, store, child, firstOpts...) + } else { + spendAndCreate(t, ctx, store, child) + } + + cone = append(cone, *child.TxIDChainHash()) + prev = child + } + + return cone +} + +// A walk that stops on its budget still has the counter cone it reached +// mined-checked: every member of it descends from a counter-spender, so a mined +// one is proof, whichever of the walk's two passes ran out. The walk runs the +// counter-spenders' cone first and the conflicting txs' own cone second, on one +// budget. +func Test_checkCounterConflictingOnCurrentChainBatch_overBudgetWalkChecksTheCounterConeItReached(t *testing.T) { + ctx := context.Background() + minedInfo := utxo.MinedBlockInfo{BlockID: 77, BlockHeight: 100} + blockIds := map[uint32]bool{77: true} + + // requireProven checks the walk's premise, then that the batch names the + // mined counter-cone member. + requireProven := func(t *testing.T, store utxo.Store, tSettings *settings.Settings, counter *bt.Tx, txs []chainhash.Hash, cone []chainhash.Hash, minedMember chainhash.Hash, counterConeComplete bool) { + t.Helper() + + cones, _, walkErr := utxo.GetConflictingChildrenFromRoots(ctx, store, [][]chainhash.Hash{{*counter.TxIDChainHash()}, txs}, tSettings.UtxoStore.ConflictingChildrenMaxNodes) + require.ErrorIs(t, walkErr, errors.ErrUtxoWalkLimitExceeded) + require.Contains(t, cones[0], minedMember, "the walk reached the mined member") + + if counterConeComplete { + require.ElementsMatch(t, cone, cones[0], "the counter cone was walked in full") + } else { + require.Less(t, len(cones[0]), len(cone), "the counter cone was cut short") + } + + s := &Server{logger: ulogger.NewErrorTestLogger(t), settings: tSettings, utxoStore: store} + + _, err := s.checkCounterConflictingOnCurrentChainBatch(ctx, txs, blockIds, failOnMissingRecords) + require.ErrorIs(t, err, errors.ErrTxInvalid) + require.ErrorContains(t, err, minedMember.String(), "the mined counter-cone member is named") + } + + t.Run("own cone overflows after a complete counter cone", func(t *testing.T) { + tSettings := test.CreateBaseTestSettings(t) + tSettings.UtxoStore.SpendBatcherDurationMillis = 1 + // two roots (tx1 and its counter-spender): a budget of 8. The counter + // cone, the counter-spender and 3 below it, fits; tx1's own cone of 12 + // does not. + tSettings.UtxoStore.ConflictingChildrenMaxNodes = 4 + + store := newSQLUtxoStore(t, ctx, tSettings) + counter, chain := buildConflictingChain(t, ctx, store, 12, false) + cone := buildCone(t, ctx, store, counter, 3) + + // the third member: neither a root nor a direct counter-spender + _, err := store.SetMinedMulti(ctx, []*chainhash.Hash{&cone[2]}, minedInfo) + require.NoError(t, err) + + requireProven(t, store, tSettings, counter, chain[:1], cone, cone[2], true) + }) + + t.Run("counter cone overflows after reaching a mined member", func(t *testing.T) { + tSettings := test.CreateBaseTestSettings(t) + tSettings.UtxoStore.SpendBatcherDurationMillis = 1 + // two roots: a budget of 4, against a counter cone of 10 + tSettings.UtxoStore.ConflictingChildrenMaxNodes = 2 + + store := newSQLUtxoStore(t, ctx, tSettings) + counter, chain := buildConflictingChain(t, ctx, store, 1, false) + cone := buildCone(t, ctx, store, counter, 10) + + _, err := store.SetMinedMulti(ctx, []*chainhash.Hash{&cone[1]}, minedInfo) + require.NoError(t, err) + + requireProven(t, store, tSettings, counter, chain, cone, cone[1], false) + }) +} + // A tx-invalid verdict proven from records that were read is final: the block // is rejected whatever else in the batch could not be checked. A processing // error is returned only when nothing is proven. Otherwise one tx whose parent @@ -132,32 +224,21 @@ func Test_checkCounterConflictingOnCurrentChainBatch_provenTxInvalidWins(t *test requireRejected(t, store, tSettings, []chainhash.Hash{chain[0], minedConflict}) }) - t.Run("a mined member the walk reached before its budget ran out is proof", func(t *testing.T) { + t.Run("a mined member the walk reached before a store fault stopped it is proof", func(t *testing.T) { const coneLength = 10 tSettings := test.CreateBaseTestSettings(t) tSettings.UtxoStore.SpendBatcherDurationMillis = 1 - // two roots (tx1 and its counter-spender): a budget of 4, against a - // counter-spender cone of 10 whose first member is mined - tSettings.UtxoStore.ConflictingChildrenMaxNodes = 2 store := newSQLUtxoStore(t, ctx, tSettings) counter, chain := buildConflictingChain(t, ctx, store, 1, false) - prev := counter - for i := 0; i < coneLength; i++ { - child := spendingChild(t, prev, 0) - - if i == 0 { - spendAndCreate(t, ctx, store, child, mined) - } else { - spendAndCreate(t, ctx, store, child) - } - - prev = child - } + // a counter-spender cone whose first member is mined, and whose fourth + // cannot be read: the walk stops there + cone := buildCone(t, ctx, store, counter, coneLength, mined) + faulty := &faultyGetStore{Store: store, fail: cone[3]} - requireRejected(t, store, tSettings, chain) + requireRejected(t, faulty, tSettings, chain) }) t.Run("a frozen counter-spender does not hide a mined counter-spender", func(t *testing.T) { @@ -218,7 +299,7 @@ func Test_processTxMetaUsingStore_passedConflictCheckKeepsEntries(t *testing.T) _, err := s.processTxMetaUsingStore(ctx, txHashes, txMetaSlice, map[uint32]bool{77: true}, true, true, true) require.ErrorIs(t, err, errors.ErrThresholdExceeded) - require.ErrorIs(t, err, errors.ErrContextCanceled, "the threshold stays retryable") + require.NotErrorIs(t, err, errors.ErrContextCanceled, "ValidateSubtreeInternal retries the threshold by its own code") require.NotErrorIs(t, err, errors.ErrTxInvalid) require.True(t, txMetaSlice[0].isSet, "the conflicting tx passed the check, so the retry need not decorate and check it again") @@ -408,13 +489,12 @@ func Test_blessMissingTransaction_provenTxInvalidWins(t *testing.T) { require.ErrorContains(t, err, counter.TxIDChainHash().String(), "the mined counter-spender is named") } -// processTxMetaUsingStore wraps every failure in CONTEXT_CANCELED, and block -// validation retries what carries that code. A proven conflict is the one -// exception: it is the block's verdict and leaves with TX_INVALID on top and no -// CONTEXT_CANCELED in its chain, from both the batched and the unbatched branch. -// Every other failure keeps the wrap, and a proven conflict keeps it too once ctx -// is done. TestValidateBlock_ProvenConflictIsNotACancellation in blockvalidation -// checks how the callers classify the result. +// A proven conflict is the block's verdict: processTxMetaUsingStore returns it +// with TX_INVALID on top and no CONTEXT_CANCELED in its chain, from both the +// batched and the unbatched branch. A missing record keeps the CONTEXT_CANCELED +// wrap block validation needs to retry it as this node's own failure (see +// keepsCancellation). TestValidateBlock_ProvenConflictIsNotACancellation in +// blockvalidation checks how the callers classify every failure. func Test_processTxMetaUsingStore_provenConflictIsNotACancellation(t *testing.T) { ctx := context.Background() mined := utxo.WithMinedBlockInfo(utxo.MinedBlockInfo{BlockID: 77, BlockHeight: 100}) @@ -458,14 +538,124 @@ func Test_processTxMetaUsingStore_provenConflictIsNotACancellation(t *testing.T) }) } - t.Run("once ctx is done a proven conflict keeps the wrap", func(t *testing.T) { - done, cancel := context.WithCancel(ctx) - cancel() + t.Run("a TX_INVALID under another code is not a proven conflict", func(t *testing.T) { + err := settledError(ctx, errProcessTxMetaSettle, errors.NewProcessingError("not proven", errors.NewTxInvalidError("deeper in the chain"))) + require.ErrorIs(t, err, errors.ErrContextCanceled, "only TX_INVALID on top is a proven conflict") + }) - err := settledError(done, "[processTxMetaUsingStore]", errors.NewTxInvalidError("proven")) - require.ErrorIs(t, err, errors.ErrContextCanceled) + t.Run("a store fault whose text names a deadline is not a cancellation while ctx is live", func(t *testing.T) { + err := settledError(ctx, errProcessTxMetaSettle, errors.NewStorageError("store read", context.DeadlineExceeded)) + require.NotErrorIs(t, err, errors.ErrContextCanceled) + require.ErrorIs(t, err, errors.ErrStorageError) + }) - err = settledError(ctx, "[processTxMetaUsingStore]", errors.NewProcessingError("not proven", errors.NewTxInvalidError("deeper in the chain"))) - require.ErrorIs(t, err, errors.ErrContextCanceled, "only TX_INVALID on top is a proven conflict") + // ValidateSubtreeInternal returns the error as it is, so a failure that is not + // wrapped as a cancellation still carries the prefix, under its own code. + t.Run("a failure that is not wrapped keeps the prefix and its code", func(t *testing.T) { + for _, failure := range []*errors.Error{ + errors.NewTxInvalidError("proven"), + errors.NewStorageError("store down"), + errors.NewProcessingError("walk", errors.NewUtxoWalkLimitExceededError("over budget")), + errors.NewThresholdExceededError("threshold"), + } { + err := settledError(ctx, errProcessTxMetaSettle, failure) + + var top *errors.Error + require.True(t, errors.As(err, &top)) + require.Equal(t, failure.Code(), top.Code(), "%v", err) + require.Equal(t, errProcessTxMetaSettle, top.Message()) + require.ErrorIs(t, err, failure) + require.NotErrorIs(t, err, errors.ErrContextCanceled) + } + }) +} + +// cancelOnCheckStore cancels the caller's ctx when the conflict check makes its +// first read (phase 1 asks for the tx's inpoints), and serves that read and every +// later one anyway: a store whose reads had already completed when the +// cancellation arrived. +type cancelOnCheckStore struct { + utxo.Store + cancel context.CancelFunc +} + +func (c *cancelOnCheckStore) Get(_ context.Context, hash *chainhash.Hash, fs ...fields.FieldName) (*meta.Data, error) { + if slices.Contains(fs, fields.TxInpoints) { + c.cancel() + } + + return c.Store.Get(context.Background(), hash, fs...) +} + +func (c *cancelOnCheckStore) GetMeta(_ context.Context, hash *chainhash.Hash, data *meta.Data) error { + return c.Store.GetMeta(context.Background(), hash, data) +} + +// A proven conflict is proven by records that were read, so it is TX_INVALID on +// every path even when ctx is cancelled while the check runs. The fresh path +// used to return it wrapped as a cancellation in that case, while the cached path +// and blessMissingTransaction rejected: the same block was retried or rejected +// depending on which path reached its tx. +func Test_provenConflictWithCancelledCtxIsTxInvalidOnEveryPath(t *testing.T) { + mined := utxo.WithMinedBlockInfo(utxo.MinedBlockInfo{BlockID: 77, BlockHeight: 100}) + blockIds := map[uint32]bool{77: true} + + // newStore returns a store holding tx1, conflicting, whose counter-spender is + // mined on the candidate chain, and a ctx the store cancels once the check + // starts reading. + newStore := func(t *testing.T) (context.Context, *settings.Settings, *cancelOnCheckStore) { + t.Helper() + + tSettings := test.CreateBaseTestSettings(t) + tSettings.UtxoStore.SpendBatcherDurationMillis = 1 + + store := newSQLUtxoStore(t, context.Background(), tSettings) + buildConflictingChain(t, context.Background(), store, 1, false, mined) + + ctx, cancel := context.WithCancel(context.Background()) + t.Cleanup(cancel) + + return ctx, tSettings, &cancelOnCheckStore{Store: store, cancel: cancel} + } + + requireProvenAfterCancel := func(t *testing.T, ctx context.Context, err error) { + t.Helper() + + require.Error(t, ctx.Err(), "ctx was cancelled during the check") + require.ErrorIs(t, err, errors.ErrTxInvalid) + require.NotErrorIs(t, err, errors.ErrContextCanceled) + } + + for _, batched := range []bool{true, false} { + t.Run(fmt.Sprintf("fresh path, batched=%t", batched), func(t *testing.T) { + ctx, tSettings, store := newStore(t) + s := &Server{logger: ulogger.NewErrorTestLogger(t), settings: tSettings, utxoStore: store} + + txHashes := []chainhash.Hash{*tx1.TxIDChainHash()} + _, err := s.processTxMetaUsingStore(ctx, txHashes, make([]metaSliceItem, len(txHashes)), blockIds, batched, false, true) + + requireProvenAfterCancel(t, ctx, err) + require.True(t, provenConflict(err), "TX_INVALID on top: %v", err) + }) + } + + t.Run("cached path", func(t *testing.T) { + ctx, tSettings, store := newStore(t) + s, block, cachedSubtrees := newCachedCheckServer(t, store, tSettings, []chainhash.Hash{*tx1.TxIDChainHash()}, []uint32{77}) + + requireProvenAfterCancel(t, ctx, s.checkCachedSubtreesAgainstCandidateChain(ctx, block, cachedSubtrees)) + }) + + t.Run("bless", func(t *testing.T) { + ctx, tSettings, store := newStore(t) + s := &Server{ + logger: ulogger.NewErrorTestLogger(t), + settings: tSettings, + utxoStore: store, + validatorClient: conflictingValidator{}, + } + + _, err := s.blessMissingTransaction(ctx, chainhash.Hash{}, chainhash.Hash{}, tx1, 123, blockIds, validator.ProcessOptions()) + requireProvenAfterCancel(t, ctx, err) }) } diff --git a/services/subtreevalidation/processTxMetaUsingStore.go b/services/subtreevalidation/processTxMetaUsingStore.go index 6dbbe5e9e6..20c9642717 100644 --- a/services/subtreevalidation/processTxMetaUsingStore.go +++ b/services/subtreevalidation/processTxMetaUsingStore.go @@ -26,6 +26,12 @@ import ( const errProcessTxMetaContextDone = "[processTxMetaUsingStore] context done" +// errProcessTxMetaSettle prefixes every settle failure settledError returns, on +// both branches, under a cancellation wrap or under the failure's own code. It +// does not say the context is done: the cancellation wrap also covers a missing +// record or a decode fault while ctx is live (keepsCancellation). +const errProcessTxMetaSettle = "[processTxMetaUsingStore]" + var TxMetaFieldsForDecorate = []fields.FieldName{fields.Fee, fields.SizeInBytes, fields.TxInpoints, fields.Conflicting, fields.BlockIDs, fields.Creating} // txMetaFieldsForBlockValidation is TxMetaFieldsForDecorate without TxInpoints. @@ -354,7 +360,7 @@ func (u *Server) processTxMetaUsingStoreAllowing(ctx context.Context, txHashes [ } if err := u.settleCollectedConflictingTxs(ctx, g.Wait(), &conflicting, txMetaSlice, blockIds); err != nil { - return int(missed.Load()), settledError(ctx, "[processTxMetaUsingStore]", err) + return int(missed.Load()), settledError(ctx, errProcessTxMetaSettle, err) } return int(missed.Load()), nil @@ -440,7 +446,7 @@ func (u *Server) processTxMetaUsingStoreAllowing(ctx context.Context, txHashes [ } if err := u.settleCollectedConflictingTxs(ctx, g.Wait(), &conflicting, txMetaSlice, blockIds); err != nil { - return int(missed.Load()), settledError(ctx, errProcessTxMetaContextDone, err) + return int(missed.Load()), settledError(ctx, errProcessTxMetaSettle, err) } return int(missed.Load()), nil @@ -448,17 +454,72 @@ func (u *Server) processTxMetaUsingStoreAllowing(ctx context.Context, txHashes [ } // settledError is the error processTxMetaUsingStore returns for what -// settleCollectedConflictingTxs returned. Every failure keeps the ContextCanceled -// wrap it has always had, and so its class at every caller, except a proven -// conflict: that is a verdict on the block, and under ContextCanceled block -// validation took it for this node's own cancellation and retried the block -// (blockvalidation's isRecoverableBlockMessageError, and catchup's -// releaseCatchupLock). Once ctx is done the caller has given up, and even a -// proven conflict keeps the wrap. +// settleCollectedConflictingTxs returned. +// +// A proven conflict is a verdict on the block and leaves as TX_INVALID. Under a +// ContextCanceled wrap block validation took it for this node's own cancellation +// and retried the block (blockvalidation's isRecoverableBlockMessageError, and +// catchup's releaseCatchupLock). It leaves as TX_INVALID even once ctx is done, as +// on the cached path and in blessMissingTransaction: the verdict needs a member +// whose record was read and lists a block of the candidate chain, and a read that +// failed, cancelled or not, is never counted as mined. +// +// Any other failure is this node's own, and is wrapped as a cancellation only +// where block validation needs the wrap to treat it as one (keepsCancellation). +// +// Every failure leaves with message on top, as ValidateSubtreeInternal expects. +// One that is not wrapped as a cancellation keeps its own code there (prefixed). func settledError(ctx context.Context, message string, err error) error { - if ctx.Err() == nil && provenConflict(err) { + if !provenConflict(err) && keepsCancellation(ctx, err) { + return errors.NewContextCanceledError(message, err) + } + + return prefixed(message, err) +} + +// prefixed puts message on top of err under err's own code, so errors.Is and +// every caller's classification of err are unchanged. An err that is not itself +// an *errors.Error leaves as it is: a wrap would have to flatten it to text. +func prefixed(message string, err error) error { + var top *errors.Error + if !errors.As(err, &top) || error(top) != err { return err } - return errors.NewContextCanceledError(message, err) + return errors.New(top.Code(), message, top) +} + +// keepsCancellation reports whether settledError wraps a failure that is not a +// proven conflict as a cancellation. It reads codes, never error text, so a store +// fault whose text names a deadline while ctx is live leaves as a store fault. +// +// - ctx.Err(): ctx is done, so the failure is this node's own cancellation. +// - ERR_CONTEXT_CANCELED: the failure is already one, from a read that gave up +// on a done errgroup ctx. +// - ERR_TX_NOT_FOUND: a record this node no longer holds, not the peer's fault. +// - ERR_TX_INVALID under another code: a record this node could not decode. +// +// Without the wrap, block validation already retries a store fault +// (ERR_STORAGE_ERROR, or a local service fault), the threshold +// (ERR_THRESHOLD_EXCEEDED), and a processing error carrying no code it reads as +// the peer's: the walk limit, a frozen output, an input out of range. The wrap only +// mislabelled those as cancellations. The last two arms still need it: +// +// - A missing record (ERR_TX_NOT_FOUND). ValidateBlock turns it into a transient +// block-incomplete error, a code catch-up reads as the peer's, so without the +// wrap catch-up would signal a peer failure, which rotates the sync peer, and +// fetch the block from other peers, for a record this node pruned. +// - A TX_INVALID under another code, such as a record the aerospike store read +// but could not decode. The wrap keeps the Kafka path retrying the block and +// keeps catch-up from charging the peer or reporting it malicious. It does not +// keep the peer from being struck: ValidateBlock's infrastructure guard +// matches context.Canceled, not ERR_CONTEXT_CANCELED, so the decode fault +// still gets a corrupt verdict and penalizeCorruptBlockPeer, on the Kafka +// path and in catch-up alike. That is pre-existing, and +// TestValidateBlock_ProvenConflictIsNotACancellation pins it. +func keepsCancellation(ctx context.Context, err error) bool { + return ctx.Err() != nil || + errors.Is(err, errors.ErrContextCanceled) || + errors.Is(err, errors.ErrTxNotFound) || + errors.Is(err, errors.ErrTxInvalid) } diff --git a/settings/utxostore_settings.go b/settings/utxostore_settings.go index 47cb8d62b1..0e86d5217f 100644 --- a/settings/utxostore_settings.go +++ b/settings/utxostore_settings.go @@ -68,7 +68,7 @@ type UtxoStoreSettings struct { // batchers. OutpointBatcherMaxConcurrent int `key:"utxostore_outpointBatcherMaxConcurrent" desc:"Per-batcher concurrency override for the outpoint (decorate) batcher" default:"0" category:"UtxoStore" usage:"0 = inherit utxostore_batcherMaxConcurrent" type:"int" longdesc:"### Purpose\nOverrides the maximum concurrent in-flight Aerospike batch callbacks for the outpoint batcher used by PreviousOutputsDecorate / BatchPreviousOutputsDecorate. This is the heavy parent-output read fan-out during legacy block sync and the primary target of #1187: it can be capped independently of the write batchers so a single Aerospike node is not saturated on very large blocks.\n\n### Values\n- **0** (default) - inherit utxostore_batcherMaxConcurrent\n- **> 0** - cap this batcher at the given value (validated safe floor 8 on a single-node stack per #1187)"` QueryIdleTimeoutSeconds int `key:"utxostore_queryIdleTimeoutSeconds" desc:"Idle timeout for Aerospike partition queries" default:"60" category:"UtxoStore" usage:"Seconds with no records before aborting a stalled query" type:"int" longdesc:"### Purpose\nDetects stalled Aerospike connections during partition scans (e.g. unmined transaction loading).\n\n### How It Works\nDuring parallel partition queries, each worker monitors the record stream. If no record arrives within this duration, the worker assumes the Aerospike connection is dead (e.g. node restart mid-query) and aborts with an error. This prevents block assembly and pruner from hanging indefinitely on a dead connection.\n\n### Values\n- **Default 60s** - Conservative; normal scans produce records continuously\n- **120s** - For very slow Aerospike clusters under heavy load\n- **0** - Disables idle detection (NOT RECOMMENDED — returns to hang-forever behavior)\n\n### Important\nThis is NOT a total query timeout. The total scan can take hours — this only fires when data stops flowing. During normal operation, Aerospike streams records continuously so 60s of silence means the connection is dead."` - ConflictingChildrenMaxNodes int `key:"utxostore_conflictingChildrenMaxNodes" desc:"Node budget per root of a conflicting-descendant walk" default:"100000" category:"UtxoStore" usage:"Fail-closed cap on the conflicting-children BFS; <=0 disables" type:"int" longdesc:"### Purpose\nBounds the conflicting-descendant graph walks (GetConflictingChildren BFS) that subtree validation runs to check conflicting transactions against the candidate chain: as one batch per check, whether a block's conflicting transactions are checked together or a single missing transaction is blessed. Prevents a large counter-conflicting descendant cone from stalling block validation for the lifetime of the request context (issue 1391: mainnet fleet wedged at 960,827 by an O(N^2) walk).\n\n### How It Works\nThe walk counts every visited transaction, including the root. A walk from one root may visit this many. When subtree validation checks conflicting transactions (CheckBlockSubtrees on cached subtrees, the batched store lookup of a fresh subtree, and blessing a missing transaction, as a batch of one), it walks all of them as one batch on a shared visited set (issue 2037), from N roots (the conflicting transactions and their counter-spenders), and the batch may visit this budget times N. That equals the old aggregate bound of N separate walks, so a block whose every per-transaction walk fit still passes. The costs: the visited set can reach N times this budget entries in the worst case, and a single runaway cone fails after N times this budget reads rather than this budget. When the count would exceed this budget the walk stops. If what it reached already includes a transaction mined on the candidate chain, the block is rejected with TX_INVALID on the first attempt, and raising this setting does not change that. Otherwise the block fails validation closed with ERR_UTXO_WALK_LIMIT_EXCEEDED and is re-fetched and retried, so raising this setting and restarting resumes progress. The conflict-demotion path (ProcessConflicting) is intentionally NOT budgeted and always walks to completion.\n\n### Values\n- **Default 100000** - Far above realistic descendant cones (thousands); a fully linear 100k walk completes in minutes\n- **<=0** - Unlimited (operator escape hatch to unwedge a node whose legitimate cone exceeds the budget)\n\n### Recommendations\n- Alert on ERR_UTXO_WALK_LIMIT_EXCEEDED in subtree-validation logs; it means a block was rejected on this budget\n- Raise only when a legitimate cone exceeds the cap; the walk cost is one store read per transaction in the cone"` + ConflictingChildrenMaxNodes int `key:"utxostore_conflictingChildrenMaxNodes" desc:"Node budget per root of a conflicting-descendant walk" default:"100000" category:"UtxoStore" usage:"Fail-closed cap on the conflicting-children BFS; <=0 disables" type:"int" longdesc:"### Purpose\nBounds the conflicting-descendant graph walks (GetConflictingChildren BFS) that subtree validation runs to check conflicting transactions against the candidate chain: as one batch per check, whether a block's conflicting transactions are checked together or a single missing transaction is blessed. Prevents a large counter-conflicting descendant cone from stalling block validation for the lifetime of the request context (issue 1391: mainnet fleet wedged at 960,827 by an O(N^2) walk).\n\n### How It Works\nThe walk counts every visited transaction, including the root. A walk from one root may visit this many. When subtree validation checks conflicting transactions (CheckBlockSubtrees on cached subtrees, the batched store lookup of a fresh subtree, and blessing a missing transaction, as a batch of one), it walks all of them as one batch on a shared visited set (issue 2037), from N roots (the conflicting transactions and their counter-spenders), and the batch may visit this budget times N. That equals the old aggregate bound of N separate walks, so a block whose every per-transaction walk fit still passes. The costs: the visited set can reach N times this budget entries in the worst case, and a single runaway cone fails after N times this budget reads rather than this budget. When the count would exceed this budget the walk stops. If what it reached already includes a counter-conflicting transaction (a conflicting transaction, a counter-spender or a counter-spender's descendant) mined on the candidate chain, the block is rejected with TX_INVALID on the first attempt, and raising this setting does not change that. The conflicting transactions' own descendants are only checked for frozen outputs. Otherwise the block fails validation closed with ERR_UTXO_WALK_LIMIT_EXCEEDED and is re-fetched and retried, so raising this setting and restarting resumes progress. The conflict-demotion path (ProcessConflicting) is intentionally NOT budgeted and always walks to completion.\n\n### Values\n- **Default 100000** - Far above realistic descendant cones (thousands); a fully linear 100k walk completes in minutes\n- **<=0** - Unlimited (operator escape hatch to unwedge a node whose legitimate cone exceeds the budget)\n\n### Recommendations\n- Alert on ERR_UTXO_WALK_LIMIT_EXCEEDED in subtree-validation logs; it means a block was rejected on this budget\n- Raise only when a legitimate cone exceeds the cap; the walk cost is one store read per transaction in the cone"` DisableDAHCleaner bool `key:"utxostore_disableDAHCleaner" desc:"Disable the DAH cleaner process" default:"false" category:"UtxoStore" usage:"Use for debugging DAH issues" type:"bool" longdesc:"### Purpose\nDisables Phase 2 DAH (Delete-At-Height) pruning process.\n\n### Two-Phase Pruning\n- **Phase 1** - Preserves parent transactions of old unmined transactions (CRITICAL - must succeed)\n- **Phase 2** - Deletes records where DeleteAtHeight <= currentHeight (runs only if Phase 1 succeeds)\n\nWhen disabled, Phase 1 (parent preservation) still runs.\n\n### Values\n- **Default false** - Keep enabled in production for normal operation\n- **true** - Only for specific debugging scenarios\n\n### When to Disable\n- Temporarily debugging pruning issues (investigate why records pruned unexpectedly)\n- Data recovery scenarios (restore from backup without losing recent data)\n- Testing/development (keep full transaction history)\n\n### Warning\n**Disabling causes unbounded database growth.** Only disable temporarily for debugging. Re-enable after fixing root cause."` // Per-batcher tick interval (go-batcher SetTickInterval, fixed-cadence flushing).