Skip to content

Commit 226e82e

Browse files
shruggrsirdeggenclaude
authored
feat(wallet-toolbox): run StorageKnex on Postgres (#719)
* fix(wallet-toolbox): accept Postgres as a storage dbtype Add 'Postgres' to DBType, TableSettings.dbtype and the remote settings validator. Postgres dates use Date objects, as on MySQL. * fix(wallet-toolbox): run Knex migrations on Postgres - determineDBType returns 'Postgres' for the postgresql dialect without sending the MySQL VERSION() probe. - The initial migration skips binary(len) widening on Postgres. - Quote activeStorage in the 2025-02-22-001 raw update. - Create sync_transfer_parts.bytes as bytea on Postgres. * fix(wallet-toolbox): run StorageKnex queries on Postgres - Return generated ids with RETURNING on Postgres. - Alias count(*) in getCount and convert it with Number(). - Use ?? bindings for camelCase identifiers in raw SQL fragments. - Bind spendable as a boolean in purgeData. - Take the prepared-BEEF epoch and sync-state row locks on Postgres too. - Add Setup.createPostgresKnex and Setup.postgresAfterCreate, which parse int8 values as numbers per connection. * fix(wallet-toolbox): portable session merges and Postgres duplicate keys KnexSessionManager no longer compares boolean columns with integers when merging equal-timestamp updates. The merge result is unchanged. Treat Postgres error 23505 as a duplicate key in the session and payment replay stores. * test(wallet-toolbox): run Knex suites against Postgres with RUNPOSTGRES Add pg as a dev dependency. RUNPOSTGRES and POSTGRES_CONNECTION add a Postgres store wherever the suites add a MySQL store under RUNMYSQL; missing test databases are created. Add a Postgres test for the Knex remoting stores. * docs(wallet-toolbox): document Postgres storage * fix(wallet-toolbox): parse Postgres int8 per connection in StorageKnex * fix(wallet-toolbox): keep Postgres transactions usable under READ COMMITTED * test(wallet-toolbox): pass the Knex suites with a second backend * fix(wallet-toolbox): keep rejecting Postgres in ChaintracksStorageKnex * feat(wallet-toolbox): run adminStats and admin req review on Postgres * feat(wallet-toolbox): accept a Postgres connection in MonitorDaemon * feat(wallet-toolbox): build Postgres migration indexes concurrently * ci(wallet-toolbox): run the Knex storage suites against Postgres * chore(wallet-infra): sync the KnexPaymentReplayStore runtime copy * chore(wallet-toolbox): prepare 2.15.0 release notes for Postgres storage * docs(wallet-toolbox): regenerate API docs * docs(wallet-toolbox): describe Postgres migrations and monitor setup * test(wallet-toolbox): use new rows in the TxLabelMap mismatch test * test(wallet-toolbox): age the req review fixture with the server clock The MySQL driver writes Dates in the client time zone, so a client and server in different zones shifted hoursOld by the offset. * test(wallet-toolbox): trim trailing spaces from createAction2 database names MySQL rejects database names that end in a space. * fix(wallet-toolbox): order paged StorageKnex queries by key Postgres does not keep row order across LIMIT/OFFSET queries without an ORDER BY. Against a restored production wallet database, getSyncChunk for a large user returned duplicate provenTxReqs and outputTagMaps and skipped others. Paged find queries and the get*ForUser sync queries now order by the table key; getCount clears the order before counting. * docs(wallet-toolbox): explain identifier bindings in raw Postgres SQL * chore(wallet-toolbox): drop release metadata and regenerated docs from the Postgres change * revert(wallet-toolbox): drop MonitorDaemon and operator Postgres options * revert(wallet-toolbox): build migration indexes as before on Postgres Drops the CREATE INDEX CONCURRENTLY path and per-migration transaction config. Keeps bytea, the quoted activeStorage update and skipping the binary widening on Postgres. * revert(wallet-toolbox): keep admin req review MySQL-only * fix(wallet-toolbox): narrow StorageKnex Postgres handling int8 parsing is private to StorageKnex; migrate uses migrate.latest; adminStats stays MySQL-only; Postgres inserts in a transaction always use a savepoint; the allocateChangeInput reservation re-check only runs on Postgres and leaves the other dialects' code unchanged. * fix(wallet-toolbox): reject Postgres in ChaintracksStorageKnex after dialect detection * docs(wallet-toolbox): short Postgres note; drop @types/pg * fix(ci): clear new Sonar findings in StorageKnex Postgres support - KnexMigrations: replace nested ternary for the bytes column type with a lookup (S3358) - TestUtilsWalletStorage: use replaceAll for identifier quoting (S7781) - TestUtilsWalletStorage: advance sequences concurrently instead of awaiting in a loop (S9382) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Deggen <d.kellenschwiler@bsvassociation.org> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
1 parent 4d0ea63 commit 226e82e

61 files changed

Lines changed: 1172 additions & 303 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/workflows/ci.yml‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -969,6 +969,19 @@ jobs:
969969
- { id: shard-4, shard: 4 }
970970
- { id: sync-http-0, latency: 0 }
971971
- { id: sync-http-1000, latency: 1000 }
972+
services:
973+
# The Knex storage suites also run against Postgres (RUNPOSTGRES).
974+
postgres:
975+
image: postgres@sha256:d5daad18926b71c3d663f358af0aea798670cb79fb550c106d19662a9d1627ef # 17-alpine
976+
env:
977+
POSTGRES_PASSWORD: ${{ github.run_id }}-${{ github.run_attempt }}
978+
ports:
979+
- 5432:5432
980+
options: >-
981+
--health-cmd "pg_isready --host 127.0.0.1 --username postgres"
982+
--health-interval 5s
983+
--health-timeout 5s
984+
--health-retries 30
972985
steps:
973986
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
974987
- uses: pnpm/action-setup@ea17c68df8912ef543352723c149a84f56e3d413 # v6.1.0
@@ -989,11 +1002,16 @@ jobs:
9891002
env:
9901003
WALLET_SHARD: ${{ matrix.shard }}
9911004
SYNC_LATENCY: ${{ matrix.latency }}
1005+
POSTGRES_PASSWORD: ${{ github.run_id }}-${{ github.run_attempt }}
9921006
run: |
9931007
# The two real-HTTP latency scenarios run in independent processes;
9941008
# their global transport spies must never overlap in one Jest worker.
9951009
args=(--coverage --coverageDirectory=coverage/${{ matrix.id }})
9961010
if [ -n "$WALLET_SHARD" ]; then
1011+
export RUNPOSTGRES=1
1012+
POSTGRES_CONNECTION="$(jq -cn --arg password "$POSTGRES_PASSWORD" \
1013+
'{host: "127.0.0.1", port: 5432, user: "postgres", password: $password, database: "postgres"}')"
1014+
export POSTGRES_CONNECTION
9971015
args+=(--testPathIgnorePatterns='man.test.ts|bench.test.ts|test/monitor/Monitor.test.ts|client/test|mobile/test|src/storage/sync/syncTransferHttp.test.ts')
9981016
args+=(--shard="$WALLET_SHARD/4")
9991017
else

‎infra/wallet-infra/src/KnexPaymentReplayStore.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,7 @@ function isDuplicate(error: unknown): boolean {
1010
value.code === 'ER_DUP_ENTRY' ||
1111
value.code === 'SQLITE_CONSTRAINT_PRIMARYKEY' ||
1212
value.code === 'SQLITE_CONSTRAINT_UNIQUE' ||
13+
value.code === '23505' ||
1314
value.errno === 1062
1415
)
1516
}

‎packages/wallet/wallet-toolbox/README.md‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -63,6 +63,20 @@ store already left with unjournaled schema objects by an older version. Preserve
6363
the database and verified backups and reconcile the exact schema and migration
6464
journal before recovery; do not delete journal rows or wallet data blindly.
6565

66+
### Postgres storage
67+
68+
`StorageKnex` runs on Postgres through the knex `pg` client. Install `pg` in the
69+
application and pass a Postgres knex:
70+
71+
```ts
72+
import { knex as makeKnex } from 'knex'
73+
74+
const knex = makeKnex({ client: 'pg', connection: { host, port, user, password, database } })
75+
const storage = new StorageKnex({ ...StorageKnex.defaultOptions(), chain: 'main', knex })
76+
```
77+
78+
`ChaintracksStorageKnex` and `adminStats` do not support Postgres.
79+
6680
## Overview
6781

6882
The Wallet Toolbox is the reference implementation of the BRC-100 wallet interface. It connects the BSV SDK's cryptographic primitives to real storage backends, network services, and signing flows so that application developers don't have to wire these layers together themselves.

‎packages/wallet/wallet-toolbox/package.json‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -111,6 +111,7 @@
111111
"fs-extra": "^11.4.0",
112112
"jest": "^30.5.1",
113113
"oxlint": "^1.83.0",
114+
"pg": "^8.23.0",
114115
"picocolors": "^1.1.1",
115116
"ts-jest": "^29.4.12",
116117
"ts-node": "^10.9.2",

‎packages/wallet/wallet-toolbox/src/services/chaintracker/chaintracks/Storage/ChaintracksStorageKnex.ts‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@ import { DBType } from '../../../../storage/StorageReader'
1414
import { BulkHeaderFileInfo } from '../util/BulkHeaderFile'
1515
import { HeightRange } from '../util/HeightRange'
1616
import { Chain } from '../../../../sdk/types'
17-
import { WERR_INVALID_OPERATION, WERR_INVALID_PARAMETER } from '../../../../sdk/WERR_errors'
17+
import { WERR_INVALID_OPERATION, WERR_INVALID_PARAMETER, WERR_NOT_IMPLEMENTED } from '../../../../sdk/WERR_errors'
1818
import { determineDBType } from '../../../../storage/schema/KnexMigrations'
1919
import { normalizeBulkHeaderFileInfo, normalizeBulkHeaderFileSequence } from '../util/BulkFileDataManager'
2020

@@ -94,6 +94,9 @@ export class ChaintracksStorageKnex extends ChaintracksStorageBase implements Ch
9494
}
9595
if (!this.isAvailable) {
9696
this._dbtype = await determineDBType(this.knex)
97+
// Chaintracks storage supports SQLite and MySQL only.
98+
if (this._dbtype === 'Postgres')
99+
throw new WERR_NOT_IMPLEMENTED('ChaintracksStorageKnex does not support Postgres.')
97100
await super.makeAvailable()
98101
// Connect the bulk data file manager to the table provided by this storage class.
99102
await this.bulkManager.setStorage(this, this.log)

0 commit comments

Comments
 (0)