sqldb: reuse one migrate driver across migrations to avoid pool exhaustion

[Vandit1604]

Fixes #11007.

Problem

On a fresh Postgres database, lnd hangs during startup migrations when db.postgres.maxconnections is small. Each migration step creates a new golang-migrate driver with pgx_migrate.WithInstance, and that driver checks out one connection from the pool and holds it until it is closed. The drivers are never closed, so every migration keeps another connection for the rest of startup. There are 18 migrations, so a fresh database can need around 20 connections. Once the pool is used up, the migration-tracker transaction waits for a connection that never frees, with no timeout or error, and startup never finishes.

@ziggie1984 diagnosed this in #11007 and already wrote a fix in #10426 (commit 214af56). That PR also changes the default connection count and has been open a while, so this is the connection-reuse part on its own, plus the regression test asked for there.

Fix

ApplyAllMigrations now creates one migrate driver and reuses it for every step, so all migrations share a single connection instead of one each. The driver is not closed, because closing it would close the shared *sql.DB the store keeps using, so it holds one connection for the life of the store. That is the trade-off for not exhausting the pool.

With the fix a migration run needs two connections: one for the shared driver and one for the migration-tracker transaction. So maxconnections=1 would still deadlock. PostgresConfig.Validate now rejects maxconnections=1 (0 still means unlimited), so that fails at startup with a clear message instead of hanging.

The per-call ExecuteMigrations / GetSchemaVersion / SetSchemaVersion methods are kept because the test harness uses them, with a note not to call them in a loop.

I did not touch the sqlite path. Its WithInstance keeps the *sql.DB directly and its lock is an in-process flag, so it never holds a connection open and is not affected by this bug.

Testing

TestPostgresMigrationSmallConnPool applies all migrations with maxconnections=2 under a timeout. It deadlocks (times out) on master and passes with this change. The full sqldb package passes.

[github-actions]

🔴 PR Severity: CRITICAL

Rule-based classification | 4 files | 128 additions, 1 deletion

🔴 Critical (2 files)

• sqldb/config.go - sqldb/* package; database migration/connection config, always classified critical
• sqldb/postgres.go - sqldb/* package; changes migration driver lifecycle and connection pooling used during schema migrations, always classified critical

🟢 Low (2 files)

• sqldb/migrations_test.go - test-only change (regression test for the connection pool deadlock)
• docs/release-notes/release-notes-0.22.0.md - release notes update

Analysis

This PR modifies sqldb/postgres.go and sqldb/config.go, which fall under the database migration path (sqldb/*) that is always treated as critical severity regardless of size. The change alters how ApplyAllMigrations creates and reuses a golang-migrate driver across all migration steps (to avoid Postgres connection-pool exhaustion during startup), and adds validation rejecting maxconnections=1. This touches core migration/connection-lifecycle behavior for the Postgres backend, so expert review is warranted despite the modest diff size (no file-count or line-count bump was needed — already at the top tier).

---

_{To override, add a severity-override-{critical,high,medium,low} label.}

[Lrifton92]

Verified the mechanism: with maxconnections=2 on a fresh DB, master leaks one sql.Conn per driver (GetSchemaVersion + first ExecuteMigrations) and executor.ExecTx blocks on BeginTx with no deadline, so the new test does time out on master and pass here. Sequential use only, and the driver's Lock/Unlock are balanced by Migrate.Migrate, so sharing it is safe. The SQLite driver keeps the *sql.DB directly, so leaving it alone is correct.

One behavioural point I think is worth fixing before merge, plus a few smaller ones:

1. Steady-state regression. On master, a fully-migrated DB creates zero drivers (GetDatabaseVersion succeeds, the loop skips everything, so neither GetSchemaVersion nor ExecuteMigrations is ever called). With this PR ApplyAllMigrations always builds the driver, so every restart now pins one pool connection for the life of the process even when there is nothing to migrate. Two options: (a) create the driver lazily on first use inside the executor; or (b) open a dedicated *sql.DB for the migrate driver (sql.Open("pgx", s.cfg.Dsn)) and defer driver.Close() at the end of ApplyAllMigrations — pgx_migrate.Postgres.Close() closes both the conn and its own db, so nothing is reserved afterwards and nothing leaks past store.Close(). (b) also removes the need for the maxconnections == 1 guard.
2. PostgresConfig.Validate is shared with the kvdb-postgres backend and with skipmigrations=true, where maxconnections=1 does not hit this deadlock. If the guard stays, it belongs in ApplyAllMigrations where the requirement actually exists.
3. Nit: "0 (unlimited)" is not accurate for the native store — NewPostgresStore maps 0 to defaultMaxConns (25).
4. Nit: TestPostgresMigrationSmallConnPool never closes store.

Follow-up (not for this PR): sqldb/v2/postgres.go has the same per-call driver pattern.

[Lrifton92]

On master a fully-migrated DB creates no driver at all (GetDatabaseVersion hits, the loop skips). This now reserves one pool connection on every startup, migration or not. Either create the driver lazily on first use, or give it its own *sql.DB and defer driver.Close() here — pgx_migrate.Postgres.Close() closes the conn and the db it was built on, so with a private db that is exactly what we want and nothing stays reserved.

[Lrifton92]

This config also drives kvdb-postgres and the skipmigrations path, where maxconnections=1 was accepted and doesn't deadlock. Consider checking this in ApplyAllMigrations instead. Also "0 (unlimited)" isn't right for the native store: NewPostgresStore turns 0 into defaultMaxConns (25).

[Lrifton92]

t.Cleanup(func() { require.NoError(t, store.DB.Close()) }) to match NewTestPostgresDB.

[Vandit1604]

Thanks for the careful review, and good catch on the steady-state regression. You're right that a fully-migrated DB built no driver before, and my version pinned a connection on every restart.

Went with your option (b): the driver now runs on its own sql.Open("pgx", dsn) handle with defer driver.Close(), so nothing stays reserved on the store's pool once migrations finish, migration or no migration. That also let me drop the maxconnections=1 guard entirely, so config.go is untouched now. Agree it didn't belong in the shared validation. Added the t.Cleanup to close the store too.

Left sqldb/v2/postgres.go alone as you suggested, happy to do that as a follow-up.