[marcusrbrown]

Adds docs/runbooks/umami-postgres-major-upgrade.md. It is Unit 2 of the Umami Postgres 18 plan: a runbook with copy-paste commands that covers pre-flight, an encrypted backup, staging on the workstation, a gated cutover, supervised retention on PG18, rollback, and finalize.

• Every hard gate is a command with Expect:/STOP:. Every destructive step lists its rollback next to it.
• The integrity manifest prints the row count and an md5 of the row content for each table. \gexec generates the per-table queries, and the session's text-output settings are pinned so PG15 and PG18 produce the same output. The gate is an exact diff.
• Dumps stream over SSH into age -r on the workstation, so the droplet never holds a dump. Decrypt tests write plaintext only to umask 077 temp dirs and delete it right after the test.
• Staging runs on the workstation as linux/amd64 with throwaway secrets, an internal network, and no published ports.
• Each gate fails closed. Negative tests with a tampered manifest, a failing gh, the wrong age identity, and a bad checksum each exit non-zero before any restart.

I validated the runbook against disposable linux/amd64 PG15 and PG18 containers:

• The manifest diff stayed empty across a real 15 → 18 dump and restore. Changing one character changed the checksum.
• The layout, SCRAM, and collation gates passed: PGDATA=/var/lib/postgresql/18/docker on a single named volume at /var/lib/postgresql.
• The TCP password probe works.
• The age round-trip and the PG15 rollback rehearsal passed.

Commands that run on the droplet or need the umami image (Prisma, retention) were not exercised here. The staging step runs them first.

[fro-bot]

Verdict: CONDITIONAL

Blocking issues

• [P2] Preflight failures can return success. In docs/runbooks/umami-postgres-major-upgrade.md:355, placing the strict subshell on the left of && disables Bash's errexit throughout that subshell. If the remote disk-space assertion fails, execution still reaches sentinel_check; a successful sentinel check makes P3 succeed. The same pattern at lines 378–380 suppresses failures in P4, including the rollback-anchor lookup and checksum pipeline. pipefail does not force an exit in this context. Run these strict blocks outside conditional lists, capture/check their status afterward, or explicitly propagate every failure. Reproduced locally with the same Bash control-flow structure: a failing child command followed by a successful sentinel probe returned success.

Non-blocking concerns

None

Missing tests

Add failure-injection validation for P3's remote failure/low-disk path and P4's missing-anchor/checksum-failure paths; each must return nonzero before later commands or parent-shell assignments execute. No production migration commands were run during review.

Risk assessment (MED) + rationale

Documentation-only merge has no immediate deploy impact, but these copy-paste commands govern a production database migration. Silently bypassed preflight checks undermine the stated fail-closed recovery guarantees.

---

Run Summary

Field	Value
Event	pull_request
Repository	marcusrbrown/infra
Run ID	40064091704
Cache	hit
Session	ses_e93be3374ffekFiKoWeZoSLeaM

[fro-bot]

Verdict: CONDITIONAL

Blocking issues

• [P2] Previous preflight failure-propagation issue remains unresolved. docs/runbooks/umami-postgres-major-upgrade.md:355 still places the strict subshell on the left of &&, disabling Bash's errexit inside it. A failed remote disk-space check followed by a successful sentinel_check returns success. Lines 378–380 have the same issue for P4's anchor lookup and checksum pipeline. Reproduced again locally. Execute the strict blocks outside conditional lists and explicitly check their resulting status, or explicitly propagate each failure. The latest keychain change does not address this blocker.

Non-blocking concerns

None

Missing tests

Failure-injection validation remains needed for P3's remote/low-disk failure and P4's missing-anchor/checksum failure, demonstrating nonzero exit before subsequent commands or image assignments.

Risk assessment (MED) + rationale

No immediate deploy change, but the runbook controls a production database migration; preflight checks can incorrectly succeed and undermine recovery guarantees. Reviewed the latest diff and revalidated the outstanding finding without running production commands.

---

Run Summary

Field	Value
Event	pull_request
Repository	marcusrbrown/infra
Run ID	40065406309
Cache	hit
Session	ses_e93be3374ffekFiKoWeZoSLeaM

[fro-bot]

Verdict: PASS

The previous blocker is resolved in 20fb57a: P3/P4 now execute strict subshells independently and check their exit status before continuing. Local failure-injection checks of the revised control-flow patterns confirmed remote failure, missing anchor, and checksum-pipeline failure stop subsequent commands and image assignments. git diff --check passed.

Blocking issues

None

Non-blocking concerns

None

Missing tests

None

Risk assessment (MED) + rationale

The merge changes documentation only, but its commands govern a production database migration. Staging, restore-integrity gates, and the preserved PG15 volume limit operational risk. Production commands were not executed during review.

---

Run Summary

Field	Value
Event	pull_request
Repository	marcusrbrown/infra
Run ID	40067146180
Cache	hit
Session	ses_e93be3374ffekFiKoWeZoSLeaM