feat(cli): cache the shadow database baseline as a PGDATA snapshot (CLI-2191) - #6184
feat(cli): cache the shadow database baseline as a PGDATA snapshot (CLI-2191)#6184avallete wants to merge 105 commits into
Conversation
…lta-next # Conflicts: # apps/cli-go/cmd/db.go # apps/cli/docs/go-cli-porting-status.md # apps/cli/src/legacy/commands/db/diff/SIDE_EFFECTS.md # apps/cli/src/legacy/commands/db/diff/diff.handler.ts # apps/cli/src/legacy/commands/db/diff/diff.integration.test.ts # apps/cli/src/legacy/commands/db/diff/diff.layers.ts # apps/cli/src/legacy/commands/db/pull/SIDE_EFFECTS.md # apps/cli/src/legacy/commands/db/pull/pull.handler.ts # apps/cli/src/legacy/commands/db/pull/pull.integration.test.ts # apps/cli/src/legacy/commands/db/pull/pull.layers.ts # apps/cli/src/legacy/commands/db/push/SIDE_EFFECTS.md # apps/cli/src/legacy/commands/db/reset/SIDE_EFFECTS.md # apps/cli/src/legacy/commands/db/schema/declarative/declarative.orchestrate.integration.test.ts # apps/cli/src/legacy/commands/db/schema/declarative/declarative.orchestrate.ts # apps/cli/src/legacy/commands/db/schema/declarative/generate/generate.handler.ts # apps/cli/src/legacy/commands/db/schema/declarative/generate/generate.integration.test.ts # apps/cli/src/legacy/commands/db/schema/declarative/sync/SIDE_EFFECTS.md # apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.handler.ts # apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.integration.test.ts # apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.seam.layer.ts # apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.seam.service.ts # apps/cli/src/legacy/shared/db-bootstrap/db-setup.ts # apps/cli/src/legacy/shared/legacy-db-connection.sql-pg.layer.ts # apps/cli/src/legacy/shared/legacy-http-errors.ts # apps/cli/src/legacy/shared/legacy-migration-apply.ts # apps/cli/src/legacy/shared/legacy-migration-apply.unit.test.ts # apps/cli/src/legacy/shared/legacy-pgdelta.cache.ts # packages/api/src/effect.ts # packages/api/src/internal/client.ts # packages/api/src/internal/client.unit.test.ts
…lta-next # Conflicts: # apps/cli/src/legacy/commands/db/diff/SIDE_EFFECTS.md # apps/cli/src/legacy/commands/db/diff/diff.handler.ts # apps/cli/src/legacy/commands/db/diff/diff.integration.test.ts # apps/cli/src/legacy/commands/db/pull/SIDE_EFFECTS.md # apps/cli/src/legacy/commands/db/shared/legacy-shadow-source.ts
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9baed89bbc
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
The export now writes data/SUPABASE_BASELINE into the stopped shadow's PGDATA right before the outbound docker cp, and warm-hit validation requires it alongside data/PG_VERSION — a valid bare-cluster tar is discarded as suspect instead of restoring into baselinePresent: true and skipping setup. Only snapshotBaseline stamps, strictly after the platform baseline, so a snapshot taken too early can never carry the marker (regression guard). Uppercase name keeps the entry at the front of docker cp's sorted tar so warm validations stay a first-blocks read. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4d494f4d96
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…tants The baseline marker's content is now the snapshot's own cache key and warm validation verifies it, so a valid snapshot copied over another key's filename is discarded (wrong-key verdict, distinct from a missing marker) instead of restoring a mismatched baseline. The baseline digest (renamed baseline_embedded_digest) now also hashes the Realtime seed constants persisted by the one-shot job (tenant id, encryption key, db user/name/port), so editing them without an image bump invalidates stale tars. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b382f94904
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…cli-pr-conflicts-529506 # Conflicts: # apps/cli/src/legacy/shared/db-bootstrap/shadow-database.ts
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9134f2e9f5
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Docker-aware gate (describeDockerLive), ephemeral shadow port instead of a fixed one, best-effort network removal in the scope finalizer, and scoped SUPABASE_HOME/SUPABASE_SHADOW_CACHE overrides restored on every outcome. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5ca12af032
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
One runSupabaseLive golden path: init + minimal start, then db diff --local --use-pg-delta twice against the same SUPABASE_HOME — the cold run must export the tar (shadow-debug: baseline-export), the warm run must restore it (baseline-restore, refreshed mtime, identical stdout). Replaces the in-process legacyAcquireShadowDatabase calls, which could stay green while the command wiring or env propagation broke; mechanics coverage lives in the integration suite. Shadow port comes from SUPABASE_DB_SHADOW_PORT with a bounded retry on bind conflicts — true reservation is impossible since Docker must bind the port itself. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4ba00ac249
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…n's pid Eight candidates from a pid-seeded base in the IANA dynamic range replace the fixed 54987/54988 pair, so independently concurrent runs start from different bases and an occupied port costs one retry step. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9d277539bb
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
avallete
left a comment
avallete
left a comment
There was a problem hiding this comment.
We should have SUPABASE_SHADOW_CACHE false by default.
So we can have a soaking / testing / opt-in period before releasing to all consumers.
Gate SUPABASE_SHADOW_CACHE through the viper helper, tighten tar retention, drop the one-off debug channel, and reuse existing helpers.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7fd00b925a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
The platform image still installs pgjwt, so RESTRICT drop of pgcrypto fails during CLI-owned shadow prep.
…erge-conflicts-848c2e # Conflicts: # apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.layer.ts # apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.layer.unit.test.ts
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: bb21b82611
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
The next engine ignores schema_paths when building its migrations baseline, so the declared-schema drift diffed to nothing. Also fix the invalid 'analytics' exclude (logflare). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3c8ac90d5e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
- Revalidate roles.sql before publishing a cold baseline tar; a mid-provision edit skips the export instead of publishing under a stale key. - Share one JWKS memo across provisionPlan's two shadows so their snapshot keys hash identical bytes. - Scope legacyWaitForShadowReady's lastFailure per evaluation (Effect.suspend). - Narrow the platform-error reason with Predicate.isTagged. - Relocate the shadow-cache/session-semantics notes from the frozen go-cli-divergences.md into the SIDE_EFFECTS.md files. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…omments Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e31bb57d1b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2 participants
db diff/db pull/ declarative sync rebuild the shadow's platform baseline (initdb + init SQL + the PG15+ one-shot realtime/storage/auth jobs, ~15s) on every run, before user migrations even start. This PR makes that a once-per-config cost. Measured on a default-services PG17 project: ~3.3s warm provision vs ~15.5s cold.The cache is opt-in for soak (
SUPABASE_SHADOW_CACHE=1/true; unset is off). After soak it can flip to default-on.What changed
legacyWaitForShadowReadypolls container state plus a short authenticated connect (500ms spacing, 2s connect timeout). Same timeout budget, error shape, and log dump. The healthcheck config and the long-runningdbcontainer's wait are untouched.exec'd postgres entrypoint. PID 1 wassh, which swallowed SIGTERM, so everydocker stopburned the 10s grace period. All three entrypoint variants nowexec; stops take ~1s (also benefitssupabase stop).SUPABASE_SHADOW_CACHEis set). When enabled, the cold path stops the shadow after the platform baseline and streams PGDATA to~/.supabase/cache/shadow-baseline/shadow-baseline-<key>.tar(~90MB;SUPABASE_HOMEoverrides the root). A matching later run unpacks that tar into a fresh container beforedocker start— initdb and the baseline are skipped, migrations apply directly.Cache contract
docker cp -. The directory form ofdocker cpresets ownership to root and Postgres refuses the data dir..partialfiles older than 5 minutes are swept. Warm hits refresh mtime.pg_netpolicy).sync --no-cachebypasses the cache for that invocation.--rm(Docker destroys an AutoRemove container ondocker stop). Release still removes every shadow; a SIGKILLed CLI leaves at most one stopped project-labeled container, swept bysupabase stop.Still uncached:
db diff --use-pgadmin,db pull --declarative's raw shadow,migration squash.