Parse → Supabase DB foundation: read-scope schema, sample importer, local dev

.github/workflows/ci.yml

@@ -35,4 +35,17 @@ jobs:
         env:
           BLOOM_PARSE_APP_ID_PROD: ${{ secrets.BLOOM_PARSE_APP_ID_PROD }}
           BLOOM_PARSE_APP_ID_DEV: ${{ secrets.BLOOM_PARSE_APP_ID_DEV }}
-          BLOOM_PARSE_APP_ID_UNIT_TEST: ${{ secrets.BLOOM_PARSE_APP_ID_UNIT_TEST }}
+          BLOOM_PARSE_APP_ID_UNIT_TEST: ${{ secrets.BLOOM_PARSE_APP_ID_UNIT_TEST }}
+
+      # Keep in sync with the supabase version pinned in package.json.
+      - uses: supabase/setup-cli@v1
+        with:
+          version: 2.109.1
+
+      # Prove the SQL migrations apply cleanly from scratch (catches broken
+      # or order-dependent migrations before they reach staging).
+      - name: Validate migrations apply from scratch
+        run: |
+          supabase start -x logflare,vector,studio,imgproxy,realtime,mailpit,edge-runtime,storage-api
+          supabase db reset
+          supabase stop --no-backup

README.md

@@ -12,7 +12,7 @@ pnpm install
 cp .env.example .env.local
 # Edit .env.local with your Parse Server credentials
 
-# Start local development (requires Docker)
+# Start local development (requires a container runtime — see Prerequisites)
 pnpm dev
 
 # Run tests
@@ -21,17 +21,54 @@ pnpm test
 
 Test the fs function:
 ```bash
-curl "http://127.0.0.1:54321/functions/v1/fs/harvest/VuebFgcL0R/Ososi.bloompub" -o test.bloompub
+curl "http://127.0.0.1:44321/functions/v1/fs/harvest/VuebFgcL0R/Ososi.bloompub" -o test.bloompub
 ```
 
 ## 📋 Prerequisites
 
 - [pnpm](https://pnpm.io/) v11+ (pins its own version via `packageManager` and downloads
   Node.js v22.20.0 via `devEngines.runtime` in `package.json` — no Volta or manual Node install needed)
 - [Deno](https://deno.com/) v2.9+ (runs and tests the edge functions)
-- [Docker Desktop](https://docs.docker.com/desktop/) (local development only)
+- A container runtime (local development only): [Podman](https://podman.io/) (see
+  [Windows + Podman setup](#-windows--podman-setup) below) or
+  [Docker Desktop](https://docs.docker.com/desktop/)
 - [Supabase account](https://supabase.com/)
 
+## 🪟 Windows + Podman setup
+
+Podman is the supported non-Docker-Desktop way to run the local stack (verified 2026-07):
+
+```powershell
+winget install RedHat.Podman
+podman machine init
+podman machine set --rootful   # required: rootless port forwarding doesn't reach the Windows host
+podman machine start
+```
+
+Then start the stack. If Docker Desktop is also installed, point the CLI at Podman's pipe
+explicitly, and exclude the analytics services (on Windows they require a TCP-exposed
+Docker daemon, which Podman doesn't provide):
+
+```powershell
+$env:DOCKER_HOST = "npipe:////./pipe/podman-machine-default"
+pnpm exec supabase start -x logflare,vector
+```
+
+Gotchas we hit so you don't have to:
+
+- **Local ports are 443xx, not Supabase's default 543xx** (API `44321`, DB `44322`,
+  Studio `44323`, Mailpit `44324`). Windows reserves semi-random "excluded port ranges"
+  for Hyper-V/WSL inside the dynamic range (49152+), and Supabase's defaults landed inside
+  one — every service unreachable from the host, with no error anywhere. Ports below 49152
+  can't be dynamically excluded. Check yours with
+  `netsh interface ipv4 show excludedportrange protocol=tcp`.
+- Podman (unlike Docker) doesn't auto-create missing bind-mount sources; the repo now
+  commits `supabase/snippets/` and `supabase/seed.sql` so `supabase start` has everything
+  it needs.
+- `supabase stop`'s volume prune trips over a Podman/Docker API difference
+  ("all" is an invalid volume filter) — harmless; use `supabase db reset` to get a truly
+  fresh database.
+
 ### Dependency policy
 
 Both package resolvers enforce a **7-day cooldown**: a version published less than 7 days
@@ -48,17 +85,37 @@ lockfile changes.
 ## 📁 Project Structure
 
 ```
-supabase/functions/
-├── _shared/              # Shared utilities
-│   ├── BloomParseServer.ts
-│   └── utils.ts
-├── fs/                   # S3 proxy function (streams book files)
-│   ├── index.ts
-│   ├── BookData.ts
-│   └── README.md
-└── tests/               # Deno tests
+supabase/
+├── migrations/           # SQL migrations (the database schema: books, languages, tags, ...)
+├── seed.sql              # Local-dev seed (real data comes from packages/sync-tool)
+└── functions/
+    ├── _shared/          # Shared utilities
+    │   ├── BloomParseServer.ts
+    │   └── utils.ts
+    ├── fs/               # S3 proxy function (streams book files)
+    │   ├── index.ts
+    │   ├── BookData.ts
+    │   └── README.md
+    └── tests/            # Deno tests
+packages/
+└── sync-tool/            # Parse -> Supabase data import (v0: sample importer)
+docs/
+└── db/                   # Database migration docs (field mapping, plan review, roadmap)
 ```
 
+## 📚 Importing sample data (local dev)
+
+With the local stack running, pull ~100 real books (plus their languages, tags, uploaders,
+and relatedBooks) from the production Parse server into your local database:
+
+```bash
+pnpm --filter @bloom/sync-tool import-sample
+```
+
+Idempotent — re-run any time to refresh. It refuses to write to a non-localhost Supabase
+unless you set `SYNC_ALLOW_REMOTE=1` (env vars are `SYNC_*`-prefixed on purpose; see
+`packages/sync-tool/src/import-sample.mjs`).
+
 ## 🔧 Scripts
 
 ```bash
@@ -78,13 +135,13 @@ Streams book files from S3 without exposing bucket details.
 **Example**:
 ```bash
 # Get thumbnail
-curl "http://localhost:54321/functions/v1/fs/dev-harvest/ZWI7FUQnDd/thumbnails/thumbnail-256.png"
+curl "http://localhost:44321/functions/v1/fs/dev-harvest/ZWI7FUQnDd/thumbnails/thumbnail-256.png"
 
 # Download book (streams, no buffering)
-curl "http://localhost:54321/functions/v1/fs/harvest/VuebFgcL0R/Ososi.bloompub" -o book.bloompub
+curl "http://localhost:44321/functions/v1/fs/harvest/VuebFgcL0R/Ososi.bloompub" -o book.bloompub
 
 # Range request
-curl -H "Range: bytes=0-1023" "http://localhost:54321/functions/v1/fs/harvest/VuebFgcL0R/Ososi.bloompub"
+curl -H "Range: bytes=0-1023" "http://localhost:44321/functions/v1/fs/harvest/VuebFgcL0R/Ososi.bloompub"
 ```
 
 See [`supabase/functions/fs/README.md`](supabase/functions/fs/README.md) for details.

docs/db/README.md

@@ -0,0 +1,98 @@
+# The database: Parse → Supabase migration notes
+
+Status (2026-07-17): the **"Local Supabase + Blorg" milestone** is implemented — the
+read-scope schema exists as a migration, `packages/sync-tool` imports a ~100-book sample
+from production Parse, and blorg (branch `SupabaseMigration`) can browse it anonymously
+against a local stack. Parse remains the production system of record; nothing here is
+serving production traffic yet.
+
+See `FUNCTIONS-MIGRATION-PLAN.md` at the repo root for the *functions* migration (Azure → Edge
+Functions, phases F0–F8). This doc covers the *database* side.
+
+## Schema conventions
+
+- **IDs**: `id TEXT PRIMARY KEY` preserving legacy Parse objectIds. New rows get a
+  DB-generated legacy-style 10-char alphanumeric id (`generate_legacy_style_id()`).
+- **Names**: Parse camelCase → snake_case (`bookInstanceId` → `book_instance_id`).
+  The conversion is capital-run-aware, so `bloomPUBVersion` → `bloom_pub_version` with no
+  special-casing. (The `analytics_*` fields already carry underscores in Parse.)
+- **Types**: Parse String → `text`, Number → `integer`/`numeric`/`bigint` (timestamps),
+  Boolean → `boolean`, Date → `timestamptz`, Array-of-strings → `text[]`,
+  Object / Array-of-objects → `jsonb` (`show`, `internet_limits`, `tools`).
+- **Field authority**: the private repo `BloomBooks/bloom-parser-server-schema`
+  (`schema/production.json`), NOT `setupTables` in bloom-parse-server's cloud code (its
+  header admits it's out of date). `publisher_book_id` postdates the dump (bloom-parse-server
+  PR #76).
+- **Relations**: `books.lang_pointers text[]` keeps the raw language ids for sync fidelity,
+  AND `book_languages` (junction, FKs) exists so PostgREST/supabase-js can embed language
+  records with books — the equivalent of Parse's `include=langPointers`:
+  `select("*, languages(*)")` (PostgREST resolves the many-to-many through the junction
+  automatically). This works because Parse's legacy `languages` array field is deliberately
+  not ported (see correction 8) — the name is reserved for the embed.
+  `books.uploader_id` → `users(id)`.
+- **RLS**: enabled everywhere; anonymous public read via `Public read` policies + table
+  grants; no client write policies (the sync tool writes with the service role, which
+  bypasses RLS). Two policies are row-gated: soft-deleted books (`is_deleted`, the future
+  sync's tombstones) are never served, and a `users` row is only readable while the user
+  has at least one visible book — uploader emails stay embeddable for book display without
+  the table being a listable email directory.
+- **Deliberately absent (post-milestone work)**: derivation triggers replacing Parse's
+  `beforeSave` (search string, tag normalization, moderator-field preservation — these must
+  be gated off for sync connections when they arrive), `updated_at` triggers (same gating
+  problem — see correction 7), write policies, Firebase third-party
+  auth wiring, and the classes `apiAccount` (needed for the opds function),
+  `appSpecification`/`appDetailsInLanguage`/`booksInApp` (verify they're used at all before
+  porting), `downloadHistory`, `version`, `bookDeletion` tombstones.
+
+## Corrections to the earlier draft plans
+
+The `supabase/*.md` docs in bloom-parse-server were a useful starting point; things fixed
+or superseded here:
+
+1. `uploader_id REFERENCES auth.users(id)` was wrong — under Firebase third-party auth,
+   `auth.users` stays empty. It references `public.users` (TEXT legacy ids).
+2. The draft DDL was missing `edition`, `upload_pending_timestamp`, `banner_image_url`
+   (languages), and the whole `apiAccount` class; and misnamed `analytics_started_count`.
+3. `language.isoCode` is NOT unique in production (multiple rows per code) — the draft's
+   UNIQUE constraint would have broken the import.
+4. `tools` can hold objects — `jsonb`, not `text[]`.
+5. The draft's legacy-id generator could produce <10 chars (base64 of 8 bytes minus
+   stripped symbols); ours draws 24 bytes.
+6. The auth plan (custom login endpoint + bespoke session tokens) is superseded by
+   Supabase's native **third-party Firebase auth** (`[auth.third_party.firebase]`,
+   supabase-js `accessToken`, `auth.jwt()` in RLS; requires a `role: 'authenticated'`
+   custom claim on Firebase users — a backfill task when we get there).
+7. The sync plan's derivation-trigger idea conflicts with sync writes: triggers re-deriving
+   `search`/tags would fight the Parse-computed values the sync delivers. When triggers
+   arrive they must be gated (e.g. a `bloom.sync` session flag) so sync writes pass through
+   verbatim — which also gives us a parity test (replay a synced row through the triggers,
+   diff against Parse's output). This bit immediately: the v0 schema shipped `updated_at`
+   triggers stamping `now()` on update, which meant every importer re-run (an upsert's
+   update path) silently replaced Parse's real timestamps with the import time. They're
+   removed until the gating mechanism exists.
+8. Parse's legacy `books.languages` array is not ported. Production evidence (2026-07-18):
+   only 146 books have the field, none created after Jan 2015, and every value is `[]` —
+   it carries zero information. Keeping it would also have collided with the natural
+   PostgREST embed name: a `languages` *column* on `books` and an embedded `languages`
+   *relation* can't both appear in one response, forcing every consumer to alias the embed
+   forever. `lang_pointers` + `book_languages` carry the real language data.
+
+## Roadmap after this milestone (summary)
+
+1. **Full sync tool**: watermark-based incremental sync (order: users → languages → tags →
+   books), tombstones for book deletions (the ONE change bloom-parse-server needs: an
+   `afterDelete("books")` trigger), scheduled runs + validation harness (sampled deep-diffs,
+   count checks, lag alerts).
+2. **Auth**: enable Firebase third-party auth; `role` claim backfill; RLS policies for
+   uploader/moderator writes (email-verified, username==email semantics from
+   `bloomFirebaseAuthAdapter`).
+3. **Read-path flips**: edge functions (opds, stats, books-GET) switch from
+   `BloomParseServer.ts` to Postgres, one at a time behind env switches, staging first;
+   byte-diff outputs against the Parse-sourced versions.
+4. **blorg**: contract tests green on both backends, production flip of reads+writes
+   together at cutover.
+5. **Write cutover** (joint with F8 books/upload): freeze Parse writes → final sync →
+   flip → 1–2 week read-only rollback window → decommission Parse + MongoDB Atlas.
+
+Key risks: search-semantics parity (Mongo `$text` vs Postgres), the untested `beforeSave`
+behaviors, Parse Pointer-shaped JSON in frozen API contracts, and Firebase claim coverage.

packages/sync-tool/package.json

@@ -0,0 +1,13 @@
+{
+  "name": "@bloom/sync-tool",
+  "version": "0.1.0",
+  "private": true,
+  "type": "module",
+  "description": "Parse -> Supabase data import. v0: imports a diverse sample of production books into a local Supabase for the 'Local Supabase + Blorg' milestone. Grows into the full incremental sync tool.",
+  "scripts": {
+    "import-sample": "node src/import-sample.mjs"
+  },
+  "dependencies": {
+    "@supabase/supabase-js": "2.110.2"
+  }
+}