diff --git a/.changeset/cli-anonymous-telemetry.md b/.changeset/cli-anonymous-telemetry.md new file mode 100644 index 00000000..4d3befc6 --- /dev/null +++ b/.changeset/cli-anonymous-telemetry.md @@ -0,0 +1,28 @@ +--- +'stash': minor +--- + +Add anonymous, opt-out usage analytics to the `stash` CLI, plus a +`stash telemetry [status|enable|disable]` command to manage it. + +Only coarse events are collected — command name, CLI version, OS/arch, Node +version, success/failure, duration, and a coarse caller class (e.g. +`claude-code`, `cursor`, `interactive`) derived from environment markers so we +can gauge agent- vs human-driven usage. Events carry a random install +identifier (a locally generated UUID, not derived from any machine or user +attribute) used only to de-duplicate events in aggregate. Plaintext, schema, +table/column names, +connection strings, argument values, and any session/trace identifier are never +collected — enforced by a property-key allowlist at the emitter boundary plus +closed-vocabulary coercion of every argv- or error-derived value (unrecognised +commands, subcommands, and error class names all collapse to ``). A +one-time notice is shown on first run, and nothing is sent on that run. + +Telemetry is off by default in CI and can be disabled with `DO_NOT_TRACK=1` +(the cross-tool standard), `STASH_TELEMETRY_DISABLED=1`, or +`stash telemetry disable` (persisted to `~/.cipherstash/telemetry.json`). + +Events are sent via a first-party proxy and never block or slow the CLI. The +feature ships dormant — no events are sent until a PostHog project key is +embedded at release. Updates the `stash-cli` skill to document the command and +opt-out controls. diff --git a/.changeset/drizzle-legacy-readme-pointer.md b/.changeset/drizzle-legacy-readme-pointer.md new file mode 100644 index 00000000..50cd60eb --- /dev/null +++ b/.changeset/drizzle-legacy-readme-pointer.md @@ -0,0 +1,7 @@ +--- +'@cipherstash/drizzle': patch +--- + +Docs: the README's "for new projects" pointer named the removed +`@cipherstash/stack/drizzle` subpath; it now points at the separate +`@cipherstash/stack-drizzle` package (EQL v3 on its `/v3` subpath). diff --git a/.changeset/eql-v3-json-selector.md b/.changeset/eql-v3-json-selector.md new file mode 100644 index 00000000..86646264 --- /dev/null +++ b/.changeset/eql-v3-json-selector.md @@ -0,0 +1,26 @@ +--- +'@cipherstash/stack-drizzle': minor +'stash': patch +--- + +Add EQL v3 JSON **selector-with-constraint** querying to the Drizzle integration +(#623). `ops.selector(col, '$.path')` returns comparison methods bound to a +JSONPath into a `types.Json` column — `eq`/`ne`/`gt`/`gte`/`lt`/`lte` — emitting +`col->'' ` over the encrypted document. Its unique power +over `contains` is **ordering at a path** (`col->'$.age' > 21`), which +containment cannot express. + +Complements the existing `contains` (JSONB `@>`) containment operator. Core +`@cipherstash/stack` needs no change — the selector hash and comparison entry are +produced by `encryptQuery`/`encrypt` on the existing `types.Json` surface. v1 +supports dot-notation object paths; array-index/wildcard paths are rejected with +a clear error. The Supabase adapter is tracked separately. + +The right-hand comparison operand is currently a storage-encrypted needle (its +ste_vec entry carries the ordering term), pending a ciphertext-free ordering +query needle from protect-ffi (cipherstash/protectjs-ffi#137); until then the +value's ciphertext appears in the WHERE clause. + +The bundled `stash-encryption` and `stash-drizzle` skills document the new +`ops.selector(...)` surface (they previously said JSONPath selector queries were +not yet implemented). diff --git a/.changeset/eql-v3-sole-docs.md b/.changeset/eql-v3-sole-docs.md new file mode 100644 index 00000000..1e1a7202 --- /dev/null +++ b/.changeset/eql-v3-sole-docs.md @@ -0,0 +1,16 @@ +--- +'stash': patch +'@cipherstash/stack': patch +--- + +Docs: EQL v3 is now the sole documented approach. The `stash-encryption`, +`stash-drizzle`, and `stash-supabase` skills and the `@cipherstash/stack` +README teach only the v3 typed surface (`EncryptionV3`, `types.*` concrete +domains, `@cipherstash/stack-drizzle/v3`, `encryptedSupabaseV3`); EQL v2 +shrinks to one short Legacy section per document. Two explicit exceptions are +called out: DynamoDB still requires the v2 schema surface (#657), and the +encrypt rollout tooling (`stash encrypt backfill`/`cutover`, +`@cipherstash/migrate`) currently targets v2 columns (#648) — its guidance is +kept under a version callout. Also corrects the legacy `@cipherstash/drizzle` +README's pointer to the removed `@cipherstash/stack/drizzle` subpath (now the +separate `@cipherstash/stack-drizzle` package). diff --git a/.changeset/pre.json b/.changeset/pre.json new file mode 100644 index 00000000..40e6d09f --- /dev/null +++ b/.changeset/pre.json @@ -0,0 +1,64 @@ +{ + "mode": "pre", + "tag": "rc", + "initialVersions": { + "@cipherstash/e2e": "0.0.2", + "@cipherstash/basic-example": "1.2.13", + "@cipherstash/prisma-next-example": "0.0.5", + "@cipherstash/supabase-worker-example": "0.0.0", + "@cipherstash/bench": "0.0.4", + "stash": "0.17.1", + "@cipherstash/drizzle": "3.0.3", + "@cipherstash/migrate": "0.2.0", + "@cipherstash/nextjs": "4.1.1", + "@cipherstash/prisma-next": "0.3.2", + "@cipherstash/protect": "12.0.1", + "@cipherstash/protect-dynamodb": "12.0.1", + "@cipherstash/schema": "3.0.1", + "@cipherstash/stack": "0.19.0", + "@cipherstash/stack-drizzle": "0.0.0", + "@cipherstash/stack-supabase": "0.0.0", + "@cipherstash/test-kit": "0.0.0", + "@cipherstash/wizard": "0.4.0" + }, + "changesets": [ + "adapter-package-split", + "adapter-split-skills", + "cli-eql-v3-single-bundle", + "eql-v3-adapter-type-robustness", + "eql-v3-bigint-domains", + "eql-v3-bundle-from-package", + "eql-v3-cli-install", + "eql-v3-drizzle-encrypt-query", + "eql-v3-drizzle-fail-open-guards", + "eql-v3-drizzle", + "eql-v3-ffi-0-28-concrete-types", + "eql-v3-ga-rebaseline", + "eql-v3-json-skills", + "eql-v3-json", + "eql-v3-public-domains", + "eql-v3-rename-contains-to-matches", + "eql-v3-supabase-adapter", + "eql-v3-text-search", + "eql-v3-typed-client", + "eql-v3-typed-schema", + "eql-v3-wasm-inline", + "prisma-next-0-14", + "remove-secrets-leftovers", + "rename-db-install-to-eql-install", + "schema-stevec-standard-pin", + "stack-1-0-0-rc", + "stack-adapter-kit", + "stash-cli-eql-v3-default", + "stash-cli-skill-refresh", + "stash-drizzle-skill-encrypt-query", + "stash-skills-contains-to-matches", + "stash-supabase-contains-substrings", + "supabase-encryption-error", + "supabase-in-list-operands", + "supabase-is-null-operands", + "supabase-or-string-parser", + "supabase-v3-order-by-ope-term", + "wizard-allow-env-templates" + ] +} diff --git a/.changeset/stack-1-0-0-rc.md b/.changeset/stack-1-0-0-rc.md new file mode 100644 index 00000000..555ae5c8 --- /dev/null +++ b/.changeset/stack-1-0-0-rc.md @@ -0,0 +1,13 @@ +--- +'@cipherstash/stack': major +'@cipherstash/stack-drizzle': major +'@cipherstash/stack-supabase': major +'stash': major +--- + +CipherStash Stack 1.0 (release candidate). + +This is the first 1.0-line release of `@cipherstash/stack`, the first published +release of the split-out EQL v3 adapters `@cipherstash/stack-drizzle` and +`@cipherstash/stack-supabase`, and moves the `stash` CLI to 1.0 alongside them. +These four packages now version together as the Stack 1.0 family. diff --git a/.changeset/supabase-v3-json-querying.md b/.changeset/supabase-v3-json-querying.md new file mode 100644 index 00000000..93a8e046 --- /dev/null +++ b/.changeset/supabase-v3-json-querying.md @@ -0,0 +1,28 @@ +--- +'@cipherstash/stack-supabase': minor +'@cipherstash/stack': minor +'stash': patch +--- + +Encrypted-JSON querying on the v3 Supabase surface (#650). A `types.Json` +column now supports exact encrypted containment — `contains(col, subDocument)` +(ste_vec `@>` via PostgREST `cs`, with the sub-document storage-encrypted +against the column) — and JSONPath selector predicates: `selectorEq(col, path, +value)` and `selectorNe(col, path, value)` (dot-notation paths; `ne` includes +rows where the path is absent, mirroring the Drizzle selector's semantics). +Raw `.filter(col, 'cs', subDocument)` and `not(col, 'contains', …)` route +through the same encrypted path. Selector ordering is not expressible over +PostgREST yet (needs an EQL-bundle overload — see +cipherstash/encrypt-query-language#407); the Drizzle integration's +`ops.selector()` covers ordering today. + +In core, `QueryTypesForColumn` gains the `searchableJson` arm (a `types.Json` +column no longer resolves to `never`, so typed adapter key sets can include +it), and the JSONPath selector-path helpers the Drizzle adapter introduced in +#651 moved to `@cipherstash/stack/adapter-kit` so both adapters share one +validation surface (`@cipherstash/stack-drizzle` re-exports them unchanged). + +The bundled `stash-supabase` and `stash-encryption` skills are updated to +document the new querying surface (including the array-leaf and SQL-NULL +semantics, and the operand-exposure caveat) — skills ship inside the `stash` +tarball, hence the patch. diff --git a/.changeset/wizard-analytics-privacy.md b/.changeset/wizard-analytics-privacy.md new file mode 100644 index 00000000..c4799b7d --- /dev/null +++ b/.changeset/wizard-analytics-privacy.md @@ -0,0 +1,11 @@ +--- +'@cipherstash/wizard': patch +--- + +Align the wizard's analytics with the `stash` CLI's telemetry privacy contract. +The wizard now honors `DO_NOT_TRACK`, `STASH_TELEMETRY_DISABLED`, and CI +auto-detection; uses a random per-session identifier instead of one derived +from username@hostname; disables IP→geo resolution; and reports error events as +fixed labels / error class names instead of raw messages (which could embed +schema names or connection details). Analytics remain dormant unless a PostHog +key is configured at build time. diff --git a/.github/workflows/integration-drizzle.yml b/.github/workflows/integration-drizzle.yml index 9e5454a3..0b2a9b83 100644 --- a/.github/workflows/integration-drizzle.yml +++ b/.github/workflows/integration-drizzle.yml @@ -58,6 +58,18 @@ jobs: integration: name: Drizzle v3 integration (db=${{ matrix.db }}) runs-on: blacksmith-4vcpu-ubuntu-2404 + # Serialize every job that brings up the SAME docker-compose stack, so no two + # bind the same fixed host ports on a shared runner at once (the "address + # already in use" flake). Keyed by the compose variant, NOT the ref (two PRs + # contend for a host port just as much as two pushes to one PR), so it + # serializes across refs. The `supabase` leg uses the SAME supabase compose + # (55430 / 55433) as the Supabase workflow and shares its key + # (`integration-live-db-supabase`), so the two workflows queue rather than + # collide; the `postgres` leg (55432) has its own key and still runs in + # parallel with them. + concurrency: + group: integration-live-db-${{ matrix.db }} + cancel-in-progress: false # Fork PRs have no secrets. Skip cleanly rather than fail on something the # contributor cannot fix — `tests.yml` still gives them a green signal. if: ${{ github.event_name == 'push' || github.event.pull_request.head.repo.full_name == github.repository }} @@ -134,6 +146,14 @@ jobs: client-key: ${{ secrets.CS_CLIENT_KEY }} client-access-key: ${{ secrets.CS_CLIENT_ACCESS_KEY }} + # Belt-and-braces for the concurrency guard: a run that was hard-killed + # (runner crash / forced cancel) can skip its `down` and leak a container + # holding the host port onto a REUSED runner. Tear any prior stack down + # before starting a fresh one. `|| true` so a clean runner (nothing to + # remove) is not an error. + - name: Clear any leaked containers from a prior run + run: docker compose -f local/docker-compose.${{ matrix.db }}.yml down -v --remove-orphans || true + - name: Start ${{ matrix.db }} run: docker compose -f local/docker-compose.${{ matrix.db }}.yml up -d --wait diff --git a/.github/workflows/integration-supabase.yml b/.github/workflows/integration-supabase.yml index 953e19ad..64252ff5 100644 --- a/.github/workflows/integration-supabase.yml +++ b/.github/workflows/integration-supabase.yml @@ -50,6 +50,18 @@ jobs: integration: name: Supabase v3 integration (db=${{ matrix.db }}) runs-on: blacksmith-4vcpu-ubuntu-2404 + # Serialize every job that brings up the SAME docker-compose stack, so no two + # bind the same fixed host ports (55430 / 55433) on a shared runner at once — + # the "address already in use" flake. The group is keyed by the compose + # variant, NOT the ref: two different PRs contend for the same host port just + # as much as two pushes to one PR, so it must serialize across refs. It is the + # SAME key the Drizzle workflow's matching leg uses (`integration-live-db-`), + # so those two workflows also queue against each other rather than colliding. + # `cancel-in-progress: false` queues (a live-DB job that shares one ZeroKMS + # workspace should finish, not be interrupted mid-run). + concurrency: + group: integration-live-db-${{ matrix.db }} + cancel-in-progress: false # Fork PRs have no secrets. Skip cleanly rather than fail loudly on something # the contributor cannot fix — `tests.yml` still gives them a green signal. if: ${{ github.event_name == 'push' || github.event.pull_request.head.repo.full_name == github.repository }} @@ -106,6 +118,14 @@ jobs: client-key: ${{ secrets.CS_CLIENT_KEY }} client-access-key: ${{ secrets.CS_CLIENT_ACCESS_KEY }} + # Belt-and-braces for the concurrency guard: a run that was hard-killed + # (runner crash / forced cancel) can skip its `down` and leak a container + # holding the host port onto a REUSED runner. Tear any prior stack down + # before starting a fresh one. `|| true` so a clean runner (nothing to + # remove) is not an error. + - name: Clear any leaked containers from a prior run + run: docker compose -f local/docker-compose.${{ matrix.db }}.yml down -v --remove-orphans || true + - name: Start ${{ matrix.db }} run: docker compose -f local/docker-compose.${{ matrix.db }}.yml up -d --wait diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 3f0e1987..947c0dc6 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -69,3 +69,10 @@ jobs: # changesets/action writes a token .npmrc that shadows OIDC and # every publish fails with E404 (see npm/cli#8976). GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + # Embeds the CLI's PostHog project key at build time (see + # packages/cli/tsup.config.ts). A repo *variable*, not a secret: the + # key is public and write-only (like a web SDK key). Unset until GA, so + # every release before it is flipped ships telemetry-dormant. This is + # the single go-live switch — set it with: + # gh variable set STASH_POSTHOG_KEY --repo cipherstash/stack --body '' + STASH_POSTHOG_KEY: ${{ vars.STASH_POSTHOG_KEY }} diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml index 0a8c6143..6bbc3cf3 100644 --- a/.github/workflows/tests.yml +++ b/.github/workflows/tests.yml @@ -116,6 +116,17 @@ jobs: - name: Type tests (test-kit — enforces v3 domain coverage) run: pnpm --filter @cipherstash/test-kit run test:types + # The adapter packages carry their own `.test-d.ts` type-contract guards + # (the M1 client-surface / #622 erasure guards, the Supabase key-set + # gating). They moved here from `@cipherstash/stack` in the #627 split but + # were never wired into CI, so type-level regressions in them went + # undetected — run them explicitly. + - name: Type tests (stack-drizzle) + run: pnpm --filter @cipherstash/stack-drizzle run test:types + + - name: Type tests (stack-supabase) + run: pnpm --filter @cipherstash/stack-supabase run test:types + - name: Lint — no hardcoded package-manager runners run: pnpm run lint:runners diff --git a/AGENTS.md b/AGENTS.md index 0b0b4a02..fd6e9cda 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -147,8 +147,8 @@ Three rules to remember when editing CI or pnpm config: ## Key Concepts and APIs -- **Initialization**: `Encryption({ schemas })` returns an initialized `EncryptionClient`. Provide at least one `encryptedTable`. -- **Schema**: Define tables/columns with `encryptedTable` and `encryptedColumn` from `@cipherstash/stack/schema`. Add `.freeTextSearch().equality().orderAndRange()` to enable searchable encryption on PostgreSQL. Use `.searchableJson()` for encrypted JSONB queries. Use `encryptedField` for nested object encryption (DynamoDB). +- **Initialization**: `EncryptionV3({ schemas })` returns the typed EQL v3 client. (`Encryption({ schemas })` is the legacy v2 client.) Provide at least one `encryptedTable`. +- **Schema (EQL v3, the documented approach)**: Define tables/columns with `encryptedTable` and the `types.*` concrete-domain factories from `@cipherstash/stack/eql/v3` (`types.TextSearch`, `types.IntegerOrd`, `types.Json`, …) — each domain's query capabilities are fixed by its type; there are no chainable capability tuners. Build the client with `EncryptionV3` from `@cipherstash/stack/v3`. (Legacy EQL v2 — `Encryption` + `encryptedColumn(...).equality().freeTextSearch().orderAndRange().searchableJson()` from `@cipherstash/stack/schema` — remains for existing deployments and is what DynamoDB (`encryptedField`) still requires; see #657.) - **Operations** (all return Result-like objects and support chaining `.withLockContext(lockContext)` and `.audit()` when applicable): - `encrypt(plaintext, { table, column })` - `decrypt(encryptedPayload)` diff --git a/e2e/CHANGELOG.md b/e2e/CHANGELOG.md index 2ab6bf48..c213cf7b 100644 --- a/e2e/CHANGELOG.md +++ b/e2e/CHANGELOG.md @@ -1,5 +1,49 @@ # @cipherstash/e2e +## 0.0.3-rc.0 + +### Patch Changes + +- Updated dependencies [31ca318] +- Updated dependencies [31ca318] +- Updated dependencies [229ce59] +- Updated dependencies [c4787c0] +- Updated dependencies [66a0e02] +- Updated dependencies [cfd46ee] +- Updated dependencies [0b9b192] +- Updated dependencies [7eba32d] +- Updated dependencies [0ebf57e] +- Updated dependencies [d73a03c] +- Updated dependencies [89b903f] +- Updated dependencies [229ce59] +- Updated dependencies [82f2e69] +- Updated dependencies [50c0a9c] +- Updated dependencies [63ca540] +- Updated dependencies [5d23e80] +- Updated dependencies [1aa9a11] +- Updated dependencies [af2d04e] +- Updated dependencies [b8a3d20] +- Updated dependencies [a0f3b2c] +- Updated dependencies [f23f952] +- Updated dependencies [0b9b192] +- Updated dependencies [7c7dbca] +- Updated dependencies [5411a13] +- Updated dependencies [e25eb22] +- Updated dependencies [1a9d190] +- Updated dependencies [161f17b] +- Updated dependencies [e40c3da] +- Updated dependencies [58d7439] +- Updated dependencies [99f8b0a] +- Updated dependencies [fd33aad] +- Updated dependencies [8cd485d] +- Updated dependencies [9b65ae8] +- Updated dependencies [9c673bb] + - @cipherstash/stack@1.0.0-rc.0 + - stash@1.0.0-rc.0 + - @cipherstash/wizard@0.5.0-rc.0 + - @cipherstash/drizzle@3.0.4-rc.0 + - @cipherstash/protect@12.0.2-rc.0 + ## 0.0.2 ### Patch Changes diff --git a/e2e/package.json b/e2e/package.json index 7ca2f69d..800cec15 100644 --- a/e2e/package.json +++ b/e2e/package.json @@ -1,6 +1,6 @@ { "name": "@cipherstash/e2e", - "version": "0.0.2", + "version": "0.0.3-rc.0", "private": true, "description": "End-to-end tests that exercise built CipherStash binaries and cross-package behaviour.", "type": "module", diff --git a/examples/basic/CHANGELOG.md b/examples/basic/CHANGELOG.md index b35e63dd..e53d0faa 100644 --- a/examples/basic/CHANGELOG.md +++ b/examples/basic/CHANGELOG.md @@ -1,5 +1,38 @@ # @cipherstash/basic-example +## 1.2.14-rc.0 + +### Patch Changes + +- Updated dependencies [31ca318] +- Updated dependencies [c4787c0] +- Updated dependencies [66a0e02] +- Updated dependencies [cfd46ee] +- Updated dependencies [7eba32d] +- Updated dependencies [0ebf57e] +- Updated dependencies [d73a03c] +- Updated dependencies [89b903f] +- Updated dependencies [229ce59] +- Updated dependencies [50c0a9c] +- Updated dependencies [63ca540] +- Updated dependencies [e40c3da] +- Updated dependencies [5d23e80] +- Updated dependencies [1aa9a11] +- Updated dependencies [af2d04e] +- Updated dependencies [b8a3d20] +- Updated dependencies [a0f3b2c] +- Updated dependencies [f23f952] +- Updated dependencies [7c7dbca] +- Updated dependencies [5411a13] +- Updated dependencies [2fd4985] +- Updated dependencies [99f8b0a] +- Updated dependencies [fd33aad] +- Updated dependencies [8cd485d] +- Updated dependencies [9b65ae8] + - @cipherstash/stack@1.0.0-rc.0 + - @cipherstash/stack-drizzle@1.0.0-rc.0 + - @cipherstash/stack-supabase@1.0.0-rc.0 + ## 1.2.13 ### Patch Changes diff --git a/examples/basic/package.json b/examples/basic/package.json index 8a2f8e24..7bbf20cf 100644 --- a/examples/basic/package.json +++ b/examples/basic/package.json @@ -1,7 +1,7 @@ { "name": "@cipherstash/basic-example", "private": true, - "version": "1.2.13", + "version": "1.2.14-rc.0", "type": "module", "scripts": { "start": "tsx index.ts" diff --git a/examples/prisma/CHANGELOG.md b/examples/prisma/CHANGELOG.md index f815e347..79cb7ad6 100644 --- a/examples/prisma/CHANGELOG.md +++ b/examples/prisma/CHANGELOG.md @@ -1,5 +1,36 @@ # @cipherstash/prisma-next-example +## 0.0.6-rc.0 + +### Patch Changes + +- Updated dependencies [31ca318] +- Updated dependencies [c4787c0] +- Updated dependencies [66a0e02] +- Updated dependencies [cfd46ee] +- Updated dependencies [7eba32d] +- Updated dependencies [0ebf57e] +- Updated dependencies [d73a03c] +- Updated dependencies [89b903f] +- Updated dependencies [229ce59] +- Updated dependencies [50c0a9c] +- Updated dependencies [63ca540] +- Updated dependencies [5d23e80] +- Updated dependencies [1aa9a11] +- Updated dependencies [af2d04e] +- Updated dependencies [b8a3d20] +- Updated dependencies [a0f3b2c] +- Updated dependencies [d6d23be] +- Updated dependencies [f23f952] +- Updated dependencies [7c7dbca] +- Updated dependencies [5411a13] +- Updated dependencies [99f8b0a] +- Updated dependencies [fd33aad] +- Updated dependencies [8cd485d] +- Updated dependencies [9b65ae8] + - @cipherstash/stack@1.0.0-rc.0 + - @cipherstash/prisma-next@0.4.0-rc.0 + ## 0.0.5 ### Patch Changes diff --git a/examples/prisma/package.json b/examples/prisma/package.json index 1fbe8ebb..e8e5169f 100644 --- a/examples/prisma/package.json +++ b/examples/prisma/package.json @@ -1,7 +1,7 @@ { "name": "@cipherstash/prisma-next-example", "private": true, - "version": "0.0.5", + "version": "0.0.6-rc.0", "description": "End-to-end example of @cipherstash/prisma-next: searchable application-layer encryption for Postgres with Prisma Next, using @cipherstash/stack as the SDK.", "type": "module", "scripts": { diff --git a/packages/bench/CHANGELOG.md b/packages/bench/CHANGELOG.md index 4652e87f..283dabc7 100644 --- a/packages/bench/CHANGELOG.md +++ b/packages/bench/CHANGELOG.md @@ -1,5 +1,36 @@ # @cipherstash/bench +## 0.0.5-rc.0 + +### Patch Changes + +- Updated dependencies [31ca318] +- Updated dependencies [c4787c0] +- Updated dependencies [66a0e02] +- Updated dependencies [cfd46ee] +- Updated dependencies [7eba32d] +- Updated dependencies [0ebf57e] +- Updated dependencies [d73a03c] +- Updated dependencies [89b903f] +- Updated dependencies [229ce59] +- Updated dependencies [50c0a9c] +- Updated dependencies [63ca540] +- Updated dependencies [e40c3da] +- Updated dependencies [5d23e80] +- Updated dependencies [1aa9a11] +- Updated dependencies [af2d04e] +- Updated dependencies [b8a3d20] +- Updated dependencies [a0f3b2c] +- Updated dependencies [f23f952] +- Updated dependencies [7c7dbca] +- Updated dependencies [5411a13] +- Updated dependencies [99f8b0a] +- Updated dependencies [fd33aad] +- Updated dependencies [8cd485d] +- Updated dependencies [9b65ae8] + - @cipherstash/stack@1.0.0-rc.0 + - @cipherstash/stack-drizzle@1.0.0-rc.0 + ## 0.0.4 ### Patch Changes diff --git a/packages/bench/package.json b/packages/bench/package.json index 62318993..ada4786a 100644 --- a/packages/bench/package.json +++ b/packages/bench/package.json @@ -1,6 +1,6 @@ { "name": "@cipherstash/bench", - "version": "0.0.4", + "version": "0.0.5-rc.0", "private": true, "description": "Performance / index-engagement benchmarks for stack integrations (Drizzle, encryptedSupabase, Prisma).", "type": "module", diff --git a/packages/cli/AGENTS.md b/packages/cli/AGENTS.md index c3ba4d68..73795c3b 100644 --- a/packages/cli/AGENTS.md +++ b/packages/cli/AGENTS.md @@ -81,11 +81,20 @@ exercise the same code paths. test actually asserts on the string — premature extraction is worse than copy-paste here. For literals tests don't touch (e.g. command names like `init`, `eql install`), keep them inline. -- **Telemetry.** The CLI source no longer imports `posthog-node` (analytics - moved to `packages/wizard`). The dep is still listed in `package.json` - and should be removed in a follow-up. If you re-introduce telemetry to - the CLI, gate construction on an env var (the wizard's - `getClient()` pattern) so E2E tests can no-op it. +- **Telemetry.** The CLI has anonymous, opt-out usage analytics in + `src/telemetry/` (posthog-node, loaded lazily only when an event is + actually sent). It ships dormant — a real project key is embedded only by + the release build (`STASH_POSTHOG_KEY` repo variable → tsup define) — and + is force-disabled in every pty e2e child via `STASH_TELEMETRY_DISABLED=1` + in `tests/helpers/pty.ts`. Two contracts to preserve when touching it: + (1) event VALUES are closed vocabularies — `classifyCommand` / + `classifyErrorType` in `src/telemetry/classify-command.ts` must wrap + anything argv- or error-derived before emit; (2) never intercept + `process.exit` with a thrown signal — @clack/core exits from keypress + handlers and several commands have broad catches, so interception breaks + cancel flows (tried and reverted; see `src/cli/exit.ts`). Commands that + terminate via deep `process.exit()` are simply not tracked; cooperative + exits use `throw new CliExit(code)` from verified-unwindable sites only. ## Plan and rationale diff --git a/packages/cli/CHANGELOG.md b/packages/cli/CHANGELOG.md index c676c613..782edb2e 100644 --- a/packages/cli/CHANGELOG.md +++ b/packages/cli/CHANGELOG.md @@ -1,5 +1,159 @@ # @cipherstash/cli +## 1.0.0-rc.0 + +### Major Changes + +- 7c7dbca: CipherStash Stack 1.0 (release candidate). + + This is the first 1.0-line release of `@cipherstash/stack`, the first published + release of the split-out EQL v3 adapters `@cipherstash/stack-drizzle` and + `@cipherstash/stack-supabase`, and moves the `stash` CLI to 1.0 alongside them. + These four packages now version together as the Stack 1.0 family. + +### Minor Changes + +- 229ce59: `stash eql install --eql-version 3` now installs the eql-3.0.0 GA bundle, + vendored from the pinned `@cipherstash/eql` package (sha256-verified). + + Since eql-3.0.0 one artifact installs everywhere: the operator-class + statements self-skip when the role lacks superuser (managed Postgres, + Supabase) and the bundle disables the ORE-backed encrypted domains it cannot + support. The separate v3 Supabase bundle variant is gone — `--supabase` and + `--exclude-operator-family` no longer select a different v3 file (the role + GRANTs for `eql_v3` / `eql_v3_internal` still apply with `--supabase`). + + The bundled skills are also refreshed for the eql-3.0.0 naming convention + (`public.eql_v3_` column domains) and the EQL v3 typed-schema surface. + +- 0b9b192: Add an EQL v3 install path to `stash eql install` via a new `--eql-version <2|3>` + flag (default `2`). v3 installs the native concrete-domain schema (`public.*` + type domains, `eql_v3` operators, `eql_v3_internal` constructors) from bundles + vendored into `packages/cli/src/sql` by `scripts/build-eql-v3-sql.mjs` (full + bundle + a Supabase variant with the two superuser-only operator-class chunks + stripped). v3 currently supports the direct install path only — + `--drizzle`/`--migration`/`--migrations-dir`/`--latest` are rejected — and the + installer keys `isInstalled`/version checks and Supabase grants to the `eql_v3` + schema. +- 0b9b192: Rename `stash db install` to `stash eql install`. The command scaffolds + `stash.config.ts` and installs the EQL extensions, so it now lives under a + dedicated `eql` command group. `stash db install` keeps working as a + deprecated alias that prints a warning pointing at the new name. All help + text, hints, generated migration headers, and wizard steps now reference + `stash eql install`. +- e25eb22: Default EQL to v3 and stop the CLI recommending `stash db push` (#585). + + - **EQL v3 is now the default.** `stash eql install` and `stash eql upgrade` target v3 (the native `eql_v3.*` domain schema) without `--eql-version 3`. The v2-only paths — `--drizzle`, `--migration`, `--migrations-dir`, and `--latest` — now require an explicit `--eql-version 2` and error with clear guidance otherwise (v3 installs via the direct path only). `stash init` pins v2 automatically when it drives the Drizzle migration flow. **Note:** for a Supabase project, `stash init` now runs a v3 direct install rather than offering the v2 migration-file flow; run `stash eql install --supabase --migration --eql-version 2` if you want a checked-in migration file. + - **`stash db push` is no longer recommended in CLI output.** `db push` writes the `public.eql_v2_configuration` table, which is a v2 + CipherStash Proxy artifact — EQL v3 has no configuration table (config lives in each column's `eql_v3.*` type) and nothing in the v3 stack reads it. The push recommendations are removed from `eql status`, the help banner, and the init/plan/cutover guidance. `db push` (and `db activate`) remain available for EQL v2 + Proxy users; they're now labelled as such. + - **`eql status` is v3-aware.** On a v3-only database it reports that encrypt config lives in the column types instead of hitting a "table not found" dead-end that told users to run `db push` (which neither creates that table nor applies to v3). + - **`stash db push` guards a v3-only database** with a clear "not needed under EQL v3" message instead of a raw `relation "public.eql_v2_configuration" does not exist` error. + +### Patch Changes + +- 31ca318: Update the bundled `stash-drizzle`, `stash-supabase`, and `stash-encryption` agent + skills (and the stack README / Supabase reference doc) for the adapter package + split: the Drizzle and Supabase integrations import from `@cipherstash/stack-drizzle` + (+ `/v3`) and `@cipherstash/stack-supabase` respectively, installed alongside + `@cipherstash/stack`, rather than from `@cipherstash/stack/{drizzle,supabase,eql/v3/drizzle}` + subpaths. Skills ship inside the `stash` tarball, so the stale import paths would + otherwise become wrong guidance in a user's project. +- 82f2e69: Document EQL v3 JSON columns in the bundled skills: `types.Json` in the + `stash-encryption` typed-schema catalog (capability suffix, family, and an + encrypted-JSONB query section), and `contains(col, subObject)` JSON containment + on the v3 Drizzle operators in `stash-drizzle`. +- f23f952: Remove the leftovers from the secrets removal (`1929c8fe`), which deleted + `packages/stack/src/secrets/` but left its export, build entry, skill, and docs + behind. Secrets tooling is not ready; nothing here was functional. + + - **Drop the dead `@cipherstash/stack/secrets` subpath export.** It pointed at + `./dist/secrets/index.js`, which has no source and is not in the tarball, so + `import '@cipherstash/stack/secrets'` has been throwing `ERR_MODULE_NOT_FOUND` + for every consumer since the source was removed. Also drops the dangling + `src/secrets/index.ts` entry from `tsup.config.ts`. Removing an export that + cannot resolve breaks nothing. + - **Remove the `stash-secrets` agent skill** and its references in `AGENTS.md` + and the init setup-prompt skill index. It was never installed by `stash init` + (it is absent from `SKILL_MAP`), so no user project ever received it. + - **Remove the secrets documentation** from both published READMEs: the + `Secrets` class API and the `npx stash secrets` command reference in + `@cipherstash/stack`, and the `npx stash secrets` section in `stash`. The CLI + command does not exist — `stash secrets` returns `Unknown command`. + +- 1a9d190: Refresh the bundled `stash-cli` agent skill and the CLI README against the current + command surface. The skills directory ships inside the `stash` tarball and is copied + into the user's `.claude/skills/` / `.codex/skills/` (or inlined into `AGENTS.md`) at + handoff time, so a stale skill becomes stale guidance in the user's project. + + - **New `Start here` and `Authentication` sections.** Setup is driven through the CLI: + agents read `stash manifest --json` first, then trigger `stash auth login --json` and + surface the verification URL for a human to approve, then run `stash init`. Authenticating + before `init` matters — `init`'s auth step is interactive and would otherwise try to open + a browser on the agent's host. + - **New `Never read these` invariant**, mirrored into the `AGENTS.md` doctrine: agents must + never read `~/.cipherstash/secretkey.json`, `~/.cipherstash/auth.json`, anything under + `~/.cipherstash/workspaces/`, or `.env*`. The wizard already blocks these paths in code; + the other handoff targets had no written rule. + - **Documents `manifest`, `doctor`, `wizard`, and `auth regions`**, which the skill omitted + entirely, plus the non-interactive interface (per-command escape hatches, exit codes, the + `DATABASE_URL` resolution order, the `auth login --json` NDJSON event contract). + - **Corrects the `db` → `eql` move.** `db install`, `db upgrade`, and `db status` are + deprecated aliases that warn and forward; `db push`, `db activate`, `db validate`, + `db test-connection`, and `db migrate` remain in the `db` group. + - **Scopes `db push` / `db activate` as EQL v2 + CipherStash Proxy only**, in both the skill + and the README's recommended flow. SDK users hold their encryption config in application + code and don't need them. + - Adds the missing `--database-url`, `--eql-version`, `--prisma-next`, `--proxy`/`--no-proxy`, + and `--region` flags; corrects six programmatic API signatures; fixes the README's claim + that `stash init` ends in an agent-handoff menu (that belongs to `stash plan` / `stash impl`); + and marks `stash env` as the non-functional stub it currently is. + +- 161f17b: Correct the `stash-drizzle` skill: `inArray` / `notInArray` now encrypt the whole + list in a single `encryptQuery` batch crossing (the `bulkEncrypt`/concurrency + fallback was removed when v3 query operands moved to `encryptQuery` — #622). The + skill ships inside the `stash` tarball, so this keeps the bundled guidance in step + with the adapter's behaviour. +- e40c3da: Update the `stash-drizzle` and `stash-supabase` skills for the EQL v3 + `contains()` → `matches()` rename (#617): the encrypted free-text operator is now + `matches()` (fuzzy bloom token matching), `contains()` is reserved for exact + containment, and Supabase `like()`/`ilike()` on encrypted columns are documented + as an approximate compatibility shim delegating to `matches()`. Skills ship inside + the `stash` tarball, so they must track the adapter surface. +- 58d7439: Correct the bundled `stash-supabase` agent skill: EQL v3 `contains()` matches + substrings. The skill previously carried the reverse — that `contains()` matched + only exact values because the query's bloom filter appended the whole search term + as an extra token. That was never true: `include_original` is inert in + protect-ffi (the match bloom is trigram-only either way), so any substring of at + least the tokenizer's `token_length` (3 characters) matches, and shorter terms are + rejected rather than silently matching every row. The skills directory ships + inside the `stash` tarball and is copied into the user's `.claude/skills/` / + `.codex/skills/` (or inlined into `AGENTS.md`) at handoff time, so the stale + sentence was shipping wrong guidance into customer repos. +- Updated dependencies [31ca318] +- Updated dependencies [c4787c0] +- Updated dependencies [66a0e02] +- Updated dependencies [cfd46ee] +- Updated dependencies [7eba32d] +- Updated dependencies [0ebf57e] +- Updated dependencies [d73a03c] +- Updated dependencies [89b903f] +- Updated dependencies [229ce59] +- Updated dependencies [50c0a9c] +- Updated dependencies [63ca540] +- Updated dependencies [5d23e80] +- Updated dependencies [1aa9a11] +- Updated dependencies [af2d04e] +- Updated dependencies [b8a3d20] +- Updated dependencies [a0f3b2c] +- Updated dependencies [f23f952] +- Updated dependencies [7c7dbca] +- Updated dependencies [5411a13] +- Updated dependencies [99f8b0a] +- Updated dependencies [fd33aad] +- Updated dependencies [8cd485d] +- Updated dependencies [9b65ae8] + - @cipherstash/stack@1.0.0-rc.0 + - @cipherstash/migrate@1.0.0-rc.0 + ## 0.17.1 ### Patch Changes diff --git a/packages/cli/package.json b/packages/cli/package.json index dc8dd8d9..f68f7b87 100644 --- a/packages/cli/package.json +++ b/packages/cli/package.json @@ -1,6 +1,6 @@ { "name": "stash", - "version": "0.17.1", + "version": "1.0.0-rc.0", "description": "CipherStash CLI — the one stash command for auth, init, encryption schema, database setup, and secrets.", "repository": { "type": "git", @@ -66,7 +66,7 @@ "@cipherstash/auth-win32-x64-msvc": "catalog:repo" }, "peerDependencies": { - "@cipherstash/stack": ">=0.6.0" + "@cipherstash/stack": ">=1.0.0-rc.0" }, "peerDependenciesMeta": { "@cipherstash/stack": { diff --git a/packages/cli/src/bin/main.ts b/packages/cli/src/bin/main.ts index c71794a8..ee7e2ab7 100644 --- a/packages/cli/src/bin/main.ts +++ b/packages/cli/src/bin/main.ts @@ -19,6 +19,7 @@ import { readFileSync } from 'node:fs' import { dirname, join } from 'node:path' import { fileURLToPath } from 'node:url' import * as p from '@clack/prompts' +import { CliExit } from '../cli/exit.js' import { renderCommandHelp } from '../cli/help.js' // Commands that depend on @cipherstash/stack are lazy-loaded in the switch below. import { @@ -31,11 +32,22 @@ import { manifestCommand, planCommand, statusCommand, + telemetryCommand, testConnectionCommand, upgradeCommand, wizardCommand, } from '../commands/index.js' import { messages } from '../messages.js' +import { + classifyCommand, + classifyErrorType, +} from '../telemetry/classify-command.js' +import { + initTelemetry, + maybeShowFirstRunNotice, + shutdownTelemetry, + trackCommand, +} from '../telemetry/index.js' function isModuleNotFound(err: unknown): boolean { return ( @@ -72,7 +84,7 @@ async function requireStack(importFn: () => Promise): Promise { Install it with: ${prodInstallCommand(PM, '@cipherstash/stack')} Or run: ${STASH} init`, ) - process.exit(1) as never + throw new CliExit(1) } throw err } @@ -92,6 +104,7 @@ Commands: wizard AI-guided encryption setup (reads your codebase) doctor Diagnose install problems (native binaries, runtime) manifest Print the structured, versioned command surface (--json for docs/agents) + telemetry Manage anonymous usage analytics (status, enable, disable) eql install Scaffold stash.config.ts (if missing) and install EQL extensions eql upgrade Upgrade EQL extensions to the latest version @@ -228,7 +241,7 @@ async function runEqlCommand( p.log.error(`${messages.eql.unknownSubcommand}: ${sub ?? '(none)'}`) console.log() console.log(HELP) - process.exit(1) + throw new CliExit(1) } } @@ -292,7 +305,7 @@ async function runDbCommand( p.log.error(`${messages.db.unknownSubcommand}: ${sub ?? '(none)'}`) console.log() console.log(HELP) - process.exit(1) + throw new CliExit(1) } } @@ -367,7 +380,7 @@ async function runEncryptCommand( p.log.error(`Unknown encrypt subcommand: ${sub ?? '(none)'}`) console.log() console.log(HELP) - process.exit(1) + throw new CliExit(1) } } @@ -375,7 +388,7 @@ function requireValue(values: Record, key: string): string { const v = values[key] if (!v) { p.log.error(`Missing required --${key} value.`) - process.exit(1) + throw new CliExit(1) } return v } @@ -400,7 +413,7 @@ async function runSchemaCommand( p.log.error(`Unknown schema subcommand: ${sub ?? '(none)'}`) console.log() console.log(HELP) - process.exit(1) + throw new CliExit(1) } } @@ -431,6 +444,58 @@ export async function run() { return } + // Anonymous, opt-out usage analytics. The notice shows once (to stderr) and + // the run that shows it sends nothing; both are no-ops when telemetry is off. + initTelemetry(pkg.version) + maybeShowFirstRunNotice(STASH) + + const startedAt = Date.now() + let success = true + let errorType: string | undefined + let exitCode: number | undefined + + // Outcomes are tracked for commands that RETURN, THROW, or throw CliExit (the + // cooperative exit used by main.ts's own helpers and the outermost cancel + // handlers — see cli/exit.ts). Deep `process.exit()` calls terminate without + // an event by design: intercepting them globally proved unsafe (clack exits + // from keypress handlers; broad catches swallowed the signal). + try { + await dispatch(command, subcommand, commandArgs, flags, values) + } catch (err) { + if (err instanceof CliExit) { + exitCode = err.code + success = err.code === 0 + } else { + success = false + errorType = classifyErrorType(err) + // Rethrow to bootstrap's handler ("Fatal error" + exit 1). The finally + // below still runs — including the awaited flush — before propagation. + throw err + } + } finally { + // Coerce command/subcommand to a known vocabulary before emit so a free-text + // positional (e.g. a `stash wizard ""` description) never leaves. + const safe = classifyCommand(command, subcommand) + trackCommand({ + command: safe.command, + subcommand: safe.subcommand, + success, + durationMs: Date.now() - startedAt, + errorType, + }) + await shutdownTelemetry() + } + + if (exitCode !== undefined) process.exit(exitCode) +} + +async function dispatch( + command: string, + subcommand: string | undefined, + commandArgs: string[], + flags: Record, + values: Record, +) { switch (command) { case 'init': await initCommand(flags, values) @@ -473,6 +538,9 @@ export async function run() { // the native binary is missing. manifestCommand({ json: flags.json, version: pkg.version }) break + case 'telemetry': + await telemetryCommand(subcommand) + break case 'wizard': { // Forward everything after `stash wizard` verbatim. The wizard package // owns its own flag parsing; we don't try to interpret its surface @@ -492,6 +560,6 @@ export async function run() { default: console.error(`${messages.cli.unknownCommand}: ${command}\n`) console.log(HELP) - process.exit(1) + throw new CliExit(1) } } diff --git a/packages/cli/src/cli/exit.ts b/packages/cli/src/cli/exit.ts new file mode 100644 index 00000000..36d1a49b --- /dev/null +++ b/packages/cli/src/cli/exit.ts @@ -0,0 +1,24 @@ +/** + * Cooperative exit for first-party command code: `throw new CliExit(code)` + * instead of `process.exit(code)`. + * + * Thrown from a command, it unwinds to `run()` (bin/main.ts), which records the + * outcome for telemetry, flushes, and then performs the real `process.exit` + * with the carried code. This only works from call stacks that reach `run()`'s + * catch without an intervening broad `catch` — which is why ONLY sites verified + * to unwind cleanly use it (main.ts's own dispatch helpers, the telemetry + * command, and the outermost CancelledError handlers of init/plan/impl). + * + * Deep exits stay `process.exit()` deliberately. An earlier version intercepted + * `process.exit` globally with a thrown signal; the review showed that breaks + * code we don't control — @clack/core calls `process.exit(0)` from a keypress + * handler (no enclosing try ⇒ uncaughtException), and several command-level + * broad catches swallowed the signal, continuing past hard stops. Those + * invocations are simply not tracked; see the telemetry module doc. + */ +export class CliExit extends Error { + constructor(readonly code: number) { + super(`exit ${code}`) + this.name = 'CliExit' + } +} diff --git a/packages/cli/src/cli/registry.ts b/packages/cli/src/cli/registry.ts index c388d5ed..66f527f5 100644 --- a/packages/cli/src/cli/registry.ts +++ b/packages/cli/src/cli/registry.ts @@ -231,6 +231,20 @@ export const registry: CommandGroup[] = [ }, ], }, + { + name: 'telemetry', + summary: 'Manage anonymous usage analytics', + long: [ + 'Manage the anonymous, opt-out usage analytics the CLI collects to', + 'improve the tool. `status` (the default) reports whether telemetry is', + 'on and which setting governs it; `enable` / `disable` write your saved', + 'preference. Telemetry is also disabled by the DO_NOT_TRACK or', + 'STASH_TELEMETRY_DISABLED environment variables and automatically in CI.', + 'No plaintext, schema, table/column names, or connection details are', + 'ever collected. See https://cipherstash.com/docs/reference/cli.', + ].join('\n'), + examples: ['telemetry status', 'telemetry disable', 'telemetry enable'], + }, ], }, { diff --git a/packages/cli/src/commands/impl/index.ts b/packages/cli/src/commands/impl/index.ts index 93eb743c..86177838 100644 --- a/packages/cli/src/commands/impl/index.ts +++ b/packages/cli/src/commands/impl/index.ts @@ -1,6 +1,7 @@ import { existsSync, readFileSync } from 'node:fs' import { resolve } from 'node:path' import * as p from '@clack/prompts' +import { CliExit } from '../../cli/exit.js' import { type AgentEnvironment, detectAgents } from '../init/detect-agents.js' import { effectiveStep, @@ -291,7 +292,9 @@ export async function implCommand( } catch (err) { if (err instanceof CancelledError) { p.cancel('Cancelled.') - process.exit(0) + // Cooperative exit: unwinds to run() so the cancel is tracked and the + // telemetry flush completes before the process exits 0 (see cli/exit.ts). + throw new CliExit(0) } throw err } diff --git a/packages/cli/src/commands/index.ts b/packages/cli/src/commands/index.ts index 9069fe67..914dbfe3 100644 --- a/packages/cli/src/commands/index.ts +++ b/packages/cli/src/commands/index.ts @@ -9,4 +9,5 @@ export { initCommand } from './init/index.js' export { manifestCommand } from './manifest/index.js' export { planCommand } from './plan/index.js' export { statusCommand } from './status/index.js' +export { telemetryCommand } from './telemetry/index.js' export { wizardCommand } from './wizard/index.js' diff --git a/packages/cli/src/commands/init/index.ts b/packages/cli/src/commands/init/index.ts index 1bb19aaa..630ffdc0 100644 --- a/packages/cli/src/commands/init/index.ts +++ b/packages/cli/src/commands/init/index.ts @@ -1,4 +1,5 @@ import * as p from '@clack/prompts' +import { CliExit } from '../../cli/exit.js' import { HANDOFF_CHOICES } from '../impl/steps/how-to-proceed.js' import { planCommand } from '../plan/index.js' import { createBaseProvider } from './providers/base.js' @@ -140,7 +141,9 @@ export async function initCommand( } catch (err) { if (err instanceof CancelledError) { p.cancel('Setup cancelled.') - process.exit(0) + // Cooperative exit: unwinds to run() so the cancel is tracked and the + // telemetry flush completes before the process exits 0 (see cli/exit.ts). + throw new CliExit(0) } throw err } diff --git a/packages/cli/src/commands/plan/index.ts b/packages/cli/src/commands/plan/index.ts index b47ec37e..e780e3a6 100644 --- a/packages/cli/src/commands/plan/index.ts +++ b/packages/cli/src/commands/plan/index.ts @@ -2,6 +2,7 @@ import { existsSync } from 'node:fs' import { resolve } from 'node:path' import { readManifest } from '@cipherstash/migrate' import * as p from '@clack/prompts' +import { CliExit } from '../../cli/exit.js' import { HANDOFF_CHOICES, howToProceedStep, @@ -224,7 +225,9 @@ export async function planCommand( } catch (err) { if (err instanceof CancelledError) { p.cancel('Cancelled.') - process.exit(0) + // Cooperative exit: unwinds to run() so the cancel is tracked and the + // telemetry flush completes before the process exits 0 (see cli/exit.ts). + throw new CliExit(0) } throw err } diff --git a/packages/cli/src/commands/telemetry/index.ts b/packages/cli/src/commands/telemetry/index.ts new file mode 100644 index 00000000..b74d2b80 --- /dev/null +++ b/packages/cli/src/commands/telemetry/index.ts @@ -0,0 +1,65 @@ +import * as p from '@clack/prompts' +import { CliExit } from '../../cli/exit.js' +import { messages } from '../../messages.js' +import { + setTelemetryDisabled, + type TelemetryDisabledReason, + telemetryStatus, +} from '../../telemetry/index.js' + +/** Human-readable explanation for why telemetry is currently off. */ +function reasonText(reason: TelemetryDisabledReason): string { + switch (reason) { + case 'do-not-track': + return 'disabled by the DO_NOT_TRACK environment variable' + case 'stash-disabled': + return 'disabled by the STASH_TELEMETRY_DISABLED environment variable' + case 'ci': + return 'disabled automatically in CI' + case 'config': + return 'disabled (run `stash telemetry enable` to turn it back on)' + case 'unconfigured': + return 'not active in this build' + } +} + +function printStatus(): void { + const status = telemetryStatus() + if (status.enabled) { + p.log.info('Telemetry is enabled. Anonymous usage analytics are collected.') + return + } + p.log.info(`Telemetry is ${reasonText(status.reason)}.`) + // An env override wins over the persisted flag, so flag `enable` won't help. + if (status.reason === 'do-not-track' || status.reason === 'stash-disabled') { + p.log.info( + 'An environment variable is overriding your saved preference; unset it to re-enable.', + ) + } +} + +/** + * `stash telemetry [status|enable|disable]` — manage anonymous CLI analytics. + * `enable`/`disable` write the persisted opt-out flag; `status` (the default) + * reports the current state and which gate governs it. + */ +export async function telemetryCommand(sub: string | undefined): Promise { + switch (sub) { + case undefined: + case 'status': + printStatus() + break + case 'enable': + setTelemetryDisabled(false) + p.log.success(messages.telemetry.enabled) + printStatus() + break + case 'disable': + setTelemetryDisabled(true) + p.log.success(messages.telemetry.disabled) + break + default: + p.log.error(`${messages.telemetry.unknownSubcommand}: ${sub}`) + throw new CliExit(1) + } +} diff --git a/packages/cli/src/config/__tests__/tty.test.ts b/packages/cli/src/config/__tests__/tty.test.ts new file mode 100644 index 00000000..6d7c301b --- /dev/null +++ b/packages/cli/src/config/__tests__/tty.test.ts @@ -0,0 +1,130 @@ +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest' +import { + CALLER_ENV_VARS, + CI_ENV_VARS, + isCiEnv, + isCiEnvBroad, + resolveCaller, +} from '../tty.js' + +describe('isCiEnv vs isCiEnvBroad (the narrow/broad split)', () => { + beforeEach(() => { + for (const name of CI_ENV_VARS) vi.stubEnv(name, '') + }) + afterEach(() => { + vi.unstubAllEnvs() + }) + + it('isCiEnv is NARROW: a provider marker alone must not flip it', () => { + // The contract prompt gating depends on: a developer with JENKINS_URL or + // GITHUB_ACTIONS exported in their shell still gets interactive prompts. + // Re-broadening isCiEnv (e.g. delegating to isCiEnvBroad) breaks that and + // this test is what catches it. + for (const marker of ['GITHUB_ACTIONS', 'GITLAB_CI', 'JENKINS_URL']) { + vi.stubEnv(marker, 'true') + expect(isCiEnv()).toBe(false) + vi.stubEnv(marker, '') + } + }) + + it('isCiEnv reads only a truthy CI', () => { + expect(isCiEnv()).toBe(false) + vi.stubEnv('CI', 'true') + expect(isCiEnv()).toBe(true) + vi.stubEnv('CI', '1') + expect(isCiEnv()).toBe(true) + vi.stubEnv('CI', 'false') + expect(isCiEnv()).toBe(false) + }) + + it('isCiEnvBroad is BROAD: provider markers and CONTINUOUS_INTEGRATION count', () => { + expect(isCiEnvBroad()).toBe(false) + vi.stubEnv('JENKINS_URL', 'https://ci.example.com') + expect(isCiEnvBroad()).toBe(true) + vi.stubEnv('JENKINS_URL', '') + vi.stubEnv('CONTINUOUS_INTEGRATION', 'true') + expect(isCiEnvBroad()).toBe(true) + }) +}) + +describe('resolveCaller', () => { + // This repo's own agent harness sets some caller markers ambiently, so clear + // every consulted var before each case; otherwise the fallback tests would + // pick up e.g. CLAUDECODE from the process running the suite. + beforeEach(() => { + for (const name of CALLER_ENV_VARS) vi.stubEnv(name, '') + }) + afterEach(() => { + vi.unstubAllEnvs() + }) + + /** Force process.stdin.isTTY for the fallback cases; restored after each test. */ + function withStdinTty(isTTY: boolean, run: () => void): void { + const original = Object.getOwnPropertyDescriptor(process.stdin, 'isTTY') + Object.defineProperty(process.stdin, 'isTTY', { + value: isTTY, + configurable: true, + }) + try { + run() + } finally { + if (original) Object.defineProperty(process.stdin, 'isTTY', original) + else delete (process.stdin as { isTTY?: boolean }).isTTY + } + } + + it('detects Claude Code via CLAUDECODE', () => { + vi.stubEnv('CLAUDECODE', '1') + expect(resolveCaller()).toBe('claude-code') + }) + + it('detects Claude Code via the CLAUDE_CODE_ENTRYPOINT backup marker', () => { + vi.stubEnv('CLAUDE_CODE_ENTRYPOINT', 'cli') + expect(resolveCaller()).toBe('claude-code') + }) + + it('detects Cursor via CURSOR_TRACE_ID', () => { + vi.stubEnv('CURSOR_TRACE_ID', 'abc-123') + expect(resolveCaller()).toBe('cursor') + }) + + it('detects Codex via CODEX_SANDBOX', () => { + vi.stubEnv('CODEX_SANDBOX', 'seatbelt') + expect(resolveCaller()).toBe('codex') + }) + + it('confirmed agents outrank the editor marker (Cursor also sets TERM_PROGRAM=vscode)', () => { + vi.stubEnv('CURSOR_TRACE_ID', 'abc-123') + vi.stubEnv('TERM_PROGRAM', 'vscode') + expect(resolveCaller()).toBe('cursor') + }) + + it('the first agent marker wins when several are present', () => { + vi.stubEnv('CLAUDECODE', '1') + vi.stubEnv('CURSOR_TRACE_ID', 'abc-123') + expect(resolveCaller()).toBe('claude-code') + }) + + it('classifies a VS Code-family editor terminal as editor, not an agent', () => { + vi.stubEnv('TERM_PROGRAM', 'vscode') + expect(resolveCaller()).toBe('editor') + }) + + it('is case-insensitive on TERM_PROGRAM', () => { + vi.stubEnv('TERM_PROGRAM', 'vsCode') + expect(resolveCaller()).toBe('editor') + }) + + it('ignores whitespace-only marker values', () => { + vi.stubEnv('CLAUDECODE', ' ') + withStdinTty(true, () => expect(resolveCaller()).toBe('interactive')) + }) + + it('falls back to interactive when stdin is a TTY and no marker is set', () => { + withStdinTty(true, () => expect(resolveCaller()).toBe('interactive')) + }) + + it('falls back to non-interactive when stdin is not a TTY', () => { + withStdinTty(false, () => expect(resolveCaller()).toBe('non-interactive')) + }) +}) diff --git a/packages/cli/src/config/tty.ts b/packages/cli/src/config/tty.ts index 09fd476a..ceb3bf1b 100644 --- a/packages/cli/src/config/tty.ts +++ b/packages/cli/src/config/tty.ts @@ -4,16 +4,130 @@ */ /** - * True when `CI` is set to a common truthy spelling (`true`/`1`, - * case-insensitive) — not every CI provider sets `CI=true` exactly. Shared by - * the region resolver (`commands/auth/region.ts`) and the DATABASE_URL resolver - * (`config/database-url.ts`) so their non-interactive gating stays identical. + * Provider-specific markers for CI systems that don't reliably set `CI=true` + * (Jenkins, Azure Pipelines/`TF_BUILD`, TeamCity, …). GitHub Actions and GitLab + * both DO set `CI=true`, so their vars here are belt-and-suspenders. Detected by + * presence of any non-empty value. + */ +export const CI_PROVIDER_ENV_VARS = [ + 'GITHUB_ACTIONS', + 'GITLAB_CI', + 'CIRCLECI', + 'TRAVIS', + 'BUILDKITE', + 'JENKINS_URL', + 'TEAMCITY_VERSION', + 'TF_BUILD', // Azure Pipelines + 'APPVEYOR', + 'DRONE', + 'BITBUCKET_BUILD_NUMBER', + 'CODEBUILD_BUILD_ID', +] as const + +/** + * Every env var {@link isCiEnvBroad} consults ({@link isCiEnv} reads only `CI`). + * Exported so tests can neutralize all CI signals hermetically — clearing only + * `CI` leaves ambient provider vars (a real `GITHUB_ACTIONS=true` in this + * repo's own CI) able to flip the broad check. + */ +export const CI_ENV_VARS = [ + 'CI', + 'CONTINUOUS_INTEGRATION', + ...CI_PROVIDER_ENV_VARS, +] as const + +/** + * True when `CI` is set to a truthy spelling (`1`/`true`) — the near-universal + * marker, set by GitHub Actions, GitLab, CircleCI, and most others. This is the + * NARROW check that governs interactive prompting: the region resolver + * (`commands/auth/region.ts`), the DATABASE_URL resolver (`config/database-url.ts`), + * and {@link isInteractive}. It is intentionally conservative about suppressing a + * prompt — a developer whose shell happens to export a provider marker like + * `JENKINS_URL` should still get prompted. Telemetry uses the broader + * {@link isCiEnvBroad} instead, where erring toward "don't collect" is safer. */ export function isCiEnv(): boolean { const ciVar = process.env.CI?.trim() return ciVar !== undefined && /^(1|true)$/i.test(ciVar) } +/** + * True in CI, detected broadly: {@link isCiEnv} plus `CONTINUOUS_INTEGRATION` and + * the {@link CI_PROVIDER_ENV_VARS} markers for systems that don't set `CI` + * (Jenkins, TeamCity, Azure Pipelines, …). Used ONLY for telemetry auto-disable, + * where a false positive merely skips collection — unlike prompt gating, where a + * false positive breaks an interactive command. Keep it separate from + * {@link isCiEnv} so broadening CI-for-telemetry never changes prompt behavior. + */ +export function isCiEnvBroad(): boolean { + if (isCiEnv()) return true + if (process.env.CONTINUOUS_INTEGRATION?.trim()) return true + return CI_PROVIDER_ENV_VARS.some((name) => Boolean(process.env[name]?.trim())) +} + +/** + * Coarse classification of who invoked the CLI, for anonymous telemetry. A + * heuristic, not a guarantee: it recognises the major coding-agent harnesses by + * an env var each one sets, and otherwise falls back to whether stdin is a TTY. + * Only the fixed label ever leaves the machine (see the telemetry allowlist) — + * never the underlying env value, some of which carry session/trace IDs. Read + * the numbers as a lower bound on agent usage: a harness without a known marker + * lands in `interactive`/`non-interactive`. + */ +export type CallerKind = + | 'claude-code' + | 'cursor' + | 'codex' + | 'editor' + | 'interactive' + | 'non-interactive' + +/** + * Known coding-agent harnesses, each recognised by an env var it sets in the + * shell it drives. Ordered most-specific first; the first match wins. Extend + * this as new harnesses appear — every `kind` must stay a fixed, non-identifying + * string, because that label (not the env value) is what telemetry emits. + */ +const AGENT_MARKERS: ReadonlyArray<{ + readonly kind: CallerKind + readonly envVars: readonly string[] +}> = [ + // Claude Code sets CLAUDECODE=1; CLAUDE_CODE_ENTRYPOINT is a backup signal. + { kind: 'claude-code', envVars: ['CLAUDECODE', 'CLAUDE_CODE_ENTRYPOINT'] }, + // Cursor's agent/terminal exports a per-session trace id. + { kind: 'cursor', envVars: ['CURSOR_TRACE_ID'] }, + // OpenAI Codex CLI runs commands in a sandbox that exports CODEX_SANDBOX. + { kind: 'codex', envVars: ['CODEX_SANDBOX'] }, +] + +/** + * Every env var {@link resolveCaller} consults. Exported so tests can neutralize + * all caller signals hermetically — this repo's own agent harness sets some of + * these ambiently, which would otherwise flip the result. + */ +export const CALLER_ENV_VARS = [ + ...AGENT_MARKERS.flatMap((m) => m.envVars), + 'TERM_PROGRAM', +] as const + +/** + * Classify the caller. Confirmed agent harnesses win first; then a VS Code-family + * editor terminal (`TERM_PROGRAM=vscode`, which Cursor also sets, so it is checked + * after Cursor and kept distinct because it may be a human in an editor); then the + * plain TTY heuristic. + */ +export function resolveCaller(): CallerKind { + for (const marker of AGENT_MARKERS) { + if (marker.envVars.some((name) => Boolean(process.env[name]?.trim()))) { + return marker.kind + } + } + if (process.env.TERM_PROGRAM?.trim().toLowerCase() === 'vscode') { + return 'editor' + } + return process.stdin.isTTY ? 'interactive' : 'non-interactive' +} + /** * True when it's safe to show an interactive clack prompt: stdin is a TTY and * we're not in CI. Every prompt gate (the DATABASE_URL resolver, the config diff --git a/packages/cli/src/messages.ts b/packages/cli/src/messages.ts index 0e7e418a..9e671825 100644 --- a/packages/cli/src/messages.ts +++ b/packages/cli/src/messages.ts @@ -109,4 +109,22 @@ export const messages = { ) => `\`${pkg}\` is not installed in this project.\n\nInstall the CipherStash packages, then re-run:\n ${installCommands}\n\nOr run \`${stash} init\` to set everything up.`, }, + telemetry: { + /** + * The one-time first-run notice. Printed to stderr so it never pollutes + * piped or `--json` stdout. Nothing is sent on the run that shows it. + * `stashRef` is the runner-aware invocation (e.g. `npx stash`) so the + * opt-out command is actionable even before `stash` is on PATH. + */ + notice: (stashRef: string) => + [ + 'CipherStash collects anonymous CLI usage analytics to improve the tool.', + 'No plaintext, schema, table/column names, or connection details are ever collected.', + `We honor the DO_NOT_TRACK standard. Opt out any time: set DO_NOT_TRACK=1, or run \`${stashRef} telemetry disable\`.`, + 'Learn more: https://cipherstash.com/docs/reference/cli', + ].join('\n'), + enabled: 'Telemetry enabled.', + disabled: 'Telemetry disabled.', + unknownSubcommand: 'Unknown telemetry command', + }, } as const diff --git a/packages/cli/src/telemetry/__tests__/classify-command.test.ts b/packages/cli/src/telemetry/__tests__/classify-command.test.ts new file mode 100644 index 00000000..e2748860 --- /dev/null +++ b/packages/cli/src/telemetry/__tests__/classify-command.test.ts @@ -0,0 +1,76 @@ +import { describe, expect, it } from 'vitest' +import { classifyCommand, classifyErrorType } from '../classify-command.js' + +describe('classifyErrorType (telemetry value allowlist)', () => { + it('passes builtins and first-party error class names through', () => { + expect(classifyErrorType(new TypeError('x'))).toBe('TypeError') + expect(classifyErrorType(new Error('x'))).toBe('Error') + }) + + it('collapses a user-defined error class name to ', () => { + // The CLI runs user code in-process (stash.config.ts via jiti); a class + // named after a table/column must not leave inside errorType. + class PatientsSsnColumnMissingError extends Error {} + expect(classifyErrorType(new PatientsSsnColumnMissingError('x'))).toBe( + '', + ) + }) + + it('collapses non-Error throws to ', () => { + expect(classifyErrorType('a thrown string')).toBe('') + expect(classifyErrorType(undefined)).toBe('') + }) +}) + +describe('classifyCommand (telemetry value allowlist)', () => { + it('keeps a recognised command + subcommand path', () => { + expect(classifyCommand('eql', 'install')).toEqual({ + command: 'eql', + subcommand: 'install', + }) + expect(classifyCommand('auth', 'login')).toEqual({ + command: 'auth', + subcommand: 'login', + }) + }) + + it('keeps a recognised command with no subcommand', () => { + expect(classifyCommand('init', undefined)).toEqual({ + command: 'init', + subcommand: undefined, + }) + }) + + it('keeps the telemetry sub-verbs (not in the registry, but known-safe)', () => { + for (const verb of ['status', 'enable', 'disable']) { + expect(classifyCommand('telemetry', verb)).toEqual({ + command: 'telemetry', + subcommand: verb, + }) + } + }) + + it('coerces a free-text positional to — the wizard-prompt leak', () => { + // The core fix: `stash wizard "add encryption to patients.ssn"` must never + // send the prompt (which carries table/column names) as the subcommand value. + expect( + classifyCommand('wizard', 'add searchable encryption to patients.ssn'), + ).toEqual({ command: 'wizard', subcommand: '' }) + }) + + it('keeps the command but drops an unrecognised subcommand to ', () => { + expect(classifyCommand('eql', 'rm -rf /')).toEqual({ + command: 'eql', + subcommand: '', + }) + }) + + it('coerces an entirely unknown command to and drops its subcommand', () => { + expect(classifyCommand('definitely-not-a-command', 'secret-value')).toEqual( + { + command: '', + subcommand: undefined, + }, + ) + }) +}) diff --git a/packages/cli/src/telemetry/__tests__/state.test.ts b/packages/cli/src/telemetry/__tests__/state.test.ts new file mode 100644 index 00000000..91b52be4 --- /dev/null +++ b/packages/cli/src/telemetry/__tests__/state.test.ts @@ -0,0 +1,66 @@ +import fs from 'node:fs' +import os from 'node:os' +import path from 'node:path' +import { afterEach, beforeEach, describe, expect, it } from 'vitest' +import { readState, writeState } from '../state.js' + +let home: string +let originalHome: string | undefined + +beforeEach(() => { + originalHome = process.env.HOME + home = fs.mkdtempSync(path.join(os.tmpdir(), 'stash-telemetry-')) + process.env.HOME = home +}) + +afterEach(() => { + if (originalHome === undefined) { + delete process.env.HOME + } else { + process.env.HOME = originalHome + } + fs.rmSync(home, { recursive: true, force: true }) +}) + +const file = () => path.join(home, '.cipherstash', 'telemetry.json') + +describe('telemetry state', () => { + it('returns a fresh default (with an id) when no file exists', () => { + const state = readState() + expect(state.telemetryDisabled).toBe(false) + expect(state.noticeShownAt).toBeUndefined() + expect(state.anonymousId).toMatch(/[0-9a-f-]{36}/) + }) + + it('persists 0600 and round-trips', () => { + const written = writeState({ + anonymousId: 'anon-x', + telemetryDisabled: true, + noticeShownAt: '2026-07-14T00:00:00.000Z', + }) + expect(readState()).toEqual(written) + // Private to the user — never group/world readable. + expect(fs.statSync(file()).mode & 0o077).toBe(0) + }) + + it('recovers from a corrupt file with a fresh id, never throwing', () => { + fs.mkdirSync(path.dirname(file()), { recursive: true }) + fs.writeFileSync(file(), '{ this is not json') + const state = readState() + expect(state.anonymousId).toMatch(/[0-9a-f-]{36}/) + expect(state.telemetryDisabled).toBe(false) + }) + + it('coerces a partial/garbage object to valid defaults', () => { + fs.mkdirSync(path.dirname(file()), { recursive: true }) + fs.writeFileSync( + file(), + JSON.stringify({ telemetryDisabled: 'yes', noticeShownAt: 42 }), + ) + const state = readState() + // A non-boolean flag is not treated as opted out; a non-string ts is dropped. + expect(state.telemetryDisabled).toBe(false) + expect(state.noticeShownAt).toBeUndefined() + expect(state.anonymousId).toMatch(/[0-9a-f-]{36}/) + }) +}) diff --git a/packages/cli/src/telemetry/__tests__/telemetry-lifecycle.test.ts b/packages/cli/src/telemetry/__tests__/telemetry-lifecycle.test.ts new file mode 100644 index 00000000..ebb14be4 --- /dev/null +++ b/packages/cli/src/telemetry/__tests__/telemetry-lifecycle.test.ts @@ -0,0 +1,185 @@ +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest' +import { CI_ENV_VARS } from '../../config/tty.js' +import type { TelemetryState } from '../state.js' + +// Controllable doubles for posthog-node and the on-disk state, wired via +// vi.hoisted so the (hoisted) vi.mock factories can close over them. +const h = vi.hoisted(() => ({ + capture: vi.fn(), + shutdown: vi.fn(async () => {}), + PostHog: vi.fn(), + writeState: vi.fn(), + state: { value: undefined as TelemetryState | undefined }, +})) + +vi.mock('posthog-node', () => ({ PostHog: h.PostHog })) +vi.mock('../state.js', () => ({ + readState: () => h.state.value, + writeState: (s: TelemetryState) => { + h.writeState(s) + h.state.value = s + return s + }, +})) + +/** Re-import the module fresh so its lazily-initialised caches reset per test. */ +async function load() { + vi.resetModules() + return import('../index.js') +} + +const ENABLED: TelemetryState = { + anonymousId: 'anon-x', + telemetryDisabled: false, + noticeShownAt: '2026-01-01T00:00:00.000Z', +} + +describe('telemetry lifecycle (emitter + flush)', () => { + beforeEach(() => { + h.capture.mockReset() + h.shutdown.mockReset().mockResolvedValue(undefined) + h.PostHog.mockReset().mockImplementation(() => ({ + capture: h.capture, + shutdown: h.shutdown, + })) + h.writeState.mockReset() + h.state.value = { ...ENABLED } + // Enabled by default; neutralize every gate so status is { enabled: true }. + vi.stubEnv('STASH_POSTHOG_KEY', 'phc_test_key') + for (const name of [ + 'DO_NOT_TRACK', + 'STASH_TELEMETRY_DISABLED', + ...CI_ENV_VARS, + ]) { + vi.stubEnv(name, '') + } + }) + afterEach(() => { + vi.unstubAllEnvs() + vi.useRealTimers() + }) + + it('trackCommand builds no client and sends nothing when disabled', async () => { + vi.stubEnv('STASH_POSTHOG_KEY', '') // dormant + const t = await load() + t.trackCommand({ command: 'init', success: true, durationMs: 1 }) + await t.shutdownTelemetry() + expect(h.PostHog).not.toHaveBeenCalled() + expect(h.capture).not.toHaveBeenCalled() + }) + + it('trackCommand is a no-op on the first run (freebie)', async () => { + h.state.value = { anonymousId: 'a', telemetryDisabled: false } // no noticeShownAt + const t = await load() + t.trackCommand({ command: 'init', success: true, durationMs: 1 }) + await t.shutdownTelemetry() + expect(h.PostHog).not.toHaveBeenCalled() + }) + + it('captures one anonymous event and flushes on shutdown', async () => { + const t = await load() + t.initTelemetry('9.9.9') + t.trackCommand({ + command: 'eql', + subcommand: 'install', + success: true, + durationMs: 5, + }) + await t.shutdownTelemetry() + expect(h.PostHog).toHaveBeenCalledTimes(1) + expect(h.capture).toHaveBeenCalledTimes(1) + const arg = h.capture.mock.calls[0][0] + expect(arg.distinctId).toBe('anon-x') + expect(arg.event).toBe('command_invoked') + expect(arg.properties.command).toBe('eql') + expect(arg.properties.cliVersion).toBe('9.9.9') + expect(arg.properties.$process_person_profile).toBe(false) + expect(h.shutdown).toHaveBeenCalledTimes(1) + }) + + it('swallows a throwing capture and still shuts down cleanly', async () => { + h.capture.mockImplementation(() => { + throw new Error('boom') + }) + const t = await load() + t.trackCommand({ command: 'init', success: true, durationMs: 1 }) + await expect(t.shutdownTelemetry()).resolves.toBeUndefined() + }) + + it('shutdownTelemetry is a safe no-op when nothing was captured, even twice', async () => { + const t = await load() + await expect(t.shutdownTelemetry()).resolves.toBeUndefined() + await expect(t.shutdownTelemetry()).resolves.toBeUndefined() + expect(h.shutdown).not.toHaveBeenCalled() + }) + + /** Save/restore stderr.isTTY faithfully: when the property wasn't an own + * property (vitest pipes stderr, so isTTY is undefined and absent), restore + * ABSENCE — a defineProperty with `value: undefined` would create a read-only + * own property that breaks later `isTTY = …` assignments in sibling tests. */ + function withStderrTty(isTTY: boolean, run: () => void | Promise) { + const original = Object.getOwnPropertyDescriptor(process.stderr, 'isTTY') + Object.defineProperty(process.stderr, 'isTTY', { + value: isTTY, + configurable: true, + }) + const restore = () => { + if (original) Object.defineProperty(process.stderr, 'isTTY', original) + else delete (process.stderr as { isTTY?: boolean }).isTTY + } + const result = run() + if (result instanceof Promise) return result.finally(restore) + restore() + return result + } + + it('shows + persists the first-run notice even when stderr is NOT a TTY, and keeps noticeShownAt across setTelemetryDisabled', async () => { + h.state.value = { anonymousId: 'a', telemetryDisabled: false } // firstRun + const write = vi.spyOn(process.stderr, 'write').mockReturnValue(true) + try { + await withStderrTty(false, async () => { + const t = await load() + t.maybeShowFirstRunNotice('npx stash') + expect(write).toHaveBeenCalledTimes(1) + expect(h.writeState).toHaveBeenCalledTimes(1) + expect(h.writeState.mock.calls[0][0].noticeShownAt).toBeDefined() + // Fix: the notice reassigns the state cache, so a later opt-out write + // does not clobber the just-persisted noticeShownAt ("shows twice" bug). + t.setTelemetryDisabled(false) + const lastWrite = h.writeState.mock.calls.at(-1)?.[0] + expect(lastWrite?.noticeShownAt).toBeDefined() + }) + } finally { + write.mockRestore() + } + }) + + it('does NOT print the notice when the state write fails — no once-per-run nag loop', async () => { + // Persist-first: an unwritable HOME (sandboxed runner, read-only container) + // must not print the disclosure on every invocation forever. No persist ⇒ + // no print ⇒ no telemetry ever (firstRun never clears) — dormant, not noisy. + h.state.value = { anonymousId: 'a', telemetryDisabled: false } // firstRun + const write = vi.spyOn(process.stderr, 'write').mockReturnValue(true) + try { + const t = await load() + h.writeState.mockImplementation(() => { + throw new Error('EROFS: read-only file system') + }) + // The mocked ../state.js writeState wrapper re-throws through h.writeState. + expect(() => t.maybeShowFirstRunNotice('npx stash')).not.toThrow() + expect(write).not.toHaveBeenCalled() + } finally { + write.mockRestore() + } + }) + + it('shutdownTelemetry resolves within the flush timeout when shutdown() hangs', async () => { + // A black-holed endpoint: shutdown() never resolves. The bounded-flush + // guarantee must still let the process continue via the ~1500ms timeout. + // Real timers (fake timers don't drive the dynamic import); headroom below. + h.shutdown.mockImplementation(() => new Promise(() => {})) + const t = await load() + t.trackCommand({ command: 'init', success: true, durationMs: 1 }) + await expect(t.shutdownTelemetry()).resolves.toBeUndefined() + }, 4000) +}) diff --git a/packages/cli/src/telemetry/__tests__/telemetry.test.ts b/packages/cli/src/telemetry/__tests__/telemetry.test.ts new file mode 100644 index 00000000..fd89550c --- /dev/null +++ b/packages/cli/src/telemetry/__tests__/telemetry.test.ts @@ -0,0 +1,211 @@ +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest' +import { CI_ENV_VARS } from '../../config/tty.js' +import { + ALLOWED_PROP_KEYS, + PLACEHOLDER_KEY, + resolveStatus, + sanitize, + type TelemetryStatus, +} from '../index.js' +import type { TelemetryState } from '../state.js' + +const enabledState: TelemetryState = { + anonymousId: 'anon-1', + telemetryDisabled: false, +} + +/** The gates only run once a project key is present; otherwise it's dormant. */ +function withKey(): void { + vi.stubEnv('STASH_POSTHOG_KEY', 'phc_test_key') +} + +/** Clear every env var that resolveStatus / isCiEnvBroad consult, INCLUDING + * STASH_POSTHOG_KEY — a real value in the ambient shell (this repo sets it as a + * GitHub Actions variable) would otherwise flip the dormant tests to enabled. */ +function clearGateEnv(): void { + for (const name of [ + 'DO_NOT_TRACK', + 'STASH_TELEMETRY_DISABLED', + 'STASH_POSTHOG_KEY', + ...CI_ENV_VARS, + ]) { + vi.stubEnv(name, '') + } +} + +function reasonOf(status: TelemetryStatus): string { + return status.enabled ? 'enabled' : status.reason +} + +describe('resolveStatus gates', () => { + beforeEach(() => { + clearGateEnv() + }) + afterEach(() => { + vi.unstubAllEnvs() + }) + + it('is dormant (unconfigured) when no project key is set', () => { + // No STASH_POSTHOG_KEY and the embedded literal is still the placeholder. + expect(resolveStatus(enabledState)).toEqual({ + enabled: false, + reason: 'unconfigured', + }) + }) + + it('an un-injected build (placeholder passthrough) stays dormant', () => { + // Simulates a dev/fork/CI build: tsup baked in no key, so the resolved key + // is the placeholder sentinel. Must read as unconfigured whatever the gates. + vi.stubEnv('STASH_POSTHOG_KEY', PLACEHOLDER_KEY) + expect(resolveStatus(enabledState)).toEqual({ + enabled: false, + reason: 'unconfigured', + }) + }) + + it('a release-injected key is NOT mistaken for the placeholder → enabled', () => { + // The regression the sentinel guards against: an injected key with no env + // override must not read as unconfigured (the old `=== EMBEDDED_KEY` check + // would have). Any real key differs from the placeholder. + vi.stubEnv('STASH_POSTHOG_KEY', 'phc_realish_injected_key') + expect(resolveStatus(enabledState)).toEqual({ enabled: true }) + }) + + it('the placeholder sentinel matches the build-define identifier name', () => { + // tsup replaces the `__STASH_POSTHOG_KEY__` identifier at release; the + // sentinel must stay the identically-named string literal, or an un-injected + // build would not be detected as dormant. + expect(PLACEHOLDER_KEY).toBe('__STASH_POSTHOG_KEY__') + }) + + it('treats an empty/whitespace STASH_POSTHOG_KEY as unset → dormant', () => { + // The `?? EMBEDDED_KEY` nullish fallback would let '' slip through and flip a + // dormant build to enabled with an empty key; projectKey() trims it away. + for (const value of ['', ' ']) { + vi.stubEnv('STASH_POSTHOG_KEY', value) + expect(resolveStatus(enabledState)).toEqual({ + enabled: false, + reason: 'unconfigured', + }) + } + }) + + it('is enabled with a key and no opt-out signals', () => { + withKey() + expect(resolveStatus(enabledState)).toEqual({ enabled: true }) + }) + + it('DO_NOT_TRACK disables and outranks the persisted flag', () => { + withKey() + vi.stubEnv('DO_NOT_TRACK', '1') + // Even with telemetry "enabled" in config, the env override wins. + expect(reasonOf(resolveStatus(enabledState))).toBe('do-not-track') + }) + + it('STASH_TELEMETRY_DISABLED disables', () => { + withKey() + vi.stubEnv('STASH_TELEMETRY_DISABLED', 'true') + expect(reasonOf(resolveStatus(enabledState))).toBe('stash-disabled') + }) + + it('DO_NOT_TRACK outranks STASH_TELEMETRY_DISABLED', () => { + withKey() + vi.stubEnv('DO_NOT_TRACK', '1') + vi.stubEnv('STASH_TELEMETRY_DISABLED', '1') + expect(reasonOf(resolveStatus(enabledState))).toBe('do-not-track') + }) + + it('auto-disables in CI', () => { + withKey() + vi.stubEnv('CI', 'true') + expect(reasonOf(resolveStatus(enabledState))).toBe('ci') + }) + + it('honors a non-standard CI marker (GITHUB_ACTIONS)', () => { + withKey() + vi.stubEnv('GITHUB_ACTIONS', 'true') + expect(reasonOf(resolveStatus(enabledState))).toBe('ci') + }) + + it('the persisted flag disables when no env override is present', () => { + withKey() + expect( + reasonOf(resolveStatus({ ...enabledState, telemetryDisabled: true })), + ).toBe('config') + }) + + it('treats DO_NOT_TRACK=0 / false / empty as not opted out', () => { + withKey() + for (const value of ['0', 'false', '']) { + vi.stubEnv('DO_NOT_TRACK', value) + expect(resolveStatus(enabledState)).toEqual({ enabled: true }) + } + }) + + it('honors DO_NOT_TRACK set to any other non-empty value', () => { + withKey() + // The convention is that setting the variable at all signals opt-out. + for (const value of ['1', 'true', 'on', 'yes', 'please']) { + vi.stubEnv('DO_NOT_TRACK', value) + expect(reasonOf(resolveStatus(enabledState))).toBe('do-not-track') + } + }) + + it('auto-disables for a provider marker without CI (GitLab)', () => { + withKey() + vi.stubEnv('GITLAB_CI', 'true') + expect(reasonOf(resolveStatus(enabledState))).toBe('ci') + }) +}) + +describe('sanitize (property allowlist)', () => { + it('drops any key not on the allowlist', () => { + const out = sanitize({ + command: 'eql', + subcommand: 'install', + // Everything below is exactly what must never leave the machine. + table: 'users', + column: 'email', + databaseUrl: 'postgres://secret@host/db', + plaintext: 'alice@example.com', + argv: ['--database-url', 'postgres://…'], + }) + expect(out).toEqual({ command: 'eql', subcommand: 'install' }) + }) + + it('keeps the full coarse event shape', () => { + const event = { + command: 'encrypt', + subcommand: 'backfill', + success: true, + durationMs: 1234, + errorType: undefined, + cliVersion: '0.17.1', + os: 'darwin', + arch: 'arm64', + nodeVersion: '22.0.0', + caller: 'claude-code', + } + expect(sanitize(event)).toEqual(event) + for (const key of Object.keys(event)) { + expect(ALLOWED_PROP_KEYS.has(key)).toBe(true) + } + }) + + it('the allowlist contains no identifying or payload fields', () => { + for (const forbidden of [ + 'table', + 'column', + 'schema', + 'databaseUrl', + 'connectionString', + 'plaintext', + 'ciphertext', + 'argv', + 'args', + 'value', + ]) { + expect(ALLOWED_PROP_KEYS.has(forbidden)).toBe(false) + } + }) +}) diff --git a/packages/cli/src/telemetry/classify-command.ts b/packages/cli/src/telemetry/classify-command.ts new file mode 100644 index 00000000..5d25d1db --- /dev/null +++ b/packages/cli/src/telemetry/classify-command.ts @@ -0,0 +1,95 @@ +import { registry } from '../cli/registry.js' + +/** + * Coerce the raw, argv-derived (command, subcommand) to a fixed vocabulary + * before telemetry emit. + * + * `subcommand` is `argv[1]` (see `parseArgs`): for most commands it's a fixed + * verb, but for a few it's a free-form positional — `stash wizard "add + * encryption to patients.ssn"` puts a natural-language prompt (routinely + * containing table/column names) in that slot. The event allowlist gates + * property KEYS, not values, so without this an arbitrary user string would + * leave the machine as the `subcommand` value, breaking the "no table/column + * names, no argument values" contract. Anything not on the known-verb list + * collapses to ``, so we learn "a wizard ran" without learning what for. + */ +const OTHER = '' + +/** + * Full command paths (`"eql install"`, `"auth login"`, `"init"`, …) from the + * registry — the single source of truth that also backs `stash manifest` and + * `--help` — plus the `telemetry` sub-verbs the registry doesn't enumerate. + * Hidden (deprecated) descriptors are excluded. + */ +const KNOWN_PATHS: ReadonlySet = new Set([ + ...registry.flatMap((group) => + group.commands.filter((c) => !c.hidden).map((c) => c.name), + ), + 'telemetry status', + 'telemetry enable', + 'telemetry disable', +]) + +/** The first token of every known path — the set of recognised top-level commands. */ +const KNOWN_COMMANDS: ReadonlySet = new Set( + [...KNOWN_PATHS].map((path) => path.split(' ')[0]), +) + +/** + * Error class names allowed to leave as `errorType`. The same closed-vocabulary + * rule as commands: the CLI executes USER code in-process (stash.config.ts and + * the encrypt client, via jiti), so `err.constructor.name` is an open + * vocabulary — a user-defined `PatientsSsnColumnMissingError` would carry a + * column name off the machine. First-party classes + Node/JS builtins pass; + * everything else collapses to ``. + */ +const KNOWN_ERROR_TYPES: ReadonlySet = new Set([ + // JS builtins + 'Error', + 'TypeError', + 'RangeError', + 'SyntaxError', + 'ReferenceError', + 'EvalError', + 'URIError', + 'AggregateError', + // Node-flavoured + 'AbortError', + 'TimeoutError', + 'SystemError', + // First-party CLI errors + 'CliExit', + 'CancelledError', + 'BackfillConfigError', + // Config validation (zod) + 'ZodError', +]) + +/** Coerce an unknown thrown value to a telemetry-safe error class name. */ +export function classifyErrorType(err: unknown): string { + if (!(err instanceof Error)) return '' + return KNOWN_ERROR_TYPES.has(err.constructor.name) + ? err.constructor.name + : '' +} + +/** + * Return telemetry-safe (command, subcommand). An unrecognised command becomes + * `` (and drops its subcommand); a recognised command with an + * unrecognised subcommand keeps the command but reports `` for the + * subcommand, so a free-text positional never leaves as-is. + */ +export function classifyCommand( + command: string, + subcommand: string | undefined, +): { command: string; subcommand: string | undefined } { + if (!KNOWN_COMMANDS.has(command)) { + return { command: OTHER, subcommand: undefined } + } + if (subcommand === undefined) { + return { command, subcommand: undefined } + } + return KNOWN_PATHS.has(`${command} ${subcommand}`) + ? { command, subcommand } + : { command, subcommand: OTHER } +} diff --git a/packages/cli/src/telemetry/index.ts b/packages/cli/src/telemetry/index.ts new file mode 100644 index 00000000..eb87f816 --- /dev/null +++ b/packages/cli/src/telemetry/index.ts @@ -0,0 +1,368 @@ +import type { PostHog } from 'posthog-node' +import { isCiEnvBroad, resolveCaller } from '../config/tty.js' +import { messages } from '../messages.js' +import { readState, type TelemetryState, writeState } from './state.js' + +/** + * Anonymous, opt-out CLI usage analytics. + * + * Design (see the CLI analytics plan): + * - Opt-out by default, but **nothing is sent until the first-run notice has + * been shown once** — the run that shows it is always a freebie. + * - Four opt-out gates, any of which disables: `DO_NOT_TRACK`, + * `STASH_TELEMETRY_DISABLED`, CI auto-detection, and the persisted + * `stash telemetry disable` flag. + * - Events are anonymous (no PostHog "person" profiles) and carry only a fixed + * allowlist of coarse properties. Table/column/schema names, connection + * strings, plaintext, ciphertext, and raw argument values can never leave — + * {@link sanitize} enforces the key allowlist, and callers must coerce the + * VALUES of `command`/`subcommand` to a known vocabulary first (the raw argv + * token is a value, not a key, so sanitize alone would pass it through — see + * `classifyCommand` in `./classify-command.ts`). + * - Sending never blocks or slows the CLI: this module reads no disk and loads + * no posthog-node until a real event is actually sent (both are deferred off + * the `--version`/`--help` fast paths), flushing is bounded by a timeout, and + * every failure is swallowed. + * - KNOWN MEASUREMENT GAP: commands that terminate via a deep `process.exit()` + * (mid-flow cancels in db install, auth login failures, config-loader exits, + * clack's own keypress exit) emit no event — the process ends before the + * flush could run. Global interception was tried and reverted (it broke + * cancel flows; see `cli/exit.ts`). First-party sites verified to unwind + * cleanly use `throw new CliExit(code)` instead, which IS tracked. Read + * success rates as covering returning/throwing/CliExit paths only. + */ + +/** Default endpoint — our Cloudflare proxy, not PostHog directly, so a future + * US→EU migration is a proxy-target change with no CLI re-release. + * `STASH_POSTHOG_HOST` overrides it (testing against a real PostHog ingestion + * endpoint before the proxy is deployed, or self-hosting), symmetric with + * `STASH_POSTHOG_KEY`. */ +const DEFAULT_POSTHOG_HOST = 'https://telemetry.cipherstash.com' + +function posthogHost(): string { + return process.env.STASH_POSTHOG_HOST ?? DEFAULT_POSTHOG_HOST +} + +/** + * The un-injected sentinel. A build that has NOT embedded a key resolves to + * exactly this string, and {@link resolveStatus} treats it as "no key" → + * dormant. It is a plain string literal (not the `__STASH_POSTHOG_KEY__` + * identifier), so the build-time `define` below never rewrites it. + */ +export const PLACEHOLDER_KEY = '__STASH_POSTHOG_KEY__' + +/** + * Build-time define. The release build (`.github/workflows/release.yml`) + * replaces the `__STASH_POSTHOG_KEY__` identifier with the real, public, + * write-only PostHog project key from the `STASH_POSTHOG_KEY` repo variable, via + * tsup's esbuild `define` (see `tsup.config.ts`). Safe to embed, exactly like a + * web SDK key. EVERY other build — local dev, a contributor's checkout, a fork, + * CI unit tests — leaves the identifier undefined, so {@link EMBEDDED_KEY} falls + * back to {@link PLACEHOLDER_KEY} and telemetry stays fully dormant. The `typeof` + * guard makes the reference safe even when the identifier was never defined + * (`typeof undefinedIdent` yields `"undefined"` rather than throwing). + */ +declare const __STASH_POSTHOG_KEY__: string | undefined + +const EMBEDDED_KEY = + typeof __STASH_POSTHOG_KEY__ === 'string' && __STASH_POSTHOG_KEY__.length > 0 + ? __STASH_POSTHOG_KEY__ + : PLACEHOLDER_KEY + +/** `STASH_POSTHOG_KEY` in the environment overrides the embedded key entirely + * (testing / self-hosting); otherwise the build-time value is used. An empty or + * whitespace-only override counts as unset — otherwise `STASH_POSTHOG_KEY=''` + * would slip past the nullish fallback and flip a dormant build to "enabled" + * with an empty key (asymmetric with the `.length > 0` guard on EMBEDDED_KEY). */ +function projectKey(): string { + const override = process.env.STASH_POSTHOG_KEY?.trim() + return override ? override : EMBEDDED_KEY +} + +/** Flush is bounded so a slow or unreachable endpoint can't hang `stash`. */ +const FLUSH_TIMEOUT_MS = 1500 + +export type TelemetryDisabledReason = + | 'do-not-track' + | 'stash-disabled' + | 'ci' + | 'config' + | 'unconfigured' + +export type TelemetryStatus = + | { enabled: true } + | { enabled: false; reason: TelemetryDisabledReason } + +/** The only property keys allowed to leave the machine. Everything else is dropped. */ +export const ALLOWED_PROP_KEYS: ReadonlySet = new Set([ + 'command', + 'subcommand', + 'success', + 'durationMs', + 'errorType', + 'cliVersion', + 'os', + 'arch', + 'nodeVersion', + 'caller', +]) + +/** + * An opt-out env var is "set" when present and not an explicit off value. This + * is deliberately broad: `DO_NOT_TRACK=1`, `=true`, `=on`, or any other non-empty + * value opts out, matching the DO_NOT_TRACK convention that setting the variable + * at all signals intent. Only `''`, `'0'`, and `'false'` mean "not opted out". + */ +function envOptOut(name: string): boolean { + const raw = process.env[name]?.trim().toLowerCase() + return raw != null && raw !== '' && raw !== '0' && raw !== 'false' +} + +/** + * Resolve whether telemetry is on, in precedence order. Env overrides win over + * the persisted flag, so a user can force telemetry off in a context where the + * config file says otherwise (and never the reverse). CI detection uses the + * BROAD check ({@link isCiEnvBroad} — provider markers included), deliberately + * separate from the narrow `isCiEnv` that gates interactive prompts: for + * telemetry a false CI positive merely skips collection, whereas for prompts it + * would break an interactive command. + */ +export function resolveStatus(state: TelemetryState): TelemetryStatus { + // Compare against the sentinel, NOT EMBEDDED_KEY: once a real key is injected + // at release, EMBEDDED_KEY holds it, and `=== EMBEDDED_KEY` would then read a + // key-present build (no env override) as unconfigured. The placeholder passing + // through — no build-time key and no env override — is the "dormant" signal. + if (projectKey() === PLACEHOLDER_KEY) { + return { enabled: false, reason: 'unconfigured' } + } + if (envOptOut('DO_NOT_TRACK')) { + return { enabled: false, reason: 'do-not-track' } + } + if (envOptOut('STASH_TELEMETRY_DISABLED')) { + return { enabled: false, reason: 'stash-disabled' } + } + if (isCiEnvBroad()) return { enabled: false, reason: 'ci' } + if (state.telemetryDisabled) return { enabled: false, reason: 'config' } + return { enabled: true } +} + +/** Defence-in-depth: keep only allowlisted keys, whatever a caller passes. */ +export function sanitize( + properties: Record, +): Record { + const out: Record = {} + for (const [key, value] of Object.entries(properties)) { + if (ALLOWED_PROP_KEYS.has(key)) out[key] = value + } + return out +} + +// Module state is resolved LAZILY on first telemetry use, not at import time, so +// fast paths that never call a telemetry function (`--version`, `--help`) touch +// no disk — the state file read (and its randomUUID on a fresh machine) is +// deferred out of the CLI's hottest paths. +let stateCache: TelemetryState | undefined +let statusCache: TelemetryStatus | undefined +let firstRunCache = false + +/** + * Read the state file once per process and derive status + first-run. Returns + * the resolved values so callers never touch the (possibly stale) module caches + * directly. `firstRun` stays true for the whole process even after the notice is + * persisted, so the current run never sends — the freebie. + */ +function init(): { + state: TelemetryState + status: TelemetryStatus + firstRun: boolean +} { + if (stateCache === undefined || statusCache === undefined) { + stateCache = readState() + statusCache = resolveStatus(stateCache) + firstRunCache = stateCache.noticeShownAt === undefined + } + return { state: stateCache, status: statusCache, firstRun: firstRunCache } +} + +let cliVersion = '0.0.0' +// posthog-node is imported dynamically the first time an event is actually sent +// (see getClient), so it is NOT pulled into every `stash` invocation — only when +// telemetry is enabled AND a real command is being tracked. clientPromise is the +// "was anything ever sent this process?" signal shutdownTelemetry gates on. +let clientPromise: Promise | null = null +// The last capture()'s enqueue promise; shutdownTelemetry awaits it before +// flushing so an event fired on an exit path is delivered, not raced away. +let lastCapture: Promise | null = null + +/** Provide the CLI version for event properties. Call once at startup. */ +export function initTelemetry(version: string): void { + cliVersion = version +} + +export function telemetryStatus(): TelemetryStatus { + return init().status +} + +async function getClient(): Promise { + if (clientPromise === null) { + clientPromise = import('posthog-node').then( + ({ PostHog }) => + new PostHog(projectKey(), { + host: posthogHost(), + flushAt: 1, + flushInterval: 0, + // Fire-and-forget: a short-lived CLI must not retry a failed send. + // Retries (default 3, exponential backoff) would keep internal timers + // pending and hang the process. Drop the event instead. + fetchRetryCount: 0, + // Bound PostHog's OWN request (undici AbortSignal.timeout). Without + // this it defaults to 10s, so a black-holed endpoint keeps the socket + // — and the event loop — alive long past our flush window, defeating + // the bounded-flush guarantee. Match the flush budget. + requestTimeout: FLUSH_TIMEOUT_MS, + // Never resolve IP → geo; we don't want or store location. + disableGeoip: true, + }), + ) + } + return clientPromise +} + +function baseProps(): Record { + return { + cliVersion, + os: process.platform, + arch: process.arch, + nodeVersion: process.versions.node, + // Coarse, fixed-enum classification of the caller (agent harness vs + // interactive shell). Never the raw env value — see resolveCaller. + caller: resolveCaller(), + } +} + +export interface CommandEvent { + command: string + subcommand?: string + success: boolean + durationMs: number + /** The error constructor name on failure — a class, never a message. */ + errorType?: string +} + +/** + * Record that a command ran. A no-op when telemetry is disabled or on the + * first run. Never throws. + */ +export function trackCommand(event: CommandEvent): void { + const { state, status, firstRun } = init() + if (!status.enabled || firstRun) return + const properties = { + // Sanitize the FULL property set (event + base) so the allowlist is the + // single enforced boundary — a future prop added to baseProps() can't + // bypass it. Only the explicit PostHog control key is added afterward. + ...sanitize({ ...event, ...baseProps() }), + // Keep events anonymous: no PostHog person profiles. + $process_person_profile: false, + } + // Fire-and-forget. getClient() dynamically imports posthog-node only now, so a + // disabled/dormant run never loads it. shutdownTelemetry awaits lastCapture + // before flushing so the event is delivered even on process.exit paths. + lastCapture = getClient() + .then((client) => { + client.capture({ + distinctId: state.anonymousId, + event: 'command_invoked', + properties, + }) + }) + .catch(() => { + // Telemetry must never surface in the command path. + }) +} + +/** + * Show the one-time first-run notice (to stderr, so it never pollutes piped or + * `--json` stdout) and mark it shown. No-op when telemetry is disabled or the + * notice was already shown. The run that shows it sends nothing (the freebie). + * + * The disclosure is written on EVERY first run, TTY or not — a non-interactive + * caller (an agent harness, a piped invocation) still gets it in its stderr/log, + * and `noticeShownAt` is persisted so that machine advances past the freebie and + * starts sending on its next run. Gating persistence on a TTY (as before) left + * exactly the agent population the `caller` dimension exists to measure dormant + * forever. `stashRef` is the runner-aware invocation (e.g. `npx stash`) so the + * opt-out hint is actionable before the CLI is on PATH. + */ +export function maybeShowFirstRunNotice(stashRef: string): void { + const { state, status, firstRun } = init() + if (!status.enabled || !firstRun) return + try { + // PERSIST FIRST, print only on success. On a machine where the state write + // fails (read-only HOME, sandboxed runner) the notice would otherwise print + // on every single run forever — worse than pre-notice behaviour. Persist + // failure ⇒ no print AND no telemetry ever (firstRun never clears), which is + // the honest fallback: no disclosure, no collection. + // + // Reassign the cache: a later setTelemetryDisabled() in the same process + // must not write from a stale `state` and clobber this noticeShownAt. + stateCache = writeState({ + ...state, + noticeShownAt: new Date().toISOString(), + }) + } catch { + return + } + process.stderr.write(`${messages.telemetry.notice(stashRef)}\n`) +} + +/** Persist the opt-out flag. Surfaces write errors (the command wants to know). */ +export function setTelemetryDisabled(disabled: boolean): void { + const { state } = init() + stateCache = writeState({ ...state, telemetryDisabled: disabled }) + statusCache = resolveStatus(stateCache) +} + +/** Flush any buffered events, bounded by {@link FLUSH_TIMEOUT_MS}. Never throws. */ +export async function shutdownTelemetry(): Promise { + // clientPromise is null iff nothing was ever captured (disabled/dormant/first + // run), so there is nothing to flush and posthog-node was never loaded. + if (clientPromise === null) return + // The timer MUST be cleared once the flush wins the race: an uncleared + // pending setTimeout keeps the Node event loop alive, so the process would + // hang for the full timeout after the flush already completed. + let timer: ReturnType | undefined + // Capture the promises before the race so the finally can reset the module + // state regardless of which side wins. + const pendingCapture = lastCapture + const pendingClient = clientPromise + try { + // EVERYTHING is inside the race — including awaiting the capture enqueue + // (which contains the dynamic import of posthog-node) and the client + // resolution. If any of it stalls (a hung filesystem mid-import, a + // black-holed endpoint), the timeout still unblocks the process; nothing + // that can hang sits outside the bound. The swallow is attached to the + // flush promise ITSELF (not just the race): if the timeout wins while the + // flush is still pending, a later rejection would otherwise surface as an + // unhandledRejection crash. + const flush = (async () => { + // Wait for the capture enqueue (client construction + queueing) to + // finish, otherwise shutdown() could race ahead of the event we mean + // to deliver. Its rejections are already swallowed in trackCommand. + if (pendingCapture !== null) await pendingCapture + const client = await pendingClient + await client?.shutdown() + })().catch(() => { + // Swallow — a failed flush must not fail (or crash) the process. + }) + await Promise.race([ + flush, + new Promise((resolve) => { + timer = setTimeout(resolve, FLUSH_TIMEOUT_MS) + }), + ]) + } finally { + if (timer !== undefined) clearTimeout(timer) + clientPromise = null + lastCapture = null + } +} diff --git a/packages/cli/src/telemetry/state.ts b/packages/cli/src/telemetry/state.ts new file mode 100644 index 00000000..fab18b1b --- /dev/null +++ b/packages/cli/src/telemetry/state.ts @@ -0,0 +1,74 @@ +import { randomUUID } from 'node:crypto' +import fs from 'node:fs' +import os from 'node:os' +import path from 'node:path' + +/** + * Machine-level telemetry state, persisted alongside the auth profile in + * `~/.cipherstash/` but in its own file. We only ever read/write `telemetry.json` + * here — never the auth secrets (`auth.json`, `secretkey.json`, `workspaces/`). + * + * The file holds three things: a random anonymous id (for de-duplicating events + * in aggregate, never derivable to a person), the persisted opt-out flag written + * by `stash telemetry disable`, and the timestamp of the first-run notice so the + * banner shows exactly once per install. + */ +export interface TelemetryState { + /** Random UUID; the PostHog `distinctId`. Not tied to any identity. */ + anonymousId: string + /** Set by `stash telemetry disable`. The lowest-precedence opt-out gate. */ + telemetryDisabled: boolean + /** ISO timestamp the first-run notice was shown; unset means "never shown". */ + noticeShownAt?: string +} + +/** Resolved at call time so it always tracks the current home directory. */ +function stateDir(): string { + return path.join(os.homedir(), '.cipherstash') +} +function stateFile(): string { + return path.join(stateDir(), 'telemetry.json') +} + +/** Coerce arbitrary parsed JSON into a valid state, filling gaps with defaults. */ +function normalize(value: unknown): TelemetryState { + const o = ( + typeof value === 'object' && value !== null ? value : {} + ) as Record + return { + anonymousId: + typeof o.anonymousId === 'string' && o.anonymousId.length > 0 + ? o.anonymousId + : randomUUID(), + telemetryDisabled: o.telemetryDisabled === true, + noticeShownAt: + typeof o.noticeShownAt === 'string' ? o.noticeShownAt : undefined, + } +} + +/** + * Read the state file. Never throws: a missing or corrupt file yields a fresh + * default (with a new anonymous id), so a bad file can never break a command. + * The fresh id is only ephemeral until something calls {@link writeState}. + */ +export function readState(): TelemetryState { + try { + return normalize(JSON.parse(fs.readFileSync(stateFile(), 'utf-8'))) + } catch { + return { anonymousId: randomUUID(), telemetryDisabled: false } + } +} + +/** + * Persist state (0600, private). Returns the normalized value actually written. + * Unlike {@link readState} this may throw — callers that must not fail a command + * (the emitter) wrap it; the `stash telemetry` command lets a write error surface. + */ +export function writeState(state: TelemetryState): TelemetryState { + const normalized = normalize(state) + fs.mkdirSync(stateDir(), { recursive: true }) + fs.writeFileSync(stateFile(), `${JSON.stringify(normalized, null, 2)}\n`, { + mode: 0o600, + }) + return normalized +} diff --git a/packages/cli/tests/e2e/smoke.e2e.test.ts b/packages/cli/tests/e2e/smoke.e2e.test.ts index f059e3c3..f872b19e 100644 --- a/packages/cli/tests/e2e/smoke.e2e.test.ts +++ b/packages/cli/tests/e2e/smoke.e2e.test.ts @@ -99,6 +99,23 @@ describe('stash CLI — non-interactive smoke', () => { expect(r.output).toContain('bogus-sub') }) + it('telemetry status reports the current state and exits 0', async () => { + const r = render(['telemetry', 'status']) + const { exitCode } = await r.exit + expect(exitCode).toBe(0) + // Dev builds carry the placeholder key, so status reports the dormant build + // (the harness also sets STASH_TELEMETRY_DISABLED, but unconfigured wins). + expect(r.output).toContain('Telemetry is') + }) + + it('telemetry bogus-sub exits 1 (the CliExit cooperative-exit path)', async () => { + const r = render(['telemetry', 'bogus-sub']) + const { exitCode } = await r.exit + expect(exitCode).toBe(1) + expect(r.output).toContain(messages.telemetry.unknownSubcommand) + expect(r.output).toContain('bogus-sub') + }) + // `--migration` without `--supabase` fails flag validation before any I/O // or prompt, so these two cases can observe the install entry path // deterministically without a database. diff --git a/packages/cli/tests/helpers/pty.ts b/packages/cli/tests/helpers/pty.ts index 3d7b787e..8a559011 100644 --- a/packages/cli/tests/helpers/pty.ts +++ b/packages/cli/tests/helpers/pty.ts @@ -4,6 +4,7 @@ import { dirname, join, resolve } from 'node:path' import { fileURLToPath } from 'node:url' import { type IPty, spawn } from 'node-pty' import stripAnsi from 'strip-ansi' +import { CI_ENV_VARS } from '../../src/config/tty.js' const __dirname = dirname(fileURLToPath(import.meta.url)) @@ -98,11 +99,25 @@ export function render(args: string[], opts: RenderOptions = {}): Rendered { // debugging a failure. NO_COLOR: '1', FORCE_COLOR: '0', - // Match the convention the CLI itself uses (e.g. install.ts checks - // `process.env.CI !== 'true'`) so test runs hit the same code paths. - CI: 'true', - ...(opts.env ?? {}), } + // Strip every CI marker so the harness default below (and per-test overrides) + // fully control CI detection. Prompt gating (isCiEnv) reads only `CI`, but + // telemetry's isCiEnvBroad consults the provider markers (GITHUB_ACTIONS, + // GITLAB_CI, …), which leak in from the ambient environment — this repo's own + // CI — and would otherwise flip telemetry gating in a test that de-CIs with + // `env: { CI: '' }`. + for (const name of CI_ENV_VARS) delete env[name] + // Match the convention the CLI itself uses (e.g. install.ts checks + // `process.env.CI !== 'true'`) so test runs hit the same code paths. + env.CI = 'true' + // Hard-disable telemetry regardless of the ambient shell. Without this, a + // developer with STASH_POSTHOG_KEY exported (the documented testing override) + // running the e2e suite would spawn telemetry-ENABLED CLIs against their real + // ~/.cipherstash — sending genuine events from tests and injecting the + // first-run notice into pty output. The CI='true' default is not enough: the + // de-CI'ing tests strip it. Tests exercising telemetry itself can override. + env.STASH_TELEMETRY_DISABLED = '1' + Object.assign(env, opts.env ?? {}) // Use the absolute path to the current node binary — node-pty's // `posix_spawnp` doesn't inherit PATH lookup reliably across all macOS / diff --git a/packages/cli/tsup.config.ts b/packages/cli/tsup.config.ts index 0a85deea..2f17f343 100644 --- a/packages/cli/tsup.config.ts +++ b/packages/cli/tsup.config.ts @@ -1,6 +1,18 @@ import { cpSync, existsSync } from 'node:fs' import { defineConfig } from 'tsup' +/** + * Build-time value for the embedded PostHog project key (see + * `src/telemetry/index.ts`). Only the release workflow sets `STASH_POSTHOG_KEY` + * (from a public repo variable), so every other build — dev, forks, CI — bakes + * in an empty string and the CLI ships telemetry-dormant. Applied to both bundles + * because either may inline the telemetry module. The value must be a JS + * expression string, hence `JSON.stringify`. + */ +const posthogKeyDefine = { + __STASH_POSTHOG_KEY__: JSON.stringify(process.env.STASH_POSTHOG_KEY ?? ''), +} + export default defineConfig([ { entry: ['src/index.ts'], @@ -17,6 +29,7 @@ export default defineConfig([ ...options.logOverride, 'empty-import-meta': 'silent', } + options.define = { ...options.define, ...posthogKeyDefine } }, onSuccess: async () => { // Copy bundled SQL files into dist so they ship with the package @@ -52,5 +65,8 @@ var require = __createRequire(import.meta.url);`, sourcemap: true, skipNodeModulesBundle: true, + esbuildOptions(options) { + options.define = { ...options.define, ...posthogKeyDefine } + }, }, ]) diff --git a/packages/drizzle/CHANGELOG.md b/packages/drizzle/CHANGELOG.md index b0d308a6..f85a1c41 100644 --- a/packages/drizzle/CHANGELOG.md +++ b/packages/drizzle/CHANGELOG.md @@ -1,5 +1,13 @@ # @cipherstash/drizzle +## 3.0.4-rc.0 + +### Patch Changes + +- Updated dependencies [229ce59] + - @cipherstash/schema@3.0.2-rc.0 + - @cipherstash/protect@12.0.2-rc.0 + ## 3.0.3 ### Patch Changes diff --git a/packages/drizzle/README.md b/packages/drizzle/README.md index 3bc1d527..7a27ac83 100644 --- a/packages/drizzle/README.md +++ b/packages/drizzle/README.md @@ -5,7 +5,7 @@ Seamlessly integrate CipherStash encryption with Drizzle ORM and PostgreSQL to encrypt your data while maintaining full query capabilities—equality, range queries, text search, and sorting—all with complete TypeScript type safety. > [!TIP] -> For new projects we recommend [`@cipherstash/stack`](https://www.npmjs.com/package/@cipherstash/stack), which provides this integration as `encryptedType`, `extractEncryptionSchema`, and `createEncryptionOperators` from `@cipherstash/stack/drizzle`. See the [Drizzle docs](https://cipherstash.com/docs/stack/cipherstash/encryption/drizzle). This package documents the legacy `@cipherstash/protect`-based API. +> For new projects we recommend [`@cipherstash/stack`](https://www.npmjs.com/package/@cipherstash/stack), which provides this integration via the separate [`@cipherstash/stack-drizzle`](https://www.npmjs.com/package/@cipherstash/stack-drizzle) package (EQL v3 on its `/v3` subpath). See the [Drizzle docs](https://cipherstash.com/docs). This package documents the legacy `@cipherstash/protect`-based API. ## Features diff --git a/packages/drizzle/package.json b/packages/drizzle/package.json index 73cbfd00..6e41233e 100644 --- a/packages/drizzle/package.json +++ b/packages/drizzle/package.json @@ -1,6 +1,6 @@ { "name": "@cipherstash/drizzle", - "version": "3.0.3", + "version": "3.0.4-rc.0", "description": "CipherStash Protect.js Drizzle ORM integration for TypeScript", "keywords": [ "encrypted", @@ -44,8 +44,8 @@ "release": "tsup" }, "peerDependencies": { - "@cipherstash/protect": ">=10.5.0", - "@cipherstash/schema": ">=2.1.0", + "@cipherstash/protect": ">=12.0.2-rc.0", + "@cipherstash/schema": ">=3.0.2-rc.0", "@types/pg": "*", "drizzle-kit": ">=0.20", "drizzle-orm": ">=0.33", diff --git a/packages/migrate/CHANGELOG.md b/packages/migrate/CHANGELOG.md index a7e95846..8b991d51 100644 --- a/packages/migrate/CHANGELOG.md +++ b/packages/migrate/CHANGELOG.md @@ -1,5 +1,34 @@ # @cipherstash/migrate +## 1.0.0-rc.0 + +### Patch Changes + +- Updated dependencies [31ca318] +- Updated dependencies [c4787c0] +- Updated dependencies [66a0e02] +- Updated dependencies [cfd46ee] +- Updated dependencies [7eba32d] +- Updated dependencies [0ebf57e] +- Updated dependencies [d73a03c] +- Updated dependencies [89b903f] +- Updated dependencies [229ce59] +- Updated dependencies [50c0a9c] +- Updated dependencies [63ca540] +- Updated dependencies [5d23e80] +- Updated dependencies [1aa9a11] +- Updated dependencies [af2d04e] +- Updated dependencies [b8a3d20] +- Updated dependencies [a0f3b2c] +- Updated dependencies [f23f952] +- Updated dependencies [7c7dbca] +- Updated dependencies [5411a13] +- Updated dependencies [99f8b0a] +- Updated dependencies [fd33aad] +- Updated dependencies [8cd485d] +- Updated dependencies [9b65ae8] + - @cipherstash/stack@1.0.0-rc.0 + ## 0.2.0 ### Minor Changes diff --git a/packages/migrate/package.json b/packages/migrate/package.json index 30afb188..f0a5b52e 100644 --- a/packages/migrate/package.json +++ b/packages/migrate/package.json @@ -1,6 +1,6 @@ { "name": "@cipherstash/migrate", - "version": "0.2.0", + "version": "1.0.0-rc.0", "description": "Plaintext-to-encrypted column migration for CipherStash: resumable backfill, per-column state, and EQL lifecycle orchestration.", "repository": { "type": "git", @@ -50,7 +50,7 @@ "zod": "^3.25.76" }, "peerDependencies": { - "@cipherstash/stack": ">=0.6.0", + "@cipherstash/stack": ">=1.0.0-rc.0", "pg": ">=8" }, "devDependencies": { diff --git a/packages/prisma-next/CHANGELOG.md b/packages/prisma-next/CHANGELOG.md index c1bb5907..c1a0e7b1 100644 --- a/packages/prisma-next/CHANGELOG.md +++ b/packages/prisma-next/CHANGELOG.md @@ -1,5 +1,45 @@ # @cipherstash/prisma-next +## 0.4.0-rc.0 + +### Minor Changes + +- d6d23be: Upgrade to Prisma Next 0.14.0 (from 0.8.0). Every `@prisma-next/*` dependency is now pinned at 0.14.0; consuming apps must run Prisma Next 0.14 to use this release. + + Highlights of the upgrade: + + - The extension contract space is re-emitted in the 0.14 canonical shape: storage is namespace-enveloped (`storage.namespaces.public.entries.table`), the domain plane replaces flat `models`, and the baseline EQL-install migration is re-pinned to the new storage hash. The vendored EQL bundle SQL is unchanged byte-for-byte. + - `deriveStackSchemas` reads the namespace-enveloped contract shape emitted by Prisma Next 0.10+. + - The bulk-encrypt middleware accepts the widened insert/update AST value unions introduced through 0.9–0.11. + - README examples use the namespace-qualified ORM accessors (`db.orm.public.User`) required since Prisma Next 0.14. + +### Patch Changes + +- Updated dependencies [31ca318] +- Updated dependencies [c4787c0] +- Updated dependencies [66a0e02] +- Updated dependencies [cfd46ee] +- Updated dependencies [7eba32d] +- Updated dependencies [0ebf57e] +- Updated dependencies [d73a03c] +- Updated dependencies [89b903f] +- Updated dependencies [229ce59] +- Updated dependencies [50c0a9c] +- Updated dependencies [63ca540] +- Updated dependencies [5d23e80] +- Updated dependencies [1aa9a11] +- Updated dependencies [af2d04e] +- Updated dependencies [b8a3d20] +- Updated dependencies [a0f3b2c] +- Updated dependencies [f23f952] +- Updated dependencies [7c7dbca] +- Updated dependencies [5411a13] +- Updated dependencies [99f8b0a] +- Updated dependencies [fd33aad] +- Updated dependencies [8cd485d] +- Updated dependencies [9b65ae8] + - @cipherstash/stack@1.0.0-rc.0 + ## 0.3.2 ### Patch Changes diff --git a/packages/prisma-next/package.json b/packages/prisma-next/package.json index 36937fad..a14470ae 100644 --- a/packages/prisma-next/package.json +++ b/packages/prisma-next/package.json @@ -1,6 +1,6 @@ { "name": "@cipherstash/prisma-next", - "version": "0.3.2", + "version": "0.4.0-rc.0", "license": "MIT", "author": "CipherStash ", "description": "CipherStash extension for Prisma Next: searchable application-layer field-level encryption for Postgres, with six encrypted column types, 17 query operators, bulk encrypt/decrypt middleware, and a baseline migration that installs the vendored EQL bundle SQL byte-for-byte.", diff --git a/packages/protect-dynamodb/CHANGELOG.md b/packages/protect-dynamodb/CHANGELOG.md index 0b8b9905..6be07a47 100644 --- a/packages/protect-dynamodb/CHANGELOG.md +++ b/packages/protect-dynamodb/CHANGELOG.md @@ -1,5 +1,11 @@ # @cipherstash/protect-dynamodb +## 12.0.2-rc.0 + +### Patch Changes + +- @cipherstash/protect@12.0.2-rc.0 + ## 12.0.1 ### Patch Changes diff --git a/packages/protect-dynamodb/package.json b/packages/protect-dynamodb/package.json index 48ea68ee..2e1d99d0 100644 --- a/packages/protect-dynamodb/package.json +++ b/packages/protect-dynamodb/package.json @@ -1,6 +1,6 @@ { "name": "@cipherstash/protect-dynamodb", - "version": "12.0.1", + "version": "12.0.2-rc.0", "description": "Protect.js DynamoDB Helpers", "keywords": [ "dynamodb", diff --git a/packages/protect/CHANGELOG.md b/packages/protect/CHANGELOG.md index 88f26a07..9c79eaf4 100644 --- a/packages/protect/CHANGELOG.md +++ b/packages/protect/CHANGELOG.md @@ -1,5 +1,12 @@ # @cipherstash/protect +## 12.0.2-rc.0 + +### Patch Changes + +- Updated dependencies [229ce59] + - @cipherstash/schema@3.0.2-rc.0 + ## 12.0.1 ### Patch Changes diff --git a/packages/protect/package.json b/packages/protect/package.json index e040c99e..e2541ce2 100644 --- a/packages/protect/package.json +++ b/packages/protect/package.json @@ -1,6 +1,6 @@ { "name": "@cipherstash/protect", - "version": "12.0.1", + "version": "12.0.2-rc.0", "description": "CipherStash Protect for JavaScript", "keywords": [ "encrypted", diff --git a/packages/schema/CHANGELOG.md b/packages/schema/CHANGELOG.md index 3fb9edcf..67841486 100644 --- a/packages/schema/CHANGELOG.md +++ b/packages/schema/CHANGELOG.md @@ -1,5 +1,14 @@ # @cipherstash/schema +## 3.0.2-rc.0 + +### Patch Changes + +- 229ce59: `searchableJson()` now pins the SteVec encoding mode to `standard` explicitly. + protect-ffi 0.29 flipped the library default to `compat` (the EQL v3 + encoding); pinning keeps the v2 wire format byte-stable so existing encrypted + JSON columns stay queryable and comparable. + ## 3.0.1 ### Patch Changes diff --git a/packages/schema/package.json b/packages/schema/package.json index 2211dfc5..f718c249 100644 --- a/packages/schema/package.json +++ b/packages/schema/package.json @@ -1,6 +1,6 @@ { "name": "@cipherstash/schema", - "version": "3.0.1", + "version": "3.0.2-rc.0", "description": "CipherStash schema builder for TypeScript", "keywords": [ "encrypted", diff --git a/packages/stack-drizzle/CHANGELOG.md b/packages/stack-drizzle/CHANGELOG.md new file mode 100644 index 00000000..5cbf048f --- /dev/null +++ b/packages/stack-drizzle/CHANGELOG.md @@ -0,0 +1,108 @@ +# @cipherstash/stack-drizzle + +## 1.0.0-rc.0 + +### Major Changes + +- 7c7dbca: CipherStash Stack 1.0 (release candidate). + + This is the first 1.0-line release of `@cipherstash/stack`, the first published + release of the split-out EQL v3 adapters `@cipherstash/stack-drizzle` and + `@cipherstash/stack-supabase`, and moves the `stash` CLI to 1.0 alongside them. + These four packages now version together as the Stack 1.0 family. + +### Minor Changes + +- 31ca318: Split the Drizzle and Supabase integrations into their own packages. + + The adapters now ship as first-party packages that depend on `@cipherstash/stack`, + following the `@cipherstash/prisma-next` precedent: + + - **`@cipherstash/stack-drizzle`** — Drizzle ORM integration. EQL v2 on the package + root (`@cipherstash/stack-drizzle`: `encryptedType`, `extractEncryptionSchema`, + `createEncryptionOperators`) and EQL v3 on `@cipherstash/stack-drizzle/v3` + (`types` factories, `createEncryptionOperatorsV3`, `extractEncryptionSchemaV3`, …). + - **`@cipherstash/stack-supabase`** — Supabase integration: `encryptedSupabase` (v2) + and `encryptedSupabaseV3` (v3, connect-time introspection). + + **Breaking (`@cipherstash/stack`):** the `./drizzle`, `./supabase`, and + `./eql/v3/drizzle` subpath exports are removed. Migrate imports: + + - `@cipherstash/stack/drizzle` → `@cipherstash/stack-drizzle` + - `@cipherstash/stack/eql/v3/drizzle` → `@cipherstash/stack-drizzle/v3` + - `@cipherstash/stack/supabase` → `@cipherstash/stack-supabase` + + Add the relevant package to your dependencies alongside `@cipherstash/stack`. A new + `@cipherstash/stack/adapter-kit` subpath exposes the narrow core internals the + first-party adapters consume; it is the core↔adapter seam, not general-purpose API. + +- 7eba32d: EQL v3 Drizzle: encrypt every query operand with `encryptQuery`, not `encrypt` (#622). + + The v3 Drizzle operators (`eq`/`ne`/`gt`/`gte`/`lt`/`lte`/`between`/`notBetween`/ + `inArray`/`notInArray`/`contains`) previously encrypted their operands with + `client.encrypt`, producing a full storage envelope (including the ciphertext `c`) + cast to `::jsonb`. A WHERE-clause operand should be a query _term_, not a value to + store. Every operator now uses `client.encryptQuery`, which yields a + ciphertext-free query term cast to the column's `eql_v3.query_` type — so + predicates carry no ciphertext and reach the bundle's `(domain, query_)` + operator overloads. This unifies the scalar/text operators with the JSON + containment path (already on `encryptQuery`) and removes the previously-optional + `encryptQuery` guard: it is now a required capability of the operand client. + + `@cipherstash/stack` gains a batch `encryptQuery(terms)` overload on + `TypedEncryptionClient` (the type `EncryptionV3` returns), mirroring the nominal + `EncryptionClient`. This is additive — it lets `inArray`/`notInArray` encrypt a + whole list of query terms in one crossing. + +- e40c3da: Rename the EQL v3 encrypted free-text operator `contains()` → `matches()` (#617). + + Encrypted free-text search is fuzzy bloom-filter token matching — order- and + multiplicity-insensitive and one-sided (a `true` may be a false positive) — not + containment. The name `contains()` promised substring/containment semantics it + never had. It is renamed to `matches()` on the encrypted surface; `contains()` is + kept for genuine, exact containment: + + - **Drizzle** (`@cipherstash/stack-drizzle/v3`): `matches()` = bloom free-text on + `text_match`/`text_search` columns; `contains()` = exact encrypted-JSON `@>` on + `types.Json` (ste_vec) columns. + - **Supabase** (`@cipherstash/stack-supabase`): `.matches()` = encrypted free-text; + `.contains()` = native jsonb/array `@>` on plaintext columns (and throws on an + encrypted column, pointing to `matches()`). + + Also on the Supabase v3 surface, `like()`/`ilike()` on an encrypted column are no + longer rejected — they are delegated to `matches()` as a best-effort compatibility + shim. This is APPROXIMATE (fuzzy, case-insensitive, one-sided; anchoring and + wildcards are not honored): surrounding `%` are stripped, an internal `%` or any + `_` is rejected, and a one-time warning is emitted. A plaintext column keeps real + SQL LIKE. + + Breaking: encrypted `contains()` callers must migrate to `matches()`. The + encrypted operator has not shipped in a stable release (it lands via the EQL v3 + work), so there is no deprecation alias. + +### Patch Changes + +- Updated dependencies [31ca318] +- Updated dependencies [c4787c0] +- Updated dependencies [66a0e02] +- Updated dependencies [cfd46ee] +- Updated dependencies [7eba32d] +- Updated dependencies [0ebf57e] +- Updated dependencies [d73a03c] +- Updated dependencies [89b903f] +- Updated dependencies [229ce59] +- Updated dependencies [50c0a9c] +- Updated dependencies [63ca540] +- Updated dependencies [5d23e80] +- Updated dependencies [1aa9a11] +- Updated dependencies [af2d04e] +- Updated dependencies [b8a3d20] +- Updated dependencies [a0f3b2c] +- Updated dependencies [f23f952] +- Updated dependencies [7c7dbca] +- Updated dependencies [5411a13] +- Updated dependencies [99f8b0a] +- Updated dependencies [fd33aad] +- Updated dependencies [8cd485d] +- Updated dependencies [9b65ae8] + - @cipherstash/stack@1.0.0-rc.0 diff --git a/packages/stack-drizzle/__tests__/v3/operators.test-d.ts b/packages/stack-drizzle/__tests__/v3/operators.test-d.ts index c2c09c1d..7c04db64 100644 --- a/packages/stack-drizzle/__tests__/v3/operators.test-d.ts +++ b/packages/stack-drizzle/__tests__/v3/operators.test-d.ts @@ -53,6 +53,9 @@ describe('createEncryptionOperatorsV3 - client parameter (M1)', () => { _opts?: never, ): QueryOp & QueryOp => ({}) as never, + // Storage encryption — consumed only by the JSON selector RHS, but part of + // the operand-client contract, so a structural double must supply it too. + encrypt: (_value: never, _opts: never): QueryOp => ({}) as never, } expectTypeOf(createEncryptionOperatorsV3).toBeCallableWith(double) }) @@ -67,6 +70,9 @@ describe('createEncryptionOperatorsV3 - client parameter (M1)', () => { const erased = { encryptQuery: (_valueOrTerms: never, _opts?: never): QueryOp => ({}) as never, + // A correctly-typed `encrypt` so the ONLY reason this double is rejected is + // the `encryptQuery`-resolves-`unknown` erasure — not a missing member. + encrypt: (_value: never, _opts: never): QueryOp => ({}) as never, } // @ts-expect-error — `encryptQuery` resolving `unknown` does not satisfy the // factory's `ChainableOperation` client contract. diff --git a/packages/stack-drizzle/__tests__/v3/selector.test.ts b/packages/stack-drizzle/__tests__/v3/selector.test.ts new file mode 100644 index 00000000..e3c2296c --- /dev/null +++ b/packages/stack-drizzle/__tests__/v3/selector.test.ts @@ -0,0 +1,112 @@ +import { integer, pgTable } from 'drizzle-orm/pg-core' +import { describe, expect, it } from 'vitest' +import { + createEncryptionOperatorsV3, + EncryptionOperatorError, + parseSelectorSegments, + reconstructSelectorDocument, +} from '../../src/v3/operators.js' +import { types } from '../../src/v3/types.js' + +describe('parseSelectorSegments', () => { + it('parses $-rooted, bare, and whitespace-padded dot paths', () => { + expect(parseSelectorSegments('$.a')).toEqual(['a']) + expect(parseSelectorSegments('$.a.b')).toEqual(['a', 'b']) + expect(parseSelectorSegments('a.b')).toEqual(['a', 'b']) + expect(parseSelectorSegments(' $.a.b ')).toEqual(['a', 'b']) + }) + + it('rejects array-index and wildcard syntax', () => { + expect(() => parseSelectorSegments('$.items[0]')).toThrow(/array\/wildcard/) + expect(() => parseSelectorSegments('$.items[*].name')).toThrow( + /array\/wildcard/, + ) + }) + + it('rejects the empty / root path', () => { + expect(() => parseSelectorSegments('$')).toThrow(/addresses no field/) + expect(() => parseSelectorSegments('$.')).toThrow(/addresses no field/) + expect(() => parseSelectorSegments('')).toThrow(/addresses no field/) + }) + + it('rejects malformed paths (.. / stray dots) rather than silently collapsing', () => { + expect(() => parseSelectorSegments('$..age')).toThrow(/malformed/) + expect(() => parseSelectorSegments('$.a..b')).toThrow(/malformed/) + expect(() => parseSelectorSegments('$.a.')).toThrow(/malformed/) + }) + + it('rejects prototype-pollution keys', () => { + for (const key of ['__proto__', 'prototype', 'constructor']) { + expect(() => parseSelectorSegments(`$.${key}`)).toThrow(/forbidden key/) + expect(() => parseSelectorSegments(`$.a.${key}`)).toThrow(/forbidden key/) + } + }) +}) + +describe('reconstructSelectorDocument', () => { + it('nests the value under the segments', () => { + expect(reconstructSelectorDocument(['a'], 30)).toEqual({ a: 30 }) + expect(reconstructSelectorDocument(['a', 'b'], 'x')).toEqual({ + a: { b: 'x' }, + }) + }) + + it('serializes correctly and creates own keys (no prototype pollution)', () => { + // parseSelectorSegments rejects __proto__ upstream, but the builder must be + // safe regardless: a __proto__ segment is an OWN key, not the prototype. + const doc = reconstructSelectorDocument(['__proto__', 'age'], 1) + expect(JSON.stringify(doc)).toBe('{"__proto__":{"age":1}}') + expect(Object.getPrototypeOf(doc)).toBeNull() + expect({}.age).toBeUndefined() // global prototype untouched + }) +}) + +describe('ops.selector — up-front guards (no encryption reached)', () => { + const table = pgTable('selector_guard', { + id: integer('id').primaryKey(), + doc: types.Json('doc'), + }) + // The encrypt methods throw if reached — every case below must reject first. + const failIfCalled = () => { + throw new Error('guard should reject before encrypting') + } + const ops = createEncryptionOperatorsV3({ + encryptQuery: failIfCalled, + encrypt: failIfCalled, + } as never) + + it('rejects a non-scalar leaf value (object / array) — use contains()', async () => { + await expect(ops.selector(table.doc, '$.a').eq({ x: 1 })).rejects.toThrow( + /scalar leaf.*contains\(\)/, + ) + await expect(ops.selector(table.doc, '$.a').eq([1, 2])).rejects.toThrow( + /scalar leaf/, + ) + }) + + it('rejects ordering a boolean leaf', async () => { + await expect(ops.selector(table.doc, '$.flag').gt(true)).rejects.toThrow( + /boolean leaf has no ordering/, + ) + }) + + it('allows equality on a boolean leaf (would reach encryption)', async () => { + // eq on a boolean is permitted by the guard; it fails only because the mock + // encrypt throws — proving the guard passed it through. + await expect(ops.selector(table.doc, '$.flag').eq(true)).rejects.toThrow( + /guard should reject before encrypting/, + ) + }) + + it('surfaces path errors as EncryptionOperatorError with context', async () => { + await expect( + ops.selector(table.doc, '$.items[0]').eq(1), + ).rejects.toBeInstanceOf(EncryptionOperatorError) + await expect(ops.selector(table.doc, '$').eq(1)).rejects.toThrow( + /addresses no field/, + ) + await expect(ops.selector(table.doc, '$.__proto__').eq(1)).rejects.toThrow( + /forbidden key/, + ) + }) +}) diff --git a/packages/stack-drizzle/integration/json-selector.integration.test.ts b/packages/stack-drizzle/integration/json-selector.integration.test.ts new file mode 100644 index 00000000..e22fd4bc --- /dev/null +++ b/packages/stack-drizzle/integration/json-selector.integration.test.ts @@ -0,0 +1,169 @@ +/** + * Live JSONPath selector-with-constraint for the v3 `types.Json()` column (#623). + * Distinct from containment (`ops.contains`, `@>`): this extracts the encrypted + * leaf entry at a JSONPath and compares it, so it can express ORDERING at a path + * (`col->'$.age' > 25`) that containment cannot. Emits + * `eql_v3.(eql_v3.jsonb_path_query_first(col, ''), eql_v3.jsonb_path_query_first(''::eql_v3_json, ''))`. + * + * INTERIM (cipherstash/protectjs-ffi#137): the RHS needle is a STORAGE encryption + * of `{path: value}` — its ste_vec entry carries `c` + `op`/`hm`, which the + * comparison extracts. Once protect-ffi can mint a ciphertext-free ordering query + * needle for a ste_vec column, the RHS drops the ciphertext. + */ + +import type { JsonDocument } from '@cipherstash/stack/eql/v3' +import { EncryptionV3 } from '@cipherstash/stack/v3' +import { databaseUrl, unwrapResult } from '@cipherstash/test-kit' +import { and, asc, eq, type SQL } from 'drizzle-orm' +import { integer, pgTable, text } from 'drizzle-orm/pg-core' +import { drizzle } from 'drizzle-orm/postgres-js' +import postgres from 'postgres' +import { afterAll, beforeAll, describe, expect, it } from 'vitest' +import { + createEncryptionOperatorsV3, + extractEncryptionSchemaV3, + types, +} from '../src/v3/index.js' + +const sqlClient = postgres(databaseUrl(), { prepare: false }) + +const TABLE_NAME = 'protect_ci_v3_json_selector' +const RUN = `run-${Date.now()}-${Math.random().toString(36).slice(2, 8)}` + +const docTable = pgTable(TABLE_NAME, { + id: integer('id').primaryKey().generatedAlwaysAsIdentity(), + rowKey: text('row_key').notNull(), + testRunId: text('test_run_id').notNull(), + doc: types.Json('doc'), +}) + +const schema = extractEncryptionSchemaV3(docTable) + +// Distinct ages so ordering-at-a-selector has a definite expected set. +const DOCS: Record = { + ada: { user: 'ada@example.com', age: 30 }, + grace: { user: 'grace@example.com', age: 20 }, + zoe: { user: 'zoe@example.com', age: 40 }, + // No `$.age` — exercises absent-path semantics (excluded by eq/ordering, + // included by ne). + noage: { user: 'noage@example.com' }, +} + +type SelectRow = { rowKey: string } +let client: Awaited> +let ops: ReturnType +let db: ReturnType + +async function matching(condition: SQL): Promise { + const rows = (await db + .select({ rowKey: docTable.rowKey }) + .from(docTable) + .where(and(eq(docTable.testRunId, RUN), condition)) + .orderBy(asc(docTable.rowKey))) as SelectRow[] + return rows.map((row) => row.rowKey) +} + +beforeAll(async () => { + client = await EncryptionV3({ schemas: [schema] }) + ops = createEncryptionOperatorsV3(client) + db = drizzle({ client: sqlClient }) + + await sqlClient.unsafe(`DROP TABLE IF EXISTS ${TABLE_NAME}`) + await sqlClient.unsafe(` + CREATE TABLE ${TABLE_NAME} ( + id INTEGER PRIMARY KEY GENERATED ALWAYS AS IDENTITY, + row_key TEXT NOT NULL, + test_run_id TEXT NOT NULL, + doc public.eql_v3_json NOT NULL + ) + `) + + const rows = Object.entries(DOCS).map(([rowKey, doc]) => ({ + rowKey, + testRunId: RUN, + doc, + })) + const encrypted = unwrapResult( + await client.bulkEncryptModels(rows, schema), + ) as Array> + await db.insert(docTable).values(encrypted as never) +}, 120000) + +afterAll(async () => { + await sqlClient`DELETE FROM ${sqlClient(TABLE_NAME)} WHERE test_run_id = ${RUN}` + await sqlClient.end() +}, 30000) + +describe('v3 drizzle JSON selector-with-constraint (live pg)', () => { + it('equality at a scalar selector', async () => { + expect( + await matching(await ops.selector(docTable.doc, '$.age').eq(30)), + ).toEqual(['ada']) + }, 30000) + + it('equality at a string selector', async () => { + const condition = await ops + .selector(docTable.doc, '$.user') + .eq('zoe@example.com') + expect(await matching(condition)).toEqual(['zoe']) + }, 30000) + + it('ordering at a selector: greater-than (the form containment cannot express)', async () => { + // ages: ada 30, grace 20, zoe 40 → > 25 selects ada, zoe. + expect( + await matching(await ops.selector(docTable.doc, '$.age').gt(25)), + ).toEqual(['ada', 'zoe']) + }, 30000) + + it('ordering at a selector: less-than', async () => { + expect( + await matching(await ops.selector(docTable.doc, '$.age').lt(35)), + ).toEqual(['ada', 'grace']) + }, 30000) + + it('ordering at a selector: gte inclusive boundary', async () => { + expect( + await matching(await ops.selector(docTable.doc, '$.age').gte(30)), + ).toEqual(['ada', 'zoe']) + }, 30000) + + it('returns nothing when no leaf satisfies the constraint', async () => { + expect( + await matching(await ops.selector(docTable.doc, '$.age').gt(100)), + ).toEqual([]) + }, 30000) + + it('equality excludes rows whose document lacks the path', async () => { + // `noage` has no `$.age`, so it is not equal to 30 → excluded. + expect( + await matching(await ops.selector(docTable.doc, '$.age').eq(30)), + ).toEqual(['ada']) + }, 30000) + + it('ne includes rows whose document lacks the path', async () => { + // ne(30): grace(20), zoe(40), AND noage (no `$.age`) — "not equal to 30" + // covers "has no age". Without the IS NULL arm, noage would be dropped. + expect( + await matching(await ops.selector(docTable.doc, '$.age').ne(30)), + ).toEqual(['grace', 'noage', 'zoe']) + }, 30000) + + it('ordering excludes rows whose document lacks the path', async () => { + // noage has no age → not > 0. + expect( + await matching(await ops.selector(docTable.doc, '$.age').gt(0)), + ).toEqual(['ada', 'grace', 'zoe']) + }, 30000) + + it('rejects a non-scalar leaf value before querying', async () => { + await expect( + ops.selector(docTable.doc, '$.age').eq({ nested: 1 }), + ).rejects.toThrow(/scalar leaf/) + }, 30000) + + it('rejects array/wildcard selector paths (v1 supports object keys only)', async () => { + await expect( + ops.selector(docTable.doc, '$.items[0].name').eq('x'), + ).rejects.toThrow(/not yet supported/) + }, 30000) +}) diff --git a/packages/stack-drizzle/package.json b/packages/stack-drizzle/package.json index 540ce71d..b6685ad7 100644 --- a/packages/stack-drizzle/package.json +++ b/packages/stack-drizzle/package.json @@ -1,6 +1,6 @@ { "name": "@cipherstash/stack-drizzle", - "version": "0.0.0", + "version": "1.0.0-rc.0", "description": "CipherStash Stack Drizzle ORM integration: searchable, application-layer field-level encryption for PostgreSQL.", "keywords": [ "encrypted", diff --git a/packages/stack-drizzle/src/v3/operators.ts b/packages/stack-drizzle/src/v3/operators.ts index e8105202..0e0ed365 100644 --- a/packages/stack-drizzle/src/v3/operators.ts +++ b/packages/stack-drizzle/src/v3/operators.ts @@ -1,8 +1,12 @@ import type { Result } from '@byteslice/result' import type { AuditConfig } from '@cipherstash/stack/adapter-kit' import { + jsonPathOf, matchNeedleError, + parseSelectorSegments, + reconstructSelectorDocument, stripDomainSchema, + unsupportedLeafReason, } from '@cipherstash/stack/adapter-kit' import type { AnyEncryptedV3Column, @@ -66,8 +70,20 @@ type OperandEncryptionClient = { opts: never, ): ChainableOperation encryptQuery(terms: never): ChainableOperation + /** + * Storage encryption — the JSON selector RHS. See {@link selectorCompare}: + * pending a ciphertext-free ordering query needle from protect-ffi + * (cipherstash/protectjs-ffi#137), the right-hand operand is a STORAGE + * encryption of `{path: value}`, whose ste_vec entry at the selector carries + * the `c` + `op`/`hm` the comparison extracts. + */ + encrypt(value: never, opts: never): ChainableOperation } +// Path helpers now live in @cipherstash/stack/adapter-kit (shared with the +// Supabase adapter, #650); re-exported so existing imports keep working. +export { parseSelectorSegments, reconstructSelectorDocument } + /** * A dedicated error for v3 operator gating and operand-encryption failures, * carrying the offending column/table/operator for diagnostics. @@ -571,6 +587,159 @@ export function createEncryptionOperatorsV3( return sql`${JSON.stringify(result.data)}::eql_v3.query_jsonb` } + /** + * JSONPath selector-with-constraint on an `eql_v3_json` (`ste_vec`) column: + * `col->'path' value`. Extracts the encrypted leaf entry at `path` on both + * sides (`v3Dialect.selectorEntry` → `jsonb_path_query_first`) and compares them + * with the `eql_v3_jsonb_entry` comparators; `eq_term`/`ord_term` read only the + * entries' `hm`/`op`. + * + * - the SELECTOR (`path`): `encryptQuery(path, searchableJson)` on a string + * needle infers a `ste_vec_selector` term → the bare HMAC selector hash, the + * `text` argument of `jsonb_path_query_first`. + * - the VALUE (RHS): **interim (cipherstash/protectjs-ffi#137)** — a STORAGE + * `encrypt` of `{path: value}`, whose ste_vec entry carries `c` + `op`/`hm`. + * protect-ffi can't yet mint a ciphertext-free ordering query needle for a + * ste_vec column, so the value's ciphertext appears in the WHERE clause until + * #137 lands; the comparison itself only reads `hm`/`op`. + * + * Equality-at-selector is also expressible via `contains(col, {path: value})`; + * this path's unique power is ORDERING at a selector (`gt`/`gte`/`lt`/`lte`). + */ + async function selectorCompare( + col: SQLWrapper, + path: string, + op: EqualityOp | ComparisonOp, + value: unknown, + operator: string, + opts?: EncryptionOperatorCallOpts, + ): Promise { + const ctx = resolveContext(col, operator) + requireIndex( + ctx, + JSON_CONTAINMENT_INDEXES, + operator, + 'JSON selector (searchableJson)', + ) + requireNonNullOperand(ctx, value, operator) + + // A selector compares a scalar leaf — reject non-scalars / non-orderable + // types up front with a clear error, not a deferred DB failure. + const ordering = op !== 'eq' && op !== 'ne' + const leafReason = unsupportedLeafReason(value, ordering) + if (leafReason) { + throw new EncryptionOperatorError( + `Operator "${operator}" cannot compare column "${ctx.columnName}": ${leafReason}`, + { columnName: ctx.columnName, tableName: ctx.tableName, operator }, + ) + } + + // Surface path-validation failures as EncryptionOperatorError with context. + let segments: string[] + try { + segments = parseSelectorSegments(path) + } catch (err) { + throw new EncryptionOperatorError( + `Operator "${operator}" on column "${ctx.columnName}": ${err instanceof Error ? err.message : String(err)}`, + { columnName: ctx.columnName, tableName: ctx.tableName, operator }, + ) + } + + // INTERIM (cipherstash/protectjs-ffi#137): the RHS is a STORAGE-encrypted + // needle, not a ciphertext-free query term. protect-ffi can't yet mint an + // ordering (`op`) query needle for a ste_vec column — `encryptQuery` only + // produces the `hm` equality term. So we storage-encrypt `{path: value}`, + // whose ste_vec entry at the selector carries `c` + `op`/`hm`, and extract + // that entry on the RHS. `eq_term`/`ord_term` read only `hm`/`op`, so the + // comparison is correct; the tradeoff is the value's ciphertext appears in + // the WHERE clause. Once #137 lands, the RHS becomes a ciphertext-free term. + // + // The selector hash and the storage needle are independent → encrypt + // concurrently (mirrors `range`). + const [selResult, docResult] = await Promise.all([ + applyOperationOptions( + client.encryptQuery( + jsonPathOf(segments) as never, + { + table: ctx.table, + column: ctx.builder, + queryType: 'searchableJson', + } as never, + ), + opts, + ), + applyOperationOptions( + client.encrypt( + reconstructSelectorDocument(segments, value) as never, + { + table: ctx.table, + column: ctx.builder, + } as never, + ), + opts, + ), + ]) + if (selResult.failure) { + throw operandFailure(ctx, operator, selResult.failure.message) + } + if (docResult.failure) { + throw operandFailure(ctx, operator, docResult.failure.message) + } + + // A v3 selector term is the bare HMAC hash string; guard the shape so a + // wrapped envelope can't silently bind as a JSON blob and match no rows. + const selValue = selResult.data + if (typeof selValue !== 'string') { + throw operandFailure( + ctx, + operator, + `expected a bare selector hash, got ${typeof selValue}.`, + ) + } + + const selSql = sql`${selValue}::text` + const leftEntry = v3Dialect.selectorEntry(colSql(col), selSql) + const rightEntry = v3Dialect.selectorEntry( + sql`${JSON.stringify(docResult.data)}::public.eql_v3_json`, + selSql, + ) + if (op === 'eq') return v3Dialect.equality('eq', leftEntry, rightEntry) + if (op === 'ne') { + // Absent-path semantics: a row whose document lacks the path yields a NULL + // entry, and "not equal to X" should INCLUDE "has no X". Without the + // `IS NULL` arm, three-valued logic silently drops those rows. (`eq` and the + // ordering ops keep SQL's default: a missing path is not equal to / not + // greater than the value, so those rows are correctly excluded.) + return sql`(${v3Dialect.equality('ne', leftEntry, rightEntry)} OR ${leftEntry} IS NULL)` + } + return v3Dialect.comparison(op, leftEntry, rightEntry) + } + + /** Comparison methods bound to a `col->'path'` selector, mirroring the scalar + * operators. Async: each encrypts its operand. */ + function selectorOps(col: SQLWrapper, path: string) { + const at = + (op: EqualityOp | ComparisonOp) => + (value: unknown, opts?: EncryptionOperatorCallOpts) => + selectorCompare(col, path, op, value, `selector(${path}).${op}`, opts) + return { + /** `col->'path' = value` (encrypted equality at the selector). A row whose + * document lacks `path` is excluded (it is not equal to `value`). */ + eq: at('eq'), + /** `col->'path' <> value`, INCLUDING rows whose document lacks `path` + * ("not equal to value" covers "has no value"). */ + ne: at('ne'), + /** `col->'path' > value` (encrypted ordering at the selector). */ + gt: at('gt'), + /** `col->'path' >= value`. */ + gte: at('gte'), + /** `col->'path' < value`. */ + lt: at('lt'), + /** `col->'path' <= value`. */ + lte: at('lte'), + } + } + async function inArrayOp( left: SQLWrapper, values: unknown[], @@ -674,6 +843,13 @@ export function createEncryptionOperatorsV3( * a `ste_vec` index (a `types.Json` column). */ contains: (l: SQLWrapper, r: unknown, opts?: EncryptionOperatorCallOpts) => containsJsonOp(l, r, 'contains', opts), + /** JSONPath selector-with-constraint on a `types.Json` (`ste_vec`) column. + * Returns comparison methods bound to `col->'path'` — e.g. + * `await ops.selector(users.doc, '$.age').gt(21)` emits + * `col->'' > `. Its unique power over `contains` is ORDERING at + * a path (`gt`/`gte`/`lt`/`lte`); `eq`/`ne` are also provided. Dot-notation + * object paths only in v1. */ + selector: (l: SQLWrapper, path: string) => selectorOps(l, path), /** Membership: ORs one encrypted `eq` term per value. The whole list is * encrypted in one `encryptQuery` batch crossing. Rejects an empty list; * requires a `unique` or `ore` index. */ diff --git a/packages/stack-drizzle/src/v3/sql-dialect.ts b/packages/stack-drizzle/src/v3/sql-dialect.ts index 38a3a5f1..aac0ae0b 100644 --- a/packages/stack-drizzle/src/v3/sql-dialect.ts +++ b/packages/stack-drizzle/src/v3/sql-dialect.ts @@ -64,6 +64,22 @@ export const v3Dialect = { return sql`${left} OPERATOR(public.@>) ${enc}` }, + /** + * Extract the encrypted JSONB leaf entry at a selector: + * `eql_v3.jsonb_path_query_first(src, sel)` → `eql_v3_jsonb_entry`. `src` is + * either an `eql_v3_json` column or a storage-needle document already cast to + * `eql_v3_json`; `sel` is the selector hash bound as `text`. The returned entry + * feeds `eql_v3.{eq,neq,lt,lte,gt,gte}(jsonb_entry, jsonb_entry)`, so a selector + * comparison is `equality`/`comparison` applied to two extractions (column side + * and needle side) rather than column vs operand. + * + * The root `eql_v3_json` domain has no comparison operators (they're blocked in + * the bundle), which is why the selector must be extracted before comparing. + */ + selectorEntry(source: SQL, selector: SQL): SQL { + return sql`${fn('jsonb_path_query_first')}(${source}, ${selector})` + }, + orderBy(left: SQL, flavour: 'ope' | 'ore'): SQL { // eql-3.0.0 splits the ordering extractor by term flavour: `ord_term` // takes the OPE-backed `_ord` domains (returns eql_v3_internal.ope_cllw), diff --git a/packages/stack-supabase/CHANGELOG.md b/packages/stack-supabase/CHANGELOG.md new file mode 100644 index 00000000..a13b71b8 --- /dev/null +++ b/packages/stack-supabase/CHANGELOG.md @@ -0,0 +1,97 @@ +# @cipherstash/stack-supabase + +## 1.0.0-rc.0 + +### Major Changes + +- 7c7dbca: CipherStash Stack 1.0 (release candidate). + + This is the first 1.0-line release of `@cipherstash/stack`, the first published + release of the split-out EQL v3 adapters `@cipherstash/stack-drizzle` and + `@cipherstash/stack-supabase`, and moves the `stash` CLI to 1.0 alongside them. + These four packages now version together as the Stack 1.0 family. + +### Minor Changes + +- 31ca318: Split the Drizzle and Supabase integrations into their own packages. + + The adapters now ship as first-party packages that depend on `@cipherstash/stack`, + following the `@cipherstash/prisma-next` precedent: + + - **`@cipherstash/stack-drizzle`** — Drizzle ORM integration. EQL v2 on the package + root (`@cipherstash/stack-drizzle`: `encryptedType`, `extractEncryptionSchema`, + `createEncryptionOperators`) and EQL v3 on `@cipherstash/stack-drizzle/v3` + (`types` factories, `createEncryptionOperatorsV3`, `extractEncryptionSchemaV3`, …). + - **`@cipherstash/stack-supabase`** — Supabase integration: `encryptedSupabase` (v2) + and `encryptedSupabaseV3` (v3, connect-time introspection). + + **Breaking (`@cipherstash/stack`):** the `./drizzle`, `./supabase`, and + `./eql/v3/drizzle` subpath exports are removed. Migrate imports: + + - `@cipherstash/stack/drizzle` → `@cipherstash/stack-drizzle` + - `@cipherstash/stack/eql/v3/drizzle` → `@cipherstash/stack-drizzle/v3` + - `@cipherstash/stack/supabase` → `@cipherstash/stack-supabase` + + Add the relevant package to your dependencies alongside `@cipherstash/stack`. A new + `@cipherstash/stack/adapter-kit` subpath exposes the narrow core internals the + first-party adapters consume; it is the core↔adapter seam, not general-purpose API. + +- e40c3da: Rename the EQL v3 encrypted free-text operator `contains()` → `matches()` (#617). + + Encrypted free-text search is fuzzy bloom-filter token matching — order- and + multiplicity-insensitive and one-sided (a `true` may be a false positive) — not + containment. The name `contains()` promised substring/containment semantics it + never had. It is renamed to `matches()` on the encrypted surface; `contains()` is + kept for genuine, exact containment: + + - **Drizzle** (`@cipherstash/stack-drizzle/v3`): `matches()` = bloom free-text on + `text_match`/`text_search` columns; `contains()` = exact encrypted-JSON `@>` on + `types.Json` (ste_vec) columns. + - **Supabase** (`@cipherstash/stack-supabase`): `.matches()` = encrypted free-text; + `.contains()` = native jsonb/array `@>` on plaintext columns (and throws on an + encrypted column, pointing to `matches()`). + + Also on the Supabase v3 surface, `like()`/`ilike()` on an encrypted column are no + longer rejected — they are delegated to `matches()` as a best-effort compatibility + shim. This is APPROXIMATE (fuzzy, case-insensitive, one-sided; anchoring and + wildcards are not honored): surrounding `%` are stripped, an internal `%` or any + `_` is rejected, and a one-time warning is emitted. A plaintext column keeps real + SQL LIKE. + + Breaking: encrypted `contains()` callers must migrate to `matches()`. The + encrypted operator has not shipped in a stable release (it lands via the EQL v3 + work), so there is no deprecation alias. + +### Patch Changes + +- 2fd4985: Populate `EncryptedSupabaseError.encryptionError` on encryption failures (#626). + The query builder's catch block previously hardcoded `encryptionError: undefined`, + so the typed field was always empty and callers had to detect encryption failures + indirectly (via `status`/`statusText` or `.throwOnError()`). It now threads the + underlying `EncryptionError` through — for both the v2 and v3 dialects — when the + failure originates in an encrypt/decrypt step, and leaves it unset for plain + PostgREST/API errors. +- Updated dependencies [31ca318] +- Updated dependencies [c4787c0] +- Updated dependencies [66a0e02] +- Updated dependencies [cfd46ee] +- Updated dependencies [7eba32d] +- Updated dependencies [0ebf57e] +- Updated dependencies [d73a03c] +- Updated dependencies [89b903f] +- Updated dependencies [229ce59] +- Updated dependencies [50c0a9c] +- Updated dependencies [63ca540] +- Updated dependencies [5d23e80] +- Updated dependencies [1aa9a11] +- Updated dependencies [af2d04e] +- Updated dependencies [b8a3d20] +- Updated dependencies [a0f3b2c] +- Updated dependencies [f23f952] +- Updated dependencies [7c7dbca] +- Updated dependencies [5411a13] +- Updated dependencies [99f8b0a] +- Updated dependencies [fd33aad] +- Updated dependencies [8cd485d] +- Updated dependencies [9b65ae8] + - @cipherstash/stack@1.0.0-rc.0 diff --git a/packages/stack-supabase/__tests__/helpers/supabase-mock.ts b/packages/stack-supabase/__tests__/helpers/supabase-mock.ts index a72efd61..26d8576a 100644 --- a/packages/stack-supabase/__tests__/helpers/supabase-mock.ts +++ b/packages/stack-supabase/__tests__/helpers/supabase-mock.ts @@ -36,11 +36,16 @@ export function fakeEnvelope(value: unknown, column: string): FakeEnvelope { : typeof value === 'bigint' ? value.toString() : value + // `String(pt)` throws on a null-prototype object (no toString) — which the + // JSON selector needles deliberately are (prototype-pollution safety). Tag + // the fake ciphertext via JSON for objects instead. + const tag = + pt !== null && typeof pt === 'object' ? JSON.stringify(pt) : String(pt) return { v: 2, i: { t: 'tbl', c: column }, - c: `ct:${String(pt)}`, - hm: `hm:${String(pt)}`, + c: `ct:${tag}`, + hm: `hm:${tag}`, pt, } } diff --git a/packages/stack-supabase/__tests__/supabase-v3-builder.test.ts b/packages/stack-supabase/__tests__/supabase-v3-builder.test.ts index 48391c69..b12d2554 100644 --- a/packages/stack-supabase/__tests__/supabase-v3-builder.test.ts +++ b/packages/stack-supabase/__tests__/supabase-v3-builder.test.ts @@ -1753,18 +1753,19 @@ describe('v3 raw filter() resolves the query type from the operator', () => { ]) }) - // `encryptCollectedTerms` rejects any queryType outside the three scalar EQL - // v3 kinds. No public call path can produce a fourth — `mapFilterOpToQueryType`, + // `encryptCollectedTerms` rejects any queryType outside the four supported EQL + // v3 kinds. No public call path can produce a fifth — `mapFilterOpToQueryType`, // `queryTypeForRawOp` and `queryTypeForOrOp` are exhaustive — so this backstop // is unreachable without breaking the internal contract, which is exactly what // the subclass below does. Keep the guard: it is what a future producer - // gaining a fourth QueryTypeName would trip over. - it('rejects a query type outside the scalar EQL v3 kinds', async () => { + // gaining a new QueryTypeName would trip over. (`searchableJson` used to be + // the exemplar here until #650 made it a real, supported kind.) + it('rejects a query type outside the supported EQL v3 kinds', async () => { const supabase = createMockSupabase() class BogusQueryType extends EncryptedQueryBuilderV3Impl { protected override queryTypeForRawOp(_operator: string) { - return 'searchableJson' as never + return 'steVecSelector' as never } } @@ -1781,8 +1782,8 @@ describe('v3 raw filter() resolves the query type from the operator', () => { .filter('email', 'eq', 'a@b.com') expect(status).toBe(500) - expect(error?.message).toContain('query type "searchableJson"') - expect(error?.message).toContain('not supported on scalar EQL v3 columns') + expect(error?.message).toContain('query type "steVecSelector"') + expect(error?.message).toContain('not supported on EQL v3 columns') }) }) diff --git a/packages/stack-supabase/__tests__/supabase-v3-json.test.ts b/packages/stack-supabase/__tests__/supabase-v3-json.test.ts new file mode 100644 index 00000000..d0122ada --- /dev/null +++ b/packages/stack-supabase/__tests__/supabase-v3-json.test.ts @@ -0,0 +1,314 @@ +/** + * Encrypted-JSON querying on the v3 supabase surface (#650): `contains()` on a + * `types.Json` column (encrypted ste_vec containment), the JSONPath selector + * methods (`selectorEq`/`selectorNe`), and the guards that keep the capability- + * overloaded `cs` wire operator honest (free-text vs containment). + * + * Wire semantics (which overload PostgREST resolves, absent-path `ne` + * inclusion) are proven live in `integration/json.integration.test.ts`; this + * suite pins the ADAPTER's behaviour — operand routing, encryption calls, and + * rejection surfaces — against mocks. + */ + +import { encryptedTable, types } from '@cipherstash/stack/eql/v3' +import { describe, expect, it } from 'vitest' +import { EncryptedQueryBuilderV3Impl } from '../src/query-builder-v3' +import { + createMockEncryptionClient, + createMockSupabase, +} from './helpers/supabase-mock' + +const events = encryptedTable('events', { + payload: types.Json('payload'), + name: types.TextSearch('name'), +}) + +const EVENTS_ALL_COLUMNS = ['id', 'payload', 'name', 'note'] + +function makeBuilder(resultData: unknown = []) { + const supabase = createMockSupabase(resultData) + const encryptionClient = createMockEncryptionClient() + const builder = new EncryptedQueryBuilderV3Impl( + 'events', + events, + encryptionClient as never, + supabase.client as never, + EVENTS_ALL_COLUMNS, + ) + return { supabase, builder } +} + +/** The recorded wire call for a column's containment filter, parsed. */ +function containsCall( + supabase: ReturnType, + method: 'filter' | 'not', +) { + const call = supabase.calls.find((c) => c.method === method) + expect(call).toBeDefined() + return call as { method: string; args: unknown[] } +} + +describe('contains() on an encrypted types.Json column', () => { + it('storage-encrypts the sub-document and emits the cs wire operator', async () => { + const { supabase, builder } = makeBuilder() + await builder.select('id').contains('payload', { user: { role: 'admin' } }) + + const call = containsCall(supabase, 'filter') + expect(call.args[0]).toBe('payload') + expect(call.args[1]).toBe('cs') + // The operand is the JSON.stringify'd storage envelope of the WHOLE + // sub-document — encrypted via encrypt/bulkEncrypt, never encryptQuery. + const envelope = JSON.parse(call.args[2] as string) + expect(envelope.pt).toEqual({ user: { role: 'admin' } }) + expect(envelope.i.c).toBe('payload') + }) + + it('accepts an array sub-document', async () => { + const { supabase, builder } = makeBuilder() + await builder.select('id').contains('payload', [{ tag: 'vip' }]) + const call = containsCall(supabase, 'filter') + expect(JSON.parse(call.args[2] as string).pt).toEqual([{ tag: 'vip' }]) + }) + + it('rejects a scalar operand with a sub-document steer', () => { + const { builder } = makeBuilder() + expect(() => builder.contains('payload', 'admin' as never)).toThrow( + /takes a sub-document/, + ) + expect(() => builder.contains('payload', null as never)).toThrow( + /takes a sub-document/, + ) + }) + + it('rejects empty needles — {} and [] match every row', () => { + const { builder } = makeBuilder() + expect(() => builder.contains('payload', {})).toThrow(/matches every row/) + expect(() => builder.contains('payload', [])).toThrow(/matches every row/) + }) + + it('rejects non-plain objects (Date/Map) that would serialize to scalars or {}', () => { + const { builder } = makeBuilder() + expect(() => builder.contains('payload', new Date() as never)).toThrow( + /takes a sub-document/, + ) + expect(() => + builder.contains('payload', new Map([['a', 1]]) as never), + ).toThrow(/takes a sub-document/) + }) + + it('still rejects contains() on an encrypted TEXT column (that is matches())', () => { + const { builder } = makeBuilder() + expect(() => builder.contains('name', { any: 'thing' })).toThrow( + /native \(exact\) containment.*matches\(\)/s, + ) + }) + + it('still passes contains() through natively on a plaintext column', async () => { + const { supabase, builder } = makeBuilder() + await builder.select('id').contains('note', ['x']) + // Plaintext containment takes the base path: the operand is formatted as a + // native containment literal (never encrypted — no envelope in the operand). + const call = containsCall(supabase, 'filter') + expect(call.args[0]).toBe('note') + expect(call.args[1]).toBe('cs') + expect(String(call.args[2])).not.toContain('pt') + }) +}) + +describe('matches() vs encrypted JSON', () => { + it('rejects matches() on a types.Json column with a contains/selector steer', () => { + const { builder } = makeBuilder() + expect(() => builder.matches('payload', 'admin')).toThrow( + /does not apply to encrypted JSON column .*contains\(.*selectorEq\(/s, + ) + }) +}) + +describe('selectorEq()', () => { + it('reconstructs the path-shaped needle and emits encrypted cs', async () => { + const { supabase, builder } = makeBuilder() + await builder.select('id').selectorEq('payload', '$.user.role', 'admin') + + const call = containsCall(supabase, 'filter') + expect(call.args[1]).toBe('cs') + expect(JSON.parse(call.args[2] as string).pt).toEqual({ + user: { role: 'admin' }, + }) + }) + + it('accepts a bare dot path (no $ prefix)', async () => { + const { supabase, builder } = makeBuilder() + await builder.select('id').selectorEq('payload', 'user.age', 30) + const call = containsCall(supabase, 'filter') + expect(JSON.parse(call.args[2] as string).pt).toEqual({ user: { age: 30 } }) + }) + + it('rejects invalid paths with the shared validation errors', () => { + const { builder } = makeBuilder() + expect(() => builder.selectorEq('payload', '$', 'x')).toThrow( + /addresses no field/, + ) + expect(() => builder.selectorEq('payload', '$.a[0]', 'x')).toThrow( + /array\/wildcard syntax/, + ) + expect(() => builder.selectorEq('payload', '$.a..b', 'x')).toThrow( + /malformed/, + ) + expect(() => builder.selectorEq('payload', '$.a.__proto__', 'x')).toThrow( + /forbidden key/, + ) + }) + + it('rejects non-scalar leaves and null', () => { + const { builder } = makeBuilder() + expect(() => + builder.selectorEq('payload', '$.user', { role: 'admin' } as never), + ).toThrow(/scalar leaf.*contains\(\)/s) + expect(() => + builder.selectorEq('payload', '$.tags', ['vip'] as never), + ).toThrow(/scalar leaf/) + expect(() => + builder.selectorEq('payload', '$.user.role', null as never), + ).toThrow(/non-null scalar leaf.*is\(column, null\)/s) + }) + + it('rejects Date/bigint leaves with the serialization steer (JSON scalars only)', () => { + const { builder } = makeBuilder() + expect(() => + builder.selectorEq('payload', '$.user.joined', new Date() as never), + ).toThrow(/got a Date — pass date\.toISOString\(\)/) + expect(() => + builder.selectorEq('payload', '$.user.balance', 10n as never), + ).toThrow(/JSON scalar.*got bigint/s) + }) + + it('rejects a non-JSON column', () => { + const { builder } = makeBuilder() + expect(() => builder.selectorEq('name', '$.a', 'x')).toThrow( + /requires an encrypted JSON \(types\.Json\) column/, + ) + expect(() => builder.selectorEq('note', '$.a', 'x')).toThrow( + /requires an encrypted JSON \(types\.Json\) column/, + ) + }) +}) + +describe('selectorNe()', () => { + it('emits the IS-NULL-inclusive or: payload.is.null, payload.not.cs.', async () => { + const { supabase, builder } = makeBuilder() + await builder.select('id').selectorNe('payload', '$.user.role', 'admin') + + // NULL-document parity with Drizzle's ne: a bare not.cs would drop rows + // whose payload column is SQL NULL (three-valued logic), so selectorNe + // compiles to a structured OR whose containment arm is encrypted. + const call = supabase.calls.find((c) => c.method === 'or') + expect(call).toBeDefined() + const orString = String(call?.args[0]) + expect(orString).toContain('payload.is.null') + expect(orString).toContain('payload.not.cs.') + // The needle in the or-string is the ENCRYPTED envelope (quote-escaped by + // the or-value formatter), not a plaintext containment literal. + expect(orString).toContain('\\"pt\\"') + expect(orString).toContain('admin') + }) + + it('shares selectorEq validation (path + leaf + column kind)', () => { + const { builder } = makeBuilder() + expect(() => builder.selectorNe('payload', '$.a[*]', 'x')).toThrow( + /array\/wildcard/, + ) + expect(() => builder.selectorNe('name', '$.a', 'x')).toThrow( + /requires an encrypted JSON/, + ) + }) +}) + +describe('the resolver operand boundary (non-method spellings)', () => { + it("raw .filter(col, 'cs', string) throws the sub-document steer, not a silent scalar query", async () => { + // Pre-#650 this failed on capability; the searchableJson resolution must + // not turn it into a silently-empty containment of a JSON string scalar. + const { builder } = makeBuilder() + const { error, status } = await builder + .select('id') + .filter('payload', 'cs', '{"role":"admin"}') + expect(status).toBe(500) + expect(error?.message).toMatch(/takes a sub-document/) + }) + + it("not(col, 'matches', …) on a JSON column throws the steer", () => { + const { builder } = makeBuilder() + expect(() => builder.not('payload', 'matches', 'admin')).toThrow( + /does not apply to encrypted JSON column/, + ) + }) + + it('an .or() string condition with a raw cs term is rejected loudly', async () => { + const { builder } = makeBuilder() + const { error, status } = await builder.select('id').or('payload.cs.admin') + expect(status).toBe(500) + expect(error?.message).toMatch(/takes a sub-document/) + }) + + it('a structured .or() contains condition with a sub-document is encrypted', async () => { + const { supabase, builder } = makeBuilder() + await builder + .select('id') + .or([{ column: 'payload', op: 'contains', value: { a: 1 } }]) + const call = supabase.calls.find((c) => c.method === 'or') + const orString = String(call?.args[0]) + expect(orString).toContain('payload.cs.') + expect(orString).toContain('\\"pt\\"') + }) + + it('a structured .or() contains condition with an empty needle is rejected', async () => { + const { builder } = makeBuilder() + const { error, status } = await builder + .select('id') + .or([{ column: 'payload', op: 'contains', value: {} }]) + expect(status).toBe(500) + expect(error?.message).toMatch(/matches every row/) + }) +}) + +describe('raw filter routing on a JSON column', () => { + it("accepts .filter(col, 'cs', subdoc) — the raw containment spelling", async () => { + const { supabase, builder } = makeBuilder() + await builder.select('id').filter('payload', 'cs', { user: { age: 30 } }) + const call = containsCall(supabase, 'filter') + expect(call.args[1]).toBe('cs') + expect(JSON.parse(call.args[2] as string).pt).toEqual({ user: { age: 30 } }) + }) + + it("rejects scalar ops (.filter(col, 'eq', …)) by capability", async () => { + const { builder } = makeBuilder() + const { error, status } = await builder + .select('id') + .filter('payload', 'eq', { a: 1 }) + expect(status).toBe(500) + expect(error?.message).toMatch(/does not support equality queries/) + }) + + it('not(col, "contains", …) is allowed on a JSON column (exact negated containment)', async () => { + const { supabase, builder } = makeBuilder() + await builder.select('id').not('payload', 'contains', { a: 1 }) + const call = containsCall(supabase, 'not') + expect(call.args[1]).toBe('cs') + // The operand must be the ENCRYPTED envelope — a plaintext containment + // literal here would mean the term skipped encryption entirely. + expect(JSON.parse(call.args[2] as string).pt).toEqual({ a: 1 }) + }) + + it('rejects a whitespace-at-boundary selector path', () => { + const { builder } = makeBuilder() + expect(() => + builder.selectorEq('payload', '$.user .role', 'admin'), + ).toThrow(/whitespace at a segment boundary/) + }) + + it('not(col, "contains", …) stays rejected on an encrypted TEXT column', () => { + const { builder } = makeBuilder() + expect(() => builder.not('name', 'contains', 'x')).toThrow( + /not.*fuzzy free-text matching/s, + ) + }) +}) diff --git a/packages/stack-supabase/__tests__/supabase-v3.test-d.ts b/packages/stack-supabase/__tests__/supabase-v3.test-d.ts index 10ab39b2..51239b44 100644 --- a/packages/stack-supabase/__tests__/supabase-v3.test-d.ts +++ b/packages/stack-supabase/__tests__/supabase-v3.test-d.ts @@ -355,3 +355,79 @@ describe('encryptedSupabaseV3 untyped surface (no schemas)', () => { supabase.from('users').select() }) }) + +// --------------------------------------------------------------------------- +// Encrypted JSON querying (#650): contains on types.Json + selector methods +// --------------------------------------------------------------------------- + +const docs = encryptedTable('docs', { + payload: types.Json('payload'), + email: types.TextEq('email'), +}) + +declare const docsBuilder: EncryptedQueryBuilderV3< + typeof docs, + InferPlaintext & { note: string; meta: Record } +> + +describe('encrypted JSON keys and operands (#650)', () => { + it('contains() accepts a sub-document on the types.Json column', () => { + docsBuilder.contains('payload', { user: { role: 'admin' } }) + docsBuilder.contains('payload', [{ tag: 'vip' }]) + // A raw-string operand is the PLAINTEXT native form only; the encrypted + // JSON overload requires an object/array sub-document. + // @ts-expect-error — string operand is not a sub-document + docsBuilder.contains('payload', '{"user":{}}') + }) + + it('a types.Json column is queryable via its OWN methods, not scalar predicates', () => { + // Pre-#650, QueryTypesForColumn resolved types.Json to never, putting it in + // NonQueryableV3Keys and rejecting EVERY filter mention at compile time. + // Post-#650 the JSON methods are open… + docsBuilder.selectorEq('payload', '$.user.role', 'admin') + docsBuilder.filter('payload', 'cs', { user: { role: 'admin' } }) + docsBuilder.not('payload', 'contains', { user: { role: 'admin' } }) + // …but the scalar predicates stay compile-excluded: an encrypted document + // has no scalar terms, so eq/gt/in against it can only fail at runtime. + // @ts-expect-error — eq on a JSON column has no term to compare + docsBuilder.eq('payload', { a: 1 }) + // @ts-expect-error — gt on a JSON column has no ordering term + docsBuilder.gt('payload', { a: 1 }) + // @ts-expect-error — in-lists have no equality terms on a JSON column + docsBuilder.in('payload', [{ a: 1 }]) + // @ts-expect-error — the raw filter overload for JSON keys admits only 'cs' + docsBuilder.filter('payload', 'eq', { a: 1 }) + }) + + it('selectorEq/selectorNe accept exactly the JSON scalar leaves', () => { + docsBuilder.selectorEq('payload', '$.user.age', 30) + docsBuilder.selectorNe('payload', '$.user.active', true) + docsBuilder.selectorEq('payload', '$.user.role', 'admin') + // @ts-expect-error — a JsonDocument cannot contain a Date; pass toISOString() + docsBuilder.selectorEq('payload', '$.user.joined', new Date()) + // @ts-expect-error — a JsonDocument cannot contain a bigint + docsBuilder.selectorEq('payload', '$.user.balance', 10n) + // @ts-expect-error — objects are contains(), not a selector leaf + docsBuilder.selectorEq('payload', '$.user', { role: 'admin' }) + // @ts-expect-error — null is not a comparable leaf + docsBuilder.selectorNe('payload', '$.user.role', null) + }) + + it('selector methods reject non-JSON columns at compile time', () => { + // @ts-expect-error — email is a TextEq column, not types.Json + docsBuilder.selectorEq('email', '$.a', 'x') + // @ts-expect-error — note is a plaintext column + docsBuilder.selectorNe('note', '$.a', 'x') + }) + + it('matches() still rejects the JSON column at compile time', () => { + // @ts-expect-error — payload carries searchableJson, not freeTextSearch + docsBuilder.matches('payload', 'admin') + }) + + it('plaintext contains() is unaffected', () => { + docsBuilder.contains('meta', { a: 1 }) + // @ts-expect-error — scalar plaintext column has no containment operand + docsBuilder.contains('note', 'x') + }) +}) diff --git a/packages/stack-supabase/integration/json.integration.test.ts b/packages/stack-supabase/integration/json.integration.test.ts new file mode 100644 index 00000000..1c502b48 --- /dev/null +++ b/packages/stack-supabase/integration/json.integration.test.ts @@ -0,0 +1,236 @@ +/** + * Live encrypted-JSON querying for the v3 supabase adapter (#650): real crypto, + * real PostgREST (as `anon`), real `public.eql_v3_json` domain. + * + * What this uniquely proves (the unit suite runs against mocks): + * - `contains()` reaches the `eql_v3."@>"(eql_v3_json, eql_v3_json)` overload + * through PostgREST's column-type cast of the `cs.` operand, with a real + * storage-encrypted needle (probe-verified wire form; see #650). + * - `selectorEq`/`selectorNe` — equality-at-a-path compiled to containment of + * the reconstructed needle — return the right ROWS, including the documented + * `ne` semantics: rows whose document lacks the path entirely are INCLUDED. + * - The `anon` grants cover the ste_vec functions the operator expands to. + * + * Selector ORDERING is deliberately absent: not expressible over PostgREST + * until cipherstash/encrypt-query-language#407 lands. Drizzle's + * `json-selector.integration.test.ts` proves ordering semantics. + */ + +import type { JsonDocument } from '@cipherstash/stack/eql/v3' +import { encryptedTable, types } from '@cipherstash/stack/eql/v3' +import { databaseUrl } from '@cipherstash/test-kit' +import postgres from 'postgres' +import { afterAll, beforeAll, describe, expect, it } from 'vitest' +import { encryptedSupabaseV3 } from '../src/index.js' +import { makePostgrestClient, reloadSchemaCache } from './helpers/pgrest' + +const TABLE = 'protect_ci_v3_supabase_json' +const RUN = `run-${Date.now()}-${Math.random().toString(36).slice(2, 8)}` + +const sql = postgres(databaseUrl(), { prepare: false }) + +const docs = encryptedTable(TABLE, { + payload: types.Json('payload'), +}) + +const DOCS: Record = { + ada: { user: { email: 'ada@example.com', role: 'admin' }, age: 30 }, + grace: { user: { email: 'grace@example.com', role: 'admin' }, age: 20 }, + zoe: { user: { email: 'zoe@example.com', role: 'analyst' }, age: 40 }, + // No `$.age` and no `$.user.role` — exercises absent-path semantics + // (excluded by contains/selectorEq, INCLUDED by selectorNe). + norole: { user: { email: 'norole@example.com' } }, + // ARRAY-VALUED path: pins the OBSERVED ste_vec semantics — a scalar-leaf + // needle does NOT match an array at the path (protect-ffi encodes array + // elements under their own selectors, not the parent path's), so selectorEq + // treats an array leaf as not-equal and selectorNe includes it. The docs + // carry this caveat; this fixture keeps it honest against the real FFI+EQL. + multi: { user: { email: 'multi@example.com', role: ['admin', 'analyst'] } }, +} + +/** A row whose payload COLUMN is SQL NULL — the maximally-absent document. + * selectorNe must include it (the `is.null` arm of its or-compilation). */ +const NULL_ROW_KEY = 'nulldoc' + +type Instance = Awaited> +let instance: Instance + +// `payload` is nullable: the suite seeds a SQL-NULL document row (the +// selectorNe is.null arm), so the type must not promise non-null. +type Row = { + row_key: string + payload: JsonDocument | null + test_run_id: string +} + +function from() { + return instance.from(TABLE).select('row_key').eq('test_run_id', RUN) +} + +async function rowKeys( + q: PromiseLike<{ data: unknown; error: { message: string } | null }>, +): Promise { + const { data, error } = await q + if (error) throw new Error(error.message) + return ((data as { row_key: string }[]) ?? []).map((r) => r.row_key).sort() +} + +beforeAll(async () => { + await sql.unsafe(` + CREATE TABLE IF NOT EXISTS public.${TABLE} ( + id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY, + row_key text NOT NULL, + test_run_id text NOT NULL, + payload public.eql_v3_json + ) + `) + await sql.unsafe( + `GRANT SELECT, INSERT ON public.${TABLE} TO anon, authenticated`, + ) + await reloadSchemaCache(sql, TABLE) + + instance = await encryptedSupabaseV3(makePostgrestClient(), { + schemas: { [TABLE]: docs } as never, + databaseUrl: databaseUrl(), + }) + + const models = Object.entries(DOCS).map(([rowKey, payload]) => ({ + row_key: rowKey, + test_run_id: RUN, + payload, + })) + const { error } = await instance.from(TABLE).insert(models) + if (error) throw new Error(`seed insert: ${error.message}`) + + // NULL-payload row: inserted directly (the adapter's insert path encrypts + // model values; a SQL NULL column needs no encryption). + await sql.unsafe( + `INSERT INTO public.${TABLE} (row_key, test_run_id, payload) VALUES ('${NULL_ROW_KEY}', '${RUN}', NULL)`, + ) +}, 120_000) + +afterAll(async () => { + await sql.unsafe(`DELETE FROM public.${TABLE} WHERE test_run_id = '${RUN}'`) + await sql.end() +}) + +describe('decrypt round-trip (the read path)', () => { + it('a filtered row decrypts back to the original document', async () => { + const { data, error } = await instance + .from(TABLE) + .select('row_key, payload') + .eq('test_run_id', RUN) + .selectorEq('payload', '$.user.role', 'analyst') + if (error) throw new Error(error.message) + expect(data).toHaveLength(1) + expect(data?.[0].payload).toEqual(DOCS.zoe) + }) +}) + +describe('encrypted containment (contains)', () => { + it('matches rows containing a nested sub-document', async () => { + expect( + await rowKeys(from().contains('payload', { user: { role: 'admin' } })), + ).toEqual(['ada', 'grace']) + }) + + it('not(col, "contains", …) is the bare negated containment (excludes the NULL row)', async () => { + expect( + await rowKeys(from().not('payload', 'contains', { age: 30 })), + ).toEqual(['grace', 'multi', 'norole', 'zoe']) + }) + + it('a multi-leaf needle requires EVERY leaf to match', async () => { + expect( + await rowKeys( + from().contains('payload', { + user: { role: 'admin' }, + age: 30, + }), + ), + ).toEqual(['ada']) + }) + + it('matches nothing for an absent value', async () => { + expect(await rowKeys(from().contains('payload', { age: 99 }))).toEqual([]) + }) + + it('raw .filter(col, "cs", subdoc) is the same query', async () => { + expect(await rowKeys(from().filter('payload', 'cs', { age: 40 }))).toEqual([ + 'zoe', + ]) + }) +}) + +describe('selector equality (selectorEq)', () => { + it('matches the row carrying the value at the path', async () => { + expect(await rowKeys(from().selectorEq('payload', '$.age', 30))).toEqual([ + 'ada', + ]) + }) + + it('works on nested paths and string leaves', async () => { + expect( + await rowKeys(from().selectorEq('payload', '$.user.role', 'analyst')), + ).toEqual(['zoe']) + }) + + it('ARRAY-LEAF: a scalar needle does NOT match an array at the path', async () => { + // multi.user.role = ['admin','analyst'] is NOT matched by the scalar + // needle 'admin' — array elements are encoded under their own selectors. + expect( + await rowKeys(from().selectorEq('payload', '$.user.role', 'admin')), + ).toEqual(['ada', 'grace']) + }) + + it('ARRAY-LEAF: the full array as a contains() needle matches the row', async () => { + expect( + await rowKeys( + from().contains('payload', { user: { role: ['admin', 'analyst'] } }), + ), + ).toEqual(['multi']) + }) + + it('an absent path matches nothing', async () => { + expect( + await rowKeys(from().selectorEq('payload', '$.user.missing', 'x')), + ).toEqual([]) + }) +}) + +describe('selector inequality (selectorNe)', () => { + it('includes different-value rows, absent-path rows, AND SQL-NULL documents', async () => { + // Drizzle-parity semantics: NOT-contains OR IS NULL. `norole` lacks the + // path; `nulldoc` lacks the whole document (the case a bare not.cs drops + // under three-valued logic); `multi` is included because its role ARRAY is + // not-equal to the scalar needle (the array-leaf caveat). + expect( + await rowKeys(from().selectorNe('payload', '$.user.role', 'admin')), + ).toEqual(['multi', 'norole', NULL_ROW_KEY, 'zoe']) + }) + + it('ne against an absent-everywhere value returns every row', async () => { + expect(await rowKeys(from().selectorNe('payload', '$.age', 99))).toEqual([ + 'ada', + 'grace', + 'multi', + 'norole', + NULL_ROW_KEY, + 'zoe', + ]) + }) +}) + +describe('guards hold on the live surface', () => { + it('matches() on the JSON column throws the steer', () => { + expect(() => from().matches('payload', 'admin')).toThrow( + /encrypted JSON column/, + ) + }) + + it('scalar ops on the JSON column are rejected by capability', async () => { + const { error, status } = await from().filter('payload', 'eq', { a: 1 }) + expect(status).toBe(500) + expect(error?.message).toMatch(/does not support equality/) + }) +}) diff --git a/packages/stack-supabase/package.json b/packages/stack-supabase/package.json index 00835851..91470564 100644 --- a/packages/stack-supabase/package.json +++ b/packages/stack-supabase/package.json @@ -1,6 +1,6 @@ { "name": "@cipherstash/stack-supabase", - "version": "0.0.0", + "version": "1.0.0-rc.0", "description": "CipherStash Stack Supabase integration: transparent, searchable field-level encryption for Supabase.", "keywords": [ "encrypted", diff --git a/packages/stack-supabase/src/helpers.ts b/packages/stack-supabase/src/helpers.ts index d3469741..7d2ae5fc 100644 --- a/packages/stack-supabase/src/helpers.ts +++ b/packages/stack-supabase/src/helpers.ts @@ -237,9 +237,11 @@ export function mapFilterOpToQueryType(op: FilterOp): QueryTypeName { return 'equality' case 'like': case 'ilike': - // `matches` is the encrypted free-text (bloom) operator. `contains` remains - // for completeness/v2 but is plaintext-only on the v3 surface, so it never - // reaches term encryption there (a plaintext operand is not encrypted). + // `matches` is the encrypted free-text (bloom) operator. `contains` is + // plaintext-native on scalar columns, but on an encrypted `types.Json` + // column it IS the encrypted ste_vec containment (#650) — the v3 dialect's + // capability resolver re-types the collected term to `searchableJson` when + // the column carries that capability instead of `freeTextSearch`. case 'contains': case 'matches': return 'freeTextSearch' diff --git a/packages/stack-supabase/src/query-builder-v3.ts b/packages/stack-supabase/src/query-builder-v3.ts index 608661ef..781ae50b 100644 --- a/packages/stack-supabase/src/query-builder-v3.ts +++ b/packages/stack-supabase/src/query-builder-v3.ts @@ -2,6 +2,9 @@ import { DATE_LIKE_CASTS, EncryptedV3Column, logger, + parseSelectorSegments, + reconstructSelectorDocument, + unsupportedLeafReason, } from '@cipherstash/stack/adapter-kit' import type { EncryptionClient } from '@cipherstash/stack/encryption' import type { AnyV3Table } from '@cipherstash/stack/eql/v3' @@ -50,10 +53,56 @@ type V3ColumnLike = { equality: boolean orderAndRange: boolean freeTextSearch: boolean + /** Optional: only `public.eql_v3_json` (`types.Json`) carries it. */ + searchableJson?: boolean } build(): ColumnSchema } +/** + * Validate an encrypted-JSON containment operand: a NON-EMPTY plain object or a + * non-empty array. Everything else is rejected with an actionable steer: + * + * - Scalars/strings: the caller meant free-text (`matches` on a text column) or + * a selector — a raw JSON string is NOT parsed, by design (parsing would make + * `'{"a":1}'` and `{a:1}` silently different queries on other surfaces). + * - Non-plain objects (`Date`, `Map`, `RegExp`, class instances): these JSON- + * serialize to scalars or `{}` — not the sub-document the caller believes. + * - `{}` and `[]`: jsonb containment holds for EVERY document (`doc @> '{}'`), + * so an accidentally-empty needle would silently return (and decrypt) the + * whole table. The Drizzle adapter rejects the same needle for the same + * reason — the two first-party adapters must agree that this is an error. + */ +function assertJsonContainmentOperand(column: string, value: unknown): void { + const isPlainObject = + value !== null && + typeof value === 'object' && + !Array.isArray(value) && + (Object.getPrototypeOf(value) === Object.prototype || + Object.getPrototypeOf(value) === null) + if (!isPlainObject && !Array.isArray(value)) { + // Array.isArray is false on this branch by construction, so the label only + // distinguishes null / non-plain object / scalar. + const got = + value === null + ? 'null' + : typeof value === 'object' + ? (value as object).constructor?.name || 'a non-plain object' + : typeof value + throw new Error( + `[supabase v3]: encrypted JSON containment on column "${column}" takes a sub-document (plain object or array) to match, got ${got}.`, + ) + } + const empty = Array.isArray(value) + ? value.length === 0 + : Object.keys(value as object).length === 0 + if (empty) { + throw new Error( + `[supabase v3]: encrypted JSON containment on column "${column}" cannot take an empty ${Array.isArray(value) ? 'array' : 'object'} needle: it matches every row. Pass a non-empty sub-document, or omit the predicate to select all rows.`, + ) + } +} + /** * Reject a declared property name that is also a DIFFERENT physical column. * @@ -402,19 +451,44 @@ export class EncryptedQueryBuilderV3Impl< */ private assertTermQueryable(term: ScalarQueryTerm): V3ColumnLike { const column = term.column as unknown as V3ColumnLike - const queryType = term.queryType ?? 'equality' + let queryType = term.queryType ?? 'equality' + const capabilities = column.getQueryCapabilities() + + // The `cs` wire operator is capability-overloaded: bloom free-text on a + // match/search TEXT column, encrypted ste_vec containment on a `types.Json` + // DOCUMENT column. Both arrive here as `freeTextSearch` (contains/matches/ + // raw `cs` all map there); resolve to the capability the column actually + // carries. The two are mutually exclusive by construction, so this can + // never reinterpret a real free-text column. + if ( + queryType === 'freeTextSearch' && + !capabilities.freeTextSearch && + capabilities.searchableJson + ) { + queryType = 'searchableJson' + // THE single enforced operand boundary for encrypted-JSON containment. + // Terms reach this resolver from every spelling — contains(), raw + // .filter(col,'cs',…), not(col,'contains'|'matches',…), and .or() + // string/structured conditions — and only contains() has a method-level + // guard. Without this check a raw string (e.g. a free-text term ported + // from a text column, or an .or() condition value, which is always a + // string) would be storage-encrypted as a JSON SCALAR and silently match + // nothing; pre-#650 every such spelling failed loudly on capability. + assertJsonContainmentOperand(column.getName(), term.value) + } if ( queryType !== 'equality' && queryType !== 'orderAndRange' && - queryType !== 'freeTextSearch' + queryType !== 'freeTextSearch' && + queryType !== 'searchableJson' ) { throw new Error( - `[supabase v3]: query type "${queryType}" is not supported on scalar EQL v3 columns`, + `[supabase v3]: query type "${queryType}" is not supported on EQL v3 columns`, ) } - if (!column.getQueryCapabilities()[queryType]) { + if (!capabilities[queryType]) { throw new Error( `[supabase v3]: column "${column.getName()}" (${column.getEqlType()}) does not support ${queryType} queries — declare the column with a domain that carries that capability`, ) @@ -571,13 +645,27 @@ export class EncryptedQueryBuilderV3Impl< return Boolean(this.v3Columns[column]) } + /** True when `column` is an encrypted `types.Json` document column. */ + private isSearchableJsonColumn(column: string): boolean { + const builder: V3ColumnLike | undefined = this.v3Columns[column] + return Boolean(builder?.getQueryCapabilities().searchableJson) + } + /** - * `contains` on the v3 surface is EXACT containment only — native jsonb/array - * `@>` on a plaintext column. On an encrypted match/search column containment - * is not the operation (that is the fuzzy `matches`), so refuse loudly rather + * `contains` on the v3 surface is EXACT containment: native jsonb/array `@>` + * on a plaintext column, ENCRYPTED ste_vec `@>` on a `types.Json` column (the + * sub-document operand is storage-encrypted whole; every leaf must match at + * its path — #650). On an encrypted match/search TEXT column containment is + * not the operation (that is the fuzzy `matches`), so refuse loudly rather * than silently emit a bloom match under a name that promises exactness. */ override contains(column: string, value: unknown): this { + if (this.isSearchableJsonColumn(column)) { + // Same validator the term resolver enforces — failing here just surfaces + // the error at the call site instead of at execution. + assertJsonContainmentOperand(column, value) + return super.contains(column, value) + } if (this.isEncryptedV3Column(column)) { throw new Error( `[supabase v3]: contains() is native (exact) containment and does not apply to encrypted column "${column}". Use matches() for encrypted free-text search.`, @@ -590,9 +678,18 @@ export class EncryptedQueryBuilderV3Impl< * `matches` is the encrypted free-text operator: fuzzy bloom-filter token * matching, one-sided (may false-positive), NOT containment. It requires an * encrypted match/search column; on a plaintext column, `contains` (native - * `@>`) is what the caller means. + * `@>`) is what the caller means — and on an encrypted JSON column, + * `contains`/`selectorEq` are (matching a document is containment, not + * free-text). Guarded here because both spellings collect the same + * `freeTextSearch` term, which the capability resolver would otherwise + * silently accept as containment of the raw string. */ override matches(column: string, value: unknown): this { + if (this.isSearchableJsonColumn(column)) { + throw new Error( + `[supabase v3]: matches() is encrypted free-text search and does not apply to encrypted JSON column "${column}". Use contains("${column}", subDocument) or selectorEq("${column}", path, value).`, + ) + } if (!this.isEncryptedV3Column(column)) { throw new Error( `[supabase v3]: matches() is encrypted free-text search and requires an encrypted column; "${column}" is not one. Use contains() for native containment.`, @@ -602,22 +699,122 @@ export class EncryptedQueryBuilderV3Impl< } /** - * `not(col, 'contains', …)` on an encrypted column would negate a fuzzy bloom - * match under the `contains` name — the exact confusion #617 removes — because - * the base `not()` path rewrites the `contains` spelling to the `cs` wire - * operator. Reject it and steer to the `matches` spelling (or the raw `cs` - * operator, which is honest about the wire op). Plaintext columns keep native - * negated containment, and every other operator is delegated unchanged. + * `not(col, 'contains', …)` on an encrypted TEXT column would negate a fuzzy + * bloom match under the `contains` name — the exact confusion #617 removes — + * because the base `not()` path rewrites the `contains` spelling to the `cs` + * wire operator. Reject it and steer to the `matches` spelling (or the raw + * `cs` operator, which is honest about the wire op). + * + * On an encrypted JSON column negated containment IS the honest exact + * operation (`not.cs` over ste_vec containment — {@link selectorNe} compiles + * to it), so it passes through. Plaintext columns keep native negated + * containment, and every other operator is delegated unchanged. */ override not(column: string, operator: string, value: unknown): this { - if (operator === 'contains' && this.isEncryptedV3Column(column)) { + if ( + operator === 'contains' && + this.isEncryptedV3Column(column) && + !this.isSearchableJsonColumn(column) + ) { throw new Error( `[supabase v3]: not("${column}", 'contains', …) does not apply to encrypted column "${column}" — that is fuzzy free-text matching, not containment. Use not("${column}", 'matches', …) or the raw 'cs' operator.`, ) } + // Mirror of the matches() guard: a `matches` spelling on a JSON column + // would otherwise resolve to containment (the two share the `cs` wire op), + // silently negating an EXACT operation under a name that promises FUZZY. + if (operator === 'matches' && this.isSearchableJsonColumn(column)) { + throw new Error( + `[supabase v3]: not("${column}", 'matches', …) does not apply to encrypted JSON column "${column}" — matches() is free-text search. Use not("${column}", 'contains', subDocument) or selectorNe("${column}", path, value).`, + ) + } return super.not(column, operator, value) } + /** + * Validate + reconstruct a selector needle: `('$.user.role', 'admin')` → + * `{user: {role: 'admin'}}`. Shared by {@link selectorEq}/{@link selectorNe}; + * throws with column context for a non-JSON column, an invalid path, or a + * non-scalar leaf. + */ + private selectorNeedle( + method: string, + column: string, + path: string, + value: unknown, + ): Record { + if (!this.isSearchableJsonColumn(column)) { + throw new Error( + `[supabase v3]: ${method}() requires an encrypted JSON (types.Json) column; "${column}" is not one.`, + ) + } + // Selector comparisons compare a scalar LEAF (null included in the shared + // helper's rejection; eq/ne arm — `ordering: false`; + // PostgREST cannot express selector ordering yet, see + // cipherstash/encrypt-query-language#407). + const leafReason = unsupportedLeafReason(value, false) + if (leafReason) { + throw new Error( + `[supabase v3]: ${method}("${column}", "${path}", …): ${leafReason}`, + ) + } + // Stricter than the shared helper (whose Date/bigint arms serve the Drizzle + // surface): a stored JsonDocument leaf is a JSON scalar, so a Date/bigint + // needle could never match one — reject with the serialization steer + // instead of running a query that structurally returns nothing. + if ( + typeof value !== 'string' && + typeof value !== 'number' && + typeof value !== 'boolean' + ) { + throw new Error( + `[supabase v3]: ${method}("${column}", "${path}", …): a JSON document leaf is a JSON scalar (string/number/boolean); got ${value instanceof Date ? 'a Date — pass date.toISOString() (or the stored form)' : typeof value}.`, + ) + } + let segments: string[] + try { + segments = parseSelectorSegments(path) + } catch (err) { + throw new Error( + `[supabase v3]: ${method}("${column}", …): ${err instanceof Error ? err.message : String(err)}`, + ) + } + return reconstructSelectorDocument(segments, value) + } + + /** + * Encrypted JSONPath-selector equality: matches rows whose document carries + * exactly `value` at `path`. Equality at a path IS containment of the + * path-shaped needle (`{user: {role: 'admin'}}`), so this compiles to + * {@link contains} — the ste_vec entry at the selector matches on its + * equality/ordering term. Selector ORDERING (`gt`/`lt`/…) is not expressible + * over PostgREST until the bundle grows a needle-comparison overload + * (cipherstash/encrypt-query-language#407); the Drizzle adapter's + * `ops.selector()` supports it today. + */ + selectorEq(column: string, path: string, value: unknown): this { + const needle = this.selectorNeedle('selectorEq', column, path, value) + return super.contains(column, needle) + } + + /** + * Encrypted JSONPath-selector inequality: rows whose document does NOT carry + * `value` at `path` — INCLUDING rows where the path is absent AND rows whose + * document column is SQL NULL, matching the Drizzle selector's `ne` (whose + * `OR entry IS NULL` arm covers both absence cases). A bare `not.cs` would + * drop NULL documents under three-valued logic (`NOT (NULL @> x)` is NULL), + * so this compiles to a structured OR: + * `column.is.null, column.not.cs.` — the containment condition's + * operand is encrypted through the normal or-condition term path. + */ + selectorNe(column: string, path: string, value: unknown): this { + const needle = this.selectorNeedle('selectorNe', column, path, value) + return super.or([ + { column, op: 'is', value: null }, + { column, op: 'contains', negate: true, value: needle }, + ]) + } + /** * `like`/`ilike` on an ENCRYPTED column are a best-effort compatibility shim, * delegated to `matches`. EQL v3 free-text search is fuzzy bloom token @@ -693,6 +890,12 @@ export class EncryptedQueryBuilderV3Impl< */ protected override queryTypeForOrOp(op: FilterOp): QueryTypeName { if (op === 'matches') return 'freeTextSearch' + // Structured conditions may carry the `contains` METHOD spelling (the wire + // token becomes `cs` in rebuildOrString). It maps to the same capability + // gate as `cs`; on a JSON column the term resolver then re-types it to + // searchableJson and validates the operand. selectorNe's IS-NULL-inclusive + // or-form relies on this arm. + if (op === 'contains') return 'freeTextSearch' return this.queryTypeForRawOp(op) } diff --git a/packages/stack-supabase/src/types.ts b/packages/stack-supabase/src/types.ts index df5fe786..c29165cd 100644 --- a/packages/stack-supabase/src/types.ts +++ b/packages/stack-supabase/src/types.ts @@ -91,14 +91,43 @@ export type NonQueryableV3Keys = { }[Extract, string>] /** - * Row keys a v3 builder accepts in filter methods: every row key except the - * table's storage-only encrypted columns. Plaintext (non-schema) columns pass - * through untouched, exactly as in v2. + * The capability names a SCALAR predicate (`eq`/`neq`/`gt`/`gte`/`lt`/`lte`/ + * `in`) can exercise. `searchableJson` is deliberately absent: an encrypted + * JSON document has no scalar terms, so a scalar predicate against it can only + * ever fail at runtime. + */ +type ScalarQueryTypeName = 'equality' | 'orderAndRange' | 'freeTextSearch' + +/** + * JS property names of a v3 table's columns that support NO scalar predicate: + * storage-only columns (no capability at all) AND `types.Json` columns (whose + * only capability is `searchableJson` — containment/selector querying, reached + * via `contains()`/`selectorEq()`/`selectorNe()` or the dedicated + * `filter(col, 'cs', …)` / `not(col, 'contains', …)` overloads, never via a + * scalar predicate). + */ +type NonScalarQueryableV3Keys
= { + [K in Extract, string>]: [ + Extract< + QueryTypesForColumn[K]>, + ScalarQueryTypeName + >, + ] extends [never] + ? K + : never +}[Extract, string>] + +/** + * Row keys a v3 builder accepts in SCALAR filter methods: every row key except + * the table's encrypted columns with no scalar capability (storage-only + * columns, and `types.Json` documents — see {@link NonScalarQueryableV3Keys}; + * before #650's `searchableJson` arm the two sets coincided). Plaintext + * (non-schema) columns pass through untouched, exactly as in v2. */ export type V3FilterableKeys< Table extends AnyV3Table, Row extends Record, -> = Exclude, NonQueryableV3Keys
> +> = Exclude, NonScalarQueryableV3Keys
> /** * JS property names of a v3 table's columns that carry NO `freeTextSearch` @@ -145,6 +174,57 @@ export type V3EncryptedFreeTextKeys< > & Extract +/** + * JS property names of a v3 table's columns that carry NO `searchableJson` + * capability — i.e. every domain but `public.eql_v3_json`. Mirror of + * {@link NonFreeTextSearchV3Keys} for the encrypted-JSON capability. + */ +type NonSearchableJsonV3Keys
= { + [K in Extract< + keyof V3ColumnsOfTable
, + string + >]: 'searchableJson' extends QueryTypesForColumn[K]> + ? never + : K +}[Extract, string>] + +/** + * Row keys the encrypted-JSON query methods accept (`contains()` on an + * encrypted column, `selectorEq()`, `selectorNe()`): ONLY the table's ENCRYPTED + * columns whose domain is `public.eql_v3_json` (`types.Json`). Plaintext + * columns are excluded — on those, `contains()` is PostgREST-native containment + * and the selector methods do not apply. Mirror of + * {@link V3EncryptedFreeTextKeys} for the `searchableJson` capability. + */ +export type V3SearchableJsonKeys< + Table extends AnyV3Table, + Row extends Record, +> = Exclude< + Extract, string>, + NonSearchableJsonV3Keys
+> & + Extract + +/** + * The operand `contains()` accepts on an ENCRYPTED `types.Json` column: a + * sub-document (object or array). The whole operand is storage-encrypted + * against the column and compared via encrypted ste_vec containment — never a + * raw PostgREST string form, which cannot be encrypted. + */ +export type EncryptedJsonContainsValue = + | Record + | readonly unknown[] + +/** + * The scalar leaf value the selector methods compare at a JSONPath: exactly the + * JSON scalar kinds a stored {@link import('@cipherstash/stack/eql/v3').JsonValue} + * can carry. `Date` and `bigint` are deliberately absent — a `JsonDocument` + * cannot contain them, so such a needle could never match a stored leaf + * (serialize to the stored form first, e.g. `date.toISOString()`). Objects and + * arrays are rejected too (that shape is `contains()`). + */ +export type SelectorLeafValue = string | number | boolean + /** * The operand `contains()` accepts on a PLAINTEXT column, mirroring * postgrest-js's own untyped `contains` overload: a jsonb literal, an array, or @@ -172,46 +252,6 @@ type PlaintextContainsValue = V extends readonly unknown[] ? V | string : never -/** - * The `contains()` operand for column `K`. - * - * A DECLARED encrypted column reaching `contains` necessarily carries a - * `freeTextSearch` capability — only `public.eql_v3_text_match` and - * `public.eql_v3_text_search` do, and both `cast_as` to `string` — so its operand is - * the string to tokenize into a bloom-filter query term. - * - * Any other key is a plaintext passthrough, where `contains` means PostgREST's - * native jsonb/array containment and the operand follows the column - * ({@link PlaintextContainsValue}). A blanket `value: string` made that half of - * {@link V3FreeTextSearchableKeys} unreachable from TypeScript; a blanket - * {@link NativeContainsValue} then over-corrected, admitting an array on a - * plaintext scalar. - * - * A UNION key is only as permissive as its strictest member: if ANY member is a - * declared column, the operand is the string term. `[K] extends [declared]` gets - * this backwards — a mixed `'email' | 'tags'` fails that test and falls to the - * plaintext branch, so an array typechecks for a key that may resolve to the - * encrypted `email` at runtime. Nothing downstream catches it: the operand goes - * straight to `encrypt()`, which has no plaintext-type guard. - * - * So test the INTERSECTION instead, wrapped in a tuple to stop the naked type - * parameter distributing (which would rebuild the same permissive union). - * - * `PlaintextContainsValue` DOES distribute over `Row[K]`, which is safe only - * because the tuple guard above has already excluded every encrypted member: a - * union of plaintext keys (`'tags' | 'meta'`) must accept the operands of each. - * The residual imprecision is a plaintext scalar unioned with a container column - * (`'note' | 'tags'`), where an array still typechecks — and yields a loud 42883 - * rather than a silent mis-encryption. - */ -export type V3ContainsValue< - Table extends AnyV3Table, - Row extends Record, - K extends string, -> = [Extract, string>>] extends [never] - ? PlaintextContainsValue - : string - /** * JS property names of a v3 table's columns that `order()` cannot sort by. Two * cases: @@ -310,6 +350,70 @@ export interface EncryptedQueryBuilderV3< column: K, value: PlaintextContainsValue, ): EncryptedQueryBuilderV3 + /** ENCRYPTED (exact) JSON containment on a `types.Json` column: the + * sub-document operand is storage-encrypted against the column and compared + * via the encrypted ste_vec `@>`. Every leaf of the operand must match at its + * path. Note the operand's ciphertext appears in the request's filter string + * (the same tradeoff as every v3 filter operand — see the class doc). */ + contains & StringKeyOf>( + column: K, + value: EncryptedJsonContainsValue, + ): EncryptedQueryBuilderV3 + /** Encrypted JSONPath-selector equality on a `types.Json` column: + * `selectorEq('doc', '$.user.role', 'admin')` matches rows whose document + * carries that value at that path, compiled to encrypted containment of the + * reconstructed `{user: {role: 'admin'}}` needle. ARRAY-LEAF CAVEAT (pinned + * by the live integration suite): an ARRAY at the path does NOT match a + * scalar needle — ste_vec encodes array elements under their own selectors — + * so `{a:[40,30]}` is NOT matched by `selectorEq('$.a', 30)`; to match an + * array-valued path, pass the full array through `contains()`. Paths are + * dot-notation object keys (`'$.a.b'`); array/wildcard steps are + * rejected. */ + selectorEq & StringKeyOf>( + column: K, + path: string, + value: SelectorLeafValue, + ): EncryptedQueryBuilderV3 + /** Encrypted JSONPath-selector inequality. Matches rows whose document does + * NOT carry `value` at `path` — INCLUDING rows where the path is absent and + * rows whose document column is SQL NULL (compiled to + * `payload IS NULL OR NOT contains`, matching the Drizzle selector's `ne` + * semantics for both absence cases). The array-leaf caveat on + * {@link selectorEq} carries over: an ARRAY at `path` is never equal to a + * scalar needle, so such rows are INCLUDED here. Selector ordering + * (`gt`/`lt`/…) is not yet expressible over PostgREST — see + * cipherstash/encrypt-query-language#407. */ + selectorNe & StringKeyOf>( + column: K, + path: string, + value: SelectorLeafValue, + ): EncryptedQueryBuilderV3 + /** Raw containment spelling on an encrypted `types.Json` column — the ONLY + * `filter()` form a JSON column supports (scalar operators have no terms to + * compare; they are compile-excluded via {@link V3FilterableKeys}). */ + filter & StringKeyOf>( + column: K, + operator: 'cs', + value: EncryptedJsonContainsValue, + ): EncryptedQueryBuilderV3 + filter & StringKeyOf>( + column: K, + operator: string, + value: Row[K], + ): EncryptedQueryBuilderV3 + /** Negated exact containment on an encrypted `types.Json` column (PostgREST + * `not.cs`). Note the bare form EXCLUDES rows whose document column is SQL + * NULL (three-valued logic) — {@link selectorNe} adds the IS NULL arm. */ + not & StringKeyOf>( + column: K, + operator: 'contains', + value: EncryptedJsonContainsValue, + ): EncryptedQueryBuilderV3 + not & StringKeyOf>( + column: K, + operator: string, + value: Row[K], + ): EncryptedQueryBuilderV3 } /** @@ -338,11 +442,26 @@ export interface EncryptedQueryBuilderV3Untyped< column: K, value: string, ): EncryptedQueryBuilderV3Untyped - /** Native jsonb/array containment on a plaintext column (PostgREST `cs`). */ + /** Native jsonb/array containment on a plaintext column (PostgREST `cs`), + * or ENCRYPTED ste_vec containment on an encrypted `types.Json` column — the + * runtime resolves the column kind and picks the encoding. */ contains>( column: K, value: NativeContainsValue, ): EncryptedQueryBuilderV3Untyped + /** Encrypted JSONPath-selector equality on an encrypted `types.Json` column + * (runtime-guarded on the untyped surface). */ + selectorEq>( + column: K, + path: string, + value: SelectorLeafValue, + ): EncryptedQueryBuilderV3Untyped + /** Encrypted JSONPath-selector inequality (includes absent-path rows). */ + selectorNe>( + column: K, + path: string, + value: SelectorLeafValue, + ): EncryptedQueryBuilderV3Untyped } /** Untyped instance (no `schemas`): rows default to `Record` diff --git a/packages/stack/CHANGELOG.md b/packages/stack/CHANGELOG.md index 4a97e92c..477d7c59 100644 --- a/packages/stack/CHANGELOG.md +++ b/packages/stack/CHANGELOG.md @@ -1,5 +1,536 @@ # @cipherstash/stack +## 1.0.0-rc.0 + +### Major Changes + +- 7c7dbca: CipherStash Stack 1.0 (release candidate). + + This is the first 1.0-line release of `@cipherstash/stack`, the first published + release of the split-out EQL v3 adapters `@cipherstash/stack-drizzle` and + `@cipherstash/stack-supabase`, and moves the `stash` CLI to 1.0 alongside them. + These four packages now version together as the Stack 1.0 family. + +### Minor Changes + +- 31ca318: Split the Drizzle and Supabase integrations into their own packages. + + The adapters now ship as first-party packages that depend on `@cipherstash/stack`, + following the `@cipherstash/prisma-next` precedent: + + - **`@cipherstash/stack-drizzle`** — Drizzle ORM integration. EQL v2 on the package + root (`@cipherstash/stack-drizzle`: `encryptedType`, `extractEncryptionSchema`, + `createEncryptionOperators`) and EQL v3 on `@cipherstash/stack-drizzle/v3` + (`types` factories, `createEncryptionOperatorsV3`, `extractEncryptionSchemaV3`, …). + - **`@cipherstash/stack-supabase`** — Supabase integration: `encryptedSupabase` (v2) + and `encryptedSupabaseV3` (v3, connect-time introspection). + + **Breaking (`@cipherstash/stack`):** the `./drizzle`, `./supabase`, and + `./eql/v3/drizzle` subpath exports are removed. Migrate imports: + + - `@cipherstash/stack/drizzle` → `@cipherstash/stack-drizzle` + - `@cipherstash/stack/eql/v3/drizzle` → `@cipherstash/stack-drizzle/v3` + - `@cipherstash/stack/supabase` → `@cipherstash/stack-supabase` + + Add the relevant package to your dependencies alongside `@cipherstash/stack`. A new + `@cipherstash/stack/adapter-kit` subpath exposes the narrow core internals the + first-party adapters consume; it is the core↔adapter seam, not general-purpose API. + +- c4787c0: Restore the EQL v3 envelope and `Result` types the adapters were erasing. + + Both v3 adapters typed their operand-encryption paths as `unknown` and dropped + the `Result` wrapper, so the query-type encoding and the failure channel were + invisible to the type system: + + - `eql/v3/drizzle/operators.ts` typed the client's `encrypt`/`bulkEncrypt` as + returning `unknown`, collapsed the operation's `Result` to + `{ data?: unknown; failure?: { message } }`, and cast the bulk response to + `Array<{ data: unknown }>`. + - `supabase/query-builder-v3.ts` returned `Promise` from + `encryptCollectedTerms`, `bulkEncryptGroup` and `encryptGroupPerTerm`, and the + base `query-builder.ts` did the same. + + These now carry the SDK's real types — `Encrypted` (the storage envelope union, + which includes every v3 per-domain payload), `BulkEncryptedData`, and + `EncryptedQueryResult` — threaded through a properly-typed operation surface that + resolves `Result`. The Supabase divergence the erasure hid is + now explicit: the v2 path yields `encryptQuery` composite literals and the v3 + path yields `JSON.stringify`'d envelope strings, and both are `EncryptedQueryResult`. + + Bumped `minor`, not `patch`: `createEncryptionOperatorsV3` is a public export + (`@cipherstash/stack/eql/v3/drizzle`), and tightening its client contract from + `unknown` to a typed operation surface is a compile-time breaking change — a + downstream consumer passing a loosely-typed (`unknown`-returning) client double + will now fail `tsc`. That tightening has teeth: `operators.test-d.ts` pins it + with a negative type-test asserting an `unknown`-returning `{ encrypt }` double + is rejected (a positive "correctly-typed double is accepted" assertion cannot + catch a re-erasure, since a correct value is assignable to `unknown`). + + Behaviour is otherwise unchanged, with one addition: the Supabase v3 bulk path + now rejects a `null` envelope returned by `bulkEncrypt` (the restored + `Encrypted | null` type makes that arm reachable, and a `null` would otherwise + be `JSON.stringify`'d to the literal `"null"` and sent as a filter operand). + +- 66a0e02: Add the EQL v3 bigint domain family to the public DSL: `types.Bigint`, + `types.BigintEq`, `types.BigintOrdOre`, and `types.BigintOrd`, backed by the + `public.bigint*` concrete domains. Plaintext is a JS `bigint`, round-tripped + losslessly across the protect-ffi 0.28 boundary (i64 bounds enforced at the + FFI — out-of-range values surface as encryption errors). Index emission follows + the numeric rule: `bigint_eq` → unique (hm); `bigint_ord`/`bigint_ord_ore` → + ore (equality answered via ob). +- 7eba32d: EQL v3 Drizzle: encrypt every query operand with `encryptQuery`, not `encrypt` (#622). + + The v3 Drizzle operators (`eq`/`ne`/`gt`/`gte`/`lt`/`lte`/`between`/`notBetween`/ + `inArray`/`notInArray`/`contains`) previously encrypted their operands with + `client.encrypt`, producing a full storage envelope (including the ciphertext `c`) + cast to `::jsonb`. A WHERE-clause operand should be a query _term_, not a value to + store. Every operator now uses `client.encryptQuery`, which yields a + ciphertext-free query term cast to the column's `eql_v3.query_` type — so + predicates carry no ciphertext and reach the bundle's `(domain, query_)` + operator overloads. This unifies the scalar/text operators with the JSON + containment path (already on `encryptQuery`) and removes the previously-optional + `encryptQuery` guard: it is now a required capability of the operand client. + + `@cipherstash/stack` gains a batch `encryptQuery(terms)` overload on + `TypedEncryptionClient` (the type `EncryptionV3` returns), mirroring the nominal + `EncryptionClient`. This is additive — it lets `inArray`/`notInArray` encrypt a + whole list of query terms in one crossing. + +- 0ebf57e: Close two fail-open paths in the EQL v3 Drizzle adapter. + + `ops.contains()` now throws `EncryptionOperatorError` for a search term that + tokenizes to nothing: the empty string, or a term shorter than the match index + tokenizer's `token_length` (3 by default). Such a term produces an empty bloom + filter, and `stored_bf @> '{}'` is true for every row — so a user searching + `"ad"` silently received the entire table. Measured live, the terms `"ad"`, + `"a"` and `"x"` each returned every seeded row, including one in which `"x"` + did not appear. + + The floor counts Unicode codepoints, matching the tokenizer. A UTF-16 length + check would wave through an astral-plane term — `"👍👍"` is 4 code units but + only 2 codepoints, yields no trigram, and matched every row. + + **Breaking for callers passing short terms:** `contains()` calls that previously + returned every row now throw. Terms of 3+ codepoints are unaffected. + + `v3FromDriver()` now throws the new `EqlV3CodecError` on a payload that is not + an EQL envelope, instead of surfacing a raw `SyntaxError` for malformed JSON and + passing a bare scalar through unchecked — `v3FromDriver('5')` previously returned + `5` typed as `Encrypted`, which then reached `decrypt` as garbage. The guard + accepts both scalar envelopes (ciphertext at `c`) and SteVec documents + (ciphertext at `sv[0].c`). A SteVec's `sv` must be a non-empty array: `sv[0]` is + the decryption root, so `sv: []` carries a ciphertext key but no ciphertext, and + is now rejected rather than passed to `decrypt`. `EqlV3CodecError` is exported + from `@cipherstash/stack/eql/v3/drizzle` so callers can catch it. + + Also removes an unreachable branch in `inArray`/`notInArray`, whose empty-list + guard already throws before it. + + Note: the v2 Drizzle adapter's `like`/`ilike` path builds the same bloom filters + and has the same short-term fail-open. It is **not** fixed here — v2 terms carry + SQL wildcards, so the floor must be measured against what its tokenizer actually + receives before the shared guard can be reused. Tracked separately. + +- d73a03c: Add EQL v3 Drizzle support at `@cipherstash/stack/eql/v3/drizzle`. A Drizzle-native + `types` namespace (same PascalCase names as `@cipherstash/stack/eql/v3`) declares + encrypted columns whose Postgres type is the semantic `public.`; the concrete + type drives the legal query operators. `createEncryptionOperatorsV3` provides + capability-checked `eq`/`ne`/`gt`/`gte`/`lt`/`lte`/`between`/`contains`/`inArray`/ + `asc`/`desc`/`and`/`or` that emit the latest two-argument `eql_v3` SQL functions with + full-envelope operands, and + `extractEncryptionSchemaV3` rebuilds the schema for `EncryptionV3`. The existing v2 + `@cipherstash/stack/drizzle` integration is unchanged. + + The v3 text-search helper is `contains`; obsolete `like`/`ilike` helpers are not + exposed because v3 free-text search is token containment rather than SQL wildcard + matching. + +- 89b903f: Upgrade `@cipherstash/protect-ffi` to 0.28.0 and update EQL v3 concrete Postgres domain names to match the SQL fixture (`integer*`, `smallint*`, `bool`, `real*`, and `double*`). The public factories remain semantic (`Integer`, `Smallint`, `Boolean`, `Real`, `Double`) while their concrete domains change, so this is a minor release. +- 229ce59: Re-baseline EQL v3 on the eql-3.0.0 GA release and protect-ffi 0.29. + + - **Breaking (v3 preview surface):** the EQL v3 column domains follow the + eql-3.0.0 naming convention — flat, prefixed names in `public` + (`public.eql_v3_text_search`, `public.eql_v3_integer_ord`, …) instead of the + alpha-era bare names. Databases installed from an alpha bundle must be + re-installed (`stash eql install --eql-version 3` replaces the schema). + - `encryptQuery` under `eqlVersion: 3` now returns EQL v3 query operands + (protect-ffi 0.29): term-only scalar operands for the `eql_v3.query_` + domains, the `eql_v3.query_jsonb` containment needle, and bare selector + hashes for JSON path queries — v3 scalar and selector queries no longer + throw `EQL_V3_QUERY_UNSUPPORTED` (the code is gone). + - v2 `searchableJson()` columns now pin the SteVec encoding to `standard` + explicitly. protect-ffi 0.29 flipped the library default to `compat` + (EQL v3's encoding); without the pin, v2 JSON containment queries would + silently match nothing and newly written rows would not be comparable with + existing ones. + - The EQL v3 test/install SQL is sourced from the pinned `@cipherstash/eql` + package (3.0.0) instead of a hand-vendored fixture. + +- 50c0a9c: Add EQL v3 JSON columns. `types.Json('col')` declares a `public.eql_v3_json` + column that encrypts a JSON document to an ste_vec `SteVecDocument` and + round-trips it losslessly through `encrypt`/`decrypt` and the model path. A new + `searchableJson` query capability emits the ste_vec index; the index uses + `mode: 'compat'`, which eql-3.0.0's `eql_v3_json` requires (it orders ste_vec + entries by the CLLW-OPE `op` term, so v2's `'standard'`/CLLW-`oc` terms are + rejected). + + The Drizzle integration's `contains(col, subObject)` now answers encrypted-JSONB + containment on a `types.Json` column, emitting the `@>` operator with a + `query_jsonb` needle (from `encryptQuery`). The ste_vec index indexes array + elements by identity but not position, so containment is a true subset test + (`{ roles: ['x'] }` matches any document whose `roles` array contains `x`, + regardless of index). + +- 5d23e80: Add `encryptedSupabaseV3` — the EQL v3 dialect of the Supabase adapter. It is + now a connect-time-async factory: `await encryptedSupabaseV3(url, key)` (or + `(client)`) introspects the database over `DATABASE_URL`, detects EQL v3 columns + by their Postgres domain (`information_schema.columns.domain_name`), and derives + each column's encryption config from its domain — callers no longer pass a + schema to `from()`. `select('*')` is supported (expanded from the introspected + column list, and aliased back to each declared column's JS property name so a + property→DB rename round-trips). A column using an EQL v3 domain this SDK version does not model + (e.g. `public.json`, `*_ord_ope`) throws at construction rather than silently + passing through. Supplying `schemas` remains optional and adds compile-time + types plus startup verification of the declared tables against the database. + Requires a Postgres connection for introspection (`pg` is a new optional peer), + so it cannot run in a Worker or the browser. + + Every column name a query carries — filters, `match`, `not`, raw `filter`, + `or()`, `order()`, and the `onConflict` option — is now resolved from its JS + property name to its DB column name in a single pass before the query is built, + so a declared rename round-trips everywhere rather than only on the paths that + remembered to translate. + + `order()` on ANY encrypted v3 column is now rejected — at compile time when + `schemas` is supplied, and at runtime otherwise. The EQL v3 domains are + `DOMAIN … AS jsonb` and the bundle declares no btree operator class on them, so + `ORDER BY col` resolves through jsonb's default `jsonb_cmp` and sorts by the + envelope's byte structure: a stable, plausible-looking, meaningless row order, + with no error. Correct ordering is `ORDER BY eql_v3.ord_term(col)`, which + PostgREST's `order=` cannot express. Order by a plaintext column, expose + `eql_v3.ord_term()` as a generated column or view, or use the EQL v3 Drizzle + integration, which emits `ord_term` directly. Note `gte`/`lte` filters remain + correct: the comparison operators _are_ declared on the ord domains, and only + sorting resolves through the missing operator class. + + `.or()` now understands PostgREST's `column.not..` negation. It was + previously parsed as `{ op: 'not', value: '.' }`, so on an encrypted + column `or('nickname.not.in.(ada,grace)')` encrypted the literal string + `in.(ada,grace)` as a single plaintext and produced a filter that silently + matched nothing. + + Free-text search on the v3 builder is `contains(column, value)`. `like`/`ilike` + are not exposed, because EQL v3 free-text search is token containment over a + bloom filter (`@>`, backed by `eql_v3.contains`) rather than SQL wildcard + matching — `%` is tokenized like any other character, so a `like` pattern is a + category error. This matches the v3 Drizzle integration, which omits them for + the same reason. On an encrypted column `like`/`ilike` now throw and name + `contains`; on a plaintext column they remain ordinary PostgREST filters. + + `contains` is narrowed at compile time to columns whose domain carries the + `freeTextSearch` capability (`public.text_match`, `public.text_search`), and + guarded at runtime for the untyped surface. A raw `filter(column, operator, …)` + on an encrypted v3 column now derives its query type from the operator instead + of always encrypting an equality term, so `filter('bio', 'cs', …)` on a + `public.text_match` column works rather than being rejected, and an unsupported + operator throws instead of silently encrypting the wrong term. + + Substring `contains` matches any needle whose trigrams are all present in the + stored value; needles shorter than the tokenizer's window (3 characters) bloom to + nothing and are rejected rather than silently matching every row. The v3 match + index now emits `include_original: false` — the flag is inert in protect-ffi (the + bloom is trigram-only either way), so this moves no ciphertext and only pins the + value a substring-search domain wants. + + v2 (`encryptedSupabase`) is unchanged: it keeps `like`/`ilike` (`eql_v2.like`, + `~~`) and its raw-`filter` query-type mapping, so no v2 ciphertext moves. + +- 1aa9a11: Add the EQL v3 `text_search` authoring DSL on a new `@cipherstash/stack/eql/v3` + subpath (`types.TextSearch`, v3 `encryptedTable` / `buildEncryptConfig`). The v3 + builders emit the existing `EncryptConfig` shape, so encryption, payloads, and + query paths are unchanged at runtime. + + Also widens the public client types (`EncryptionClientConfig.schemas`, + `EncryptOptions`, `SearchTerm`/`EncryptQueryOptions`) to a structural contract so + both v2 and v3 builders are accepted by `Encryption` / `encrypt` / `decrypt` / + `encryptQuery`. This is a backward-compatible widening — existing v2 usage is + unaffected. The structural contracts themselves (`BuildableColumn`, + `BuildableQueryColumn`, `BuildableV3QueryableColumn`, `BuildableTable`, + `BuildableTableColumns`) and the `encryptModel` return-type mapper + (`EncryptedFromBuildableTable`) are exported from `@cipherstash/stack/types` so + consumers can name them. + +- af2d04e: Add a strongly-typed EQL v3 client surface on a new `@cipherstash/stack/v3` + subpath (`EncryptionV3`, `typedClient`, `TypedEncryptionClient`). It re-exports + the v3 `types` namespace and table API (from `@cipherstash/stack/eql/v3`), so a + single import provides everything needed to author and use a v3 schema. + + Every method derives its types from the concrete `table` / `column` builder + arguments: + + - `encrypt` / `encryptQuery` pin the plaintext to the column's domain type + (`text → string`, `timestamp → Date`, …). + - `encryptQuery` constrains `queryType` to the column's capabilities and rejects + storage-only columns at compile time. + - `encryptModel` / `bulkEncryptModels` validate schema-column fields against their + inferred plaintext type (passthrough fields are untouched) and return a precise + encrypted model. + - `decryptModel` / `bulkDecryptModels` return the precise plaintext model, + reconstructing `Date` values from the encrypt-config `cast_as`. + + Because the typed methods bind to the concrete branded v3 classes, a hand-rolled + structural table/column is rejected — closing the soundness gap where a non-branded + table could be encrypted at runtime while typed as plaintext. + + Runtime behaviour is unchanged: the encrypt/query paths return the same operations + as the base client; only the model-decrypt paths add a per-column `Date` + reconstruction step. The v2 client surface (`Encryption`) is untouched. + +- b8a3d20: Add EQL v3 schema builders for supported generated SQL domains under `@cipherstash/stack/eql/v3`, exposed as the `types` namespace (one member per supported EQL v3 domain, e.g. `types.TextEq` / `types.IntegerOrd` / `types.Timestamp`), including explicit query capability metadata (`getQueryCapabilities()` / `isQueryable()`) and v3 table support in model encryption helpers (`encryptModel` / `bulkEncryptModels`). + + Also widen the accepted plaintext input type for `encrypt` / `encryptQuery` to include `Date` (via the new `Plaintext` type), so v3 `date` / `timestamp` domains can be encrypted and queried with their natural JavaScript values. + +- a0f3b2c: `@cipherstash/stack/wasm-inline` is now EQL v3 (#614). + + The WASM entry (Deno / Bun / Cloudflare Workers / Supabase Edge) previously + created a client pinned to the FFI's EQL v2 wire format, so a v3 schema + (concrete `eql_v3_*` domains) failed every encrypt on the edge. It now targets + EQL v3 exclusively: + + - The factory constructs the WASM client with `eqlVersion: 3`, so v3 schemas + encrypt/decrypt correctly on the edge. + - The entry re-exports the **v3** authoring surface (`types`, `encryptedTable`, + the column classes, `buildEncryptConfig`, and the inference helpers) — the + same API as `@cipherstash/stack/eql/v3` — so an Edge Function authors and runs + v3 from one import: + + ```ts + import { + Encryption, + encryptedTable, + types, + } from "@cipherstash/stack/wasm-inline"; + + const patients = encryptedTable("patients", { + email: types.TextSearch("email"), + }); + const client = await Encryption({ schemas: [patients], config }); + ``` + + The v2 schema builders (`encryptedColumn` / `encryptedField` / the v2 + `encryptedTable`) are no longer exported from this entry, and passing a v2 table + throws a clear error. The WASM path was never announced or documented for v2 and + had no known users; EQL v2 remains fully supported on the native + `@cipherstash/stack` entry. + +- 5411a13: Add the `@cipherstash/stack/adapter-kit` subpath — a narrow support surface for + the first-party adapter packages (`@cipherstash/stack-supabase`, + `@cipherstash/stack-drizzle`) being split out of this package (#627). It + re-exports exactly the core internals the adapters consume (the logger, + `AuditConfig`, the v3 column model + `DATE_LIKE_CASTS`, the domain registry, the + match-index guard, and the model→composite helpers) so those imports resolve + across the package boundary without leaking six internal module paths. This is the + core↔adapter seam, not general-purpose public API. +- 99f8b0a: Fix encrypted `in`-list operands in the Supabase adapter, and widen the `is` / + `contains` type surfaces. + + **`in()` on an encrypted column produced a request PostgREST rejects.** Every + encrypted operand is a serialized envelope, dense with `"` and `,`. postgrest-js + wraps a comma-bearing element as `"…"` but never escapes the quotes already + inside it, so `.in('email', […])` emitted + + ``` + in.("{"v":1,"c":"…"}",…) + ^ PostgREST ends the value here → PGRST100 + ``` + + Encrypted lists are now emitted through `filter(col, 'in', …)` with each element + quoted and escaped, matching what the `.or()` path already did. This affects + **v2 as well as v3** — v2's `("a@b.com")` composite literal is itself + quote-bearing and was equally broken. + + **`not(col, 'in', […])` encrypted the whole list as a single ciphertext**, so + the filter silently matched nothing, and emitted an unparenthesized + `not.in.a,b`. Each element is now encrypted separately and the operand is + rendered as `not.in.(…)`. Passing a PostgREST list literal (`'(a,b)'`) for an + encrypted column now throws instead of silently matching nothing — pass an + array. + + **`filter(col, 'in', […])` encrypted the whole list as a single ciphertext.** + The raw `.filter()` path reached `in` with none of the element-splitting the + `in()`, `not(…, 'in', …)` and `.or()` paths perform, so the entire list operand + was encrypted as one equality term. The two wire formats then failed + differently, which is why this went unnoticed: **v2**'s `("json")` composite + literal is already parenthesized, so PostgREST parsed it as a one-element list + and answered `200 []` — a filter that silently matched nothing. **v3**'s bare + `{…}` envelope is not, so PostgREST rejected the request outright with + `PGRST100 (failed to parse filter)`. + + Each element is now encrypted separately and the operand rendered as a quoted + PostgREST list literal. As on the `not` path, passing a list literal + (`'(a,b)'`) for an encrypted column now throws instead — pass an array. + + Plaintext columns are unaffected, including the pre-existing quirk that + postgrest-js renders `.filter(col, 'in', [array])` as an unparenthesized + `in.a,b` that PostgREST rejects; pass a list literal there, or use `.in()`. + + **`is(col, null)` is now allowed on every column**, including storage-only + encrypted ones (`types.Boolean`, `types.Integer`, …). `is` is never encrypted + and a NULL plaintext is stored as a SQL NULL, so `IS NULL` is not merely legal + there but the only predicate those columns support. `is(col, true)` remains a + compile error on encrypted columns. + + **`contains()` accepts native operands on plaintext array and jsonb columns.** A + plaintext jsonb/array column falls through to PostgREST's native containment, so + `contains('tags', ['vip'])` and `contains('meta', { plan: 'pro' })` now + typecheck. A plaintext SCALAR column does not: `@>` is undefined on `text`, so + the operand type follows the column's own shape and a scalar rejects every + containment operand. Encrypted match columns still take a `string` token. + Relatedly, `.or([{ op: 'contains' }])` now emits PostgREST's `cs` operator for + plaintext columns too — previously only encrypted conditions were translated, so + a plaintext containment reached the wire as `.contains.` and failed to parse. + + **Direct `contains()` / `not(col, 'contains', …)` now serialize their operand.** + postgrest-js builds an array operand as `cs.{a,b}` with no element quoting, so + `contains('tags', ['with,comma'])` reached Postgres as two elements; and its + `not()` stringifies the operand outright, emitting `not.contains.with,comma` + (no braces, and the wrong operator token) or `[object Object]` for a jsonb + operand. Both paths now build the containment literal the `.or()` path already + built, and emit the `cs` token. + + **`.or()` no longer drops a condition after an unbalanced brace or paren.** A + scalar operand containing `{` left the parser's depth counter stranded above + zero, so no later comma separated a condition and everything behind it was + swallowed into that operand. With a plaintext column first, the group was then + forwarded verbatim — running the swallowed condition against a ciphertext column + with a plaintext operand. Braces are now quoted on emit (they are structural to + PostgREST inside `or=(…)`), and the parser falls back to quote-only splitting + when its depth tracking does not balance. + + **`is(col, true)` is now rejected on every encrypted column, not just the + storage-only ones.** The boolean form was gated on the filterable keys, which + exclude storage-only columns but keep queryable encrypted ones — so + `is(emailTextSearchColumn, true)` compiled and emitted `IS TRUE` against a jsonb + ciphertext. + + **In-list operands encrypt in one crossing per column.** The element-wise `in` / + `not.in` encoding above spent one ZeroKMS round-trip per element; terms are now + grouped by column and each group takes a single `bulkEncrypt` call, matching the + Drizzle v3 path. Falls back to per-term encryption for clients without + `bulkEncrypt`, and rejects a bulk response whose length does not match the list + rather than silently truncating the predicate. + +- 9b65ae8: **`order()` now works on EQL v3 encrypted ordering columns in the Supabase + adapter.** It was rejected outright on every encrypted column. + + A bare `ORDER BY col` on an EQL v3 domain really is wrong — the bundle declares + no btree operator class on any domain, so the sort falls through to jsonb's + default `jsonb_cmp` and compares the envelope's keys in storage order, starting + at the random ciphertext `c`. Measured over ten rows it returns + `r00,r04,r08,r01,…` where the plaintext order is `r00..r09`. No error, a stable + and plausible-looking meaningless order. + + But the correct sort key is reachable without a function call. `eql_v3.ord_term` + returns the domain's `op` term, and OPE is order-preserving, so ordering by the + term reproduces the plaintext order. PostgREST cannot emit + `ORDER BY eql_v3.ord_term(col)`, but it can emit a jsonb path. The builder now + emits `order=col->op` for an encrypted ordering column, verified against a live + PostgREST for `integer_ord` and `text_search` in both directions. + + The guard is now on the ordering FLAVOUR, not on encryption: + + - **`ope` present → supported.** Every plain `*_ord` domain, plus `text_ord` and + `text_search`. + - **`ore` present → rejected.** The `ob` term is an array of ORE blocks whose + comparison needs the superuser-only operator class, which no jsonb path can + reach. (Such a column cannot hold data on managed Postgres anyway: its domain + CHECK raises `ore_domain_unavailable`.) ORE columns are now excluded from + `order()` at COMPILE time too, not only at runtime — `.order(oreColumn)` is a + type error, matching the rejection. + - **neither → rejected.** Storage-only, equality-only and match-only columns + carry no ordering term. + + The path is `col->op` (jsonb), not `col->>op` (text). Neither avoids the + database collation — Postgres compares jsonb strings with `varstr_cmp` under the + default collation, exactly as it does text. What makes the ordering + collation-independent is the term's encoding: lowercase hex, fixed-width for + numeric and date domains, and per-character (16 hex chars each) for text, so + lexicographic order reproduces plaintext order including the prefix case + (`ada` < `adam`). `ope-term.integration.test.ts` pins that shape. + + `V3OrderableKeys` widens to admit OPE-backed ordering columns (`*_ord`, + `text_ord`, `text_search`) while still excluding ORE (`*_ord_ore`) columns, so + `order()` typechecks exactly where it works. `is(col, true)` is unaffected — it + stays plaintext-only, and now has its own `V3PlaintextKeys` rather than + borrowing the orderable set. + +### Patch Changes + +- cfd46ee: Source the EQL v3 install bundle from `@cipherstash/eql@3.0.0-alpha.3` instead of a hand-vendored 43k-line SQL fixture committed to the test tree. The package publishes its SQL and its TypeScript wire types from the same `eql-bindings` commit, so the bundle is now pinned to a released EQL version rather than tracked by convention. + + Test-and-tooling only — `@cipherstash/eql` is a `devDependency` and no public API changes. + + The staleness check in the v3 install helper now compares `eql_v3.version()` against the pinned release instead of probing for a hand-picked sentinel domain. The previous sentinel (`public.timestamp`) exists in both the old and new bundles, so it would have reported a stale install as current and left the suite silently running the wrong SQL. + +- 63ca540: Re-vendor the EQL v3 SQL bundle and align the v3 DSL to it: encrypted type domains now live in the `public` schema (`public.text`, `public.integer`, …) rather than `eql_v3`, and the boolean domain is `public.boolean` (was `eql_v3.bool`). The `eql_v3` schema now holds only the operator-backing functions, and the index-term constructors (`hmac_256`, `ore_block_256`, `bloom_filter`) moved to `eql_v3_internal`. This keeps the SDK's emitted domain names byte-matched to the installed bundle so `CREATE TABLE`/cast resolution succeeds. +- f23f952: Remove the leftovers from the secrets removal (`1929c8fe`), which deleted + `packages/stack/src/secrets/` but left its export, build entry, skill, and docs + behind. Secrets tooling is not ready; nothing here was functional. + + - **Drop the dead `@cipherstash/stack/secrets` subpath export.** It pointed at + `./dist/secrets/index.js`, which has no source and is not in the tarball, so + `import '@cipherstash/stack/secrets'` has been throwing `ERR_MODULE_NOT_FOUND` + for every consumer since the source was removed. Also drops the dangling + `src/secrets/index.ts` entry from `tsup.config.ts`. Removing an export that + cannot resolve breaks nothing. + - **Remove the `stash-secrets` agent skill** and its references in `AGENTS.md` + and the init setup-prompt skill index. It was never installed by `stash init` + (it is absent from `SKILL_MAP`), so no user project ever received it. + - **Remove the secrets documentation** from both published READMEs: the + `Secrets` class API and the `npx stash secrets` command reference in + `@cipherstash/stack`, and the `npx stash secrets` section in `stash`. The CLI + command does not exist — `stash secrets` returns `Unknown command`. + +- fd33aad: Fix the Supabase adapter encrypting `is` and `null` filter operands. + + `is` is a SQL predicate — PostgREST accepts only `null`/`true`/`false` after it + — and a `null` operand is SQL NULL, never a value to search for. Only the direct + `.is()` filter skipped encryption; `not()`, `or()`, `match()`, raw `filter()`, + and the `in()` element list all encrypted whatever they were handed. So + `or('age.is.null')` emitted `age.is."("null")"` and `eq('email', null)` emitted + `email=("null")` — operands PostgREST rejects. A null plaintext is stored as a + NULL column rather than ciphertext, so it is found with an unencrypted + `IS NULL`; encrypting the operand could never match. + + A single `isEncryptableTerm(operator, value)` predicate now guards every term + collector. Affects both `encryptedSupabase` (v2) and `encryptedSupabaseV3`. On + v3 this additionally removes a spurious `does not support equality queries` + error, which `is` raised because it maps to the `equality` query type and so hit + the column-capability guard — `or('active.is.null')` on a storage-only column + threw rather than querying. + + Relatedly, an `or()` string is now rebuilt whenever a condition _references_ an + encrypted column, not only when one of its values was encrypted. An `is` on an + encrypted column encrypts nothing, and the old condition sent it down the + verbatim path, forwarding the caller's JS property name to a database that only + knows the column's DB name. + +- 8cd485d: Fix the Supabase adapter's `.or()` string parser mis-splitting conditions, and pin `contains()` on a mixed union column key to the encrypted operand. + + An `.or()` string is only rebuilt from its parse when it references an encrypted column — otherwise the caller's string is forwarded verbatim — so each of these corrupts precisely the mixed encrypted/plaintext case. + + **Quotes were tracked only at brace depth 0.** A `}` inside a quoted array element or jsonb string value closed the literal early, and the next `"` re-opened quoting, so the following top-level comma never split: `.or('tags.cs.{"a}b"},email.eq.secret')` parsed as a single condition and silently absorbed `email.eq.secret` into the operand. Quotes are now opaque at every depth. + + **A stray `}` or `)` drove the depth counter negative**, after which no comma split again. `}` and `)` are not PostgREST reserved characters, so `a}b` is a valid unquoted operand and `.or('nickname.eq.a}b,id.eq.1')` dropped `id.eq.1`. Depth now floors at zero. + + **`in`-list elements were split on every comma, ignoring quotes.** `.or('email.in.("a,b",c)')` parsed as three elements with the quotes still embedded; on an encrypted column each fragment was encrypted as its own term, so the intended element never matched. Elements are now split on top-level commas and unquoted, the inverse of what the rebuild emits. + + **A parenthesized operand was read as a list for every operator.** Only `in` and the range operators (`ov`, `sl`, `sr`, `nxr`, `nxl`, `adj`) take a paren-delimited operand; elsewhere `(` is an ordinary character. `email.eq.(foo)` parsed as `['foo']` and encrypted a JS array rather than the string, matching nothing. + + **A string operand spelling `null`, `true` or `false` is now quoted.** PostgREST reads a bare `null` as SQL NULL, so `.or([{ column: 'name', op: 'eq', value: 'null' }])` emitted `name.eq.null` and compared against NULL instead of the three-character string. + + **`contains(col, …)` where `col` is a union spanning an encrypted and a plaintext column** accepted an array or object operand. The union is now only as permissive as its strictest member: any declared encrypted column in the union pins the operand to `string`. A literal column argument was never affected. + ## 0.19.0 ### Minor Changes diff --git a/packages/stack/README.md b/packages/stack/README.md index bd0953f4..71f11165 100644 --- a/packages/stack/README.md +++ b/packages/stack/README.md @@ -6,7 +6,7 @@ The all-in-one TypeScript SDK for the CipherStash data security stack. [![License: MIT](https://img.shields.io/npm/l/@cipherstash/stack.svg?style=for-the-badge&labelColor=000000)](https://github.com/cipherstash/stack/blob/main/LICENSE.md) [![TypeScript](https://img.shields.io/badge/TypeScript-first-blue?style=for-the-badge&labelColor=000000)](https://www.typescriptlang.org/) --- +--- ## Table of Contents @@ -23,11 +23,11 @@ The all-in-one TypeScript SDK for the CipherStash data security stack. - [Error Handling](#error-handling) - [API Reference](#api-reference) - [Subpath Exports](#subpath-exports) -- [Migration from @cipherstash/protect](#migration-from-cipherstashprotect) +- [Legacy: EQL v2](#legacy-eql-v2) - [Requirements](#requirements) - [License](#license) --- +--- ## Install @@ -54,17 +54,19 @@ The wizard will authenticate you, walk you through choosing a database connectio ### 2. Encrypt and decrypt +Define a table with concrete EQL v3 column types, build the typed client, and encrypt: + ```typescript -import { Encryption } from "@cipherstash/stack" -import { encryptedTable, encryptedColumn } from "@cipherstash/stack/schema" +import { EncryptionV3 } from "@cipherstash/stack/v3" +import { encryptedTable, types } from "@cipherstash/stack/eql/v3" -// Define a schema +// Define a schema — the column type fixes its query capabilities const users = encryptedTable("users", { - email: encryptedColumn("email").equality().freeTextSearch(), + email: types.TextSearch("email"), // equality + order/range + free-text search }) -// Create a client -const client = await Encryption({ schemas: [users] }) +// Create a typed client +const client = await EncryptionV3({ schemas: [users] }) // Encrypt a value const encrypted = await client.encrypt("hello@example.com", { @@ -72,114 +74,169 @@ const encrypted = await client.encrypt("hello@example.com", { table: users, }) +// Every operation returns `{ data } | { failure }`. Narrow on `.failure` and +// return/throw before reading `.data` — the failure branch has no `data`. if (encrypted.failure) { - console.error("Encryption failed:", encrypted.failure.message) -} else { - console.log("Encrypted payload:", encrypted.data) + throw new Error(`Encryption failed: ${encrypted.failure.message}`) } +console.log("Encrypted payload:", encrypted.data) // Decrypt the value const decrypted = await client.decrypt(encrypted.data) - if (decrypted.failure) { - console.error("Decryption failed:", decrypted.failure.message) -} else { - console.log("Plaintext:", decrypted.data) // "hello@example.com" + throw new Error(`Decryption failed: ${decrypted.failure.message}`) } +console.log("Plaintext:", decrypted.data) // "hello@example.com" ``` +The client is typed from your schemas: passing the wrong plaintext type for a column (`client.encrypt(42, { column: users.email, ... })`) is a compile error. + ## Features - **Field-level encryption** - Every value encrypted with its own unique key via [ZeroKMS](https://cipherstash.com/products/zerokms), backed by AWS KMS. -- **Searchable encryption** - Exact match, free-text search, order/range queries, and encrypted JSONB queries in PostgreSQL. +- **Searchable encryption** - Exact match, free-text search, order/range queries, and encrypted JSON queries in PostgreSQL, driven by concrete EQL v3 column types. +- **Type-safe by construction** - Each encrypted column is a concrete Postgres domain; its query capabilities are fixed by the type you pick and enforced at compile time by the typed client. - **Bulk operations** - Encrypt or decrypt thousands of values in a single ZeroKMS call (`bulkEncrypt`, `bulkDecrypt`, `bulkEncryptModels`, `bulkDecryptModels`). -- **Identity-aware encryption** - Tie encryption to a user's JWT via `LockContext`, so only that user can decrypt. +- **Identity-aware encryption** - Tie encryption to a user's JWT via `OidcFederationStrategy` and `.withLockContext()`, so only that user can decrypt. - **CLI (`stash`)** - Initialize projects and set up encryption from the terminal. - **TypeScript-first** - Strongly typed schemas, results, and model operations with full generics support. ## Schema Definition -Define which tables and columns to encrypt using `encryptedTable` and `encryptedColumn` from `@cipherstash/stack/schema`. +Define which tables and columns to encrypt using `encryptedTable` and the `types` namespace from `@cipherstash/stack/eql/v3`. Each factory in `types` maps 1:1 to a **concrete Postgres domain** named `public.eql_v3_` — the naming rule is: strip the `eql_v3_` prefix and PascalCase each underscore-separated segment. So `types.TextSearch` builds a `public.eql_v3_text_search` column, `types.IntegerOrd` builds `public.eql_v3_integer_ord`. + +There are **no chainable capability methods** — the concrete type fully describes what a column can do. ```typescript -import { encryptedTable, encryptedColumn } from "@cipherstash/stack/schema" +import { encryptedTable, types } from "@cipherstash/stack/eql/v3" const users = encryptedTable("users", { - email: encryptedColumn("email") - .equality() // exact-match queries - .freeTextSearch() // full-text search queries - .orderAndRange(), // sorting and range queries -}) - -const documents = encryptedTable("documents", { - metadata: encryptedColumn("metadata") - .searchableJson(), // encrypted JSONB queries (JSONPath + containment) + email: types.TextSearch("email"), // equality + order/range + free-text search + age: types.IntegerOrd("age"), // equality + order/range + balance: types.Bigint("balance"), // storage only — encrypt/decrypt, no queries + metadata: types.Json("metadata"), // encrypted JSON: containment + JSONPath selectors }) ``` -### Index Types +The returned table is also a column accessor (`users.email`). The JS property name and the DB column name may differ: `createdOn: types.Timestamp("created_at")` reads and writes the `createdOn` property on models but targets the `created_at` column in the database. + +### Capability Suffixes + +The suffix on the type name encodes the query capability: + +| Suffix | Capabilities | Query types | +|---|---|---| +| _(none)_ | Storage only — encrypt/decrypt, no queries | — | +| `Eq` | Equality | `'equality'` | +| `Ord` | Equality + ordering/range (OPE-backed) | `'equality'`, `'orderAndRange'` | +| `OrdOre` | Equality + ordering/range (block-ORE-backed — the ORE operator class is superuser-only and unavailable on managed Postgres such as Supabase) | `'equality'`, `'orderAndRange'` | +| `Match` (text only) | Free-text search only | `'freeTextSearch'` | +| `Search` (text only, as `TextSearch`) | Equality + ordering/range + free-text | all three | +| `Json` | Encrypted JSON containment + JSONPath selector queries | `'searchableJson'` | + +Prefer the plain `Ord` domains unless you know your database supports the ORE operator class. -| Method | Purpose | Query Type | -|----|-----|------| -| `.equality()` | Exact match lookups | `'equality'` | -| `.freeTextSearch()` | Full-text / fuzzy search | `'freeTextSearch'` | -| `.orderAndRange()` | Sorting, comparison, range queries | `'orderAndRange'` | -| `.searchableJson()` | Encrypted JSONB path and containment queries | `'searchableJson'` | -| `.dataType(cast)` | Set the plaintext data type (`'string'`, `'number'`, `'boolean'`, `'date'`, `'bigint'`, `'json'`) | N/A | +### Domain Families and Plaintext Types -Methods are chainable - call as many as you need on a single column. +| Family | Factories | Plaintext (TypeScript) type | +|---|---|---| +| `Integer`, `Smallint`, `Numeric`, `Real`, `Double` | base, `Eq`, `Ord`, `OrdOre` | `number` | +| `Bigint` | base, `Eq`, `Ord`, `OrdOre` | `bigint` (native JS bigint, full i64 range) | +| `Date` | base, `Eq`, `Ord`, `OrdOre` | `Date` (calendar date; time-of-day truncated) | +| `Timestamp` | base, `Eq`, `Ord`, `OrdOre` | `Date` (time-of-day preserved) | +| `Text` | base, `Eq`, `Match`, `Ord`, `OrdOre`, `Search` | `string` | +| `Boolean` | base only | `boolean` | +| `Json` | `Json` only | a JSON *document* (object, array, or null — not a top-level scalar) | + +### Database Setup + +Install the EQL v3 SQL into your database with the stash CLI: + +```bash +npx stash eql install --eql-version 3 +# On Supabase, add --supabase to grant the anon/authenticated/service_role +# roles access to the eql_v3 schemas — without it, encrypted queries fail with +# "permission denied for schema eql_v3_internal": +npx stash eql install --eql-version 3 --supabase +``` + +In migrations, declare each encrypted column as its domain type: + +```sql +CREATE TABLE users ( + id BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY, + email public.eql_v3_text_search, + age public.eql_v3_integer_ord, + balance public.eql_v3_bigint, + metadata public.eql_v3_json +); +``` ## Encryption and Decryption ### Single Values ```typescript -// Encrypt +// Encrypt — plaintext is pinned to the column's domain type const encrypted = await client.encrypt("secret@example.com", { column: users.email, table: users, }) -// Decrypt +// Decrypt (narrow on `.failure` before reading `.data`) +if (encrypted.failure) throw new Error(encrypted.failure.message) const decrypted = await client.decrypt(encrypted.data) ``` ### Model Operations -Encrypt or decrypt an entire object. Only fields matching your schema are encrypted; other fields pass through unchanged. +Encrypt or decrypt an entire object. Only fields matching your schema are encrypted; other fields pass through unchanged. Schema fields are validated against their inferred plaintext type at compile time. -The return type is **schema-aware**: fields matching the table schema are typed as `Encrypted`, while other fields retain their original types. For best results, let TypeScript infer the type parameters from the arguments: +`decryptModel` takes the **table as a second argument** and returns the precise plaintext model: `Date` columns are reconstructed to real `Date` instances, and `bigint` columns round-trip as native `bigint`. ```typescript -type User = { id: string; email: string; createdAt: Date } - const user = { - id: "user_123", - email: "alice@example.com", // defined in schema -> encrypted - createdAt: new Date(), // not in schema -> unchanged + id: "user_123", // not in schema -> passes through + email: "alice@example.com", // TextSearch -> encrypted as string + age: 30, // IntegerOrd -> encrypted as number + balance: 100_000n, // Bigint -> encrypted as bigint } -// Let TypeScript infer the return type from the schema const encryptedResult = await client.encryptModel(user, users) -// encryptedResult.data.email -> Encrypted -// encryptedResult.data.id -> string -// encryptedResult.data.createdAt -> Date +// encryptedResult.data.email -> Encrypted +// encryptedResult.data.id -> string -// Decrypt a model -const decryptedResult = await client.decryptModel(encryptedResult.data) +if (encryptedResult.failure) throw new Error(encryptedResult.failure.message) +const decryptedResult = await client.decryptModel(encryptedResult.data, users) +// decryptedResult.data.email -> string +// decryptedResult.data.balance -> bigint ``` ### Bulk Operations All bulk methods make a single call to ZeroKMS regardless of the number of records, while still using a unique key per value. +#### Bulk Encrypt / Decrypt Models + +```typescript +const userModels = [ + { id: "1", email: "alice@example.com", age: 30, balance: 100_000n }, + { id: "2", email: "bob@example.com", age: 41, balance: 250_000n }, +] + +const encrypted = await client.bulkEncryptModels(userModels, users) +if (encrypted.failure) throw new Error(encrypted.failure.message) +const decrypted = await client.bulkDecryptModels(encrypted.data, users) +``` + #### Bulk Encrypt / Decrypt (raw values) +`bulkEncrypt` / `bulkDecrypt` are untyped passthroughs for raw value arrays: + ```typescript const plaintexts = [ { id: "u1", plaintext: "alice@example.com" }, { id: "u2", plaintext: "bob@example.com" }, - { id: "u3", plaintext: "charlie@example.com" }, ] const encrypted = await client.bulkEncrypt(plaintexts, { @@ -189,7 +246,9 @@ const encrypted = await client.bulkEncrypt(plaintexts, { // encrypted.data = [{ id: "u1", data: EncryptedPayload }, ...] +if (encrypted.failure) throw new Error(encrypted.failure.message) const decrypted = await client.bulkDecrypt(encrypted.data) +if (decrypted.failure) throw new Error(decrypted.failure.message) // Each item has either { data: "plaintext" } or { error: "message" } for (const item of decrypted.data) { @@ -201,175 +260,91 @@ for (const item of decrypted.data) { } ``` -#### Bulk Encrypt / Decrypt Models - -```typescript -const userModels = [ - { id: "1", email: "alice@example.com" }, - { id: "2", email: "bob@example.com" }, -] - -const encrypted = await client.bulkEncryptModels(userModels, users) -const decrypted = await client.bulkDecryptModels(encrypted.data) -``` - ## Searchable Encryption -Encrypt a query term so you can search encrypted data in PostgreSQL. +Encrypt a query term so you can search encrypted data in PostgreSQL. The typed client only accepts queryable columns, and `queryType` is constrained to the column's capabilities — equality and range queries run through the domain's own SQL operators. ```typescript -// Equality query +// Equality const eqQuery = await client.encryptQuery("alice@example.com", { column: users.email, table: users, queryType: "equality", }) -// Free-text search -const matchQuery = await client.encryptQuery("alice", { +// Free-text search — queryType is REQUIRED for a match term (see gotcha below) +const matchQuery = await client.encryptQuery("ali", { column: users.email, table: users, queryType: "freeTextSearch", }) // Order and range -const rangeQuery = await client.encryptQuery("alice@example.com", { - column: users.email, +const rangeQuery = await client.encryptQuery(30, { + column: users.age, table: users, queryType: "orderAndRange", }) ``` -### Searchable JSON +> **Gotcha — `TextSearch` defaults to equality.** A `TextSearch` column carries all three indexes, and `encryptQuery` with **no explicit `queryType` builds an equality term, not a free-text match**. A substring like `"joh"` then matches nothing. Always pass `queryType: 'freeTextSearch'` for substring/token search. -For columns using `.searchableJson()`, the query type is auto-inferred from the plaintext: +Free-text search is fuzzy bloom-filter token matching, surfaced as `matches` in the Drizzle and Supabase adapters — it is order- and multiplicity-insensitive and one-sided (a match may be a false positive, a non-match never is). It is not SQL `LIKE`; don't pass `%` wildcards. -```typescript -// String -> JSONPath selector query -const pathQuery = await client.encryptQuery("$.user.email", { - column: documents.metadata, - table: documents, -}) +### Encrypted JSON -// Object/Array -> containment query -const containsQuery = await client.encryptQuery({ role: "admin" }, { - column: documents.metadata, - table: documents, -}) -``` +A `types.Json` column encrypts a whole JSON document (an object, array, or null — not a top-level scalar) to a `public.eql_v3_json` value. Two query patterns are supported: -### Batch Query Encryption - -Encrypt multiple query terms in one call: +**Exact containment** (jsonb `@>` semantics, no false positives). Pass a sub-object or sub-array needle; array containment is a subset test regardless of element position — `{ roles: ["admin"] }` matches any document whose `roles` array includes `"admin"`: ```typescript -const terms = [ - { value: "alice@example.com", column: users.email, table: users, queryType: "equality" as const }, - { value: "bob", column: users.email, table: users, queryType: "freeTextSearch" as const }, -] +const events = encryptedTable("events", { metadata: types.Json("metadata") }) -const results = await client.encryptQuery(terms) -``` - -### Query Result Formatting (`returnType`) - -By default `encryptQuery` returns an `Encrypted` object (the raw EQL JSON payload). Use `returnType` to change the output format: - -| `returnType` | Output | Use case | -|---|---|---| -| `'eql'` (default) | `Encrypted` object | Parameterized queries, ORMs accepting JSON | -| `'composite-literal'` | `string` | Supabase `.eq()`, string-based APIs | -| `'escaped-composite-literal'` | `string` | Embedding inside another string or JSON value | - -```typescript -// Get a composite literal string for use with Supabase -const term = await client.encryptQuery("alice@example.com", { - column: users.email, - table: users, - queryType: "equality", - returnType: "composite-literal", -}) - -// term.data is a string — use directly with .eq() -await supabase.from("users").select().eq("email", term.data) +const containsQuery = await client.encryptQuery( + { roles: ["admin"] }, + { column: events.metadata, table: events }, // queryType inferred: 'searchableJson' +) ``` -Each term in a batch can have its own `returnType`. +**JSONPath selectors** — equality and ordering at a path (`$.a`, `$.a.b` dot-notation object paths): -### Ordering Encrypted Data +- **Drizzle**: `ops.selector(events.metadata, "$.age")` returns comparison methods bound to the path — `eq`, `ne`, `gt`, `gte`, `lt`, `lte` (e.g. `await ops.selector(events.metadata, "$.age").gt(21)`). Its unique power over containment is *ordering* at a path; equality at a path is equivalently `contains(col, { age: 21 })`. +- **Supabase**: `selectorEq(col, path, value)` and `selectorNe(col, path, value)`. -**`ORDER BY` on encrypted columns requires operator family support in the database.** +Two semantics to know: -On databases without operator families (e.g. Supabase, or when EQL is installed with `--exclude-operator-family`), sorting on encrypted columns is not currently supported — regardless of the client or ORM used. Sort application-side after decrypting the results as a workaround. +- **`ne` includes absent paths.** A "not equal at path" query also matches rows where the path does not exist at all. +- **Array-leaf caveat:** a scalar needle does not match an array at the path. `selectorEq("payload", "$.roles", "admin")` does *not* match `{ roles: ["admin", "analyst"] }` — use containment for membership tests. -Operator family support for Supabase is being developed in collaboration with the Supabase and CipherStash teams and will be available in a future release. +`types.Json` carries no equality or ordering on the document itself, so applying `eq` / `gt` / `asc` directly to a `Json` column throws. -### PostgreSQL / Drizzle Integration Pattern - -Encrypted data is stored as an [EQL](https://github.com/cipherstash/encrypt-query-language) JSON payload. Install the EQL extension in PostgreSQL to enable searchable queries, then store encrypted data in `eql_v2_encrypted` columns. +### Batch Query Encryption -The `@cipherstash/stack-drizzle` module provides `encryptedType` for defining encrypted columns and `createEncryptionOperators` for querying them: +Encrypt multiple query terms in one call: ```typescript -import { pgTable, integer, timestamp } from "drizzle-orm/pg-core" -import { encryptedType, extractEncryptionSchema, createEncryptionOperators } from "@cipherstash/stack-drizzle" -import { Encryption } from "@cipherstash/stack" - -// Define schema with encrypted columns -const usersTable = pgTable("users", { - id: integer("id").primaryKey().generatedAlwaysAsIdentity(), - email: encryptedType("email", { - equality: true, - freeTextSearch: true, - orderAndRange: true, - }), - profile: encryptedType<{ name: string; bio: string }>("profile", { - dataType: "json", - searchableJson: true, - }), -}) - -// Initialize -const usersSchema = extractEncryptionSchema(usersTable) -const client = await Encryption({ schemas: [usersSchema] }) -const ops = createEncryptionOperators(client) - -// Query with auto-encrypting operators -const results = await db.select().from(usersTable) - .where(await ops.eq(usersTable.email, "alice@example.com")) +const terms = [ + { value: "alice@example.com", column: users.email, table: users, queryType: "equality" as const }, + { value: "bob", column: users.email, table: users, queryType: "freeTextSearch" as const }, +] -// JSONB queries on encrypted JSON columns -const jsonResults = await db.select().from(usersTable) - .where(await ops.jsonbPathExists(usersTable.profile, "$.bio")) +const results = await client.encryptQuery(terms) ``` -#### Drizzle `encryptedType` Config Options - -| Option | Type | Description | -|---|---|---| -| `dataType` | `"string"` \| `"number"` \| `"json"` | Plaintext data type (default: `"string"`) | -| `equality` | `boolean` \| `TokenFilter[]` | Enable equality index | -| `freeTextSearch` | `boolean` \| `MatchIndexOpts` | Enable free-text search index | -| `orderAndRange` | `boolean` | Enable ORE index for sorting/range queries | -| `searchableJson` | `boolean` | Enable JSONB path queries (requires `dataType: "json"`) | - -#### Drizzle JSONB Operators +### Ordering Encrypted Data -For columns with `searchableJson: true`, three JSONB operators are available: +`ORDER BY` works on encrypted ordering columns via the domain's order term: -| Operator | Description | -|---|---| -| `jsonbPathExists(col, selector)` | Check if a JSONB path exists (boolean, use in `WHERE`) | -| `jsonbPathQueryFirst(col, selector)` | Extract first value at a JSONB path | -| `jsonbGet(col, selector)` | Get value using the JSONB `->` operator | +- **Drizzle**: `ops.asc(col)` / `ops.desc(col)` emit `ORDER BY eql_v3.ord_term(col)` (or `eql_v3.ord_term_ore` for ORE domains). +- **Supabase**: `.order()` works on OPE-backed ordering columns — every plain `*Ord` domain plus `TextSearch`. -These operators encrypt the JSON path selector using the `steVecSelector` query type and cast it to `eql_v2_encrypted` for use with the EQL PostgreSQL functions. +The one limitation is the ORE-backed `*OrdOre` domains: their ordering term needs the superuser-only ORE operator class, which is unavailable on managed Postgres (e.g. Supabase) — the Supabase adapter rejects `order()` on those columns with a clear error. Prefer the plain `*Ord` (OPE) domains for anything you need to sort in a managed environment. -### EQL v3 Concrete-Type Columns (Drizzle) +### Drizzle Integration -The `@cipherstash/stack-drizzle/v3` module targets EQL v3, where each encrypted column is a **concrete Postgres domain type** (`eql_v3.text_eq`, `eql_v3.integer_ord`, …). Instead of toggling capability flags, you pick a concrete type from the `types` namespace and its query capabilities are fixed by that choice. It is the `/v3` subpath of the same `@cipherstash/stack-drizzle` package whose v2 surface is shown above. +The `@cipherstash/stack-drizzle/v3` subpath (of the separate `@cipherstash/stack-drizzle` package) provides Drizzle-native column factories, schema extraction, and auto-encrypting, capability-checked query operators. -Declare a Drizzle table using the `types` factories. The suffix encodes the capability: `*Eq` (equality), `*Ord` (order + range, which also covers equality), `*Match` / `TextSearch` (free-text search), and the bare name (e.g. `Text`, `Bigint`) is storage-only. +Declare a Drizzle table using the `types` factories — each factory emits its domain as the column's SQL type, so `drizzle-kit generate` produces `ADD COLUMN email public.eql_v3_text_search` etc.: ```ts import { pgTable, integer } from "drizzle-orm/pg-core" @@ -386,12 +361,12 @@ const users = pgTable("users", { id: integer("id").primaryKey().generatedAlwaysAsIdentity(), email: types.TextEq("email"), // equality: eq / ne / inArray age: types.IntegerOrd("age"), // order + range: gt/gte/lt/lte, between, asc/desc - bio: types.TextMatch("bio"), // free-text search: contains + bio: types.TextMatch("bio"), // free-text search: matches balance: types.Bigint("balance"), // storage only (no query capability) }) ``` -Derive the v3 schema from the table, build the typed client, and create the capability-checked operators: +Derive the v3 schema from the table, build the typed client, and create the operators: ```ts const usersSchema = extractEncryptionSchemaV3(users) @@ -420,9 +395,9 @@ const midBand = await db.select().from(users) const listed = await db.select().from(users) .where(await ops.inArray(users.email, ["alice@example.com", "bob@example.com"])) -// Free-text token containment — bio is TextMatch +// Free-text token match — bio is TextMatch const coffee = await db.select().from(users) - .where(await ops.contains(users.bio, "coffee")) + .where(await ops.matches(users.bio, "coffee")) ``` Rows are **pre-encrypted** with `client.bulkEncryptModels(...)` before they reach `db.insert(...).values(...)` — Drizzle never sees plaintext. `Bigint` columns take a native JS `bigint`: @@ -442,10 +417,26 @@ await db.insert(users).values(rows.data) Notes: -- **Free-text search uses `ops.contains`** (token containment on a `match` index), not SQL `like` / `ilike`. It matches whole indexed tokens, so `contains(users.bio, "coffee")` finds rows whose `bio` contains the token `coffee`. -- **The concrete type defines the legal operators.** `TextEq` supports `eq` / `ne` / `inArray` / `notInArray`; `*Ord` types add `gt` / `gte` / `lt` / `lte` / `between` / `notBetween` and `asc` / `desc`; `*Match` and `TextSearch` add `contains`; a bare `Text` / `Integer` / `Bigint` column is storage-only. Using an unsupported operator throws `EncryptionOperatorError`. +- **Free-text search is `ops.matches`** (fuzzy bloom token matching on `TextMatch` / `TextSearch` columns), not SQL `like` / `ilike` — those operators do not exist on the v3 surface. `ops.contains` is a *different* operator: exact encrypted-JSON containment on `types.Json` columns. +- **The concrete type defines the legal operators.** `TextEq` supports `eq` / `ne` / `inArray` / `notInArray`; `*Ord` types add `gt` / `gte` / `lt` / `lte` / `between` / `notBetween` and `asc` / `desc`; `TextMatch` and `TextSearch` add `matches`; `Json` supports `contains` and `selector(col, '$.path').{eq,ne,gt,gte,lt,lte}`; a bare `Text` / `Integer` / `Bigint` column is storage-only. Using an unsupported operator throws `EncryptionOperatorError`. - Combine conditions with `ops.and` / `ops.or`, and do NULL checks with `ops.isNull` / `ops.isNotNull` (the where-clause operators are `async` and must be `await`ed; `ops.asc` / `ops.desc` are synchronous). +### Supabase Integration + +`encryptedSupabaseV3` from the separate `@cipherstash/stack-supabase` package wraps a Supabase client and **introspects the database at connect time** — it detects EQL v3 columns by their Postgres domain and builds the encryption client internally: + +```typescript +import { encryptedSupabaseV3 } from "@cipherstash/stack-supabase" + +const es = await encryptedSupabaseV3(supabaseUrl, supabaseKey) + +await es.from("users").insert({ email: "a@b.com", age: 30 }) +await es.from("users").select("id, email").eq("email", "a@b.com") +await es.from("users").select("id, age").gte("age", 18).order("age") +``` + +Encrypted free-text search is `matches()` (fuzzy bloom token search — `contains()` stays native, exact containment for plaintext columns), encrypted-JSON path queries are `selectorEq()` / `selectorNe()`, and `order()` works on OPE-backed ordering columns. Pass optional declared `schemas` for compile-time row types. See the `stash-supabase` skill or the [docs](https://cipherstash.com/docs) for the full guide. + ## Authentication The client authenticates to ZeroKMS through `config.authStrategy`. Leave it @@ -461,7 +452,8 @@ explicit strategies cover the other cases: into a CipherStash service token: ```typescript -import { Encryption, OidcFederationStrategy } from "@cipherstash/stack" +import { OidcFederationStrategy } from "@cipherstash/stack" +import { EncryptionV3 } from "@cipherstash/stack/v3" // The callback is re-invoked on every (re-)federation and must return the // CURRENT third-party OIDC JWT. @@ -471,7 +463,7 @@ const strategy = OidcFederationStrategy.create( ) if (strategy.failure) throw new Error(strategy.failure.error.message) -const client = await Encryption({ +const client = await EncryptionV3({ schemas: [users], config: { authStrategy: strategy.data }, }) @@ -494,6 +486,7 @@ const IDENTITY = { identityClaim: ["sub"] } const encrypted = await client .encrypt("sensitive data", { column: users.email, table: users }) .withLockContext(IDENTITY) +if (encrypted.failure) throw new Error(encrypted.failure.message) const decrypted = await client .decrypt(encrypted.data) @@ -510,6 +503,8 @@ same claim must be supplied to encrypt and decrypt. Lock contexts work with all operations: `encrypt`, `decrypt`, `encryptModel`, `decryptModel`, `bulkEncryptModels`, `bulkDecryptModels`, `bulkEncrypt`, `bulkDecrypt`, `encryptQuery`. `.withLockContext()` also accepts a `LockContext` instance. +On the typed client, `decryptModel` / `bulkDecryptModels` take the lock +context as an optional third argument instead of chaining. > **Deprecated: `LockContext.identify()`.** Per-operation CTS tokens were removed > in `protect-ffi` 0.25; the token `identify()` fetches is no longer used by @@ -541,16 +536,18 @@ npx stash init --supabase The wizard will: 1. Authenticate with CipherStash (device code flow) -2. Bind your device to the default Keyset +2. Introspect your database and install the EQL v3 SQL 3. Choose your database connection method (Drizzle ORM, Supabase JS, Prisma, or Raw SQL) 4. Build an encryption schema interactively or use a placeholder, then generate the encryption client file 5. Install `stash` as a dev dependency for database tooling -After init, run `npx stash db setup` to configure your database. +`init` installs EQL for you — no separate `eql install` step is needed afterward. | Flag | Description | |------|-------------| -| `--supabase` | Use Supabase-specific setup flow | +| `--supabase` / `--drizzle` / `--prisma-next` | Target a specific integration's setup flow | +| `--proxy` / `--no-proxy` | Opt in/out of the CipherStash Proxy path | +| `--region ` | Workspace region (env `STASH_REGION`); **required for non-interactive init when not already logged in** | ## Configuration @@ -576,10 +573,10 @@ See the [Going to Production](https://cipherstash.com/docs/stack/deploy/going-to Pass config directly when initializing the client: ```typescript -import { Encryption } from "@cipherstash/stack" +import { EncryptionV3 } from "@cipherstash/stack/v3" import { users } from "./schema" -const client = await Encryption({ +const client = await EncryptionV3({ schemas: [users], config: { workspaceCrn: "crn:ap-southeast-2.aws:your-workspace-id", @@ -596,7 +593,7 @@ const client = await Encryption({ Isolate encryption keys per tenant using keysets: ```typescript -const client = await Encryption({ +const client = await EncryptionV3({ schemas: [users], config: { keyset: { id: "123e4567-e89b-12d3-a456-426614174000" }, @@ -604,7 +601,7 @@ const client = await Encryption({ }) // or by name -const client2 = await Encryption({ +const client2 = await EncryptionV3({ schemas: [users], config: { keyset: { name: "Company A" }, @@ -661,28 +658,38 @@ if (result.failure) { ## API Reference -### `Encryption(config)` - Initialize the client +### `EncryptionV3(config)` - Initialize the typed client ```typescript -function Encryption(config: EncryptionClientConfig): Promise +function EncryptionV3(config: { + schemas: AnyV3Table[] + config?: ClientConfig +}): Promise ``` -### `EncryptionClient` Methods +The wire format is pinned to EQL v3 — you don't set it yourself. `typedClient(client, ...schemas)` (same subpath) wraps an already-built `EncryptionClient` in the typed surface. + +### `TypedEncryptionClient` Methods + +Method signatures are derived from your schemas: plaintext arguments are pinned to each column's domain type, query methods only accept queryable columns, and `queryType` is constrained to the column's capabilities. | Method | Signature | Returns | |----|------|-----| | `encrypt` | `(plaintext, { column, table })` | `EncryptOperation` (thenable) | | `decrypt` | `(encryptedData)` | `DecryptOperation` (thenable) | | `encryptQuery` | `(plaintext, { column, table, queryType?, returnType? })` | `EncryptQueryOperation` (thenable) | + +`returnType` controls the encrypted query term's shape: `'eql'` (default, the EQL JSON payload for the ORM adapters), `'composite-literal'` (a Postgres composite string for `.eq()`/string-based APIs), or `'escaped-composite-literal'` (the same, escaped for embedding). Most users take the default; the adapters set it as needed. | `encryptQuery` | `(terms: ScalarQueryTerm[])` | `BatchEncryptQueryOperation` (thenable) | -| `encryptModel` | `(model, table)` | `EncryptModelOperation>` (thenable) | -| `decryptModel` | `(encryptedModel)` | `DecryptModelOperation` (thenable) | +| `encryptModel` | `(model, table)` | `EncryptModelOperation` (thenable) | +| `decryptModel` | `(encryptedModel, table, lockContext?)` | `Promise>` | +| `bulkEncryptModels` | `(models, table)` | `BulkEncryptModelsOperation` (thenable) | +| `bulkDecryptModels` | `(encryptedModels, table, lockContext?)` | `Promise>` | | `bulkEncrypt` | `(plaintexts, { column, table })` | `BulkEncryptOperation` (thenable) | | `bulkDecrypt` | `(encryptedPayloads)` | `BulkDecryptOperation` (thenable) | -| `bulkEncryptModels` | `(models, table)` | `BulkEncryptModelsOperation>` (thenable) | -| `bulkDecryptModels` | `(encryptedModels)` | `BulkDecryptModelsOperation` (thenable) | +| `getEncryptConfig` | `()` | The resolved encrypt config | -All operations are thenable (awaitable) and support `.withLockContext(lockContext)` for identity-aware encryption. +The thenable operations support `.withLockContext(lockContext)` for identity-aware encryption. `decryptModel` / `bulkDecryptModels` return a plain `Promise` instead — pass the lock context as the optional third argument. `decrypt` of a single value cannot be strongly typed (a lone ciphertext carries no column identity), and `encryptQuery` rejects storage-only columns at compile time. ### `LockContext` (legacy) @@ -695,38 +702,76 @@ and is no longer used by encryption. ### Schema Builders ```typescript -import { encryptedTable, encryptedColumn, csValue } from "@cipherstash/stack/schema" +import { encryptedTable, types } from "@cipherstash/stack/eql/v3" -encryptedTable(tableName, columns) -encryptedColumn(columnName) // returns EncryptedColumn -csValue(valueName) // returns ProtectValue (for nested values) +encryptedTable(tableName, columns) // columns: Record +types.TextSearch("email") // one factory per public.eql_v3_* domain +``` + +Type inference helpers live on the same subpath: + +```typescript +import type { InferPlaintext, InferEncrypted } from "@cipherstash/stack/eql/v3" + +type UserPlaintext = InferPlaintext +// { email: string; age: number; balance: bigint; metadata: JsonDocument } + +type UserEncrypted = InferEncrypted +// { email: Encrypted; age: Encrypted; ... } ``` ## Subpath Exports | Import Path | Provides | |-------|-----| -| `@cipherstash/stack` | `Encryption` function (main entry point) | -| `@cipherstash/stack/schema` | `encryptedTable`, `encryptedColumn`, `csValue`, schema types | +| `@cipherstash/stack/v3` | `EncryptionV3` typed client factory, `typedClient`, plus re-exports of the EQL v3 authoring DSL | +| `@cipherstash/stack/eql/v3` | EQL v3 authoring DSL: `encryptedTable`, the `types` namespace, `buildEncryptConfig`, inference types (`InferPlaintext`, `InferEncrypted`, ...) | +| `@cipherstash/stack` | `Encryption` function (legacy v2 entry point), auth strategies | +| `@cipherstash/stack/schema` | Legacy v2 schema builders (see [Legacy: EQL v2](#legacy-eql-v2)) | | `@cipherstash/stack/identity` | `LockContext` class and identity types | | `@cipherstash/stack/client` | Client-safe exports (schema builders and types only - no native FFI) | | `@cipherstash/stack/types` | All TypeScript types (`Encrypted`, `Decrypted`, `ClientConfig`, `EncryptionClientConfig`, query types, etc.) | -| `@cipherstash/stack/v3` | `EncryptionV3` typed client plus the EQL v3 authoring DSL (`encryptedTable`, `types`, v3 type helpers) | The Drizzle and Supabase integrations are **separate first-party packages** that depend on `@cipherstash/stack` (they are no longer subpaths of it): | Package | Provides | |-------|-----| -| `@cipherstash/stack-drizzle` | Drizzle ORM integration (EQL v2): `encryptedType`, `extractEncryptionSchema`, `createEncryptionOperators` | | `@cipherstash/stack-drizzle/v3` | EQL v3 Drizzle integration: `types` column factories, `createEncryptionOperatorsV3`, `extractEncryptionSchemaV3`, `makeEqlV3Column`, `EncryptionOperatorError` | -| `@cipherstash/stack-supabase` | Supabase integration: `encryptedSupabase` (v2) and `encryptedSupabaseV3` (v3) | - -## Migration from @cipherstash/protect - -If you are migrating from `@cipherstash/protect`, the following table maps the old API to the new one: - -| `@cipherstash/protect` | `@cipherstash/stack` | Import Path | +| `@cipherstash/stack-supabase` | Supabase integration: `encryptedSupabaseV3` (and the legacy v2 `encryptedSupabase`) | +| `@cipherstash/stack-drizzle` | Legacy EQL v2 Drizzle integration (root subpath): `encryptedType`, `extractEncryptionSchema`, `createEncryptionOperators` | + +## Legacy: EQL v2 + +Before the concrete-domain types above, encrypted columns were declared with +chainable capability builders and stored in a single `eql_v2_encrypted` +composite column type. That surface remains fully supported for existing +deployments, but new work should use EQL v3: + +- **Client and schema**: `Encryption` from `@cipherstash/stack` with + `encryptedColumn("email").equality().freeTextSearch().orderAndRange()` and + `.searchableJson()` from `@cipherstash/stack/schema`. v2 and v3 tables cannot + be mixed in one client. +- **Query formatting**: v2 query terms can be rendered as strings with + `returnType: 'composite-literal'` / `'escaped-composite-literal'` for + string-based APIs. +- **Integrations**: the v2 Drizzle surface is the root of + `@cipherstash/stack-drizzle` (`encryptedType`, `extractEncryptionSchema`, + `createEncryptionOperators`); the v2 Supabase surface is `encryptedSupabase`. +- **DynamoDB still requires v2**: `encryptedDynamoDB` from + `@cipherstash/stack/dynamodb` works with the v2 API only — v3 support is + tracked in [#657](https://github.com/cipherstash/stack/issues/657). + +Full v2 documentation lives at [cipherstash.com/docs](https://cipherstash.com/docs). + +### Migrating from @cipherstash/protect + +`@cipherstash/protect` users land on the legacy v2 surface first — the mapping +below is 1:1, and method signatures on the encryption client (`encrypt`, +`decrypt`, `encryptModel`, etc.) and the `Result` pattern (`data` / `failure`) +are unchanged. From there, adopt EQL v3 for new tables: + +| `@cipherstash/protect` | `@cipherstash/stack` (legacy v2) | Import Path | |------------|-----------|-------| | `protect(config)` | `Encryption(config)` | `@cipherstash/stack` | | `csTable(name, cols)` | `encryptedTable(name, cols)` | `@cipherstash/stack/schema` | @@ -734,12 +779,11 @@ If you are migrating from `@cipherstash/protect`, the following table maps the o | `import { LockContext } from "@cipherstash/protect/identify"` | `import { LockContext } from "@cipherstash/stack/identity"` | `@cipherstash/stack/identity` | | N/A | CLI | `npx stash` | -All method signatures on the encryption client (`encrypt`, `decrypt`, `encryptModel`, etc.) remain the same. The `Result` pattern (`data` / `failure`) is unchanged. - ## Requirements -- **Node.js** >= 18 -- The package includes a native FFI module (`@cipherstash/protect-ffi`) written in Rust and embedded via [Neon](https://github.com/neon-bindings/neon). You must opt out of bundling this package in tools like Webpack, esbuild, or Next.js (`serverExternalPackages`). +- **Node.js** >= 22 +- The default entry includes a native FFI module (`@cipherstash/protect-ffi`). On a Node server, externalize it from bundling (e.g. Next.js `serverExternalPackages`). +- For bundled or non-Node runtimes (Deno, Bun, Cloudflare Workers, Supabase Edge Functions), import `@cipherstash/stack/wasm-inline` instead — it inlines the WASM build, so no externalization is needed. See the [bundling guide](https://cipherstash.com/docs/stack/deploy/bundling). ## License diff --git a/packages/stack/__tests__/v3-matrix/matrix.test-d.ts b/packages/stack/__tests__/v3-matrix/matrix.test-d.ts index bbd23891..750abcdc 100644 --- a/packages/stack/__tests__/v3-matrix/matrix.test-d.ts +++ b/packages/stack/__tests__/v3-matrix/matrix.test-d.ts @@ -14,6 +14,7 @@ import { describe, expectTypeOf, it } from 'vitest' import { encryptedTable, type InferPlaintext, + type JsonDocument, type QueryableColumnsOf, type QueryTypesForColumn, } from '@/eql/v3' @@ -29,6 +30,9 @@ const records = encryptedTable('records', { createdAt: V3_MATRIX['public.eql_v3_timestamp_ord'].builder('created_at'), // date email: V3_MATRIX['public.eql_v3_text_search'].builder('email'), // string, full-text active: V3_MATRIX['public.eql_v3_boolean'].builder('active'), // boolean, storage-only + // Pinned so core's OWN suite guards the `searchableJson` arm rather than + // leaving it to a downstream adapter's typecheck (#650). + payload: V3_MATRIX['public.eql_v3_json'].builder('payload'), // JsonDocument, containment/selector only }) describe('eql_v3 type-driven matrix (types)', () => { @@ -41,6 +45,7 @@ describe('eql_v3 type-driven matrix (types)', () => { createdAt: Date email: string active: boolean + payload: JsonDocument }>() }) @@ -66,6 +71,13 @@ describe('eql_v3 type-driven matrix (types)', () => { expectTypeOf< QueryTypesForColumn >().toEqualTypeOf() + // The #650 arm: exactly 'searchableJson' — never the scalar kinds (a + // regression to `never` re-erases JSON columns from every typed adapter + // key set; leaking a scalar kind would open scalar predicates that cannot + // succeed at runtime). + expectTypeOf< + QueryTypesForColumn + >().toEqualTypeOf<'searchableJson'>() }) it('excludes storage-only columns from the queryable set', () => { diff --git a/packages/stack/package.json b/packages/stack/package.json index 1106aaaa..c71b75a1 100644 --- a/packages/stack/package.json +++ b/packages/stack/package.json @@ -1,6 +1,6 @@ { "name": "@cipherstash/stack", - "version": "0.19.0", + "version": "1.0.0-rc.0", "description": "CipherStash Stack for TypeScript and JavaScript", "keywords": [ "encrypted", diff --git a/packages/stack/src/adapter-kit.ts b/packages/stack/src/adapter-kit.ts index 8dcf6504..2d1fad59 100644 --- a/packages/stack/src/adapter-kit.ts +++ b/packages/stack/src/adapter-kit.ts @@ -43,6 +43,17 @@ export { stripDomainSchema, } from './eql/v3/domain-registry.js' +// Shared JSONPath-selector path handling (parse/validate, needle-document +// reconstruction, scalar-leaf guard) for encrypted-JSON querying — used by the +// Drizzle selector operators (#651) and the Supabase selector filters (#650) so +// the validation rules cannot drift between adapters. +export { + jsonPathOf, + parseSelectorSegments, + reconstructSelectorDocument, + unsupportedLeafReason, +} from './eql/v3/selector-path.js' + // Shared match-index guard (short-needle rejection), reused by both adapters. export { matchNeedleError } from './schema/match-defaults.js' // Shared structured logger (adapters log rejections through the same instance). diff --git a/packages/stack/src/eql/v3/columns.ts b/packages/stack/src/eql/v3/columns.ts index 6647088a..bd8895af 100644 --- a/packages/stack/src/eql/v3/columns.ts +++ b/packages/stack/src/eql/v3/columns.ts @@ -765,7 +765,7 @@ export type PlaintextForColumn = * The user-facing `queryType` names a v3 column supports, derived 1:1 from its * capability flags. Resolves to `never` for a storage-only column (all flags * `false`) and for any non-v3 value. The names mirror the {@link QueryCapabilities} - * keys and the first three {@link import('@/types').QueryTypeName} members. + * keys, each of which is also a {@link import('@/types').QueryTypeName} member. */ export type QueryTypesForColumn = C extends EncryptedV3Column @@ -777,6 +777,13 @@ export type QueryTypesForColumn = | (D['capabilities']['freeTextSearch'] extends true ? 'freeTextSearch' : never) + // The flag is optional (absent means "not a JSON document column"), so + // only a literal `true` contributes. Without this arm a `types.Json` + // column resolved to `never`, and every typed adapter key set derived + // from this type excluded encrypted-JSON columns entirely (#650). + | (D['capabilities']['searchableJson'] extends true + ? 'searchableJson' + : never) : never /** diff --git a/packages/stack/src/eql/v3/selector-path.ts b/packages/stack/src/eql/v3/selector-path.ts new file mode 100644 index 00000000..8e17843a --- /dev/null +++ b/packages/stack/src/eql/v3/selector-path.ts @@ -0,0 +1,131 @@ +/** + * Shared JSONPath-selector path handling for the first-party adapters + * (`@cipherstash/stack-drizzle`, `@cipherstash/stack-supabase`), exposed via + * `@cipherstash/stack/adapter-kit`. + * + * Both adapters express "compare the value at `$.a.b`" over an encrypted + * `eql_v3_json` column, and both need the same three pieces: parse + validate + * the dot-notation path, reconstruct the `{ a: { b: value } }` needle document + * whose ste_vec entry at the path carries the comparison terms, and reject + * non-scalar leaves up front. Originally private to the Drizzle v3 operators + * (#651); moved here when the Supabase adapter grew the same querying (#650) so + * the validation rules cannot drift between adapters. + */ + +/** + * Object keys that are prototype-pollution vectors — rejected outright (mirrors + * core's `FORBIDDEN_KEYS`), so a selector can never address them. + */ +const FORBIDDEN_SEGMENTS: ReadonlySet = new Set([ + '__proto__', + 'prototype', + 'constructor', +]) + +/** + * Parse a dot-notation JSONPath into its object-key segments. Rejects, each with + * a clear message: array-index/wildcard syntax (v1 is object-keys-only), the + * empty/root path, malformed paths (`..`, stray/leading/trailing dots, so we + * never silently query a *different* path), and prototype-pollution keys. + * `'$.a.b'` / `' a.b '` → `['a', 'b']`. + */ +export function parseSelectorSegments(path: string): string[] { + const trimmed = path.trim() + let body = trimmed.startsWith('$') ? trimmed.slice(1) : trimmed + if (body.startsWith('.')) body = body.slice(1) + if (body === '') { + throw new Error( + `JSON selector path "${path}" addresses no field — use e.g. "$.a" or "$.a.b".`, + ) + } + if (/[[\]*]/.test(body)) { + throw new Error( + `JSON selector path "${path}" uses array/wildcard syntax, which is not yet supported — use dot-notation object keys (e.g. "$.a.b").`, + ) + } + const segments = body.split('.') + for (const segment of segments) { + if (segment === '') { + throw new Error( + `JSON selector path "${path}" is malformed (empty segment / ".." / stray dot) — use dot-notation object keys (e.g. "$.a.b").`, + ) + } + // Leading/trailing whitespace in a segment is almost certainly spacing + // around a dot ('$.user .role'), which would silently address a DIFFERENT + // key ('user '). Interior whitespace stays legal — 'a b' is a valid JSON + // key a caller may genuinely have. + if (segment !== segment.trim()) { + throw new Error( + `JSON selector path "${path}" has whitespace at a segment boundary ("${segment}") — it would address a different key. Remove the spacing around the dot.`, + ) + } + if (FORBIDDEN_SEGMENTS.has(segment)) { + throw new Error( + `JSON selector path "${path}" addresses the forbidden key "${segment}".`, + ) + } + } + return segments +} + +/** `$`-rooted JSONPath for `encryptQuery`'s selector needle. */ +export function jsonPathOf(segments: string[]): string { + return `$.${segments.join('.')}` +} + +/** + * A selector compares a single scalar LEAF. Returns a reason string when `value` + * is unsupported — a non-scalar (object/array → that's `contains`), or a boolean + * under an ordering operator (no ordering term) — else `null`. Callers raise it + * as their adapter's operator error with column context, so a bad value is an + * actionable SDK error rather than a deferred, opaque DB failure. + */ +export function unsupportedLeafReason( + value: unknown, + ordering: boolean, +): string | null { + // Explicit null arm: `typeof null === 'object'`, so without it a null leaf + // would get the actively-wrong "got an object — use contains()" steer. + if (value == null) { + return 'a selector compares a non-null scalar leaf, but got null/undefined — SQL NULL never equals anything; use is(column, null) for null checks.' + } + const isScalar = + value instanceof Date || + typeof value === 'number' || + typeof value === 'bigint' || + typeof value === 'string' || + typeof value === 'boolean' + if (!isScalar) { + return `a selector compares a scalar leaf, but got ${Array.isArray(value) ? 'an array' : 'an object'} — use contains() for sub-object matching.` + } + if (ordering && typeof value === 'boolean') { + return 'a boolean leaf has no ordering — use eq/ne (or contains()).' + } + return null +} + +/** + * Nest `value` under the segments: `['a','b']` → `{ a: { b: value } }`. The + * storage-needle document whose ste_vec entry at the path supplies the + * ciphertext-bearing comparison entry. + */ +export function reconstructSelectorDocument( + segments: string[], + value: unknown, +): Record { + // Null-prototype objects: a segment like `__proto__` must become an OWN key, + // not invoke the prototype setter (which would drop it and mis-serialize the + // needle). JSON.stringify ignores the [[Prototype]], so this serializes fine. + const root: Record = Object.create(null) + let cursor = root + segments.forEach((segment, index) => { + if (index === segments.length - 1) { + cursor[segment] = value + } else { + const next: Record = Object.create(null) + cursor[segment] = next + cursor = next + } + }) + return root +} diff --git a/packages/test-kit/CHANGELOG.md b/packages/test-kit/CHANGELOG.md new file mode 100644 index 00000000..07cab3ed --- /dev/null +++ b/packages/test-kit/CHANGELOG.md @@ -0,0 +1,30 @@ +# @cipherstash/test-kit + +## 0.0.1-rc.0 + +### Patch Changes + +- Updated dependencies [31ca318] +- Updated dependencies [c4787c0] +- Updated dependencies [66a0e02] +- Updated dependencies [cfd46ee] +- Updated dependencies [7eba32d] +- Updated dependencies [0ebf57e] +- Updated dependencies [d73a03c] +- Updated dependencies [89b903f] +- Updated dependencies [229ce59] +- Updated dependencies [50c0a9c] +- Updated dependencies [63ca540] +- Updated dependencies [5d23e80] +- Updated dependencies [1aa9a11] +- Updated dependencies [af2d04e] +- Updated dependencies [b8a3d20] +- Updated dependencies [a0f3b2c] +- Updated dependencies [f23f952] +- Updated dependencies [7c7dbca] +- Updated dependencies [5411a13] +- Updated dependencies [99f8b0a] +- Updated dependencies [fd33aad] +- Updated dependencies [8cd485d] +- Updated dependencies [9b65ae8] + - @cipherstash/stack@1.0.0-rc.0 diff --git a/packages/test-kit/package.json b/packages/test-kit/package.json index f12c580c..a5e80f16 100644 --- a/packages/test-kit/package.json +++ b/packages/test-kit/package.json @@ -1,8 +1,8 @@ { "name": "@cipherstash/test-kit", - "version": "0.0.0", + "version": "0.0.1-rc.0", "private": true, - "description": "Shared EQL v3 test harness: the domain catalog, the plaintext oracle, and the integration-suite driver. Consumed as TypeScript source \u2014 no build step.", + "description": "Shared EQL v3 test harness: the domain catalog, the plaintext oracle, and the integration-suite driver. Consumed as TypeScript source — no build step.", "type": "module", "exports": { "./integration-clerk": "./src/integration/clerk.ts", diff --git a/packages/wizard/CHANGELOG.md b/packages/wizard/CHANGELOG.md index aef9d034..b2194362 100644 --- a/packages/wizard/CHANGELOG.md +++ b/packages/wizard/CHANGELOG.md @@ -1,5 +1,33 @@ # @cipherstash/wizard +## 0.5.0-rc.0 + +### Minor Changes + +- 0b9b192: Rename `stash db install` to `stash eql install`. The command scaffolds + `stash.config.ts` and installs the EQL extensions, so it now lives under a + dedicated `eql` command group. `stash db install` keeps working as a + deprecated alias that prints a warning pointing at the new name. All help + text, hints, generated migration headers, and wizard steps now reference + `stash eql install`. + +### Patch Changes + +- 9c673bb: Stop the agent guard from blocking `.env.example`. + + `SENSITIVE_FILE_PATTERNS` matched `/\.env($|\.)/`, which tests true against + `.env.example`. Because the guard covers `Edit` and `Write` as well as `Read`, + the wizard's agent was blocked from creating or editing the very file the + CipherStash doctrine tells it to write ("New env keys go in `.env.example` with + placeholders"). Committed env templates carry placeholder key names, not values. + + `.env.example`, `.env.sample` and `.env.template` are now readable and writable. + Everything else is unchanged: `.env`, `.env.local`, `.env.production`, and + value-bearing files that merely start with a template name + (`.env.example.local`, `.env.example.bak`) stay blocked, as do `auth.json`, + `secretkey.json` and credential files. Bash access to any env file — including + the templates — remains blocked; `Read`/`Write` is the sanctioned path. + ## 0.4.0 ### Minor Changes diff --git a/packages/wizard/package.json b/packages/wizard/package.json index 70a6d071..7cea274f 100644 --- a/packages/wizard/package.json +++ b/packages/wizard/package.json @@ -1,6 +1,6 @@ { "name": "@cipherstash/wizard", - "version": "0.4.0", + "version": "0.5.0-rc.0", "description": "AI-powered encryption setup for CipherStash. Reads your codebase, picks columns to encrypt, and wires everything up.", "repository": { "type": "git", diff --git a/packages/wizard/src/lib/analytics.ts b/packages/wizard/src/lib/analytics.ts index a8e78853..f58cdf35 100644 --- a/packages/wizard/src/lib/analytics.ts +++ b/packages/wizard/src/lib/analytics.ts @@ -3,43 +3,67 @@ * * Tracks wizard interactions, framework detection, completion rates, and errors. * Analytics are non-blocking — failures are silently ignored. + * + * Honors the same opt-outs as the `stash` CLI's telemetry: `DO_NOT_TRACK` + * (cross-tool standard), `STASH_TELEMETRY_DISABLED`, and CI auto-detection. + * The wizard is usually spawned by `stash wizard`, whose first-run notice + * promises those opt-outs on behalf of the whole CLI surface — this module + * must not make that promise false. */ +import { randomUUID } from 'node:crypto' import { PostHog } from 'posthog-node' import { POSTHOG_API_KEY, POSTHOG_HOST } from './constants.js' import type { Integration, WizardSession } from './types.js' let client: PostHog | undefined +/** + * An opt-out env var is "set" when present and not an explicit off value — + * matching the DO_NOT_TRACK convention (and the CLI's `envOptOut`) that setting + * the variable at all signals intent. + */ +function envOptOut(name: string): boolean { + const raw = process.env[name]?.trim().toLowerCase() + return raw != null && raw !== '' && raw !== '0' && raw !== 'false' +} + +function analyticsDisabled(): boolean { + return ( + envOptOut('DO_NOT_TRACK') || + envOptOut('STASH_TELEMETRY_DISABLED') || + Boolean(process.env.CI?.trim()) + ) +} + function getClient(): PostHog | undefined { if (!POSTHOG_API_KEY) return undefined + if (analyticsDisabled()) return undefined if (!client) { client = new PostHog(POSTHOG_API_KEY, { host: POSTHOG_HOST, flushAt: 1, flushInterval: 0, + // Never resolve IP → geo. + disableGeoip: true, }) } return client } -import { createHash } from 'node:crypto' -import { hostname, userInfo } from 'node:os' +/** + * Per-process random identifier. Deliberately NOT derived from anything + * identifying: the previous sha256(username@hostname) was reversible by + * brute-forcing common username/hostname pairs. Session-stable is enough for + * funnel analysis within one wizard run; cross-run identity is not worth an + * identifying hash. + */ +const SESSION_ID = randomUUID() -/** Generate a stable anonymous identifier for the session. */ function getDistinctId(): string { - try { - const user = userInfo().username - const host = hostname() - return createHash('sha256') - .update(`${user}@${host}`) - .digest('hex') - .slice(0, 16) - } catch { - return 'anonymous' - } + return SESSION_ID } // --- Event tracking --- @@ -103,6 +127,11 @@ export function trackWizardCompleted( }) } +/** + * `error` must be a FIXED LABEL or an error class name — never a raw + * message. Agent/DB error messages routinely embed file paths, table/column + * names, and connection-string fragments; those must not leave the machine. + */ export function trackWizardError(error: string, integration?: Integration) { getClient()?.capture({ distinctId: getDistinctId(), diff --git a/packages/wizard/src/run.ts b/packages/wizard/src/run.ts index 87259688..fab22186 100644 --- a/packages/wizard/src/run.ts +++ b/packages/wizard/src/run.ts @@ -262,7 +262,10 @@ export async function run(options: RunOptions) { }) } } else { - trackWizardError(result.error ?? 'unknown', selectedIntegration) + // Fixed label only — result.error is the agent's raw message and can + // embed schema names or connection details; it stays local (changelog + + // stderr below), never on the wire. + trackWizardError('agent_failed', selectedIntegration) changelog.note(`Agent failed: ${result.error ?? 'unknown error'}`) await changelog.flush() p.log.error(result.error ?? 'Agent failed without a specific error.') @@ -273,7 +276,12 @@ export async function run(options: RunOptions) { } catch (error) { const message = error instanceof Error ? error.message : 'Agent execution failed.' - trackWizardError(message, selectedIntegration) + // Class name only — the message can embed schema names or connection + // details; it stays local (changelog + stderr below), never on the wire. + trackWizardError( + error instanceof Error ? error.constructor.name : 'unknown', + selectedIntegration, + ) changelog.note(`Wizard threw: ${message}`) await changelog.flush() p.log.error(message) diff --git a/skills/stash-cli/SKILL.md b/skills/stash-cli/SKILL.md index fca5c955..c7103f5f 100644 --- a/skills/stash-cli/SKILL.md +++ b/skills/stash-cli/SKILL.md @@ -16,7 +16,7 @@ Think Prisma Migrate or Drizzle Kit: a dev-time tool that prepares the database, Use this skill when: - The user wants to set up CipherStash or install EQL in a PostgreSQL database. -- Any `stash` command is being run: `init`, `plan`, `impl`, `status`, `auth`, `eql`, `db`, `encrypt`, `schema`, `manifest`, `doctor`, `wizard`, `env`. +- Any `stash` command is being run: `init`, `plan`, `impl`, `status`, `auth`, `eql`, `db`, `encrypt`, `schema`, `manifest`, `doctor`, `telemetry`, `wizard`, `env`. - A `stash.config.ts` file exists or needs to be created. - A `.cipherstash/` directory exists (`context.json`, `plan.md`, `migrations.json`, `setup-prompt.md`). - The user mentions "stash CLI", "EQL install", "encryption schema", or an encryption rollout/cutover. @@ -140,6 +140,33 @@ First hit wins: The resolved URL is returned in memory only. It is never written to disk or into `process.env`. +## Telemetry + +The CLI collects **anonymous, opt-out** usage analytics — coarse events only +(command name, CLI version, OS/arch, Node version, success/failure, duration, +and a coarse caller class such as `claude-code`/`cursor`/`interactive` derived +from environment markers). Events carry a random install identifier — a UUID +generated locally and stored in `~/.cipherstash/telemetry.json`, not derived +from any machine, user, or hardware attribute — used only to de-duplicate +events in aggregate. It **never** collects plaintext, schema, table/column +names, connection strings, argument values, or any session/trace identifier. A +one-time notice is printed on first run, and nothing is sent on that first run. + +Opt out in any of these ways (any one wins; env vars override the saved +preference): + +| Mechanism | Effect | +|---|---| +| `DO_NOT_TRACK=1` | Honors the cross-tool standard; disables telemetry | +| `STASH_TELEMETRY_DISABLED=1` | Disables telemetry | +| `CI=true` (or common CI markers) | Auto-disabled in CI | +| `npx stash telemetry disable` | Persists opt-out to `~/.cipherstash/telemetry.json` | + +`npx stash telemetry status` reports the current state and which setting governs +it; `npx stash telemetry enable` clears the saved opt-out (env overrides still +apply). State lives in `~/.cipherstash/telemetry.json` — a non-secret file +distinct from the auth credentials in that directory. + ## Configuration `stash.config.ts` in the project root: @@ -279,6 +306,7 @@ Flags below are the decision-relevant ones. Run `stash --help` for the | `status` | Rollout quest log (above) | | `manifest [--json]` | Print the structured, versioned command surface | | `doctor` | Diagnose install problems (native binaries, runtime). Runs before the CLI body loads, so it works when the native binary is broken. | +| `telemetry [status\|enable\|disable]` | Manage anonymous usage analytics (below) | | `wizard` | AI-guided encryption setup — thin wrapper over `@cipherstash/wizard` | ### Auth @@ -308,7 +336,7 @@ Gets a project from zero to installed EQL. It loads an existing `stash.config.ts | `--migration` / `--direct` | Supabase: write a migration file, or run SQL directly | | `--migrations-dir ` | Supabase migrations directory (default `supabase/migrations`) | | `--exclude-operator-family` | Skip operator families (non-superuser roles) | -| `--eql-version <2\|3>` | EQL generation. **Default `2`.** `3` is the native `public.eql_v3_*` domain schema. | +| `--eql-version <2\|3>` | EQL generation. **Default `3`** (the native `public.eql_v3_*` domain schema — the documented approach). `2` is the legacy composite schema. | | `--latest` | Fetch latest EQL from GitHub instead of the bundled copy (**v2 only**) | | `--database-url ` | One-shot install (see below) | @@ -433,7 +461,7 @@ stash encrypt cutover --table users --column email In one transaction it renames `` → `_plaintext` and `_encrypted` → ``, advances the pending config to `encrypting`, activates it, and appends a `cut_over` event. With a Proxy URL configured (`--proxy-url` or `CIPHERSTASH_PROXY_URL`) it then calls `eql_v2.reload_config()` so Proxy picks up the new shape. -> **After cutover, `` holds ciphertext — the read path is not automatic.** Wire reads through the encryption client (`decryptModel(row, table)` for Drizzle, the `encryptedSupabase` wrapper for Supabase, otherwise `decrypt` / `bulkDecryptModels`) before returning values to callers. Skip this and your read paths hand raw EQL payloads to end users. The integration skill has the exact API. **CipherStash Proxy is the one exception** — it decrypts on the wire, so Proxy users need no application change. The cutover plan written by `stash plan` includes this read-path switch as an explicit step. +> **After cutover, `` holds ciphertext — the read path is not automatic.** Wire reads through the encryption client (`decryptModel(row, table)` for Drizzle, the `encryptedSupabaseV3` wrapper for Supabase — `encryptedSupabase` on the legacy v2 surface, otherwise `decrypt` / `bulkDecryptModels`) before returning values to callers. Skip this and your read paths hand raw EQL payloads to end users. The integration skill has the exact API. **CipherStash Proxy is the one exception** — it decrypts on the wire, so Proxy users need no application change. The cutover plan written by `stash plan` includes this read-path switch as an explicit step. > **Known gap.** The pending-configuration precondition is satisfied by `stash db push`. SDK-only users (who otherwise never need `db push`) must therefore run it once before `encrypt cutover`. Decoupling this — under EQL v3 there is no configuration table at all — is tracked in [issue #585](https://github.com/cipherstash/stack/issues/585). @@ -514,7 +542,7 @@ Required: `SUPERUSER`, **or** `CREATE` on the database *and* on the `public` sch **Supabase.** Always pass `--supabase` (or `supabase: true`). It selects a compatible install script and grants `anon`, `authenticated`, and `service_role`. -**`ORDER BY` on encrypted columns doesn't work.** When EQL is installed with `--supabase` or `--exclude-operator-family`, PostgreSQL operator families aren't created, so `ORDER BY` on an encrypted column is unsupported — regardless of client or ORM. Sort application-side after decrypting. Operator-family support for Supabase is in development. +**`ORDER BY` on encrypted columns:** on EQL v3, ordering works on OPE-backed columns — Drizzle emits `ORDER BY eql_v3.ord_term(col)`, and the Supabase adapter's `order()` sorts by the `col->op` term. ORE-flavour (`*OrdOre`) domains need a superuser-only operator class (unavailable on managed Postgres/Supabase) and are rejected; storage-only and equality/match-only columns have no ordering term. For those, order by a plaintext column or sort application-side. (The legacy v2 surface — bare `eql_v2_encrypted` — cannot order encrypted columns without operator families.) **The native binary won't load.** Run `stash doctor`. diff --git a/skills/stash-drizzle/SKILL.md b/skills/stash-drizzle/SKILL.md index 2b6dc44a..9269526f 100644 --- a/skills/stash-drizzle/SKILL.md +++ b/skills/stash-drizzle/SKILL.md @@ -1,21 +1,22 @@ --- name: stash-drizzle -description: Integrate CipherStash encryption with Drizzle ORM using @cipherstash/stack-drizzle. Covers the encryptedType column type, encrypted query operators (eq, like, ilike, gt/gte/lt/lte, between, inArray, asc/desc), schema extraction, batched and/or conditions, EQL migration generation, the EQL v3 integration (@cipherstash/stack-drizzle/v3), and the complete Drizzle integration workflow. Use when adding encryption to a Drizzle ORM project, defining encrypted Drizzle schemas, or querying encrypted columns with Drizzle. +description: Integrate CipherStash encryption with Drizzle ORM using @cipherstash/stack-drizzle/v3 (EQL v3). Covers the types.* encrypted column factories (concrete Postgres domains), auto-encrypting query operators (eq, ne, gt/gte/lt/lte, between, inArray, matches, contains, JSON selector, asc/desc), schema extraction, the EncryptionV3 typed client, database setup with stash eql install, and migrating existing plaintext columns to encrypted. Use when adding encryption to a Drizzle ORM project, defining encrypted Drizzle schemas, or querying encrypted columns with Drizzle. --- # CipherStash Stack - Drizzle ORM Integration -Guide for integrating CipherStash field-level encryption with Drizzle ORM using `@cipherstash/stack-drizzle`. Provides a custom column type for encrypted fields and query operators that transparently encrypt search values. +Guide for integrating CipherStash field-level encryption with Drizzle ORM using `@cipherstash/stack-drizzle/v3` (EQL v3). Provides Drizzle-native encrypted column factories and query operators that transparently encrypt search values — Drizzle never sees plaintext in a query. + +In EQL v3 every encrypted column is a **concrete Postgres domain** (`public.eql_v3_text_search`, `public.eql_v3_integer_ord`, ...) whose query capabilities are fixed by the type you pick — there is no capability config object. See the `stash-encryption` skill's "Schema Definition" section (the `types` catalog) for the full catalog and capability suffixes (`Eq`, `Ord`/`OrdOre`, `Match`, `Search`, `Json`). ## When to Use This Skill - Adding field-level encryption to a Drizzle ORM project -- Defining encrypted columns in Drizzle table schemas -- Querying encrypted data with type-safe operators -- Sorting and filtering on encrypted columns -- Generating EQL database migrations +- Defining encrypted columns in Drizzle table schemas with the v3 `types.*` factories +- Querying encrypted data with type-safe, auto-encrypting operators +- Sorting, filtering, and encrypted-JSONB querying on encrypted columns +- Migrating an existing plaintext column to encrypted - Building Express/Hono/Next.js APIs with encrypted Drizzle queries -- Using EQL v3 typed-schema columns in Drizzle (see "EQL v3 Integration" below) ## Installation @@ -25,71 +26,55 @@ npm install @cipherstash/stack @cipherstash/stack-drizzle drizzle-orm The Drizzle integration ships as its own first-party package, `@cipherstash/stack-drizzle`, which depends on `@cipherstash/stack`. Install both. +The v3 surface documented here lives on the `@cipherstash/stack-drizzle/v3` subpath. It is distinct from the older, separate `@cipherstash/drizzle` package (which is `@cipherstash/protect`-based, with different symbol names). ## Database Setup -### Install EQL Extension - -The EQL (Encrypt Query Language) PostgreSQL extension enables searchable encryption functions. Generate a migration: +> **Runner note.** `stash init` adds `stash` to the project as a dev dependency, so `stash ` runs through whichever package manager the project uses (Bun, pnpm, Yarn, or npm) — examples in this skill show this bare form. Before init has run, prefix with your package manager's one-shot runner: `bunx`, `pnpm dlx`, `yarn dlx`, or `npx`. The CLI's behaviour is identical across all of them. -```bash -npx generate-eql-migration -# Options: -# -n, --name Migration name (default: "install-eql") -# -o, --out Output directory (default: "drizzle") -``` +### Install the EQL v3 SQL -Then apply it: +EQL (Encrypt Query Language) provides the PostgreSQL functions and domains that make encrypted columns searchable. Install version 3 directly against the database: ```bash -npx drizzle-kit migrate +stash eql install --eql-version 3 ``` +v3 installs via the direct path only — the v2 `stash eql install --drizzle` Drizzle-migration flow is **not** supported for v3 (`--drizzle`, `--migration`, `--migrations-dir`, and `--latest` are v2-only flags). EQL v3 ships one SQL bundle for every target, including Supabase. + ### Column Storage -Encrypted columns use the `eql_v2_encrypted` PostgreSQL type (installed by EQL). If not using EQL directly, use JSONB: +Each encrypted column is a concrete Postgres domain named `public.eql_v3_`: ```sql CREATE TABLE users ( - id SERIAL PRIMARY KEY, - email eql_v2_encrypted, -- with EQL extension - name jsonb NOT NULL, -- or use jsonb - age INTEGER -- non-encrypted columns are normal types + id INTEGER PRIMARY KEY GENERATED ALWAYS AS IDENTITY, + email public.eql_v3_text_search, -- equality + order/range + free-text + age public.eql_v3_integer_ord, -- equality + order/range + profile public.eql_v3_json, -- encrypted-JSONB queries + role VARCHAR(50) -- non-encrypted columns are normal types ); ``` +You don't usually hand-write this: the `types.*` factories below emit the domain as the column's SQL type, so `drizzle-kit generate` produces the `ADD COLUMN email public.eql_v3_text_search` DDL for you. + ## Schema Definition -Use `encryptedType()` to define encrypted columns in Drizzle table schemas: +Use the `types` namespace from `@cipherstash/stack-drizzle/v3` to define encrypted columns. Each factory maps 1:1 to a Postgres domain, and the column's query capabilities are fixed by the type: ```typescript import { pgTable, integer, timestamp, varchar } from "drizzle-orm/pg-core" -import { encryptedType } from "@cipherstash/stack-drizzle" +import { types } from "@cipherstash/stack-drizzle/v3" const usersTable = pgTable("users", { id: integer("id").primaryKey().generatedAlwaysAsIdentity(), - // Encrypted string with search capabilities - email: encryptedType("email", { - equality: true, // enables: eq, ne, inArray - freeTextSearch: true, // enables: like, ilike - orderAndRange: true, // enables: gt, gte, lt, lte, between, asc, desc - }), - - // Encrypted number - age: encryptedType("age", { - dataType: "number", - equality: true, - orderAndRange: true, - }), - - // Encrypted JSON object with searchable JSONB queries - profile: encryptedType<{ name: string; bio: string }>("profile", { - dataType: "json", - searchableJson: true, - }), + email: types.TextSearch("email"), // equality + order/range + free-text + age: types.IntegerOrd("age"), // equality + order/range + notes: types.Text("notes"), // storage only — encrypt/decrypt, no queries + profile: types.Json("profile"), // encrypted-JSONB containment + selector // Non-encrypted columns role: varchar("role", { length: 50 }), @@ -97,44 +82,52 @@ const usersTable = pgTable("users", { }) ``` -### `encryptedType(name, config?)` +Capability suffixes at a glance (full catalog: `stash-encryption` skill, "Schema Definition" — the `types` namespace): -| Config Option | Type | Description | +| Factory shape | Domain | Enables | |---|---|---| -| `dataType` | `"string"` \| `"number"` \| `"json"` \| `"boolean"` \| `"bigint"` \| `"date"` | Plaintext data type (default: `"string"`) | -| `equality` | `boolean` \| `TokenFilter[]` | Enable equality index | -| `freeTextSearch` | `boolean` \| `MatchIndexOpts` | Enable free-text search index | -| `orderAndRange` | `boolean` | Enable ORE index for sorting and range queries | -| `searchableJson` | `boolean` | Enable JSONB path queries (requires `dataType: "json"`) | +| `types.Text`, `types.Integer`, ... (no suffix) | `eql_v3_text`, ... | Storage only | +| `types.TextEq`, `types.IntegerEq`, ... | `eql_v3_text_eq`, ... | `eq`, `ne`, `inArray`, `notInArray` | +| `types.IntegerOrd`, `types.TimestampOrd`, ... | `eql_v3_integer_ord`, ... | equality + `gt`/`gte`/`lt`/`lte`/`between`/`asc`/`desc` | +| `types.IntegerOrdOre`, ... | `eql_v3_integer_ord_ore`, ... | as `Ord`, with block-ORE ordering (superuser-only install — see Sorting) | +| `types.TextMatch` | `eql_v3_text_match` | `matches` (fuzzy free-text) only | +| `types.TextSearch` | `eql_v3_text_search` | equality + order/range + `matches` | +| `types.Json` | `eql_v3_json` | `contains` + `selector` (encrypted JSONB) | -The generic type parameter `` sets the TypeScript type for the decrypted value. +Value families: `Integer`/`Smallint`/`Numeric`/`Real`/`Double` (`number`), `Bigint` (`bigint`), `Date`/`Timestamp` (`Date`), `Text` (`string`), `Boolean` (`boolean`, storage only), `Json` (a JSON document — object or array, not a top-level scalar). + +`makeEqlV3Column(builder)` wraps a column builder from `@cipherstash/stack/eql/v3` (e.g. `makeEqlV3Column(v3types.TextEq("email"))`) — `types.TextEq("email")` from the Drizzle subpath is shorthand for the same thing. ## Initialization ### 1. Extract Schema from Drizzle Table ```typescript -import { extractEncryptionSchema, createEncryptionOperators } from "@cipherstash/stack-drizzle" -import { Encryption } from "@cipherstash/stack" +import { extractEncryptionSchemaV3, createEncryptionOperatorsV3 } from "@cipherstash/stack-drizzle/v3" +import { EncryptionV3 } from "@cipherstash/stack/v3" -// Convert Drizzle table definition to CipherStash schema -const usersSchema = extractEncryptionSchema(usersTable) +// Convert the Drizzle table definition to a CipherStash v3 schema +const usersSchema = extractEncryptionSchemaV3(usersTable) ``` -### 2. Initialize Encryption Client +### 2. Initialize the Encryption Client ```typescript -const encryptionClient = await Encryption({ +const encryptionClient = await EncryptionV3({ schemas: [usersSchema], }) ``` +`EncryptionV3` returns a strongly-typed client: plaintext types are pinned to each column's domain, and query methods only accept queryable columns. + ### 3. Create Query Operators ```typescript -const encryptionOps = createEncryptionOperators(encryptionClient) +const ops = createEncryptionOperatorsV3(encryptionClient) ``` +`createEncryptionOperatorsV3(client, { lockContext, audit })` optionally sets defaults applied to every operand encryption; the async encrypting operators (`eq`, `ne`, `inArray`, `notInArray`, `gt`/`gte`/`lt`/`lte`, `between`/`notBetween`, `matches`, `contains`, and the comparison methods returned by `selector(...)`) also take an optional trailing `{ lockContext, audit }` argument per call. `asc`/`desc` and the passthrough operators (`isNull`, `isNotNull`, `not`, `and`, `or`, `exists`, `notExists`) encrypt nothing and take no such argument. + ### 4. Create Drizzle Instance ```typescript @@ -146,7 +139,7 @@ const db = drizzle({ client: postgres(process.env.DATABASE_URL!) }) ## Insert Encrypted Data -Encrypt models before inserting: +Rows are pre-encrypted with the client before `db.insert` — Drizzle only ever handles the encrypted EQL envelope: ```typescript // Single insert @@ -173,162 +166,175 @@ if (!encrypted.failure) { ## Query Encrypted Data +Operators auto-encrypt their plaintext operands into EQL v3 query terms — you pass plaintext, the emitted SQL compares encrypted values. Comparison operators are async (they encrypt), so `await` them (or hand them lazily to `ops.and`/`ops.or`, below). + ### Equality ```typescript -// Exact match - await the operator const results = await db .select() .from(usersTable) - .where(await encryptionOps.eq(usersTable.email, "alice@example.com")) + .where(await ops.eq(usersTable.email, "alice@example.com")) ``` -### Text Search +### Free-Text Search (`matches`) -```typescript -// Case-insensitive search -const results = await db - .select() - .from(usersTable) - .where(await encryptionOps.ilike(usersTable.email, "%alice%")) +`matches(col, needle)` is fuzzy bloom-token matching on a `TextMatch`/`TextSearch` column — **not** SQL pattern matching. There are no `like`/`ilike` operators on the v3 surface, by design; don't pass `%` wildcards. -// Case-sensitive search +```typescript const results = await db .select() .from(usersTable) - .where(await encryptionOps.like(usersTable.name, "%Smith%")) + .where(await ops.matches(usersTable.email, "alice")) ``` +Semantics to know: + +- **Fuzzy and one-sided.** The needle's downcased token set is bloom-tested as a subset of the column's — order- and multiplicity-insensitive. A match may be a false positive; a non-match never is. Re-check candidates after decryption if you need exactness. +- **Case-insensitive**, and matches substrings of 3 characters or more. +- **Short needles are rejected.** A needle shorter than the tokenizer's token length (3 by default) produces no tokens and would silently match every row, so the operator throws `EncryptionOperatorError` instead. + ### Range Queries ```typescript const results = await db .select() .from(usersTable) - .where(await encryptionOps.gte(usersTable.age, 18)) + .where(await ops.gte(usersTable.age, 18)) const results = await db .select() .from(usersTable) - .where(await encryptionOps.between(usersTable.age, 18, 65)) + .where(await ops.between(usersTable.age, 18, 65)) ``` -### Array Operators +### Array Membership ```typescript const results = await db .select() .from(usersTable) - .where(await encryptionOps.inArray(usersTable.email, [ + .where(await ops.inArray(usersTable.email, [ "alice@example.com", "bob@example.com", ])) ``` +`inArray`/`notInArray` reject an empty list and encrypt the whole list in a single batch crossing. + ### Sorting ```typescript -// Sort by encrypted column (sync - no await needed) +// Sort by encrypted column (sync — no await needed) const results = await db .select() .from(usersTable) - .orderBy(encryptionOps.asc(usersTable.age)) + .orderBy(ops.asc(usersTable.age)) const results = await db .select() .from(usersTable) - .orderBy(encryptionOps.desc(usersTable.age)) + .orderBy(ops.desc(usersTable.age)) ``` -**Note:** Sorting on encrypted columns requires operator family support in the database. On databases without operator families (e.g. Supabase, or when installed with `--exclude-operator-family`), `ORDER BY` on encrypted columns is not currently supported. Sort application-side after decrypting instead. Operator family support for Supabase is being developed with the Supabase and CipherStash teams. +`ops.asc`/`ops.desc` emit `ORDER BY eql_v3.ord_term(col)` (`ord_term_ore(col)` for the `*OrdOre` domains). The ORE-flavoured domains require a superuser install and are unavailable on managed Postgres (Supabase, RDS, etc.) — prefer the plain `Ord` domains there; ordering works everywhere EQL v3 installs. -## JSONB Queries +### Encrypted-JSONB Containment (`contains`) -Query encrypted JSON columns using JSONB operators. These require `searchableJson: true` and `dataType: "json"` in the column's `encryptedType` config. - -### Check path existence +`contains(col, subDoc)` on a `types.Json` column is **exact** encrypted containment (jsonb `@>` semantics, no false positives). The needle is a ciphertext-free `query_jsonb` term. Array containment is position-independent — `{ roles: ["admin"] }` matches any document whose `roles` array includes `"admin"`: ```typescript -// Check if a JSONB path exists in an encrypted column const results = await db .select() .from(usersTable) - .where(await encryptionOps.jsonbPathExists(usersTable.profile, "$.bio")) + .where(await ops.contains(usersTable.profile, { roles: ["admin"] })) ``` -### Extract value at path +An empty-object needle (`{}`) is rejected — `doc @> '{}'` holds for every document, so it would silently match every row. Omit the predicate if you want all rows. -```typescript -// Extract the first matching value at a JSONB path -const result = await encryptionOps.jsonbPathQueryFirst(usersTable.profile, "$.name") -``` +`types.Json` carries no equality or ordering: `eq`/`gt`/`asc` on a `Json` column throw. -### Get value with `->` operator +### JSONPath Selector-with-Constraint (`selector`) -```typescript -// Get a value using the JSONB -> operator -const result = await encryptionOps.jsonbGet(usersTable.profile, "$.name") -``` - -> **Note:** `jsonbPathExists` returns a boolean and can be used in `WHERE` clauses. `jsonbPathQueryFirst` and `jsonbGet` return encrypted values — use them in `SELECT` expressions. - -### Combine JSONB with other operators +`ops.selector(col, path)` returns comparison methods bound to the encrypted value at a JSONPath inside a `types.Json` column. Its unique power over `contains` is **ordering at a path**: ```typescript +// col->'$.age' > 25 const results = await db .select() .from(usersTable) - .where( - await encryptionOps.and( - encryptionOps.jsonbPathExists(usersTable.profile, "$.name"), - encryptionOps.eq(usersTable.email, "jane@example.com"), - ), - ) + .where(await ops.selector(usersTable.profile, "$.age").gt(25)) + +// col->'$.user' = 'zoe@example.com' +const results = await db + .select() + .from(usersTable) + .where(await ops.selector(usersTable.profile, "$.user").eq("zoe@example.com")) ``` -## Batched Conditions (and / or) +Available methods: `.eq`, `.ne`, `.gt`, `.gte`, `.lt`, `.lte`. Rules: + +- **Paths are dot-notation object keys only** (`"$.a.b"`). Array-index and wildcard syntax (`$.items[0]`) is rejected. +- **Leaves are scalars only**: `string`, `number`, `boolean`, `Date`, or `bigint`. An object or array leaf is rejected — use `contains` for sub-object matching. A `boolean` leaf is rejected under the ordering methods (booleans have no ordering). +- **A scalar needle does not match an array at the path.** `selector(col, "$.tags").eq("a")` will not match `{ tags: ["a"] }` — use `contains(col, { tags: ["a"] })` for that. +- **Absent-path semantics:** `eq` and the ordering methods exclude rows whose document lacks the path; `ne` **includes** them ("not equal to X" covers "has no X"). +- **Interim ciphertext disclosure ([cipherstash/protectjs-ffi#137](https://github.com/cipherstash/protectjs-ffi/issues/137)):** the selector's right-hand value is currently a storage-encrypted needle, so its ciphertext appears in the WHERE clause (and therefore in Postgres logs that capture query text). The comparison itself only reads the needle's index terms. Once #137 lands, the needle becomes ciphertext-free like every other operand. -Use `encryptionOps.and()` and `encryptionOps.or()` to batch multiple encrypted conditions into a single ZeroKMS call. This is more efficient than awaiting each operator individually. +### Batched Conditions (and / or) + +Use `ops.and()` and `ops.or()` to combine encrypted conditions. Pass the operators **lazily** (no `await`) so they resolve concurrently, then `await` the outer call: ```typescript -// Batched AND - all encryptions happen in one call const results = await db .select() .from(usersTable) .where( - await encryptionOps.and( - encryptionOps.gte(usersTable.age, 18), // no await needed - encryptionOps.lte(usersTable.age, 65), // lazy operators - encryptionOps.ilike(usersTable.email, "%example.com"), - eq(usersTable.role, "admin"), // mix with regular Drizzle ops + await ops.and( + ops.gte(usersTable.age, 18), // no await — lazy + ops.lte(usersTable.age, 65), + ops.matches(usersTable.email, "example"), + eq(usersTable.role, "admin"), // mix with regular Drizzle ops ), ) -// Batched OR const results = await db .select() .from(usersTable) .where( - await encryptionOps.or( - encryptionOps.eq(usersTable.email, "alice@example.com"), - encryptionOps.eq(usersTable.email, "bob@example.com"), + await ops.or( + ops.eq(usersTable.email, "alice@example.com"), + ops.eq(usersTable.email, "bob@example.com"), ), ) ``` -**Key pattern:** Pass lazy operators (no `await`) to `and()`/`or()`, then `await` the outer call. This batches all encryption into a single operation. +Both accept `undefined` conditions, which are filtered out — useful for conditional query building: + +```typescript +await ops.and( + maybeEmail ? ops.eq(usersTable.email, maybeEmail) : undefined, + ops.gte(usersTable.age, 18), +) +``` + +### NULLs and Non-Encrypted Columns + +- A `null` operand throws — use `ops.isNull(col)` / `ops.isNotNull(col)` for NULL checks. +- **No plaintext-column fallback.** Every v3 operator requires an encrypted v3 column and throws `EncryptionOperatorError` otherwise. Use regular Drizzle operators (`eq`, `gte`, ...) for non-encrypted columns — mixing the two inside `ops.and`/`ops.or` is fine. ## Decrypt Results +Selected rows hold encrypted envelopes; decrypt with the client. The v3 `decryptModel`/`bulkDecryptModels` take the schema table as the second argument: + ```typescript // Single model -const decrypted = await encryptionClient.decryptModel(results[0]) +const decrypted = await encryptionClient.decryptModel(results[0], usersSchema) if (!decrypted.failure) { console.log(decrypted.data.email) // "alice@example.com" } // Bulk decrypt -const decrypted = await encryptionClient.bulkDecryptModels(results) +const decrypted = await encryptionClient.bulkDecryptModels(results, usersSchema) if (!decrypted.failure) { for (const user of decrypted.data) { console.log(user.email) @@ -336,13 +342,15 @@ if (!decrypted.failure) { } ``` +`Date` columns are reconstructed to real `Date` instances on decrypt; `bigint` columns round-trip as native `bigint`. Non-schema fields pass through unchanged. + ## Migrating an Existing Column to Encrypted The hard case: a Drizzle table that already exists in production with live data in a plaintext column you want to encrypt. You can't just change the column type — that would drop the data and break NOT NULL constraints. CipherStash splits this into two named steps with a hard production-deploy gate between them: an **encryption rollout** (schema-add + dual-write code) and an **encryption cutover** (backfill + rename + drop). (If using CipherStash Proxy, the rollout also includes `stash db push` to register the encryption config with EQL.) The `stash-encryption` skill is the canonical reference for the lifecycle; this section walks the Drizzle-specific shape. -> **Runner note.** `stash init` adds `stash` to the project as a dev dependency, so `stash ` runs through whichever package manager the project uses (Bun, pnpm, Yarn, or npm) — examples below show this bare form. Before init has run, prefix with your package manager's one-shot runner: `bunx`, `pnpm dlx`, `yarn dlx`, or `npx`. The CLI's behaviour is identical across all of them. +> **⚠️ v3 backfill tooling status.** The CLI backfill/cutover tooling (`stash encrypt backfill`, `stash encrypt cutover`, and the underlying `@cipherstash/migrate`) currently targets **EQL v2 columns**. v3 compatibility is tracked in [cipherstash/stack#648](https://github.com/cipherstash/stack/issues/648). The lifecycle below (schema-add → dual-write → deploy gate → backfill → cutover → drop) is the correct shape for v3 either way — until #648 lands, run the backfill/rename steps with your own scripts (encrypt with `bulkEncryptModels`, write in chunks) instead of the `stash encrypt` commands. > **Where am I?** Run `stash status` first (substitute the runner per the note above). It shows you which Drizzle tables/columns are mid-rollout, which are post-deploy, and what the next move is. Re-run after every transition. @@ -370,15 +378,12 @@ Add an `email_encrypted` column **alongside** `email`. Crucially, the encrypted ```typescript // src/db/schema.ts -import { encryptedType } from '@cipherstash/stack-drizzle' +import { types } from '@cipherstash/stack-drizzle/v3' export const users = pgTable('users', { id: integer('id').primaryKey().generatedAlwaysAsIdentity(), - email: text('email').notNull(), // unchanged - email_encrypted: encryptedType('email_encrypted', { // new — nullable - freeTextSearch: true, - equality: true, - }), + email: text('email').notNull(), // unchanged + email_encrypted: types.TextSearch('email_encrypted'), // new — nullable }) ``` @@ -386,27 +391,27 @@ Update the encryption client to harvest the encrypted columns from the table: ```typescript // src/encryption/index.ts -import { Encryption } from '@cipherstash/stack' -import { extractEncryptionSchema } from '@cipherstash/stack-drizzle' +import { EncryptionV3 } from '@cipherstash/stack/v3' +import { extractEncryptionSchemaV3 } from '@cipherstash/stack-drizzle/v3' import { users } from '../db/schema' -const usersEncryptionSchema = extractEncryptionSchema(users) +const usersEncryptionSchema = extractEncryptionSchemaV3(users) -export const encryptionClient = await Encryption({ schemas: [usersEncryptionSchema] }) +export const encryptionClient = await EncryptionV3({ schemas: [usersEncryptionSchema] }) ``` -Generate the migration with `drizzle-kit generate`. The generated SQL should be a single `ALTER TABLE ... ADD COLUMN email_encrypted eql_v2_encrypted;`. Apply with `drizzle-kit migrate`. +Generate the migration with `drizzle-kit generate`. The generated SQL should be a single `ALTER TABLE ... ADD COLUMN email_encrypted public.eql_v3_text_search;`. Apply with `drizzle-kit migrate`. (This requires the EQL v3 SQL to be installed first — see Database Setup.) > **Using CipherStash Proxy?** -> +> > If your app queries encrypted data through CipherStash Proxy, register the new encryption config with EQL: -> +> > ```bash > stash db push > ``` -> +> > If this is the project's first encrypted column, `db push` writes directly to the active EQL config (nothing to rename). If an active config already exists, `db push` writes the new config as `pending` — that's expected. The pending row will be promoted to active by `stash encrypt cutover` in the cutover step. -> +> > SDK-only users can skip this step. #### Dual-writing: write to both columns from app code @@ -452,6 +457,8 @@ Once dual-writes are live in production and `cs_migrations` records `dual_writin #### Backfill: encrypt the historical rows +> Until [#648](https://github.com/cipherstash/stack/issues/648) lands, the commands in this step target v2 columns — for v3 columns, replicate the same shape with a script (chunked `bulkEncryptModels` + UPDATE inside transactions, resumable and idempotent). + ```bash stash encrypt backfill --table users --column email # (Interactive: answer 'yes' to the dual-write confirmation prompt.) @@ -462,16 +469,16 @@ Resumable, idempotent, chunked. The CLI walks the table in keyset-pagination ord If something goes wrong (e.g. you discover the dual-write code wasn't actually live when backfill ran), re-run with `--force` to re-encrypt every row regardless of current state. -> **SDK-only note:** `stash encrypt cutover` currently requires a pending EQL configuration set by `stash db push`. If you're using the SDK without Proxy, you'll hit a "No pending EQL configuration" error from cutover. **Workaround:** run `stash db push` once before `stash encrypt cutover`. [Issue #447](https://github.com/cipherstash/stack/issues/447) tracks decoupling this requirement. +> **SDK-only note:** `stash encrypt cutover` currently requires a pending EQL configuration set by `stash db push`. If you're using the SDK without Proxy, you'll hit a "No pending EQL configuration" error from cutover. **Workaround:** run `stash db push` once before `stash encrypt cutover`. #### Cutover: rename swap and activate -First, update the Drizzle schema to the post-cutover shape — switch `email` to use `encryptedType` and remove the `email_encrypted` column. +First, update the Drizzle schema to the post-cutover shape — switch `email` to the encrypted type and remove the `email_encrypted` column. > **Using CipherStash Proxy?** -> +> > If using Proxy, re-push the encryption config so EQL has a pending row that points at `email` (no `_encrypted` suffix): -> +> > ```bash > stash db push > # → writes the new config as `pending`. Active config (still pointing at @@ -492,10 +499,7 @@ The Drizzle schema you just edited now matches the physical DB shape — `email` // src/db/schema.ts (post-cutover) export const users = pgTable('users', { id: integer('id').primaryKey().generatedAlwaysAsIdentity(), - email: encryptedType('email', { - freeTextSearch: true, - equality: true, - }), + email: types.TextSearch('email'), email_plaintext: text('email_plaintext'), // temporary; dropped next }) ``` @@ -511,12 +515,12 @@ const email = rows[0].email // After const rows = await db.select().from(users).where(eq(users.id, id)) -const decrypted = await encryptionClient.decryptModel(rows[0]) +const decrypted = await encryptionClient.decryptModel(rows[0], usersEncryptionSchema) if (decrypted.failure) throw new Error(decrypted.failure.message) const email = decrypted.data.email ``` -For queries that filter on `email`, switch to the encrypted operators from `createEncryptionOperators` — `eq`, `like`, `gte`, etc. (See `## Query Encrypted Data` above.) +For queries that filter on `email`, switch to the encrypted operators from `createEncryptionOperatorsV3` — `eq`, `matches`, `gte`, etc. (See `## Query Encrypted Data` above.) #### Drop: remove the plaintext column @@ -532,10 +536,7 @@ The CLI emits a Drizzle migration file with `ALTER TABLE users DROP COLUMN email // src/db/schema.ts (final) export const users = pgTable('users', { id: integer('id').primaryKey().generatedAlwaysAsIdentity(), - email: encryptedType('email', { - freeTextSearch: true, - equality: true, - }), + email: types.TextSearch('email'), }) ``` @@ -553,148 +554,58 @@ All three are read-only. ## Complete Operator Reference +All comparison/containment operators auto-encrypt their operands and are async; `asc`/`desc` and the passthroughs are sync. + ### Encrypted Operators (async) -| Operator | Usage | Required Index | +| Operator | Usage | Required column capability (domain suffix) | |---|---|---| -| `eq(col, value)` | Equality | `equality: true` or `orderAndRange: true` | -| `ne(col, value)` | Not equal | `equality: true` or `orderAndRange: true` | -| `gt(col, value)` | Greater than | `orderAndRange: true` | -| `gte(col, value)` | Greater than or equal | `orderAndRange: true` | -| `lt(col, value)` | Less than | `orderAndRange: true` | -| `lte(col, value)` | Less than or equal | `orderAndRange: true` | -| `between(col, min, max)` | Between (inclusive) | `orderAndRange: true` | -| `notBetween(col, min, max)` | Not between | `orderAndRange: true` | -| `like(col, pattern)` | LIKE pattern match | `freeTextSearch: true` | -| `ilike(col, pattern)` | ILIKE case-insensitive | `freeTextSearch: true` | -| `notIlike(col, pattern)` | NOT ILIKE | `freeTextSearch: true` | -| `inArray(col, values)` | IN array | `equality: true` | -| `notInArray(col, values)` | NOT IN array | `equality: true` | -| `jsonbPathQueryFirst(col, selector)` | Extract first value at JSONB path | `searchableJson: true` | -| `jsonbGet(col, selector)` | Get value using JSONB `->` operator | `searchableJson: true` | -| `jsonbPathExists(col, selector)` | Check if JSONB path exists | `searchableJson: true` | +| `eq(col, value)` | Equality | equality (`Eq`, `Ord`, `OrdOre`, `TextSearch`) | +| `ne(col, value)` | Not equal | equality | +| `gt` / `gte` / `lt` / `lte` `(col, value)` | Comparison | order/range (`Ord`, `OrdOre`, `TextSearch`) | +| `between(col, min, max)` | Inclusive range | order/range | +| `notBetween(col, min, max)` | Negated range | order/range | +| `inArray(col, values)` / `notInArray(col, values)` | Membership (single-batch encryption; empty list rejected) | equality | +| `matches(col, needle)` | Fuzzy free-text token match (short needles rejected) | free-text (`TextMatch`, `TextSearch`) | +| `contains(col, subDoc)` | Exact encrypted-JSONB containment (`{}` rejected) | `Json` | +| `selector(col, path).eq/ne/gt/gte/lt/lte(value)` | JSONPath selector-with-constraint (dot-notation paths, scalar leaves) | `Json` | ### Sort Operators (sync) -| Operator | Usage | Required Index | +| Operator | Usage | Required capability | |---|---|---| -| `asc(col)` | Ascending sort | `orderAndRange: true` | -| `desc(col)` | Descending sort | `orderAndRange: true` | +| `asc(col)` | `ORDER BY eql_v3.ord_term(col)` ascending | order/range | +| `desc(col)` | `ORDER BY eql_v3.ord_term(col)` descending | order/range | -### Logical Operators (async, batched) +(`ord_term_ore` for `*OrdOre` domains — superuser-only, unavailable on managed Postgres.) -| Operator | Usage | Description | -|---|---|---| -| `and(...conditions)` | Combine with AND | Batches encryption | -| `or(...conditions)` | Combine with OR | Batches encryption | - -Both `and()` and `or()` accept `undefined` conditions, which are filtered out. This is useful for conditional query building: +### Logical Operators (async, concurrent) -```typescript -const results = await db - .select() - .from(usersTable) - .where( - await encryptionOps.and( - maybeCond ? encryptionOps.eq(usersTable.email, value) : undefined, - encryptionOps.gte(usersTable.age, 18), - ), - ) -``` +| Operator | Description | +|---|---| +| `and(...conditions)` | Conjunction — accepts lazy (un-awaited) operators and `undefined`, resolves concurrently | +| `or(...conditions)` | Disjunction — same | ### Passthrough Operators (sync, no encryption) -`exists`, `notExists`, `isNull`, `isNotNull`, `not`, `arrayContains`, `arrayContained`, `arrayOverlaps` - -These are re-exported from Drizzle and work identically. - -## Non-Encrypted Column Fallback - -All operators automatically detect whether a column is encrypted. If the column is not encrypted (regular Drizzle column), the operator falls back to the standard Drizzle operator: - -```typescript -// This works for both encrypted and non-encrypted columns -await encryptionOps.eq(usersTable.email, "alice@example.com") // encrypted -await encryptionOps.eq(usersTable.role, "admin") // falls back to drizzle eq() -``` - -## Complete Example: Express API - -```typescript -import "dotenv/config" -import express from "express" -import { eq } from "drizzle-orm" -import { drizzle } from "drizzle-orm/postgres-js" -import postgres from "postgres" -import { pgTable, integer, timestamp, varchar } from "drizzle-orm/pg-core" -import { encryptedType, extractEncryptionSchema, createEncryptionOperators, EncryptionOperatorError, EncryptionConfigError } from "@cipherstash/stack-drizzle" -import { Encryption } from "@cipherstash/stack" - -// Schema -const usersTable = pgTable("users", { - id: integer("id").primaryKey().generatedAlwaysAsIdentity(), - email: encryptedType("email", { equality: true, freeTextSearch: true }), - age: encryptedType("age", { dataType: "number", orderAndRange: true }), - role: varchar("role", { length: 50 }), - createdAt: timestamp("created_at").defaultNow(), -}) - -// Init -const usersSchema = extractEncryptionSchema(usersTable) -const encryptionClient = await Encryption({ schemas: [usersSchema] }) -const encryptionOps = createEncryptionOperators(encryptionClient) -const db = drizzle({ client: postgres(process.env.DATABASE_URL!) }) - -const app = express() -app.use(express.json()) - -// Create user -app.post("/users", async (req, res) => { - const encrypted = await encryptionClient.encryptModel(req.body, usersSchema) - if (encrypted.failure) return res.status(500).json({ error: encrypted.failure.message }) - - const [user] = await db.insert(usersTable).values(encrypted.data).returning() - res.json(user) -}) +`isNull`, `isNotNull`, `not`, `exists`, `notExists` — re-exported from Drizzle and work identically. -// Search users -app.get("/users", async (req, res) => { - const conditions = [] +### Other v3 Exports - if (req.query.email) { - conditions.push(encryptionOps.ilike(usersTable.email, `%${req.query.email}%`)) - } - if (req.query.minAge) { - conditions.push(encryptionOps.gte(usersTable.age, Number(req.query.minAge))) - } - if (req.query.role) { - conditions.push(eq(usersTable.role, req.query.role as string)) - } - - let query = db.select().from(usersTable) - if (conditions.length > 0) { - query = query.where(await encryptionOps.and(...conditions)) as typeof query - } - - const results = await query - const decrypted = await encryptionClient.bulkDecryptModels(results) - if (decrypted.failure) return res.status(500).json({ error: decrypted.failure.message }) - - res.json(decrypted.data) -}) - -app.listen(3000) -``` +`types`, `makeEqlV3Column`, `getEqlV3Column`, `isEqlV3Column`, `extractEncryptionSchemaV3`, `createEncryptionOperatorsV3`, `EncryptionOperatorError`, and the codec helpers `v3ToDriver` / `v3FromDriver` / `EqlV3CodecError` — all from `@cipherstash/stack-drizzle/v3`. ## Error Handling -Individual operators (e.g., `eq()`, `gte()`, `like()`) throw errors when invoked with invalid configuration or missing indexes: +Operators throw `EncryptionOperatorError` (exported from `@cipherstash/stack-drizzle/v3`) whenever the query cannot be answered safely: -- **`EncryptionOperatorError`** — thrown for operator-level issues (e.g., invalid arguments, unsupported operations). -- **`EncryptionConfigError`** — thrown for configuration issues (e.g., using `like` on a column without `freeTextSearch: true`). +- the column is not an encrypted v3 column (there is no plaintext fallback); +- the column's domain lacks the operator's capability (e.g. ordering a `TextEq` column, `eq` on a `Json` column); +- the operand is `null` (use `isNull`/`isNotNull`), an empty list (`inArray`), an empty object (`contains`), or a too-short needle (`matches`); +- a `selector` path is malformed / uses array syntax, or its leaf value is a non-scalar; +- operand encryption itself fails. ```typescript -import { createEncryptionOperators, EncryptionOperatorError, EncryptionConfigError } from "@cipherstash/stack-drizzle" +import { EncryptionOperatorError } from "@cipherstash/stack-drizzle/v3" class EncryptionOperatorError extends Error { context?: { @@ -703,93 +614,12 @@ class EncryptionOperatorError extends Error { operator?: string } } - -class EncryptionConfigError extends Error { - context?: { - tableName?: string - columnName?: string - operator?: string - } -} -``` - -Encryption client operations return `Result` objects with `data` or `failure`. - -## EQL v3 Integration (`@cipherstash/stack-drizzle/v3`) - -Everything above covers the v2 integration (`@cipherstash/stack-drizzle`). The **EQL v3** typed schema has its own Drizzle integration on the `@cipherstash/stack-drizzle/v3` subpath. In v3 every encrypted column is a concrete Postgres domain (`public.eql_v3_text_search`, `public.eql_v3_integer_ord`, ...) whose query capabilities are fixed by the type — there is no `equality: true` / `freeTextSearch: true` config object. See the `stash-encryption` skill's "EQL v3 Typed Schema" section for the full `types` catalog and capability suffixes (`Eq`, `Ord`/`OrdOre`, `Match`, `Search`). - -Exports: `types` (Drizzle-native column factories mirroring the `@cipherstash/stack/eql/v3` namespace), `makeEqlV3Column`, `getEqlV3Column`, `isEqlV3Column`, `extractEncryptionSchemaV3`, `createEncryptionOperatorsV3`, `EncryptionOperatorError`, and the codec helpers `v3ToDriver` / `v3FromDriver` / `EqlV3CodecError`. - -### Schema, Client, and Operators - -```typescript -import { pgTable, integer } from "drizzle-orm/pg-core" -import { EncryptionV3 } from "@cipherstash/stack/v3" -import { - types, - extractEncryptionSchemaV3, - createEncryptionOperatorsV3, -} from "@cipherstash/stack-drizzle/v3" - -const users = pgTable("users", { - id: integer("id").primaryKey().generatedAlwaysAsIdentity(), - email: types.TextSearch("email"), // equality + order/range + free-text - age: types.IntegerOrd("age"), // equality + order/range -}) - -const usersSchema = extractEncryptionSchemaV3(users) -const client = await EncryptionV3({ schemas: [usersSchema] }) -const ops = createEncryptionOperatorsV3(client) ``` -Each `types.*` factory emits its domain as the column's SQL type, so `drizzle-kit generate` produces `ADD COLUMN email public.eql_v3_text_search` etc. Install the EQL v3 SQL first with `stash eql install --eql-version 3` (direct install only for v3 — the v2 `generate-eql-migration` Drizzle path is not supported yet). - -`makeEqlV3Column(builder)` wraps a column builder from `@cipherstash/stack/eql/v3` (e.g. `makeEqlV3Column(v3types.TextEq("email"))`) — `types.TextEq("email")` from the drizzle subpath is shorthand for the same thing. +There is no `EncryptionConfigError` on the v3 path — capability problems surface as `EncryptionOperatorError` with the offending column/table/operator in `context`. -### Insert, Query, Decrypt +Encryption client operations (`encryptModel`, `bulkDecryptModels`, ...) don't throw — they return `Result` objects with `data` or `failure`. Check `.failure` before using `.data`. -The Drizzle column stores the encrypted EQL envelope, so encrypt models before insert and decrypt after select — passing the extracted schema table: +## Legacy: EQL v2 -```typescript -// Insert -const enc = await client.bulkEncryptModels( - [{ email: "alice@example.com", age: 30 }], - usersSchema, -) -if (!enc.failure) await db.insert(users).values(enc.data) - -// Query — operators auto-encrypt their plaintext operands -const rows = await db.select().from(users) - .where(await ops.and( - ops.matches(users.email, "alice"), // fuzzy free-text token match - ops.between(users.age, 18, 65), - )) - .orderBy(ops.asc(users.age)) - -// Decrypt — v3 decryptModel/bulkDecryptModels take the schema table -const dec = await client.bulkDecryptModels(rows, usersSchema) -``` - -### Operators - -| Operator | Required capability (domain suffix) | -|---|---| -| `eq`, `ne`, `inArray`, `notInArray` | equality (`Eq`, `Ord`, `OrdOre`, `TextSearch`) | -| `gt`, `gte`, `lt`, `lte`, `between`, `notBetween`, `asc`, `desc` | order/range (`Ord`, `OrdOre`, `TextSearch`) | -| `matches` | fuzzy free-text token match (`TextMatch`, `TextSearch`) | -| `contains` | exact encrypted-JSONB containment (`Json`) | -| `and`, `or` | combinators — accept lazy (un-awaited) operators and `undefined`, resolve concurrently | -| `isNull`, `isNotNull`, `not`, `exists`, `notExists` | Drizzle passthroughs, no encryption | - -Differences from the v2 operators to know about: - -- **`like` / `ilike` do not exist — by design.** v3 free-text search is tokenised bloom matching, not SQL pattern matching; `matches(col, needle)` is the free-text operator. It is fuzzy (order- and multiplicity-insensitive) and one-sided (a match may be a false positive, a non-match never is). Don't pass `%` wildcards. -- **`matches` is fuzzy free-text, `contains` is exact JSON containment — two distinct operators (#617).** `matches(col, needle)` requires a `TextMatch` / `TextSearch` column and throws `EncryptionOperatorError` (`requires free-text search`) otherwise. `contains(col, subdoc)` requires a `types.Json` column and throws (`requires JSON containment`) otherwise. -- **`contains` on a `types.Json` column** answers exact encrypted-JSONB containment: `contains(col, { roles: ['admin'] })` matches every row whose document contains that sub-object (jsonb `@>` semantics, no false positives; array containment is position-independent). It emits the `@>` operator with a `query_jsonb` needle — a `Json` column carries no `eq` / ordering, so those operators throw on it. -- **No plaintext-column fallback.** Every v3 operator requires an encrypted v3 column and throws `EncryptionOperatorError` otherwise. Use regular Drizzle operators for non-encrypted columns. -- A `null` operand throws — use `isNull()` / `isNotNull()` for NULL checks. -- `inArray` / `notInArray` reject an empty list, and encrypt the whole list in a single `encryptQuery` batch crossing. -- `matches` rejects a needle shorter than the match tokenizer's token length, and `contains` rejects an empty-object needle (either would otherwise silently match every row). -- Operators gate on the column's capabilities and throw `EncryptionOperatorError` (with `context.columnName` / `tableName` / `operator`) when the domain can't answer the operator. This `EncryptionOperatorError` is exported from `@cipherstash/stack-drizzle/v3` and is deliberately separate from the v2 class of the same name; there is no `EncryptionConfigError` on the v3 path. -- Every operator takes an optional trailing `{ lockContext, audit }` argument; `createEncryptionOperatorsV3(client, { lockContext, audit })` sets defaults applied to every operand encryption. +The original v2 integration — `encryptedType` config-flag columns, `extractEncryptionSchema`, and `createEncryptionOperators` (with `like`/`ilike`) from the `@cipherstash/stack-drizzle` package root, over the `eql_v2_encrypted` column type installed via `stash eql install --drizzle` (the older standalone `@cipherstash/drizzle` package shipped its own `generate-eql-migration` bin for the same purpose) — still exists for existing deployments and is documented at https://cipherstash.com/docs. New projects must use the `/v3` surface documented above. Note: `stash init --drizzle` currently pins EQL v2 because the v2 Drizzle-migration install path has no v3 equivalent yet. diff --git a/skills/stash-encryption/SKILL.md b/skills/stash-encryption/SKILL.md index 75726388..344c6f97 100644 --- a/skills/stash-encryption/SKILL.md +++ b/skills/stash-encryption/SKILL.md @@ -1,23 +1,48 @@ --- name: stash-encryption -description: Implement field-level encryption with @cipherstash/stack. Covers schema definition, encrypt/decrypt operations, searchable encryption (equality, free-text, range, JSON), bulk operations, model operations, identity-aware encryption with LockContext, multi-tenant keysets, the EQL v3 typed schema (concrete Postgres domain columns and the strongly-typed EncryptionV3 client), and the full TypeScript type system. Use when adding encryption to a project, defining encrypted schemas, or working with the CipherStash Encryption API. +description: Implement field-level encryption with @cipherstash/stack using the EQL v3 typed schema. Covers the types.* column catalog (concrete Postgres domains with fixed query capabilities), the strongly-typed EncryptionV3 client, encrypt/decrypt and model operations, searchable encryption (equality, free-text, range), encrypted JSON (containment and JSONPath selectors), bulk operations, identity-aware encryption with lock contexts, multi-tenant keysets, and the rollout/cutover lifecycle. Use when adding encryption to a project, defining encrypted schemas, or working with the CipherStash Encryption API. --- # CipherStash Stack - Encryption Comprehensive guide for implementing field-level encryption with `@cipherstash/stack`. Every value is encrypted with its own unique key via ZeroKMS (backed by AWS KMS). Encryption happens client-side before data leaves the application. +Encrypted columns are **EQL v3 concrete Postgres domains** (`public.eql_v3_text_search`, `public.eql_v3_integer_ord`, ...): each column's query capabilities are fixed by the domain type you pick in the schema, and the `EncryptionV3` client derives precise TypeScript types from that schema — wrong-typed plaintext is a compile error, not a runtime surprise. + +> An older schema surface (EQL v2, with chainable capability builders) still exists for existing deployments — see "Legacy: EQL v2" at the end. Everything else in this skill is the v3 surface. New projects should use v3. + ## When to Use This Skill - Adding field-level encryption to a TypeScript/Node.js project -- Defining encrypted table schemas +- Defining encrypted table schemas with the `types.*` domain catalog - Encrypting and decrypting individual values or entire models -- Implementing searchable encryption (equality, free-text, range, JSON queries) +- Implementing searchable encryption (equality, free-text, range, encrypted JSON) - Bulk encrypting/decrypting large datasets - Implementing identity-aware encryption with JWT-based lock contexts - Setting up multi-tenant encryption with keysets -- Using the EQL v3 typed schema (`@cipherstash/stack/eql/v3`) — concrete Postgres domain columns with a strongly-typed client (see "EQL v3 Typed Schema" below) -- Migrating from `@cipherstash/protect` to `@cipherstash/stack` +- Rolling encryption out to a production table with live plaintext data + +## Quick Start + +```typescript +import { EncryptionV3, encryptedTable, types } from "@cipherstash/stack/v3" + +const users = encryptedTable("users", { + email: types.TextSearch("email"), +}) + +const client = await EncryptionV3({ schemas: [users] }) + +const encrypted = await client.encrypt("secret@example.com", { + table: users, + column: users.email, +}) +if (encrypted.failure) { + throw new Error(encrypted.failure.message) +} + +const decrypted = await client.decrypt(encrypted.data) +``` ## Installation @@ -97,7 +122,7 @@ profile. ### Programmatic Config ```typescript -const client = await Encryption({ +const client = await EncryptionV3({ schemas: [users], config: { workspaceCrn: "crn:ap-southeast-2.aws:your-workspace-id", @@ -133,99 +158,138 @@ The SDK never logs plaintext data. | Import Path | Provides | |---|---| -| `@cipherstash/stack` | `Encryption` function, `encryptedTable`, `encryptedColumn`, `encryptedField` (convenience re-exports) | -| `@cipherstash/stack/schema` | `encryptedTable`, `encryptedColumn`, `encryptedField`, schema types | +| `@cipherstash/stack/v3` | `EncryptionV3` factory, `typedClient`, `TypedEncryptionClient` — plus re-exports of everything in `@cipherstash/stack/eql/v3`. The one-stop import for v3. | +| `@cipherstash/stack/eql/v3` | `encryptedTable`, the `types` namespace, `buildEncryptConfig`, inference types (`InferPlaintext`, `InferEncrypted`, `V3ModelInput`, ...) | +| `@cipherstash/stack` | `OidcFederationStrategy`, `AccessKeyStrategy`, the untyped `Encryption` function (also accepts v3 schemas), legacy v2 re-exports | | `@cipherstash/stack/identity` | `LockContext` class and identity types | -| `@cipherstash/stack-drizzle` | `encryptedType`, `extractEncryptionSchema`, `createEncryptionOperators` for Drizzle ORM | -| `@cipherstash/stack-supabase` | `encryptedSupabase` wrapper for Supabase | -| `@cipherstash/stack/dynamodb` | `encryptedDynamoDB` helper for DynamoDB | -| `@cipherstash/stack/encryption` | `EncryptionClient` class, `Encryption` function | | `@cipherstash/stack/errors` | `EncryptionErrorTypes`, `StackError`, error subtypes, `getErrorMessage` | -| `@cipherstash/stack/client` | Client-safe exports: schema builders, schema types, `EncryptionClient` type (no native FFI) | | `@cipherstash/stack/types` | All TypeScript types | -| `@cipherstash/stack/eql/v3` | EQL v3 typed schema: `encryptedTable`, `types` namespace, `buildEncryptConfig`, inference types (see "EQL v3 Typed Schema" below) | -| `@cipherstash/stack/v3` | `EncryptionV3` factory, `typedClient`, `TypedEncryptionClient` — plus re-exports of everything in `@cipherstash/stack/eql/v3` | | `@cipherstash/stack-drizzle/v3` | Drizzle ORM integration for EQL v3 schemas (see the `stash-drizzle` skill) | +| `@cipherstash/stack-supabase` | `encryptedSupabaseV3` wrapper for Supabase (see the `stash-supabase` skill) | +| `@cipherstash/stack/dynamodb` | `encryptedDynamoDB` — **still requires the legacy v2 schema surface**; see "Legacy: EQL v2" below | +| `@cipherstash/stack/schema`, `@cipherstash/stack/client`, `@cipherstash/stack/encryption` | Legacy v2 schema builders and client surface — see "Legacy: EQL v2" below | ## Schema Definition -Define which tables and columns to encrypt using `encryptedTable` and `encryptedColumn`: +Define which tables and columns to encrypt with `encryptedTable` and the `types` namespace. Every encrypted column is a **concrete Postgres domain** whose query capabilities are **fixed by the type** — there is no chainable capability tuner; every domain is fully described by its `types.*` factory. ```typescript -import { encryptedTable, encryptedColumn } from "@cipherstash/stack/schema" +import { encryptedTable, types } from "@cipherstash/stack/eql/v3" +// (also re-exported from "@cipherstash/stack/v3") const users = encryptedTable("users", { - email: encryptedColumn("email") - .equality() // exact-match queries - .freeTextSearch() // full-text / fuzzy search - .orderAndRange(), // sorting and range queries - - age: encryptedColumn("age") - .dataType("number") - .equality() - .orderAndRange(), - - address: encryptedColumn("address"), // encrypt-only, no search indexes + email: types.TextSearch("email"), // equality + order/range + free-text + username: types.TextEq("username"), // equality only + balance: types.BigintOrd("balance"), // equality + order/range + lastLogin: types.TimestampOrd("last_login"), + active: types.Boolean("active"), // storage only — encrypt/decrypt, no queries + notes: types.Text("notes"), // storage only }) -const documents = encryptedTable("documents", { - metadata: encryptedColumn("metadata") - .searchableJson(), // encrypted JSONB queries (JSONPath + containment) +const events = encryptedTable("events", { + metadata: types.Json("metadata"), // encrypted JSONB: containment + selectors }) ``` -### Index Types +The returned table is also a column accessor (`users.email`). The JS property name and the DB column name may differ: `lastLogin: types.TimestampOrd("last_login")` reads/writes the `lastLogin` property on models but targets the `last_login` column in the database. + +### The `types` Namespace + +Each factory in `types` maps 1:1 to a Postgres domain named `public.eql_v3_`. The naming rule: strip the `eql_v3_` prefix and PascalCase each underscore-separated segment. So `types.TextSearch` builds a `public.eql_v3_text_search` column, `types.IntegerOrd` builds `public.eql_v3_integer_ord`, and `types.Timestamp` builds `public.eql_v3_timestamp`. + +**Capability suffixes:** -| Method | Purpose | Query Type | +| Suffix | Capabilities | Query types | |---|---|---| -| `.equality(tokenFilters?)` | Exact match lookups. Accepts an optional array of token filters (e.g., `[{ kind: 'downcase' }]`) for case-insensitive matching. | `'equality'` | -| `.freeTextSearch(opts?)` | Full-text / fuzzy search | `'freeTextSearch'` | -| `.orderAndRange()` | Sorting, comparison, range queries | `'orderAndRange'` | -| `.searchableJson()` | Encrypted JSONB path and containment queries (auto-sets `dataType` to `'json'`) | `'searchableJson'` | -| `.dataType(cast)` | Set plaintext data type | N/A | +| _(none)_ | Storage only — encrypt/decrypt, no queries | — | +| `Eq` | Equality | `'equality'` | +| `Ord` | Equality + ordering/range (OPE-backed) | `'equality'`, `'orderAndRange'` | +| `OrdOre` | Equality + ordering/range (block-ORE flavour — see caveat below) | `'equality'`, `'orderAndRange'` | +| `Match` (text only) | Free-text containment only | `'freeTextSearch'` | +| `Search` (text only) | Equality + ordering/range + free-text | all three | +| `Json` (no suffix) | Encrypted-JSONB containment + JSONPath selector queries | `'searchableJson'` | + +> **`Ord` vs `OrdOre`:** prefer `Ord`. The `OrdOre` domains are backed by an ORE operator class whose installation requires **superuser** — unavailable on managed Postgres (including Supabase), where the install bundle skips the ORE opclass and disables the `_ord_ore` domains it cannot support. The two ordering flavours produce different, non-cross-comparable terms (`Ord`/`Search` extract via `eql_v3.ord_term`; `OrdOre` via `eql_v3.ord_term_ore`). -**Supported data types:** `'string'` (default), `'text'`, `'number'`, `'boolean'`, `'date'`, `'bigint'`, `'json'` +**Domain families and plaintext types:** -Methods are chainable - call as many as you need on a single column. +| Family | Factories | Plaintext (TypeScript) type | +|---|---|---| +| `Integer`, `Smallint`, `Numeric`, `Real`, `Double` | base, `Eq`, `Ord`, `OrdOre` | `number` | +| `Bigint` | base, `Eq`, `Ord`, `OrdOre` | `bigint` (native JS bigint; full i64 range, out-of-range values rejected client-side before the FFI) | +| `Date` | base, `Eq`, `Ord`, `OrdOre` | `Date` (calendar date; time-of-day is truncated) | +| `Timestamp` | base, `Eq`, `Ord`, `OrdOre` | `Date` (time-of-day preserved) | +| `Text` | base, `Eq`, `Match`, `Ord`, `OrdOre`, `Search` | `string` | +| `Boolean` | base only | `boolean` | +| `Json` | `Json` only | a JSON *document* (`JsonDocument`: object, array, or null — NOT a top-level scalar; nested values are any `JsonValue`) | -### Free-Text Search Options +Examples: `types.Text("notes")` (storage only), `types.TextEq("username")`, `types.BigintOrd("balance")`, `types.TimestampOrdOre("created_at")`, `types.Boolean("active")`, `types.Json("metadata")`. -```typescript -encryptedColumn("bio").freeTextSearch({ - tokenizer: { kind: "ngram", token_length: 3 }, // or { kind: "standard" } - token_filters: [{ kind: "downcase" }], - k: 6, - m: 2048, - include_original: true, -}) -``` +The match index on `Match`/`Search` columns is always emitted with the default configuration (trigram tokenizer, downcased) — there is no per-column tuning in v3. Search needles must be at least 3 characters; shorter needles tokenize to nothing and are rejected. ### Type Inference ```typescript -import type { InferPlaintext, InferEncrypted } from "@cipherstash/stack/schema" +import type { InferPlaintext, InferEncrypted } from "@cipherstash/stack/eql/v3" type UserPlaintext = InferPlaintext -// { email: string; age: string; address: string } +// { email: string; lastLogin: Date; balance: bigint; ... } type UserEncrypted = InferEncrypted -// { email: Encrypted; age: Encrypted; address: Encrypted } +// { email: Encrypted; lastLogin: Encrypted; balance: Encrypted; ... } +``` + +`V3ModelInput`, `V3EncryptedModel`, and `V3DecryptedModel` (same subpath) are the model-shape helpers the typed client uses: schema-column keys are pinned to the column's plaintext type (nullable fields stay nullable), non-schema keys pass through unchanged. + +### Database Setup + +Install the EQL v3 SQL with the stash CLI (v3 is the default): + +```bash +stash eql install --eql-version 3 + +# Supabase targets: add --supabase to apply role grants +stash eql install --eql-version 3 --supabase +``` + +EQL v3 ships **one SQL bundle for every target, including Supabase** — no separate Supabase or no-operator-family variants. For Supabase, though, still pass the `--supabase` flag: it applies the `anon`/`authenticated`/`service_role` grants on the `eql_v3` and `eql_v3_internal` schemas. Without it, encrypted queries fail with `permission denied for schema eql_v3_internal`. v3 installs via the direct path only (`--drizzle`, `--migration`, `--migrations-dir`, and `--latest` are v2-only flags and are not supported for v3). + +In migrations, declare each encrypted column as its domain type: + +```sql +CREATE TABLE users ( + id BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY, + email public.eql_v3_text_search, + last_login public.eql_v3_timestamp_ord, + balance public.eql_v3_bigint_ord +); ``` -## Client Initialization +## Client Initialization: `EncryptionV3` + +`EncryptionV3` from `@cipherstash/stack/v3` returns a `TypedEncryptionClient` whose method signatures are derived from your schemas — wrong-typed plaintext is rejected at compile time, and query methods only accept queryable columns with `queryType` constrained to the column's capabilities: ```typescript -import { Encryption } from "@cipherstash/stack" +import { EncryptionV3, encryptedTable, types } from "@cipherstash/stack/v3" -const client = await Encryption({ schemas: [users, documents] }) +const users = encryptedTable("users", { + email: types.TextSearch("email"), + lastLogin: types.TimestampOrd("last_login"), + balance: types.BigintEq("balance"), +}) + +const client = await EncryptionV3({ schemas: [users] }) ``` -The `Encryption()` function returns `Promise` and throws on error (e.g., bad credentials, missing config, invalid keyset UUID). At least one schema is required. +- The wire format is pinned to EQL v3 (`eqlVersion: 3`); you don't set it yourself. +- `EncryptionV3()` throws on init error (bad credentials, missing config, invalid keyset UUID). At least one schema is required. +- `typedClient(client, ...schemas)` (also exported from `@cipherstash/stack/v3`) wraps an already-built untyped `EncryptionClient` in the typed surface. +- The untyped `Encryption({ schemas })` from `@cipherstash/stack` also accepts v3 tables (it auto-detects them and sets the v3 wire format), but you lose the per-column typing — prefer `EncryptionV3`. **v2 and v3 tables cannot be mixed in one client** — a mixed schema set throws at init. Create separate clients if you need both. ```typescript // Error handling try { - const client = await Encryption({ schemas: [users] }) + const client = await EncryptionV3({ schemas: [users] }) } catch (error) { console.error("Init failed:", error.message) } @@ -233,18 +297,22 @@ try { ## Encrypt and Decrypt Single Values +Plaintext is pinned to the column's domain type: + ```typescript -// Encrypt +await client.encrypt("alice@example.com", { table: users, column: users.email }) // string ✓ +await client.encrypt(new Date(), { table: users, column: users.lastLogin }) // Date ✓ +// client.encrypt(42, { table: users, column: users.email }) // ✗ compile error + const encrypted = await client.encrypt("hello@example.com", { - column: users.email, table: users, + column: users.email, }) if (encrypted.failure) { - console.error(encrypted.failure.message) -} else { - console.log(encrypted.data) // Encrypted payload (opaque object) + throw new Error(encrypted.failure.message) } +console.log(encrypted.data) // Encrypted payload (opaque object) // Decrypt const decrypted = await client.decrypt(encrypted.data) @@ -254,62 +322,73 @@ if (!decrypted.failure) { } ``` -All plaintext values must be non-null. Null handling is managed at the model level by `encryptModel` and `decryptModel`. +`decrypt` of a single value cannot be strongly typed — a lone ciphertext carries no column identity. All plaintext values passed to `encrypt` must be non-null; null handling is managed at the model level by `encryptModel` and `decryptModel`. ## Model Operations -Encrypt or decrypt an entire object. Only fields matching your schema are encrypted; other fields pass through unchanged. - -The return type is **schema-aware**: fields matching the table schema are typed as `Encrypted`, while other fields retain their original types. For best results, let TypeScript infer the type parameters from the arguments rather than providing an explicit ``. +Encrypt or decrypt an entire object. Only fields matching your schema are encrypted; other fields pass through unchanged, and schema fields are validated against their inferred plaintext type at compile time. ```typescript -type User = { id: string; email: string; createdAt: Date } - -const user = { - id: "user_123", - email: "alice@example.com", // defined in schema -> encrypted - createdAt: new Date(), // not in schema -> unchanged -} - -// Encrypt model — let TypeScript infer the return type from the schema -const encResult = await client.encryptModel(user, users) -if (!encResult.failure) { - // encResult.data.email is typed as Encrypted - // encResult.data.id is typed as string - // encResult.data.createdAt is typed as Date -} +const enc = await client.encryptModel( + { id: "u1", email: "alice@example.com", lastLogin: new Date(), balance: 100n }, + users, +) +if (!enc.failure) { + // enc.data.email is Encrypted; enc.data.id stays string -// Decrypt model -const decResult = await client.decryptModel(encResult.data) -if (!decResult.failure) { - console.log(decResult.data.email) // "alice@example.com" + const dec = await client.decryptModel(enc.data, users) + if (!dec.failure) { + dec.data.email // string + dec.data.lastLogin // Date — reconstructed on decrypt + dec.data.balance // bigint + dec.data.id // string — non-schema fields pass through + } } ``` -The `Decrypted` type maps encrypted fields back to their plaintext types. +Typed-client model notes: -Passing an explicit type parameter (e.g., `client.encryptModel(...)`) still works for backward compatibility — the return type degrades to `User` in that case. +- `decryptModel` / `bulkDecryptModels` take the **table as a second argument** and return a plain `Promise>` (not a chainable operation) — pass a lock context as the optional third argument instead of chaining `.withLockContext()`. +- `Date` columns are reconstructed to real `Date` instances on decrypt; `bigint` columns round-trip as native `bigint`. +- Nullable schema fields stay nullable through the round trip. ## Bulk Operations All bulk methods make a single call to ZeroKMS regardless of record count, while still using a unique key per value. +### Bulk Encrypt / Decrypt Models + +```typescript +const userModels = [ + { id: "1", email: "alice@example.com", lastLogin: new Date(), balance: 1n }, + { id: "2", email: "bob@example.com", lastLogin: new Date(), balance: 2n }, +] + +const encrypted = await client.bulkEncryptModels(userModels, users) +if (encrypted.failure) throw new Error(encrypted.failure.message) + +const decrypted = await client.bulkDecryptModels(encrypted.data, users) +``` + ### Bulk Encrypt / Decrypt (Raw Values) +`bulkEncrypt` / `bulkDecrypt` are parity passthroughs (not v3-strengthened): + ```typescript const plaintexts = [ { id: "u1", plaintext: "alice@example.com" }, { id: "u2", plaintext: "bob@example.com" }, - { id: "u3", plaintext: "charlie@example.com" }, ] const encrypted = await client.bulkEncrypt(plaintexts, { column: users.email, table: users, }) +if (encrypted.failure) throw new Error(encrypted.failure.message) // encrypted.data = [{ id: "u1", data: EncryptedPayload }, ...] const decrypted = await client.bulkDecrypt(encrypted.data) +if (decrypted.failure) throw new Error(decrypted.failure.message) // Per-item error handling: for (const item of decrypted.data) { if ("data" in item) { @@ -320,23 +399,9 @@ for (const item of decrypted.data) { } ``` -### Bulk Encrypt / Decrypt Models - -```typescript -const userModels = [ - { id: "1", email: "alice@example.com" }, - { id: "2", email: "bob@example.com" }, -] - -const encrypted = await client.bulkEncryptModels(userModels, users) -const decrypted = await client.bulkDecryptModels(encrypted.data) -``` - ## Searchable Encryption -Encrypt query terms so you can search encrypted data in PostgreSQL. - -### Single Query Encryption +Encrypt query terms with `encryptQuery` so you can search encrypted data in PostgreSQL. On the typed client, `encryptQuery` only accepts queryable columns (storage-only columns are rejected at compile time) and constrains `queryType` to the column's capabilities. ```typescript // Equality query @@ -346,7 +411,7 @@ const eqQuery = await client.encryptQuery("alice@example.com", { queryType: "equality", }) -// Free-text search +// Free-text search (substring/token match) const matchQuery = await client.encryptQuery("alice", { column: users.email, table: users, @@ -354,28 +419,37 @@ const matchQuery = await client.encryptQuery("alice", { }) // Order and range -const rangeQuery = await client.encryptQuery(25, { - column: users.age, +const rangeQuery = await client.encryptQuery(new Date("2026-01-01"), { + column: users.lastLogin, table: users, queryType: "orderAndRange", }) +``` -// JSON path query (steVecSelector) -const pathQuery = await client.encryptQuery("$.user.email", { - column: documents.metadata, - table: documents, - queryType: "steVecSelector", -}) +### Query-Type Inference Gotcha on `types.TextSearch` + +A `TextSearch` column carries all three indexes, and `encryptQuery` with **no explicit `queryType` builds an equality term, not a free-text match** (index inference priority: unique > match > ore). A substring like `"joh"` then matches nothing. Pass `queryType: 'freeTextSearch'` explicitly for substring/token search: -// JSON containment query (steVecTerm) -const containsQuery = await client.encryptQuery({ role: "admin" }, { - column: documents.metadata, - table: documents, - queryType: "steVecTerm", +```typescript +// equality (default): exact value only +await client.encryptQuery("john@example.com", { column: users.email, table: users }) + +// free-text match: substring/token search +await client.encryptQuery("joh", { + column: users.email, + table: users, + queryType: "freeTextSearch", }) ``` -If `queryType` is omitted, it's auto-inferred from the column's configured indexes (priority: unique > match > ore > ste_vec). +### Free-Text Search Semantics + +Encrypted free-text search is **fuzzy bloom-filter token matching**, not SQL pattern matching: + +- The needle blooms to its own trigrams; a row matches when the stored value's bloom contains all of them. Case-insensitive; order- and multiplicity-insensitive. +- **One-sided:** a match may be a false positive; a non-match never is. +- Needles must be at least 3 characters (the default trigram length) — shorter needles bloom to nothing and are rejected rather than silently matching every row. +- In the Drizzle and Supabase v3 integrations the operator is **`matches`** — the adapters expose free-text search as `matches`, not `like`/`ilike` (the Supabase adapter keeps a deprecated `like`/`ilike` shim that delegates to `matches` with a warning). Don't pass `%` wildcards. ### Query Result Formatting (`returnType`) @@ -388,7 +462,6 @@ By default `encryptQuery` returns an `Encrypted` object (the raw EQL JSON payloa | `'escaped-composite-literal'` | `string` | Embedding inside another string or JSON value | ```typescript -// Get a composite literal string for use with Supabase const term = await client.encryptQuery("alice@example.com", { column: users.email, table: users, @@ -400,27 +473,9 @@ const term = await client.encryptQuery("alice@example.com", { Each term in a batch can have its own `returnType`. -### Searchable JSON - -For columns using `.searchableJson()`, the query type is auto-inferred from the plaintext: - -```typescript -// String -> JSONPath selector query -const pathQuery = await client.encryptQuery("$.user.email", { - column: documents.metadata, - table: documents, -}) - -// Object/Array -> containment query -const containsQuery = await client.encryptQuery({ role: "admin" }, { - column: documents.metadata, - table: documents, -}) -``` - ### Batch Query Encryption -Encrypt multiple query terms in one ZeroKMS call: +Encrypt multiple query terms in one ZeroKMS call (the per-term columns are heterogeneous, so the batch form takes untyped `ScalarQueryTerm`s): ```typescript const terms = [ @@ -434,6 +489,50 @@ const results = await client.encryptQuery(terms) All values in the array must be non-null. +### On the Wire: Operators and Ordering + +Scalar filters compare through each domain's `eql_v3.*` operators (`col = term`, `col > term`, ...), and `ORDER BY` on an encrypted column goes through the ordering extractors — `eql_v3.ord_term(col)` for OPE-backed (`Ord`/`Search`) domains, `eql_v3.ord_term_ore(col)` for `OrdOre`. The Drizzle v3 integration emits all of this for you (including `asc`/`desc`, which emit `ORDER BY eql_v3.ord_term(col)`). Over Supabase/PostgREST, the adapter's `order()` works on OPE-backed ordering columns (plain `*_ord`, `text_ord`, `text_search`) by sorting on the column's `col->op` term; `OrdOre`-flavour (`*_ord_ore`) domains and columns with no ordering term are rejected. See the `stash-drizzle` and `stash-supabase` skills. + +## Encrypted JSON (`types.Json`) + +A `types.Json("metadata")` column encrypts a whole JSON document to a `public.eql_v3_json` value. The plaintext is a **`JsonDocument`: an object, an array, or `null` — NOT a bare top-level scalar** (protect-ffi rejects a top-level string/number/boolean; a scalar belongs in a scalar domain like `types.TextEq` or `types.IntegerEq`). Nested scalars are fully supported. + +`types.Json` carries no equality or ordering capability — `eq` / `gt` / `asc` on it throw. It supports two query patterns: containment and JSONPath selectors. + +### Containment (exact, `@>`) + +Pass a sub-object or sub-array to `encryptQuery` with `queryType: 'searchableJson'`; it matches documents that contain the needle with jsonb `@>` semantics — **exact** containment, no false positives. Array containment is a subset test regardless of element position: `{ roles: ['admin'] }` matches any document whose `roles` array includes `admin`. + +```typescript +const events = encryptedTable("events", { metadata: types.Json("metadata") }) + +// containment: object needle +await client.encryptQuery({ roles: ["admin"] }, { column: events.metadata, table: events }) +``` + +The Drizzle and Supabase adapters reject an empty-object needle (jsonb `{} ⊆ anything` — it would silently match every row); core `client.encryptQuery` does not enforce this, so guard against empty needles yourself when composing raw SQL. + +### JSONPath Selectors (value-at-a-path) + +Selector queries constrain the value **at a path** inside the document — `metadata->'$.user.role' = 'admin'`. Their unique power over containment is **ordering at a path** (`gt`/`gte`/`lt`/`lte`), available through the Drizzle v3 integration. Paths are dot-notation object paths (`'$.a.b'`); array/wildcard steps are rejected. + +Semantics to know: + +- **`eq`** at a path excludes rows whose document lacks the path (absent is not equal). +- **`ne`** at a path **includes** rows whose document lacks the path, and — in both adapters — rows whose column is SQL NULL ("not equal to value" covers "has no value"; Drizzle adds `OR IS NULL`, Supabase an `is.null` branch). +- **Array-leaf caveat:** a scalar needle does not match an array at the path — ste_vec encodes array elements under their own selectors, so `{a: [40, 30]}` is NOT matched by a selector-eq of `30` at `$.a`. To match an array-valued path, pass the full array through containment. +- **Ciphertext in the WHERE clause (interim):** pending a ciphertext-free ordering needle from protect-ffi ([cipherstash/protectjs-ffi#137](https://github.com/cipherstash/protectjs-ffi/issues/137)), the selector's right-hand operand is a storage encryption of `{path: value}` whose ciphertext appears in the emitted SQL statement — it can surface in SQL logs and statement views. On Supabase, the selector/containment needle additionally ships as a full storage envelope because of a PostgREST operand-casting gap ([cipherstash/stack#654](https://github.com/cipherstash/stack/issues/654)) — #137 landing does not remove that exposure. + +### Adapter Matrix + +| Capability | Drizzle v3 (`ops` from `@cipherstash/stack-drizzle/v3`) | Supabase v3 (`encryptedSupabaseV3`) | +|---|---|---| +| Containment (`@>`) | `ops.contains(col, subdoc)` | `contains(col, subdoc)` / `filter(col, 'cs', ...)` | +| Selector equality | `ops.selector(col, '$.path').eq(value)` / `.ne(value)` | `selectorEq(col, '$.path', value)` / `selectorNe(...)` | +| Selector ordering | `ops.selector(col, '$.path').gt/gte/lt/lte(value)` | Not available — needs [cipherstash/encrypt-query-language#407](https://github.com/cipherstash/encrypt-query-language/issues/407) | + +See the `stash-drizzle` and `stash-supabase` skills for the full integration guides. + ## Authentication The client authenticates to ZeroKMS through `config.authStrategy`. Leave it unset for the default **auto** strategy: in local development, authenticate once with `npx stash auth login` (preferred — no credentials in your environment; `npx stash init` is the agent-assisted flow that also sets up schema and database); in CI/production, set the `CS_*` environment variables. Two explicit strategies cover the other cases: @@ -442,7 +541,10 @@ The client authenticates to ZeroKMS through `config.authStrategy`. Leave it unse - **`OidcFederationStrategy`** — authenticates the client **as the end user** by federating a third-party OIDC JWT (Clerk, Supabase, Auth0, Okta, ...) into a CipherStash service token: ```typescript -import { Encryption, OidcFederationStrategy } from "@cipherstash/stack" +import { OidcFederationStrategy } from "@cipherstash/stack" +import { EncryptionV3, encryptedTable, types } from "@cipherstash/stack/v3" + +const users = encryptedTable("users", { email: types.TextSearch("email") }) // `getJwt` is re-invoked on every (re-)federation and must return the // *current* third-party OIDC JWT. @@ -454,7 +556,7 @@ if (strategy.failure) { throw new Error(`[auth] ${strategy.failure.type}: ${strategy.failure.error.message}`) } -const client = await Encryption({ +const client = await EncryptionV3({ schemas: [users], config: { authStrategy: strategy.data }, }) @@ -462,8 +564,6 @@ const client = await Encryption({ `OidcFederationStrategy.create()` returns a `Result` — **unwrap it**. Passing the envelope straight to `authStrategy` gives the FFI an object with no `getToken()` at all. -> **Known type error (runtime is fine).** The example above works at runtime, but `authStrategy: strategy.data` does not currently typecheck. `@cipherstash/auth` 0.41 strategies declare `getToken(): Promise>`, while `@cipherstash/protect-ffi`'s exported `AuthStrategy` type still says `getToken(): Promise<{ token: string }>`. protect-ffi accepts **both** shapes at runtime (0.28+), on the Node and WASM paths alike — only its TypeScript declaration was left behind. Until it's widened, add `as unknown as AuthStrategy` or `// @ts-expect-error`. Tracked in [issue #602](https://github.com/cipherstash/stack/issues/602). - Authentication stands on its own — an OIDC-authenticated client encrypts and decrypts normally. Binding *data* to the authenticated user is a separate, optional step: the lock context, below. ## Identity-Aware Encryption (Lock Contexts) @@ -497,7 +597,11 @@ Every operation returns a `Result`. Narrow on `.failure` before touching `.data` `identityClaim` is an array of JWT claim *names*, not values: `["sub"]` (the default) or `["sub", "org_id"]`. ZeroKMS resolves each claim's value from the JWT the strategy federated. **The same claim must be supplied to encrypt and decrypt** — it is baked into the data key's tag, so decrypting without it fails with `Failed to retrieve key`. -Lock contexts work with every operation: `encrypt`, `decrypt`, `encryptModel`, `decryptModel`, `bulkEncrypt`, `bulkDecrypt`, `bulkEncryptModels`, `bulkDecryptModels`, `encryptQuery`. +Lock contexts work with every operation: `encrypt`, `decrypt`, `encryptModel`, `decryptModel`, `bulkEncrypt`, `bulkDecrypt`, `bulkEncryptModels`, `bulkDecryptModels`, `encryptQuery`. On the typed client, `decryptModel` and `bulkDecryptModels` take the lock context as their optional **third argument** (they return a plain `Promise>`, so there is no `.withLockContext()` to chain): + +```typescript +const dec = await client.decryptModel(enc.data, users, IDENTITY) +``` ### Deprecated: `LockContext.identify()` @@ -517,13 +621,13 @@ Isolate encryption keys per tenant: ```typescript // By name -const client = await Encryption({ +const client = await EncryptionV3({ schemas: [users], config: { keyset: { name: "Company A" } }, }) // By UUID -const client = await Encryption({ +const client = await EncryptionV3({ schemas: [users], config: { keyset: { id: "123e4567-e89b-12d3-a456-426614174000" } }, }) @@ -533,7 +637,7 @@ Each keyset provides full cryptographic isolation between tenants. ## Operation Chaining -All operations return thenable objects that support chaining: +Encrypt operations return thenable objects that support chaining: ```typescript const result = await client @@ -542,6 +646,8 @@ const result = await client .audit({ metadata: { action: "create" } }) // optional: audit trail ``` +(The typed client's `decryptModel` / `bulkDecryptModels` are the exception: they return a plain `Promise>` and take the lock context as an argument — see "Model Operations".) + ## Error Handling All async methods return a `Result` object - a discriminated union with either `data` (success) or `failure` (error), never both. @@ -607,234 +713,16 @@ try { ### Validation Rules - NaN and Infinity are rejected for numeric values -- `freeTextSearch` index only supports string values +- `bigint` values outside the signed 64-bit range are rejected client-side before the FFI +- Free-text search only applies to string values (`Match`/`Search` text domains); needles shorter than the trigram length (3) are rejected +- A `types.Json` column rejects a bare top-level scalar — the document must be an object, array, or null - At least one `encryptedTable` schema must be provided - Keyset UUIDs must be valid format -## Ordering Encrypted Data - -**`ORDER BY` on encrypted columns requires operator family support in the database.** - -On databases without operator families (e.g. Supabase, or when EQL is installed with `--exclude-operator-family`), sorting on encrypted columns is not currently supported — regardless of the client or ORM used. This applies to Drizzle, the Supabase JS SDK, raw SQL, and any other database client. - -**Workaround:** Sort application-side after decrypting the results. - -Operator family support for Supabase is being developed in collaboration with the Supabase and CipherStash teams and will be available in a future release. - -## PostgreSQL Storage - -Encrypted data is stored as EQL (Encrypt Query Language) JSON payloads. Install the EQL extension in PostgreSQL: - -```sql -CREATE EXTENSION IF NOT EXISTS eql_v2; - -CREATE TABLE users ( - id BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY, - email eql_v2_encrypted -); -``` - -Or store as JSONB if not using the EQL extension directly: - -```sql -CREATE TABLE users ( - id SERIAL PRIMARY KEY, - email jsonb NOT NULL -); -``` - -## EQL v3 Typed Schema - -EQL v3 is a newer schema surface where every encrypted column is a **concrete Postgres domain** (`public.eql_v3_`) and its query capabilities are fixed by the column type you pick. There is **no chainable capability tuner** — no `.equality()`, `.freeTextSearch()`, or `.orderAndRange()` — every domain is fully described by its type. The v2 surface documented above (`encryptedColumn` from `@cipherstash/stack/schema`) continues to work unchanged; v3 lives on its own subpaths. - -### Quick Start - -```typescript -import { Encryption } from "@cipherstash/stack" -import { encryptedTable, types } from "@cipherstash/stack/eql/v3" - -const users = encryptedTable("users", { - email: types.TextSearch("email"), -}) - -const client = await Encryption({ schemas: [users] }) - -const encryptResult = await client.encrypt("secret@example.com", { - column: users.email, - table: users, -}) -if (encryptResult.failure) { - // handle encryptResult.failure.message -} - -const decryptResult = await client.decrypt(encryptResult.data) -``` - -`Encryption({ schemas })` auto-detects v3 tables and sets the EQL v3 wire format. **v2 and v3 tables cannot be mixed in one client** — a mixed schema set throws at init. Create separate clients if you need both. - -The v3 `encryptedTable` intentionally shares its name with the v2 builder — the import path picks the model. The returned table is also a column accessor (`users.email`). The JS property name and the DB column name may differ: `createdOn: types.Timestamp("created_at")` reads/writes the `createdOn` property on models but targets the `created_at` column in the database. - -### The `types` Namespace - -Each factory in `types` maps 1:1 to a Postgres domain named `public.eql_v3_`. The naming rule: strip the `eql_v3_` prefix and PascalCase each underscore-separated segment. So `types.TextSearch` builds a `public.eql_v3_text_search` column, `types.IntegerOrd` builds `public.eql_v3_integer_ord`, and `types.Timestamp` builds `public.eql_v3_timestamp`. - -**Capability suffixes:** - -| Suffix | Capabilities | Query types | -|---|---|---| -| _(none)_ | Storage only — encrypt/decrypt, no queries | — | -| `Eq` | Equality | `'equality'` | -| `Ord` / `OrdOre` | Equality + ordering/range | `'equality'`, `'orderAndRange'` | -| `Match` (text only) | Free-text containment only | `'freeTextSearch'` | -| `Search` (text only) | Equality + ordering/range + free-text | all three | -| `Json` (no suffix) | Encrypted-JSONB containment queries (JSONPath selector: not yet, see #623) | `'searchableJson'` | - -**Domain families and plaintext types:** - -| Family | Factories | Plaintext (TypeScript) type | -|---|---|---| -| `Integer`, `Smallint`, `Numeric`, `Real`, `Double` | base, `Eq`, `Ord`, `OrdOre` | `number` | -| `Bigint` | base, `Eq`, `Ord`, `OrdOre` | `bigint` (native JS bigint; full i64 range, out-of-range values rejected client-side before the FFI) | -| `Date` | base, `Eq`, `Ord`, `OrdOre` | `Date` (calendar date; time-of-day is truncated) | -| `Timestamp` | base, `Eq`, `Ord`, `OrdOre` | `Date` (time-of-day preserved) | -| `Text` | base, `Eq`, `Match`, `Ord`, `OrdOre`, `Search` | `string` | -| `Boolean` | base only | `boolean` | -| `Json` | `Json` only | a JSON *document* (`JsonDocument`: object, array, or null — NOT a top-level scalar; nested values are any `JsonValue`) | - -Examples: `types.Text("notes")` (storage only), `types.TextEq("username")`, `types.BigintOrd("balance")`, `types.TimestampOrdOre("created_at")`, `types.Boolean("active")`, `types.Json("metadata")`. - -The match index on `Match`/`Search` columns is always emitted with the default configuration — there is no per-column tuning in v3. - -### Free-Text Queries on `types.TextSearch` - -A `TextSearch` column carries all three indexes, and `encryptQuery` with **no explicit `queryType` builds an equality term, not a free-text match** (index inference priority: unique > match > ore). A substring like `"joh"` then matches nothing. Pass `queryType: 'freeTextSearch'` explicitly for substring/token search: - -```typescript -// equality (default): exact value only -await client.encryptQuery("john@example.com", { column: users.email, table: users }) - -// free-text match: substring/token search -await client.encryptQuery("joh", { - column: users.email, - table: users, - queryType: "freeTextSearch", -}) -``` - -### Encrypted-JSONB Queries on `types.Json` - -A `types.Json("metadata")` column encrypts a whole JSON document (a -`JsonDocument`: object, array, or null — not a top-level scalar) to a -`public.eql_v3_json` value. - -**Containment** is the supported query today. Pass a sub-object or sub-array to -`encryptQuery` with `queryType: 'searchableJson'`; it matches documents that -contain the needle (jsonb `@>` semantics). Array containment is a subset test -regardless of element position — `{ roles: ['admin'] }` matches any document -whose `roles` array includes `admin`. - -```typescript -const events = encryptedTable("events", { metadata: types.Json("metadata") }) - -// containment: object needle -await client.encryptQuery({ roles: ["admin"] }, { column: events.metadata, table: events }) -``` - -Through the Drizzle v3 integration this is `ops.contains(col, subObject)` — see -the `stash-drizzle` skill. `types.Json` carries no equality or ordering, so -`eq` / `gt` / `asc` on it throw. - -> **Not yet implemented:** JSONPath selector-with-constraint queries -> (`metadata->'plan' = $1`, `metadata->'age' > $1`) — a distinct third pattern -> the `eql_v3_json` domain supports at the SQL level (`->` / `->>`). Neither the -> query operator nor the selector-string needle typing is wired up yet; tracked -> in [#623](https://github.com/cipherstash/stack/issues/623). - -### Strongly-Typed Client: `EncryptionV3` - -`EncryptionV3` from `@cipherstash/stack/v3` returns a `TypedEncryptionClient` whose method signatures are derived from your schemas — wrong-typed plaintext is rejected at compile time, and query methods only accept queryable columns with `queryType` constrained to the column's capabilities: - -```typescript -import { EncryptionV3, encryptedTable, types } from "@cipherstash/stack/v3" - -const users = encryptedTable("users", { - email: types.TextSearch("email"), - lastLogin: types.TimestampOrd("last_login"), - balance: types.BigintEq("balance"), -}) - -const client = await EncryptionV3({ schemas: [users] }) - -// Plaintext is pinned to the column's domain type: -await client.encrypt("alice@example.com", { table: users, column: users.email }) // string ✓ -await client.encrypt(new Date(), { table: users, column: users.lastLogin }) // Date ✓ -// client.encrypt(42, { table: users, column: users.email }) // ✗ compile error - -const enc = await client.encryptModel( - { id: "u1", email: "alice@example.com", lastLogin: new Date(), balance: 100n }, - users, -) -if (!enc.failure) { - const dec = await client.decryptModel(enc.data, users) - if (!dec.failure) { - dec.data.email // string - dec.data.lastLogin // Date — reconstructed on decrypt - dec.data.balance // bigint - dec.data.id // string — non-schema fields pass through - } -} -``` - -Typed-client notes: - -- The wire format is pinned to EQL v3 (`eqlVersion: 3`); you don't set it yourself. -- Methods: `encrypt`, `encryptQuery`, `encryptModel`, `bulkEncryptModels`, `decrypt`, `decryptModel`, `bulkDecryptModels`, plus `bulkEncrypt`/`bulkDecrypt` passthroughs and `getEncryptConfig`. -- `decryptModel` / `bulkDecryptModels` take the **table as a second argument** and return a plain `Promise>` (not a chainable operation) — pass a lock context as the optional third argument instead of chaining `.withLockContext()`. `Date` columns are reconstructed to real `Date` instances on decrypt; `bigint` columns round-trip as native `bigint`. -- `decrypt` of a single value cannot be strongly typed — a lone ciphertext carries no column identity. -- `encryptQuery` rejects storage-only columns outright at compile time. -- `typedClient(client, ...schemas)` (also exported from `@cipherstash/stack/v3`) wraps an already-built `EncryptionClient` in the typed surface. - -### Database Setup - -Install the EQL v3 SQL with the stash CLI: - -```bash -stash eql install --eql-version 3 -``` - -EQL v3 ships one SQL bundle for every target, including Supabase — no separate Supabase or no-operator-family variants. v3 currently installs via the direct path only (`--drizzle`, `--migration`, `--migrations-dir`, and `--latest` are not supported for v3 yet). - -In migrations, declare each encrypted column as its domain type: - -```sql -CREATE TABLE users ( - id BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY, - email public.eql_v3_text_search, - last_login public.eql_v3_timestamp_ord, - balance public.eql_v3_bigint_eq -); -``` - -### Type Inference - -```typescript -import type { InferPlaintext, InferEncrypted } from "@cipherstash/stack/eql/v3" - -type UserPlaintext = InferPlaintext -// { email: string; lastLogin: Date; balance: bigint } - -type UserEncrypted = InferEncrypted -// { email: Encrypted; lastLogin: Encrypted; balance: Encrypted } -``` - -`V3ModelInput`, `V3EncryptedModel`, and `V3DecryptedModel` (same subpath) are the model-shape helpers the typed client uses: schema-column keys are pinned to the column's plaintext type (nullable fields stay nullable), non-schema keys pass through unchanged. - -### Drizzle ORM - -`@cipherstash/stack-drizzle/v3` provides Drizzle-native v3 column factories, schema extraction, and auto-encrypting query operators. See the `stash-drizzle` skill for the full guide. - ## Rolling Encryption Out to Production +> **EQL v2 note ([#648](https://github.com/cipherstash/stack/issues/648)).** The backfill/cutover tooling in this section (`stash encrypt *`, `stash db push`, `@cipherstash/migrate`) currently targets **EQL v2 columns** — the `eql_v2_configuration` registry, `eql_v2_encrypted` payload columns, and v2 schemas. v3 support for the rollout lifecycle is tracked in [cipherstash/stack#648](https://github.com/cipherstash/stack/issues/648). The *process* below — rollout, deploy gate, cutover — is the model to follow either way; every `eql_v2`-named artifact it mentions is part of that v2-targeting tooling. + Adding a fresh encrypted column to a table you don't yet write to is the easy case — declare it in the schema, run the migration, start writing. The harder case is taking an **existing plaintext column with live data** and turning it into an encrypted one without dropping a write or returning the wrong value mid-cutover. CipherStash splits that into two named steps with a hard production-deploy gate between them: @@ -885,7 +773,7 @@ Once dual-writes are recorded as live in `cs_migrations`: | `stash encrypt backfill` | Walks the table in keyset-pagination order, encrypts each chunk, writes a single transactional `UPDATE` per chunk plus a `cs_migrations` checkpoint. SIGINT-safe; idempotent re-runs converge. | | Schema rename | Update the schema file: drop the `_encrypted` suffix; switch the original column declaration onto the encrypted type. | | `stash encrypt cutover` | One transaction: renames `` → `_plaintext`, `_encrypted` → ``, and promotes `pending` → `active`. Application reads of `` now return decrypted ciphertext transparently. | -| Wire reads through the encryption client | Read paths must decrypt before returning the value to callers (`decryptModel(row, table)` for Drizzle; `encryptedSupabase` wrapper for Supabase; `decrypt`/`bulkDecryptModels` otherwise). Without this step, reads return raw `eql_v2_encrypted` payloads to end users. | +| Wire reads through the encryption client | Read paths must decrypt before returning the value to callers (`decryptModel(row, table)` for Drizzle; the Supabase wrapper for Supabase; `decrypt`/`bulkDecryptModels` otherwise). Without this step, reads return raw `eql_v2_encrypted` payloads to end users. | | Remove dual-write code | The plaintext column is now `_plaintext` and is no longer authoritative. Delete the dual-write logic. | | `stash encrypt drop` | Emits a migration that removes `_plaintext`. Apply with the project's normal migration tooling. | @@ -905,7 +793,7 @@ Three sources of truth, kept separate on purpose: ### CLI sequence for a single column -> **Known limitation:** `stash encrypt cutover` currently requires a pending EQL configuration registered via `stash db push`. SDK-only users may hit a "No pending EQL configuration" error. **Workaround:** Run `stash db push` once before `stash encrypt cutover`, even if you don't use CipherStash Proxy. Decoupling cutover from EQL config for SDK users is tracked in issue [#447](https://github.com/cipherstash/stack/issues/447) follow-up work. +> **Known limitation:** `stash encrypt cutover` currently requires a pending EQL configuration registered via `stash db push`. SDK-only users may hit a "No pending EQL configuration" error. **Workaround:** Run `stash db push` once before `stash encrypt cutover`, even if you don't use CipherStash Proxy. Decoupling cutover from EQL config for SDK users is tracked separately. ```bash # Run this often — it's the canonical "where am I?" command. @@ -1017,7 +905,7 @@ await runBackfill({ }) ``` -Useful when the backfill needs to run in a worker, on a schedule, or alongside an existing job runner. +Useful when the backfill needs to run in a worker, on a schedule, or alongside an existing job runner. (Like the CLI, `runBackfill` currently targets EQL v2 columns — see the note at the top of this section.) ### Invariants the rollout preserves @@ -1027,40 +915,60 @@ Useful when the backfill needs to run in a worker, on a schedule, or alongside a - **Re-runs are safe.** Backfill is idempotent (`IS NOT NULL AND _encrypted IS NULL` guards every chunk). `cs_migrations` is append-only. - **Rollback is possible up to cutover.** Until the rename happens, the plaintext column is authoritative; aborting just leaves the encrypted twin partially populated. After cutover, rollback is a manual restore — treat cutover as the one-way door for data. -## Migration from @cipherstash/protect +## Integrations -| `@cipherstash/protect` | `@cipherstash/stack` | Import Path | +| Target | Package / entry point | Skill | |---|---|---| -| `protect(config)` | `Encryption(config)` | `@cipherstash/stack` | -| `csTable(name, cols)` | `encryptedTable(name, cols)` | `@cipherstash/stack/schema` | -| `csColumn(name)` | `encryptedColumn(name)` | `@cipherstash/stack/schema` | -| `LockContext` from `/identify` | `LockContext` from `/identity` | `@cipherstash/stack/identity` | - -All method signatures on the encryption client remain the same. The `Result` pattern is unchanged. +| Drizzle ORM | `@cipherstash/stack-drizzle/v3` — v3 column factories (each `types.*` factory emits its domain as the column's SQL type for `drizzle-kit generate`), schema extraction, auto-encrypting operators (`ops.eq`, `ops.matches`, `ops.contains`, `ops.selector`, `ops.asc`, ...) | `stash-drizzle` | +| Supabase | `encryptedSupabaseV3` from `@cipherstash/stack-supabase` — schema-aware query builder (`eq`, `matches`, `contains`, `selectorEq`/`selectorNe`, ...) that works through PostgREST, including as `anon` | `stash-supabase` | +| Prisma | `@cipherstash/prisma-next` — searchable field-level encryption for Postgres | — | +| DynamoDB | `encryptedDynamoDB` from `@cipherstash/stack/dynamodb` — **v2 schema surface only**, see the exception note under "Legacy: EQL v2" | `stash-dynamodb` | ## Complete API Reference -### EncryptionClient Methods +### TypedEncryptionClient Methods | Method | Signature | Returns | |---|---|---| -| `encrypt` | `(plaintext, { column, table })` | `EncryptOperation` | -| `decrypt` | `(encryptedData)` | `DecryptOperation` | -| `encryptQuery` | `(plaintext, { column, table, queryType?, returnType? })` | `EncryptQueryOperation` | -| `encryptQuery` | `(terms: readonly ScalarQueryTerm[])` | `BatchEncryptQueryOperation` | -| `encryptModel` | `(model, table)` | `EncryptModelOperation>` | -| `decryptModel` | `(encryptedModel)` | `DecryptModelOperation` — resolves to `Decrypted` | -| `bulkEncrypt` | `(plaintexts, { column, table })` | `BulkEncryptOperation` | -| `bulkDecrypt` | `(encryptedPayloads)` | `BulkDecryptOperation` | -| `bulkEncryptModels` | `(models, table)` | `BulkEncryptModelsOperation>` | -| `bulkDecryptModels` | `(encryptedModels)` | `BulkDecryptModelsOperation` — resolves to `Decrypted[]` | - -All operations are thenable (awaitable) and support `.withLockContext()` and `.audit()` chaining. +| `encrypt` | `(plaintext, { table, column })` — plaintext pinned to the column's domain type | `EncryptOperation` | +| `decrypt` | `(encryptedData)` — untyped (no column identity) | `DecryptOperation` | +| `encryptQuery` | `(plaintext, { table, column, queryType?, returnType? })` — queryable columns only; `queryType` constrained to the column's capabilities | `EncryptQueryOperation` | +| `encryptQuery` | `(terms: readonly ScalarQueryTerm[])` — batch form | `BatchEncryptQueryOperation` | +| `encryptModel` | `(model, table)` — schema fields validated against inferred plaintext types | `EncryptModelOperation>` | +| `decryptModel` | `(model, table, lockContext?)` | `Promise, EncryptionError>>` | +| `bulkEncryptModels` | `(models, table)` | `BulkEncryptModelsOperation>` | +| `bulkDecryptModels` | `(models, table, lockContext?)` | `Promise[], EncryptionError>>` | +| `bulkEncrypt` | `(plaintexts, { column, table })` — parity passthrough | `BulkEncryptOperation` | +| `bulkDecrypt` | `(encryptedPayloads)` — parity passthrough | `BulkDecryptOperation` | +| `getEncryptConfig` | `()` | The client's encrypt config | + +The encrypt-side operations are thenable (awaitable) and support `.withLockContext()` and `.audit()` chaining; `decryptModel`/`bulkDecryptModels` return plain promises and take the lock context as an argument. ### Schema Builders ```typescript -encryptedTable(tableName: string, columns: Record) -encryptedColumn(columnName: string) // chainable: .equality(), .freeTextSearch(), .orderAndRange(), .searchableJson(), .dataType() -encryptedField(valueName: string) // for nested encrypted fields (not searchable), chainable: .dataType() +encryptedTable(tableName: string, columns: Record) +types.(dbColumnName: string) +// e.g. types.TextSearch("email"), types.BigintOrd("balance"), types.Json("metadata") +``` + +## Legacy: EQL v2 + +Before the v3 typed schema, encrypted columns were a single composite type (`eql_v2_encrypted`) and query capabilities were enabled per column with chainable builders: + +```typescript +// Legacy v2 surface — for existing deployments only +import { Encryption } from "@cipherstash/stack" +import { encryptedTable, encryptedColumn } from "@cipherstash/stack/schema" + +const users = encryptedTable("users", { + email: encryptedColumn("email").equality().freeTextSearch().orderAndRange(), + metadata: encryptedColumn("metadata").searchableJson(), +}) + +const client = await Encryption({ schemas: [users] }) ``` + +The v2 API — `Encryption` plus the `@cipherstash/stack/schema` builders, the v2 Drizzle/Supabase integrations (`@cipherstash/stack-drizzle`, `encryptedSupabase`), and `stash eql install --eql-version 2` — **still exists and is supported for existing deployments**. Full v2 documentation lives at [cipherstash.com/docs](https://cipherstash.com/docs). Remember: v2 and v3 tables cannot be mixed in one client. (If you are migrating code from the old `@cipherstash/protect` package, its `protect`/`csTable`/`csColumn` names map onto this v2 surface.) + +> **Exception — DynamoDB.** The DynamoDB integration (`encryptedDynamoDB` from `@cipherstash/stack/dynamodb`) **still requires the v2 schema surface** — `encryptedTable`/`encryptedColumn`/`encryptedField` from `@cipherstash/stack/schema`. Do not use v3 `types.*` schemas with it. See the `stash-dynamodb` skill; v3 support is tracked in [cipherstash/stack#657](https://github.com/cipherstash/stack/issues/657). diff --git a/skills/stash-supabase/SKILL.md b/skills/stash-supabase/SKILL.md index 455d5363..1ed8a440 100644 --- a/skills/stash-supabase/SKILL.md +++ b/skills/stash-supabase/SKILL.md @@ -1,16 +1,26 @@ --- name: stash-supabase -description: Integrate CipherStash encryption with Supabase using @cipherstash/stack-supabase. Covers the encryptedSupabase wrapper, transparent encryption/decryption on insert/update/select, encrypted query filters (eq, like, ilike, gt/gte/lt/lte, in, or, match), identity-aware encryption, and the complete query builder API. Use when adding encryption to a Supabase project, querying encrypted columns, or building secure Supabase applications. +description: Integrate CipherStash encryption with Supabase using @cipherstash/stack-supabase. Covers the encryptedSupabaseV3 wrapper over native EQL v3 column domains, transparent encryption/decryption on insert/update/select, encrypted query filters (eq, matches, gt/gte/lt/lte, in, or, match), encrypted JSON querying (contains, selectorEq/selectorNe), ordering on encrypted columns, identity-aware encryption, and the complete query builder API. Use when adding encryption to a Supabase project, querying encrypted columns, or building secure Supabase applications. --- # CipherStash Stack - Supabase Integration -Guide for integrating CipherStash field-level encryption with Supabase using the `encryptedSupabase` wrapper. The wrapper provides transparent encryption on mutations and decryption on selects, with full support for querying encrypted columns. +Guide for integrating CipherStash field-level encryption with Supabase using +the `encryptedSupabaseV3` wrapper over native EQL v3 column domains. The +wrapper provides transparent encryption on mutations and decryption on +selects, with full support for querying encrypted columns — equality, range, +fuzzy free-text search, encrypted JSON containment and selectors, and +ordering. + +A legacy EQL v2 wrapper (`encryptedSupabase`) still ships for existing +deployments — see "Legacy: EQL v2" at the end. New projects should use +`encryptedSupabaseV3`. ## When to Use This Skill - Adding field-level encryption to a Supabase project -- Querying encrypted data with Supabase's query builder (eq, like, gt, in, or, etc.) +- Querying encrypted data with Supabase's query builder (eq, matches, gt, in, or, etc.) +- Querying encrypted JSON documents (containment, path selectors) - Inserting, updating, or upserting encrypted data - Using identity-aware encryption (lock contexts) with Supabase - Building applications where sensitive columns need encryption at rest and in transit @@ -24,107 +34,141 @@ npm install @cipherstash/stack @cipherstash/stack-supabase @supabase/supabase-js The Supabase integration ships as its own first-party package, `@cipherstash/stack-supabase`, which depends on `@cipherstash/stack`. Install both. -## Database Schema +## Setup + +**Credentials first:** for local development run `npx stash init` (the +agent-assisted flow — auth, schema, and database end to end) or +`npx stash auth login` (device code flow; no environment variables needed). +CI and production use the `CS_*` machine-credential environment variables — +see the `stash-encryption` skill's Configuration section. + +### 1. Install EQL v3 on the database -Encrypted columns must be stored as JSONB in your Supabase database: +```bash +stash eql install --eql-version 3 --supabase +``` + +Since eql-3.0.0 there is **one** v3 SQL artifact for every target — there is +no separate Supabase variant. The bundle's only superuser-requiring +statements (the ORE operator class/family) skip themselves when the install +role lacks the privilege, and the bundle then disables the ORE-opclass-backed +domains it cannot support. `--supabase` changes one thing: it additionally +applies the role grants for `anon` / `authenticated` / `service_role` to the +two schemas the bundle creates — `eql_v3` (the operator-backing functions) +and `eql_v3_internal` (SEM internals). Without the grants, encrypted queries +fail loudly with a permission error (e.g. `permission denied for schema +eql_v3_internal`). + +No **Exposed schemas** change is needed: the column domains and their +operators live in `public`, so bare `col = term` filters resolve under +Supabase's default PostgREST configuration. Do not expose `eql_v3_internal`. + +### 2. Database schema (per-domain columns) + +Each encrypted column is declared with a concrete `public.eql_v3_*` domain — +the domain encodes both the plaintext type and the column's query +capabilities. There is no extension to enable and no generic `jsonb` column: ```sql CREATE TABLE users ( id SERIAL PRIMARY KEY, - email jsonb NOT NULL, -- encrypted column - name jsonb NOT NULL, -- encrypted column - age jsonb, -- encrypted column (numeric) - role VARCHAR(50), -- regular column (not encrypted) + email public.eql_v3_text_search, -- eq + range + free-text search + amount public.eql_v3_integer_ord, -- eq + range + joined_at public.eql_v3_timestamp_ord, -- eq + range, decrypts to Date + payload public.eql_v3_json, -- encrypted JSON document + role VARCHAR(50), -- regular plaintext column created_at TIMESTAMPTZ DEFAULT NOW() ); ``` -For searchable encryption (equality, range, text search), install the EQL extension: +The `types.*` member name (see declared schemas below) maps to the flat +`public.eql_v3_` domain — strip the `eql_v3_` prefix and PascalCase +each `_`-separated segment: `types.TextEq` → `public.eql_v3_text_eq`, +`types.IntegerOrd` → `public.eql_v3_integer_ord`. The domains use +SQL-standard type names (`integer`, `smallint`, `real`, `double`, `boolean`, +`timestamp`). -```sql -CREATE EXTENSION IF NOT EXISTS eql_v2; -``` +### 3. Initialize the wrapper -## Setup +```typescript +import { encryptedSupabaseV3 } from "@cipherstash/stack-supabase" -**Credentials first:** for local development run `npx stash init` (the -agent-assisted flow — auth, schema, and database end to end) or -`npx stash auth login` (device code flow; no environment variables needed). -CI and production use the `CS_*` machine-credential environment variables — -see the `stash-encryption` skill's Configuration section. +// Introspects the database via options.databaseUrl or DATABASE_URL +const es = await encryptedSupabaseV3(supabaseUrl, supabaseKey) +// or wrap an existing client: await encryptedSupabaseV3(supabaseClient, options) -### 1. Define Encrypted Schema +await es.from("users").insert({ email: "a@b.com", amount: 30 }) +await es.from("users").select("id, email, amount").eq("email", "a@b.com") +``` -```typescript -import { encryptedTable, encryptedColumn } from "@cipherstash/stack/schema" +`encryptedSupabaseV3` **introspects the database at connect time**: it +detects EQL v3 columns by their Postgres domain, derives each column's +encryption config from the domain, and builds the encryption client +internally — there is no client-side schema to hand-maintain. Introspection +needs a direct Postgres connection (`options.databaseUrl`, defaulting to +`DATABASE_URL`), so the factory cannot run in a Worker or the browser. -const users = encryptedTable("users", { - email: encryptedColumn("email") - .equality() // eq, neq, in - .freeTextSearch(), // like, ilike - - name: encryptedColumn("name") - .equality() - .freeTextSearch(), - - age: encryptedColumn("age") - .dataType("number") - .equality() - .orderAndRange(), // gt, gte, lt, lte -}) -``` +Options: `{ schemas?, databaseUrl?, config? }` — `config` is the encryption +client config (e.g. `config.authStrategy`, see Authentication below). -### 2. Initialize Clients +`from(tableName)` takes only the table name — no schema argument. Column +capabilities come from the introspected domains. -```typescript -import { createClient } from "@supabase/supabase-js" -import { Encryption } from "@cipherstash/stack" -import { encryptedSupabase } from "@cipherstash/stack-supabase" +### 4. Optional declared schemas (compile-time types) -const supabase = createClient( - process.env.SUPABASE_URL!, - process.env.SUPABASE_ANON_KEY!, -) +Declaring tables is optional. Passing `schemas` — a record whose keys must +equal each table's name — adds compile-time types and verifies the declared +tables against the database at construction: -const encryptionClient = await Encryption({ schemas: [users] }) +```typescript +import { encryptedTable, types } from "@cipherstash/stack/eql/v3" +import { encryptedSupabaseV3 } from "@cipherstash/stack-supabase" -const eSupabase = encryptedSupabase({ - encryptionClient, - supabaseClient: supabase, +const users = encryptedTable("users", { + email: types.TextSearch("email"), // public.eql_v3_text_search — eq + range + free-text + amount: types.IntegerOrd("amount"), // public.eql_v3_integer_ord — eq + range + joined: types.TimestampOrd("joined_at") // public.eql_v3_timestamp_ord — eq + range, decrypts to Date }) -``` -### 3. Use the Wrapper - -All queries go through `eSupabase.from(tableName, schema)`: +const es = await encryptedSupabaseV3(supabaseUrl, supabaseKey, { + schemas: { users }, +}) -```typescript -const { data, error } = await eSupabase - .from("users", users) - .select("id, email, name") - .eq("email", "alice@example.com") +const { data } = await es.from("users").select("id, email, joined").eq("email", "a@b.com") ``` +A declared table gets a typed builder: rows infer each column's plaintext +type (`types.IntegerOrd` → `number`, `types.TimestampOrd` → `Date`), +storage-only columns are excluded from every filter method, `matches()` is +narrowed to match-indexed columns, and `order()` to orderable columns. +Undeclared tables behave exactly as with no `schemas` at all. Every v3 column +is fully described by its `types.*` factory — there are no capability or +tuning chains on v3 columns. + +A JS property may map to a different DB column name +(`joined: types.TimestampOrd("joined_at")`) — filters, selects, and results +are translated automatically, and `date`/`timestamp` columns decrypt to real +`Date` objects. + ## Insert (Encrypted Automatically) ```typescript // Single insert -const { data, error } = await eSupabase - .from("users", users) +const { data, error } = await es + .from("users") .insert({ email: "alice@example.com", // encrypted automatically - name: "Alice Smith", // encrypted automatically - age: 30, // encrypted automatically - role: "admin", // not in schema, passed through + amount: 30, // encrypted automatically + role: "admin", // plaintext column, passed through }) .select("id") // Bulk insert -const { data, error } = await eSupabase - .from("users", users) +const { data, error } = await es + .from("users") .insert([ - { email: "alice@example.com", name: "Alice", age: 30, role: "admin" }, - { email: "bob@example.com", name: "Bob", age: 25, role: "user" }, + { email: "alice@example.com", amount: 30, role: "admin" }, + { email: "bob@example.com", amount: 25, role: "user" }, ]) .select("id") ``` @@ -132,105 +176,109 @@ const { data, error } = await eSupabase ## Update (Encrypted Automatically) ```typescript -const { data, error } = await eSupabase - .from("users", users) - .update({ name: "Alice Johnson" }) // encrypted automatically +const { data, error } = await es + .from("users") + .update({ email: "alice@new.example.com" }) // encrypted automatically .eq("id", 1) - .select("id, name") + .select("id, email") ``` ## Upsert ```typescript -const { data, error } = await eSupabase - .from("users", users) +const { data, error } = await es + .from("users") .upsert( - { id: 1, email: "alice@example.com", name: "Alice", role: "admin" }, + { id: 1, email: "alice@example.com", role: "admin" }, { onConflict: "id" }, ) - .select("id, email, name") + .select("id, email") ``` ## Select (Decrypted Automatically) ```typescript -// List query - returns decrypted array -const { data, error } = await eSupabase - .from("users", users) - .select("id, email, name, role") -// data: [{ id: 1, email: "alice@example.com", name: "Alice Smith", role: "admin" }] +// All columns — select('*') (and bare select()) expands to the +// introspected column list +const { data, error } = await es.from("users").select("*") + +// Explicit columns +const { data, error } = await es + .from("users") + .select("id, email, amount, role") +// data: [{ id: 1, email: "alice@example.com", amount: 30, role: "admin" }] // Single result -const { data, error } = await eSupabase - .from("users", users) - .select("id, email, name") +const { data, error } = await es + .from("users") + .select("id, email") .eq("id", 1) .single() -// data: { id: 1, email: "alice@example.com", name: "Alice Smith" } // Maybe single (returns null if no match) -const { data, error } = await eSupabase - .from("users", users) +const { data, error } = await es + .from("users") .select("id, email") .eq("email", "nobody@example.com") .maybeSingle() // data: null ``` -**Important:** You must list columns explicitly in `select()` — using `select('*')` will throw an error. The wrapper automatically adds `::jsonb` casts to encrypted columns so PostgreSQL parses them correctly. - `select()` also accepts an optional second parameter: `select(columns, { head?: boolean, count?: 'exact' | 'planned' | 'estimated' })`. ## Query Filters -All filter values for encrypted columns are automatically encrypted before the query executes. Multiple filters are batch-encrypted in a single ZeroKMS call for efficiency. +All filter values for encrypted columns are automatically encrypted before +the query executes. Filter operands are grouped by column and each column +group takes one `bulkEncrypt` crossing — a query filtering N distinct +encrypted columns makes N ZeroKMS calls, run in parallel. ### Equality Filters ```typescript -// Exact match (requires .equality() on column) +// Exact match (requires an equality-capable domain) .eq("email", "alice@example.com") // Not equal .neq("email", "alice@example.com") -// IN array (requires .equality()) +// IN array .in("email", ["alice@example.com", "bob@example.com"]) -// NULL check (no encryption needed) +// NULL check (no encryption needed). Use this for genuine null checks — a +// null operand passed to eq/neq is not rejected; it is forwarded unencrypted. .is("email", null) ``` -### Text Search Filters +### Free-Text Search (`matches`) ```typescript -// LIKE - case sensitive (requires .freeTextSearch()) -.like("name", "%alice%") - -// ILIKE - case insensitive (requires .freeTextSearch()) -.ilike("name", "%alice%") +// Fuzzy bloom token search (requires a match-indexed domain, +// e.g. text_search / text_match). Matches substrings of >= 3 characters. +.matches("email", "alice") ``` +`like`/`ilike` on an encrypted column are a **deprecated approximate shim** +that delegates to `matches()` — see "Query behaviour on encrypted columns" +below. On plaintext columns they stay real SQL LIKE. The shim is +**runtime-only**: neither the typed nor the untyped v3 builder interface +declares `like`/`ilike`, so in TypeScript the call is a compile error (use +`matches()`) — the shim is reachable only from plain JavaScript or via a cast. + ### Range/Comparison Filters ```typescript -// Greater than (requires .orderAndRange()) -.gt("age", 21) - -// Greater than or equal -.gte("age", 18) - -// Less than -.lt("age", 65) - -// Less than or equal -.lte("age", 100) +// Requires a range-capable domain (e.g. *_ord, text_search) +.gt("amount", 21) +.gte("amount", 18) +.lt("amount", 65) +.lte("amount", 100) ``` ### Match (Multi-Column Equality) ```typescript -.match({ email: "alice@example.com", name: "Alice" }) +.match({ email: "alice@example.com", amount: 30 }) ``` ### OR Conditions @@ -263,8 +311,8 @@ Both forms encrypt values for encrypted columns automatically. ## Delete ```typescript -const { data, error } = await eSupabase - .from("users", users) +const { data, error } = await es + .from("users") .delete() .eq("id", 1) ``` @@ -274,7 +322,7 @@ const { data, error } = await eSupabase These are passed through to Supabase directly: ```typescript -.order("name", { ascending: true }) +.order("email", { ascending: true }) // encrypted columns: see behaviour below .limit(10) .range(0, 9) .csv() @@ -283,17 +331,129 @@ These are passed through to Supabase directly: .returns() ``` -### Ordering by Encrypted Columns +`order()` works on plaintext columns and on OPE-backed encrypted ordering +columns — see the `order()` bullet in the next section for exactly which +domains qualify. -**`ORDER BY` on encrypted columns is not currently supported** on databases without operator family support (including Supabase). +## Query behaviour on encrypted columns -Without operator families installed in PostgreSQL, the database cannot sort on `eql_v2_encrypted` columns. This affects all clients — the Supabase JS SDK, Drizzle, raw SQL, and any other ORM. +All envelopes (stored payloads and filter operands) are versioned `v: 3`. -**Workaround:** Sort application-side after decrypting the results. +- **`select('*')` (and bare `select()`) works** — it expands to the + introspected column list. +- **Encrypted free-text search is `matches()`, not `contains()`/`like`/`ilike`.** + The v3 domains define no LIKE operator — encrypted free-text search is fuzzy + bloom-filter token matching (PostgREST `cs` / SQL `@>`): one-sided (a match + may be a false positive, a non-match never is) and order-/multiplicity- + insensitive, where `%` is tokenized like any other character. `matches(col, + needle)` is the operator; `contains()` on an encrypted TEXT column throws an + error pointing at `matches()`. `contains()` is native (exact) jsonb/array + containment on plaintext columns — and ENCRYPTED (exact) containment on a + `types.Json` column (see "Encrypted JSON querying" below). +- **`like`/`ilike` on an encrypted column are an approximate compatibility + shim** delegating to `matches()`: leading/trailing `%` are stripped and the + residual term is fuzzy-matched (same `cs` wire, plus a one-time warning). + Results are APPROXIMATE — case-insensitive, one-sided, and anchoring is not + honored. A pattern with an internal `%` or any `_` cannot be approximated and + throws; call `matches()` directly to make the fuzzy intent explicit. On + plaintext columns `like`/`ilike` stay real SQL LIKE. Note the shim is + reachable only at runtime: `like`/`ilike` are absent from the v3 builder + interfaces (typed and untyped alike), so on the TypeScript surface the call + is a compile error — only untyped JavaScript or a cast reaches the shim. +- **`matches()` matches substrings.** The search term blooms to its own + trigrams, and a row matches when the stored value's bloom contains all of + them — so any substring of at least 3 characters (the tokenizer's + `token_length`) matches. Shorter terms bloom to nothing and would match every + row, so they are rejected with an error rather than answered. +- **INTERIM — filter operands are full storage envelopes.** EQL ships + term-only query domains (`eql_v3.query_`, which accept envelopes with + no ciphertext) and the encryption client can mint those narrowed terms, but + PostgREST has no syntax to cast a filter value — an uncast operand can only + reach the `jsonb` operator overload, which coerces it into the storage + domain, whose CHECK requires ciphertext. So the adapter still encrypts each + filter value with the full storage path. The call shape is unchanged. + + **Security caveat:** query terms are meant to be index-terms-only by + design, but a full-envelope operand carries a real decryptable ciphertext + `c` plus **all** of the column's index terms, and PostgREST filters travel + in GET query strings — so these envelopes can land in URL logs, + intermediate proxies, and Supabase request logs. The remaining gap is + PostgREST operand casting; an adapter-side fix is tracked. +- **`order()` works on OPE-backed encrypted ordering columns** (every plain + `*_ord` domain, plus `text_ord` and `text_search`). PostgREST cannot emit + `ORDER BY eql_v3.ord_term(col)`, and a bare `ORDER BY` would silently sort + the raw ciphertext envelope — so the builder instead emits `order=col->op`, + sorting by the OPE term inside the envelope, which reproduces plaintext + order (the term is fixed-width lowercase hex, so string comparison agrees + with the bytea btree; pinned by `ope-term.integration.test.ts`). ORE-flavour + columns (`*_ord_ore`) are rejected at compile time and runtime — their `ob` + term needs the superuser-only operator class no jsonb path can reach — and + columns with no ordering term (storage-only, equality-only, match-only) + reject `order()` with a clear error. For those, order by a plaintext column + or sort application-side after decrypting. +- **Storage-only domains are not filterable** (e.g. `types.Boolean`, + `types.Text`): a filter (including `.match()`) on one is a type error on a + declared table, and always a clear runtime error. `.is(column, null)` + remains available. +- **Null filter operands are forwarded unencrypted, not rejected.** A null + cannot be encrypted into an operand, so the builder skips encryption and + passes the null through to PostgREST as-is (e.g. a `col=eq.null` filter), + which is rarely what you want. Use `.is(column, null)` for genuine null + checks — the builder does not throw. -Operator family support is currently being developed in collaboration with the Supabase and CipherStash teams and will be available in a future release. +## Encrypted JSON querying (`types.Json`) -`.order()` on non-encrypted columns works normally. +A `types.Json("payload")` column (`public.eql_v3_json`) stores an encrypted +JSONB document and supports two query forms: + +```typescript +// Exact encrypted containment: every leaf of the sub-document must match at +// its path (ste_vec `@>` — PostgREST `cs`). +es.from("events").select("id").contains("payload", { user: { role: "admin" } }) + +// JSONPath selector equality / inequality at a dot-notation path: +es.from("events").select("id").selectorEq("payload", "$.user.role", "admin") +es.from("events").select("id").selectorNe("payload", "$.user.role", "admin") +``` + +- `selectorEq(col, path, value)` matches rows carrying exactly `value` at + `path` — it compiles to containment of the path-shaped needle + (`{user: {role: "admin"}}`), which is the same operation. Paths are + dot-notation object keys (`"$.a.b"` or `"a.b"`); array/wildcard steps are + rejected, and values must be JSON scalars — string, number, or boolean (an + object operand belongs to `contains()`). On Supabase a `Date` or `bigint` + leaf is rejected with a serialization steer: pass a Date as + `date.toISOString()` and a bigint as its string/number form. (The Drizzle + selector accepts Date/bigint directly.) +- `selectorNe` matches rows that do NOT carry the value — **including rows + where the path is absent entirely, and rows whose document column is SQL + NULL** (it compiles to `payload.is.null OR payload.not.cs.`, matching + the Drizzle selector's `ne` semantics for both absence cases). +- **Array-valued paths:** a scalar needle does NOT match an array at the path — + ste_vec encodes array elements under their own selectors — so + `selectorEq("payload", "$.roles", "admin")` does not match + `{roles: ["admin", "analyst"]}` (and `selectorNe` includes that row). To + match an array-valued path, pass the full array through `contains()`. +- **Selector ordering (`gt`/`gte`/`lt`/`lte`) is not available on Supabase.** + PostgREST cannot reach the entry-comparison operators (it wraps JSON arrow + paths in `to_jsonb`, and bare-column comparison is blocked by design); it + needs an EQL-bundle overload, tracked in + cipherstash/encrypt-query-language#407. The Drizzle integration's + `ops.selector()` supports ordering today — use it where ordering at a path + is required. +- `matches()` does not apply to JSON columns (it is text free-text search) and + throws with a steer; scalar filters (`eq`, `gt`, `in`, …) on the column are + rejected by capability. +- The containment/selector operand is a **full storage envelope** of the + needle document. The GET-query-string security caveat above applies with an + important difference in degree: a JSON needle carries the root decryptable + ciphertext PLUS one ciphertext-bearing entry **per node of the sub-document** + — the exposure scales with needle size (the equivalent Drizzle containment + ships no ciphertext at all). Keep needles minimal; removing the ciphertext on + this surface is tracked in cipherstash/stack#654. +- Empty needles (`{}` / `[]`) are rejected — jsonb containment holds for every + document, so an accidentally-empty filter would silently return the whole + table (the Drizzle adapter rejects the same needle). ## Authentication @@ -305,7 +465,8 @@ third-party OIDC JWT (Clerk, Supabase, Auth0, ...) with `OidcFederationStrategy`: ```typescript -import { Encryption, OidcFederationStrategy } from "@cipherstash/stack" +import { OidcFederationStrategy } from "@cipherstash/stack" +import { encryptedSupabaseV3 } from "@cipherstash/stack-supabase" const strategy = OidcFederationStrategy.create( process.env.CS_WORKSPACE_CRN!, @@ -313,11 +474,9 @@ const strategy = OidcFederationStrategy.create( ) if (strategy.failure) throw new Error(strategy.failure.error.message) -const encryptionClient = await Encryption({ - schemas: [users], +const es = await encryptedSupabaseV3(supabaseUrl, supabaseKey, { config: { authStrategy: strategy.data }, }) -const eSupabase = encryptedSupabase({ supabaseClient, encryptionClient }) ``` Authentication stands on its own — an OIDC-authenticated client runs every @@ -333,9 +492,9 @@ resolves from the federated JWT; auto/access-key auth has no user JWT to resolve claims from. Plain authentication never requires a lock context. ```typescript -const { data, error } = await eSupabase - .from("users", users) - .insert({ email: "alice@example.com", name: "Alice" }) +const { data, error } = await es + .from("users") + .insert({ email: "alice@example.com" }) .withLockContext({ identityClaim: ["sub"] }) .select("id") ``` @@ -355,9 +514,9 @@ claim must be used to encrypt and decrypt. `.withLockContext()` also accepts a Chain `.audit()` to attach metadata for ZeroKMS audit logging: ```typescript -const { data, error } = await eSupabase - .from("users", users) - .select("id, email, name") +const { data, error } = await es + .from("users") + .select("id, email") .eq("email", "alice@example.com") .audit({ metadata: { action: "user-lookup", requestId: "abc-123" } }) ``` @@ -365,41 +524,38 @@ const { data, error } = await eSupabase ## Complete Example ```typescript -import { createClient } from "@supabase/supabase-js" -import { Encryption } from "@cipherstash/stack" -import { encryptedSupabase } from "@cipherstash/stack-supabase" -import { encryptedTable, encryptedColumn } from "@cipherstash/stack/schema" +import { encryptedTable, types } from "@cipherstash/stack/eql/v3" +import { encryptedSupabaseV3 } from "@cipherstash/stack-supabase" -// Schema +// Optional declared schema — compile-time types. Introspection alone +// (no `schemas`) also works. const users = encryptedTable("users", { - email: encryptedColumn("email").equality().freeTextSearch(), - name: encryptedColumn("name").equality().freeTextSearch(), - age: encryptedColumn("age").dataType("number").equality().orderAndRange(), + email: types.TextSearch("email"), + amount: types.IntegerOrd("amount"), }) -// Clients -const supabase = createClient(process.env.SUPABASE_URL!, process.env.SUPABASE_ANON_KEY!) -const encryptionClient = await Encryption({ schemas: [users] }) -const eSupabase = encryptedSupabase({ encryptionClient, supabaseClient: supabase }) +const es = await encryptedSupabaseV3( + process.env.SUPABASE_URL!, + process.env.SUPABASE_ANON_KEY!, + { schemas: { users } }, // databaseUrl defaults to DATABASE_URL +) -// Insert -await eSupabase - .from("users", users) - .insert([ - { email: "alice@example.com", name: "Alice", age: 30 }, - { email: "bob@example.com", name: "Bob", age: 25 }, - ]) +// Insert — values encrypted automatically +await es.from("users").insert([ + { email: "alice@example.com", amount: 30 }, + { email: "bob@example.com", amount: 25 }, +]) -// Query with multiple filters -const { data } = await eSupabase - .from("users", users) - .select("id, email, name, age") - .gte("age", 18) - .lte("age", 35) - .ilike("name", "%ali%") +// Query with multiple filters — operands encrypted automatically +const { data } = await es + .from("users") + .select("id, email, amount") + .gte("amount", 18) + .lte("amount", 35) + .matches("email", "alice") // data is fully decrypted: -// [{ id: 1, email: "alice@example.com", name: "Alice", age: 30 }] +// [{ id: 1, email: "alice@example.com", amount: 30 }] ``` ## Response Type @@ -428,200 +584,87 @@ type EncryptedSupabaseError = { } ``` -## Filter to Index Mapping +## Filter to Domain Capability Mapping -| Filter Method | Required Index | Query Type | -|---|---|---| -| `eq`, `neq`, `in` | `.equality()` | `'equality'` | -| `like`, `ilike` | `.freeTextSearch()` | `'freeTextSearch'` | -| `gt`, `gte`, `lt`, `lte` | `.orderAndRange()` | `'orderAndRange'` | -| `is` | None | No encryption (NULL/boolean check) | +The column's `public.eql_v3_*` domain determines which filters it accepts: + +| Filter Method | Works On | +|---|---| +| `eq`, `neq`, `in`, `match()` | Equality-capable domains (`*_eq`, `*_ord`, `text_search`) | +| `matches()` (and the runtime-only `like`/`ilike` shim — absent from the TS interfaces) | Match-indexed domains (`text_match`, `text_search`) | +| `gt`, `gte`, `lt`, `lte` | Range-capable domains (`*_ord`, `text_search`) | +| `contains()`, `selectorEq()`, `selectorNe()` | `eql_v3_json` (encrypted); `contains()` is also native containment on plaintext jsonb/array columns | +| `order()` | OPE-backed ordering domains (plain `*_ord`, `text_ord`, `text_search`) — never `*_ord_ore` | +| `is` | Any column (no encryption; NULL check) | + +Storage-only domains (e.g. `eql_v3_text`, `eql_v3_boolean`) accept no filters +at all — only `.is(column, null)`. ## Exported Types `@cipherstash/stack-supabase` also exports the following types: -- `EncryptedSupabaseConfig` -- `EncryptedSupabaseInstance` -- `EncryptedQueryBuilder` -- `PendingOrCondition` +- `EncryptedSupabaseV3Options`, `EncryptedSupabaseV3Instance`, `TypedEncryptedSupabaseV3Instance`, `EncryptedQueryBuilderV3`, `EncryptedQueryBuilderV3Untyped`, `V3Schemas` - `SupabaseClientLike` -- `EncryptedSupabaseV3Options`, `EncryptedSupabaseV3Instance`, `TypedEncryptedSupabaseV3Instance`, `EncryptedQueryBuilderV3`, `EncryptedQueryBuilderV3Untyped`, `V3Schemas` (EQL v3) +- `EncryptedSupabaseConfig`, `EncryptedSupabaseInstance`, `EncryptedQueryBuilder`, `PendingOrCondition` (legacy EQL v2) -## EQL v3 (native `public.eql_v3_*` domains) +## Migrating an Existing Column to Encrypted -`encryptedSupabaseV3` is the EQL v3 counterpart of `encryptedSupabase`. -Instead of taking a schema, it **introspects the database at connect time**: -it detects EQL v3 columns by their Postgres domain, derives each column's -encryption config from the domain, and builds the encryption client -internally. Columns are stored in their native `public.eql_v3_*` domain (a -`DOMAIN … AS jsonb` with a CHECK) instead of the v2 composite -`eql_v2_encrypted`. +The hard case: a Supabase table that already exists with live data in a plaintext column you want to encrypt. You can't just change the column type — that would drop the data. -The query surface matches v2 — same filter methods, `withLockContext`, -`audit` — with one deliberate fork: encrypted free-text search is `matches()` -(fuzzy bloom token search); `contains()` stays native (exact) containment for -plaintext columns; and `like`/`ilike` on an encrypted column are an approximate -shim that delegates to `matches()` (see below). +CipherStash splits this into two named steps with a hard production-deploy gate between them: an **encryption rollout** (schema-add + dual-write code) and an **encryption cutover** (backfill + rename + drop). The `stash-encryption` skill is the canonical reference for the lifecycle; this section walks the Supabase-specific shape. -### Setup +> **Known gap (EQL v3 backfill).** The `stash encrypt backfill` / `stash encrypt cutover` tooling currently targets EQL v2 (`eql_v2_encrypted`) columns — EQL v3 domain columns are not yet supported end-to-end. Tracked in [cipherstash/stack#648](https://github.com/cipherstash/stack/issues/648). The lifecycle below (rollout → deploy gate → cutover) is the correct shape either way; until #648 lands, run the CLI backfill/cutover commands against a v2 encrypted twin (see "Interim path until #648" just below), or backfill v3 columns with your own script. -```typescript -import { encryptedSupabaseV3 } from "@cipherstash/stack-supabase" +> **Using CipherStash Proxy?** If you query encrypted data through [CipherStash Proxy](https://github.com/cipherstash/proxy) instead of the SDK, also run `stash db push` after schema-add and again before cutover to register the encrypted column shape with EQL. -// Introspects the database via options.databaseUrl or DATABASE_URL -const es = await encryptedSupabaseV3(supabaseUrl, supabaseKey) -// or wrap an existing client: await encryptedSupabaseV3(supabaseClient, options) +> **Runner note.** `stash init` adds `stash` to the project as a dev dependency, so `stash ` runs through whichever package manager the project uses (Bun, pnpm, Yarn, or npm) — examples below show this bare form. Before init has run, prefix with your package manager's one-shot runner: `bunx`, `pnpm dlx`, `yarn dlx`, or `npx`. The CLI's behaviour is identical across all of them. -await es.from("users").insert({ email: "a@b.com", amount: 30 }) -await es.from("users").select("id, email, amount").eq("email", "a@b.com") -await es.from("users").select("id, amount").gte("amount", 10).lte("amount", 100) -``` +> **Where am I?** Run `stash status` first (substitute the runner per the note above). It shows you which tables/columns are mid-rollout, which are post-deploy, and what the next move is. Re-run after every transition. -`from(tableName)` takes only the table name — no schema argument. Column -capabilities come from the introspected domains. Introspection needs a direct -Postgres connection (`options.databaseUrl`, defaulting to `DATABASE_URL`), so -the factory cannot run in a Worker or the browser. +### Interim path until #648: the v2 encrypted twin -### Optional declared schemas (compile-time types) +**This is the interim EQL v2 path, distinct from the v3 surface the rest of +this skill documents.** `stash encrypt backfill` / `stash encrypt cutover` +only operate on `eql_v2_encrypted` columns today, so to get the CLI-managed +backfill/cutover, make the twin a **v2** column and dual-write it through the +legacy `encryptedSupabase` (v2) wrapper. The lifecycle below (rollout → deploy +gate → cutover) is unchanged — only the twin's column type and the dual-write +client differ: -Declaring tables is optional. Passing `schemas` — a record whose keys must -equal each table's name — adds compile-time types and verifies the declared -tables against the database at construction: +```sql +-- schema-add: v2 twin instead of the v3 domain (still nullable) +ALTER TABLE users + ADD COLUMN email_encrypted eql_v2_encrypted; +``` ```typescript -import { encryptedTable, types } from "@cipherstash/stack/eql/v3" -import { encryptedSupabaseV3 } from "@cipherstash/stack-supabase" - -const users = encryptedTable("users", { - email: types.TextSearch("email"), // public.eql_v3_text_search — eq + range + free-text - amount: types.IntegerOrd("amount"), // public.eql_v3_integer_ord — eq + range - joined: types.TimestampOrd("joined_at") // public.eql_v3_timestamp_ord — eq + range, decrypts to Date -}) +// Dual-write through the legacy v2 wrapper: hand-written client-side schema, +// two-argument from(table, schema). +import { Encryption } from '@cipherstash/stack' +import { encryptedColumn, encryptedTable } from '@cipherstash/stack/schema' +import { encryptedSupabase } from '@cipherstash/stack-supabase' -const es = await encryptedSupabaseV3(supabaseUrl, supabaseKey, { - schemas: { users }, +const users = encryptedTable('users', { + email_encrypted: encryptedColumn('email_encrypted').equality().freeTextSearch(), }) -const { data } = await es.from("users").select("id, email, joined").eq("email", "a@b.com") -``` - -A declared table gets a typed builder: rows infer each column's plaintext -type (`types.IntegerOrd` → `number`, `types.TimestampOrd` → `Date`), -storage-only columns are excluded from every filter method, `matches()` is -narrowed to match-indexed columns, and `order()` to plaintext columns. -Undeclared tables behave exactly as with no `schemas` at all. Every v3 column -is fully described by its `types.*` factory — there are no capability or -tuning chains on v3 columns. - -A JS property may map to a different DB column name -(`joined: types.TimestampOrd("joined_at")`) — filters, selects, and results -are translated automatically, and `date`/`timestamp` columns decrypt to real -`Date` objects. - -### Database schema (per-domain columns) - -Each column is declared with its native domain — the `types.*` member name -maps to the flat `public.eql_v3_` domain (strip the `eql_v3_` prefix and -PascalCase each `_`-separated segment: `types.TextEq` → `public.eql_v3_text_eq`, -`types.IntegerOrd` → `public.eql_v3_integer_ord`). The domains use SQL-standard -type names (`integer`, `smallint`, `real`, `double`, `boolean`, `timestamp`): +const encryptionClient = await Encryption({ schemas: [users] }) +const esV2 = encryptedSupabase({ supabaseClient, encryptionClient }) -```sql -CREATE TABLE users ( - id SERIAL PRIMARY KEY, - email public.eql_v3_text_search, - amount public.eql_v3_integer_ord, - joined_at public.eql_v3_timestamp_ord -); +// Dual-write: BOTH the plaintext column and the encrypted twin. +export async function insertUser(email: string) { + return esV2.from('users', users).insert({ + email, // plaintext — keep writing until drop + email_encrypted: email, // v2 twin — encrypted automatically + }) +} ``` -### Install EQL v3 on Supabase - -```bash -stash eql install --eql-version 3 --supabase -``` - -Since eql-3.0.0 there is **one** v3 SQL artifact for every target — there is -no separate Supabase variant. The bundle's only superuser-requiring -statements (the ORE operator class/family) skip themselves when the install -role lacks the privilege, and the bundle then disables the ORE-opclass-backed -domains it cannot support. `--supabase` changes one thing: it additionally -applies the role grants for `anon` / `authenticated` / `service_role` to the -two schemas the bundle creates — `eql_v3` (the operator-backing functions) -and `eql_v3_internal` (SEM internals). Without the grants, encrypted queries -fail loudly with a permission error (e.g. `permission denied for schema -eql_v3_internal`). - -No **Exposed schemas** change is needed for v3: the column domains and their -operators live in `public`, so bare `col = term` filters resolve under -Supabase's default PostgREST configuration. Do not expose `eql_v3_internal`. - -### v3-specific behaviour - -All envelopes (stored payloads and filter operands) are versioned `v: 3`. - -- **`select('*')` (and bare `select()`) works on v3** — it expands to the - introspected column list. (v2 has no column list to expand, so it still - requires explicit columns.) -- **Encrypted free-text search is `matches()`, not `contains()`/`like`/`ilike`.** - The v3 domains define no LIKE operator — encrypted free-text search is fuzzy - bloom-filter token matching (PostgREST `cs` / SQL `@>`): one-sided (a match - may be a false positive, a non-match never is) and order-/multiplicity- - insensitive, where `%` is tokenized like any other character. `matches(col, - needle)` is the operator; `contains()` on an encrypted column throws an error - pointing at `matches()`. `contains()` is reserved for native (exact) - jsonb/array containment on plaintext columns, which pass through unchanged. -- **`like`/`ilike` on an encrypted column are an approximate compatibility - shim** delegating to `matches()`: leading/trailing `%` are stripped and the - residual term is fuzzy-matched (same `cs` wire, plus a one-time warning). - Results are APPROXIMATE — case-insensitive, one-sided, and anchoring is not - honored. A pattern with an internal `%` or any `_` cannot be approximated and - throws; call `matches()` directly to make the fuzzy intent explicit. On - plaintext columns `like`/`ilike` stay real SQL LIKE. -- **`matches()` matches substrings.** The search term blooms to its own - trigrams, and a row matches when the stored value's bloom contains all of - them — so any substring of at least 3 characters (the tokenizer's - `token_length`) matches. Shorter terms bloom to nothing and would match every - row, so they are rejected with an error rather than answered. -- **INTERIM — filter operands are full storage envelopes.** EQL ships - term-only query domains (`eql_v3.query_`, which accept envelopes with - no ciphertext) and the encryption client can mint those narrowed terms, but - PostgREST has no syntax to cast a filter value — an uncast operand can only - reach the `jsonb` operator overload, which coerces it into the storage - domain, whose CHECK requires ciphertext. So the adapter still encrypts each - filter value with the full storage path. The call shape is unchanged. - - **Security caveat:** query terms are meant to be index-terms-only by - design, but a full-envelope operand carries a real decryptable ciphertext - `c` plus **all** of the column's index terms, and PostgREST filters travel - in GET query strings — so these envelopes can land in URL logs, - intermediate proxies, and Supabase request logs. The remaining gap is - PostgREST operand casting; an adapter-side fix is tracked. -- **No `ORDER BY` on encrypted v3 columns** — including the range-capable - ones. PostgREST cannot emit `ORDER BY eql_v3.ord_term(col)`, and a bare - `ORDER BY` would silently sort the raw ciphertext envelope, so the builder - rejects `order()` on any encrypted column with a clear error. Range - *filtering* (`gte`/`lte`/…) works. Order by a plaintext column, or sort - application-side after decrypting. -- **Storage-only domains are not filterable** (e.g. `types.Boolean`, - `types.Text`): a filter (including `.match()`) on one is a type error on a - declared table, and always a clear runtime error. `.is(column, null)` - remains available. -- **Null filter values are rejected** with a pointer to `.is(column, null)` — - a null cannot be encrypted into an operand. - -## Migrating an Existing Column to Encrypted - -The hard case: a Supabase table that already exists with live data in a plaintext column you want to encrypt. You can't just change the column type — that would drop the data. - -CipherStash splits this into two named steps with a hard production-deploy gate between them: an **encryption rollout** (schema-add + dual-write code) and an **encryption cutover** (backfill + rename + drop). The `stash-encryption` skill is the canonical reference for the lifecycle; this section walks the Supabase-specific shape. - -> **Using CipherStash Proxy?** If you query encrypted data through [CipherStash Proxy](https://github.com/cipherstash/proxy) instead of the SDK, also run `stash db push` after schema-add and again before cutover to register the encrypted column shape with EQL. - -> **Runner note.** `stash init` adds `stash` to the project as a dev dependency, so `stash ` runs through whichever package manager the project uses (Bun, pnpm, Yarn, or npm) — examples below show this bare form. Before init has run, prefix with your package manager's one-shot runner: `bunx`, `pnpm dlx`, `yarn dlx`, or `npx`. The CLI's behaviour is identical across all of them. - -> **Where am I?** Run `stash status` first (substitute the runner per the note above). It shows you which tables/columns are mid-rollout, which are post-deploy, and what the next move is. Re-run after every transition. +With the v2 twin in place, the backfill and cutover commands in Step 2 work +as written. Alternatively, keep the twin v3 (as shown in the steps below) and +backfill it with your own script — once #648 lands, the v3 twin becomes the +CLI-supported path end to end. ### Starting state @@ -655,28 +698,22 @@ Edit the generated file to add an `email_encrypted` column **alongside** `email` ```sql -- supabase/migrations/_add_users_email_encrypted.sql ALTER TABLE users - ADD COLUMN email_encrypted eql_v2_encrypted; -- nullable + ADD COLUMN email_encrypted public.eql_v3_text_search; -- nullable ``` Apply with `supabase db reset` locally or `supabase migration up` against the remote project. -Update the encryption schema to declare the new encrypted column: +No client-side schema change is required — `encryptedSupabaseV3` introspects +the new column's domain at the next client startup. If you use declared +`schemas`, add the column so it is typed: ```typescript -// src/encryption/schema.ts -import { encryptedTable, encryptedColumn } from '@cipherstash/stack/schema' +// src/encryption/schema.ts (optional — compile-time types) +import { encryptedTable, types } from '@cipherstash/stack/eql/v3' export const users = encryptedTable('users', { - email_encrypted: encryptedColumn('email_encrypted') - .freeTextSearch() - .equality(), + email_encrypted: types.TextSearch('email_encrypted'), }) - -// src/encryption/index.ts -import { Encryption } from '@cipherstash/stack' -import { users } from './schema' - -export const encryptionClient = await Encryption({ schemas: [users] }) ``` > **Using CipherStash Proxy?** Register the new encryption config with EQL: @@ -691,31 +728,21 @@ export const encryptionClient = await Encryption({ schemas: [users] }) #### Dual-writing: write to both columns from app code -Find **every** code path that writes to `users.email` and update it to encrypt and also write to `email_encrypted`. The cleanest pattern is to keep the raw `supabase` client for the plaintext write and use the `encryptedSupabase` wrapper for the encrypted write — wrapped in a single function so callers can't forget one half: +Find **every** code path that writes to `users.email` and update it to also write the encrypted twin. With the v3 wrapper this is a single insert: `email` is a plaintext column and passes through unchanged, while `email_encrypted` is a v3 domain column the wrapper encrypts automatically. Wrap it in one function so callers can't forget one half: ```typescript // src/db/users.ts -import { supabase, encrypted } from './clients' -import { users } from '../encryption/schema' +import { es } from './clients' // encryptedSupabaseV3 instance export async function insertUser(email: string) { - // The encryptedSupabase wrapper handles the encryption call for you; - // the plaintext write is a separate `supabase` call so the rollout - // does not change read behaviour for `email` yet. - const ciphertext = await encrypted.encryptValue(email, { - table: users, - column: 'email_encrypted', - }) - if (ciphertext.failure) throw new Error(ciphertext.failure.message) - - return supabase.from('users').insert({ - email, // plaintext — keep writing - email_encrypted: ciphertext.data, // encrypted twin — new + return es.from('users').insert({ + email, // plaintext — keep writing + email_encrypted: email, // encrypted twin — new, encrypted automatically }) } ``` -Same shape for UPDATE: every site that updates `email` must also re-encrypt and update `email_encrypted` in the same statement. +Same shape for UPDATE: every site that updates `email` must also update `email_encrypted` in the same statement. **The dual-write rule.** Every persistence path that mutates this row writes both columns, in the same transaction, on every code branch. Insert sites, update sites, upserts, ON CONFLICT clauses, seeders, fixtures, edge functions, RPC functions, admin actions, background jobs, third-party webhooks — all of them. A single missed branch means rows inserted in production after deploy land in plaintext only, and backfill won't catch them. Grep for every site that touches `users.email` before declaring this step done. @@ -746,22 +773,24 @@ stash encrypt backfill --table users --column email # (CI: pass --confirm-dual-writes-deployed instead.) ``` -Resumable, idempotent, chunked. The CLI walks the table in keyset-pagination order, encrypts each chunk via the encryption client, and writes the ciphertext into `email_encrypted` inside transactions that also checkpoint to `cs_migrations`. SIGINT-safe. +Resumable, idempotent, chunked. The CLI walks the table in keyset-pagination order, encrypts each chunk via the encryption client, and writes the ciphertext into `email_encrypted` inside transactions that also checkpoint to `cs_migrations`. SIGINT-safe. (Remember the known-gap callout above: today this command targets EQL v2 columns — see #648.) If something goes wrong (e.g. you discover the dual-write code wasn't actually live when backfill ran), re-run with `--force` to re-encrypt every row regardless of current state. #### Cutover: rename swap and activate -First, update the encryption schema to the post-cutover shape — the encrypted column will live under the original column name: +First, if you use declared `schemas`, update them to the post-cutover shape — the encrypted column will live under the original column name: ```typescript // src/encryption/schema.ts (post-cutover) export const users = encryptedTable('users', { - email: encryptedColumn('email').freeTextSearch().equality(), + email: types.TextSearch('email'), }) ``` -> **Known gap (SDK-only users):** `stash encrypt cutover` currently requires a pending EQL configuration, which is set by `stash db push`. If you're using the SDK without Proxy, you'll hit a "No pending EQL configuration" error from cutover. **Workaround:** run `stash db push` once before `stash encrypt cutover`. This will be decoupled in a future release — see [issue #447](https://github.com/cipherstash/stack/issues/447). +(Without declared schemas, introspection picks up the renamed column at the next client startup.) + +> **Known gap (SDK-only users):** `stash encrypt cutover` currently requires a pending EQL configuration, which is set by `stash db push`. If you're using the SDK without Proxy, you'll hit a "No pending EQL configuration" error from cutover. **Workaround:** run `stash db push` once before `stash encrypt cutover`. This will be decoupled in a future release (tracked separately). > > **Using CipherStash Proxy?** Re-push the encryption config so EQL has a pending row that points at `email` (no `_encrypted` suffix): > @@ -779,29 +808,29 @@ stash encrypt cutover --table users --column email Inside one transaction it: (1) renames `email` → `email_plaintext` and `email_encrypted` → `email`, (2) promotes the pending EQL config to `active` (and the prior active to `inactive`), (3) records a `cut_over` event in `cs_migrations`. -App code that does `select('email')` now returns ciphertext that must be decrypted via the `encryptedSupabase` wrapper. **This is the moment that breaks read paths if they aren't going through the wrapper.** +App code that does `select('email')` now returns ciphertext that must be decrypted via the `encryptedSupabaseV3` wrapper. **This is the moment that breaks read paths if they aren't going through the wrapper.** -Update read paths to use `encryptedSupabase`: +Update read paths to use the wrapper: ```typescript // Before const { data } = await supabase.from('users').select('email').eq('id', id).single() -// After — encryptedSupabase decrypts transparently -const { data } = await encrypted.from('users').select('email').eq('id', id).single() +// After — the wrapper decrypts transparently +const { data } = await es.from('users').select('email').eq('id', id).single() ``` -For queries that filter on `email`, the `encryptedSupabase` wrapper handles the encrypted operators internally — the call site is the same shape as before (`.eq()`, `.like()`, `.ilike()`, `.gte()`, etc.), but the values are encrypted before reaching the database. See `## Query Filters` above. +For queries that filter on `email`, the wrapper handles the encrypted operators internally — the call site is the same shape as before (`.eq()`, `.matches()`, `.gte()`, etc.), but the values are encrypted before reaching the database. See `## Query Filters` above. #### Drop: remove the plaintext column -Once read paths are routing through `encryptedSupabase` and you're confident reads are decrypting correctly: +Once read paths are routing through the wrapper and you're confident reads are decrypting correctly: ```bash stash encrypt drop --table users --column email ``` -The CLI emits a Supabase migration file with `ALTER TABLE users DROP COLUMN email_plaintext;`. Review and apply with `supabase migration up` (or `supabase db reset` locally). Then remove the dual-write code from app paths — `email_plaintext` is gone; only `email` (encrypted) is written now via `encryptedSupabase`. +The CLI emits a Supabase migration file with `ALTER TABLE users DROP COLUMN email_plaintext;`. Review and apply with `supabase migration up` (or `supabase db reset` locally). Then remove the dual-write code from app paths — `email_plaintext` is gone; only `email` (encrypted) is written now, through the wrapper. ### Inspecting progress at any time @@ -812,3 +841,19 @@ stash encrypt plan # diffs your migrations.json intent vs observed state ``` All three are read-only. + +## Legacy: EQL v2 + +Earlier versions of this integration stored ciphertext in `jsonb` / +composite `eql_v2_encrypted` columns (enabled via `CREATE EXTENSION eql_v2` +or the v2 EQL bundle) and queried them through the `encryptedSupabase({ +supabaseClient, encryptionClient })` factory, which takes a hand-written +client-side schema and a two-argument `from(tableName, schema)`. That surface +still ships in `@cipherstash/stack-supabase` and is unchanged — keep using it +for existing v2 deployments — but it is not the recommended path for new +projects: use `encryptedSupabaseV3`. One current exception: the v2 wrapper is +also the interim dual-write path for CLI-managed `stash encrypt backfill` / +`stash encrypt cutover` until [cipherstash/stack#648](https://github.com/cipherstash/stack/issues/648) +lands — see "Interim path until #648: the v2 encrypted twin" in the migration +section above for a minimal recipe. For the v2 wrapper's full API and +semantics, see the docs at https://cipherstash.com/docs. diff --git a/turbo.json b/turbo.json index 64780c60..4ef8a864 100644 --- a/turbo.json +++ b/turbo.json @@ -3,7 +3,8 @@ "tasks": { "build": { "dependsOn": ["^build"], - "inputs": ["$TURBO_DEFAULT$", ".env*"] + "inputs": ["$TURBO_DEFAULT$", ".env*"], + "env": ["STASH_POSTHOG_KEY"] }, "release": { "dependsOn": ["^build"],