diff --git a/.changeset/drizzle-legacy-readme-pointer.md b/.changeset/drizzle-legacy-readme-pointer.md new file mode 100644 index 00000000..50cd60eb --- /dev/null +++ b/.changeset/drizzle-legacy-readme-pointer.md @@ -0,0 +1,7 @@ +--- +'@cipherstash/drizzle': patch +--- + +Docs: the README's "for new projects" pointer named the removed +`@cipherstash/stack/drizzle` subpath; it now points at the separate +`@cipherstash/stack-drizzle` package (EQL v3 on its `/v3` subpath). diff --git a/.changeset/eql-v3-sole-docs.md b/.changeset/eql-v3-sole-docs.md new file mode 100644 index 00000000..1e1a7202 --- /dev/null +++ b/.changeset/eql-v3-sole-docs.md @@ -0,0 +1,16 @@ +--- +'stash': patch +'@cipherstash/stack': patch +--- + +Docs: EQL v3 is now the sole documented approach. The `stash-encryption`, +`stash-drizzle`, and `stash-supabase` skills and the `@cipherstash/stack` +README teach only the v3 typed surface (`EncryptionV3`, `types.*` concrete +domains, `@cipherstash/stack-drizzle/v3`, `encryptedSupabaseV3`); EQL v2 +shrinks to one short Legacy section per document. Two explicit exceptions are +called out: DynamoDB still requires the v2 schema surface (#657), and the +encrypt rollout tooling (`stash encrypt backfill`/`cutover`, +`@cipherstash/migrate`) currently targets v2 columns (#648) — its guidance is +kept under a version callout. Also corrects the legacy `@cipherstash/drizzle` +README's pointer to the removed `@cipherstash/stack/drizzle` subpath (now the +separate `@cipherstash/stack-drizzle` package). diff --git a/.changeset/skills-identity-docs-refresh.md b/.changeset/skills-identity-docs-refresh.md new file mode 100644 index 00000000..7925bfe7 --- /dev/null +++ b/.changeset/skills-identity-docs-refresh.md @@ -0,0 +1,13 @@ +--- +'stash': patch +'@cipherstash/stack': patch +--- + +Docs: stop teaching the deprecated `LockContext.identify()` as the primary +identity-aware-encryption path (#591). The `stash-encryption` and `stash-supabase` +skills and the `@cipherstash/stack` README now lead with the current pattern — +authenticate the client with `OidcFederationStrategy`, then bind the claim per +operation with `.withLockContext({ identityClaim })` — and demote +`LockContext.identify()` to a clearly-marked deprecated note (per-operation CTS +tokens were removed in protect-ffi 0.25). Skills ship in the `stash` tarball, so +this keeps the bundled guidance correct for the 1.0 surface. diff --git a/AGENTS.md b/AGENTS.md index f01379bf..c361e441 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -147,8 +147,8 @@ Three rules to remember when editing CI or pnpm config: ## Key Concepts and APIs -- **Initialization**: `Encryption({ schemas })` returns an initialized `EncryptionClient`. Provide at least one `encryptedTable`. -- **Schema**: Define tables/columns with `encryptedTable` and `encryptedColumn` from `@cipherstash/stack/schema`. Add `.freeTextSearch().equality().orderAndRange()` to enable searchable encryption on PostgreSQL. Use `.searchableJson()` for encrypted JSONB queries. Use `encryptedField` for nested object encryption (DynamoDB). +- **Initialization**: `EncryptionV3({ schemas })` returns the typed EQL v3 client. (`Encryption({ schemas })` is the legacy v2 client.) Provide at least one `encryptedTable`. +- **Schema (EQL v3, the documented approach)**: Define tables/columns with `encryptedTable` and the `types.*` concrete-domain factories from `@cipherstash/stack/eql/v3` (`types.TextSearch`, `types.IntegerOrd`, `types.Json`, …) — each domain's query capabilities are fixed by its type; there are no chainable capability tuners. Build the client with `EncryptionV3` from `@cipherstash/stack/v3`. (Legacy EQL v2 — `Encryption` + `encryptedColumn(...).equality().freeTextSearch().orderAndRange().searchableJson()` from `@cipherstash/stack/schema` — remains for existing deployments and is what DynamoDB (`encryptedField`) still requires; see #657.) - **Operations** (all return Result-like objects and support chaining `.withLockContext(lockContext)` and `.audit()` when applicable): - `encrypt(plaintext, { table, column })` - `decrypt(encryptedPayload)` diff --git a/packages/drizzle/README.md b/packages/drizzle/README.md index 3bc1d527..7a27ac83 100644 --- a/packages/drizzle/README.md +++ b/packages/drizzle/README.md @@ -5,7 +5,7 @@ Seamlessly integrate CipherStash encryption with Drizzle ORM and PostgreSQL to encrypt your data while maintaining full query capabilities—equality, range queries, text search, and sorting—all with complete TypeScript type safety. > [!TIP] -> For new projects we recommend [`@cipherstash/stack`](https://www.npmjs.com/package/@cipherstash/stack), which provides this integration as `encryptedType`, `extractEncryptionSchema`, and `createEncryptionOperators` from `@cipherstash/stack/drizzle`. See the [Drizzle docs](https://cipherstash.com/docs/stack/cipherstash/encryption/drizzle). This package documents the legacy `@cipherstash/protect`-based API. +> For new projects we recommend [`@cipherstash/stack`](https://www.npmjs.com/package/@cipherstash/stack), which provides this integration via the separate [`@cipherstash/stack-drizzle`](https://www.npmjs.com/package/@cipherstash/stack-drizzle) package (EQL v3 on its `/v3` subpath). See the [Drizzle docs](https://cipherstash.com/docs). This package documents the legacy `@cipherstash/protect`-based API. ## Features diff --git a/packages/stack/README.md b/packages/stack/README.md index 7482e794..71f11165 100644 --- a/packages/stack/README.md +++ b/packages/stack/README.md @@ -6,7 +6,7 @@ The all-in-one TypeScript SDK for the CipherStash data security stack. [![License: MIT](https://img.shields.io/npm/l/@cipherstash/stack.svg?style=for-the-badge&labelColor=000000)](https://github.com/cipherstash/stack/blob/main/LICENSE.md) [![TypeScript](https://img.shields.io/badge/TypeScript-first-blue?style=for-the-badge&labelColor=000000)](https://www.typescriptlang.org/) --- +--- ## Table of Contents @@ -16,17 +16,18 @@ The all-in-one TypeScript SDK for the CipherStash data security stack. - [Schema Definition](#schema-definition) - [Encryption and Decryption](#encryption-and-decryption) - [Searchable Encryption](#searchable-encryption) -- [Identity-Aware Encryption](#identity-aware-encryption) +- [Authentication](#authentication) +- [Identity-Aware Encryption](#identity-aware-encryption-lock-contexts) - [CLI Reference](#cli-reference) - [Configuration](#configuration) - [Error Handling](#error-handling) - [API Reference](#api-reference) - [Subpath Exports](#subpath-exports) -- [Migration from @cipherstash/protect](#migration-from-cipherstashprotect) +- [Legacy: EQL v2](#legacy-eql-v2) - [Requirements](#requirements) - [License](#license) --- +--- ## Install @@ -53,17 +54,19 @@ The wizard will authenticate you, walk you through choosing a database connectio ### 2. Encrypt and decrypt +Define a table with concrete EQL v3 column types, build the typed client, and encrypt: + ```typescript -import { Encryption } from "@cipherstash/stack" -import { encryptedTable, encryptedColumn } from "@cipherstash/stack/schema" +import { EncryptionV3 } from "@cipherstash/stack/v3" +import { encryptedTable, types } from "@cipherstash/stack/eql/v3" -// Define a schema +// Define a schema — the column type fixes its query capabilities const users = encryptedTable("users", { - email: encryptedColumn("email").equality().freeTextSearch(), + email: types.TextSearch("email"), // equality + order/range + free-text search }) -// Create a client -const client = await Encryption({ schemas: [users] }) +// Create a typed client +const client = await EncryptionV3({ schemas: [users] }) // Encrypt a value const encrypted = await client.encrypt("hello@example.com", { @@ -71,114 +74,169 @@ const encrypted = await client.encrypt("hello@example.com", { table: users, }) +// Every operation returns `{ data } | { failure }`. Narrow on `.failure` and +// return/throw before reading `.data` — the failure branch has no `data`. if (encrypted.failure) { - console.error("Encryption failed:", encrypted.failure.message) -} else { - console.log("Encrypted payload:", encrypted.data) + throw new Error(`Encryption failed: ${encrypted.failure.message}`) } +console.log("Encrypted payload:", encrypted.data) // Decrypt the value const decrypted = await client.decrypt(encrypted.data) - if (decrypted.failure) { - console.error("Decryption failed:", decrypted.failure.message) -} else { - console.log("Plaintext:", decrypted.data) // "hello@example.com" + throw new Error(`Decryption failed: ${decrypted.failure.message}`) } +console.log("Plaintext:", decrypted.data) // "hello@example.com" ``` +The client is typed from your schemas: passing the wrong plaintext type for a column (`client.encrypt(42, { column: users.email, ... })`) is a compile error. + ## Features - **Field-level encryption** - Every value encrypted with its own unique key via [ZeroKMS](https://cipherstash.com/products/zerokms), backed by AWS KMS. -- **Searchable encryption** - Exact match, free-text search, order/range queries, and encrypted JSONB queries in PostgreSQL. +- **Searchable encryption** - Exact match, free-text search, order/range queries, and encrypted JSON queries in PostgreSQL, driven by concrete EQL v3 column types. +- **Type-safe by construction** - Each encrypted column is a concrete Postgres domain; its query capabilities are fixed by the type you pick and enforced at compile time by the typed client. - **Bulk operations** - Encrypt or decrypt thousands of values in a single ZeroKMS call (`bulkEncrypt`, `bulkDecrypt`, `bulkEncryptModels`, `bulkDecryptModels`). -- **Identity-aware encryption** - Tie encryption to a user's JWT via `LockContext`, so only that user can decrypt. +- **Identity-aware encryption** - Tie encryption to a user's JWT via `OidcFederationStrategy` and `.withLockContext()`, so only that user can decrypt. - **CLI (`stash`)** - Initialize projects and set up encryption from the terminal. - **TypeScript-first** - Strongly typed schemas, results, and model operations with full generics support. ## Schema Definition -Define which tables and columns to encrypt using `encryptedTable` and `encryptedColumn` from `@cipherstash/stack/schema`. +Define which tables and columns to encrypt using `encryptedTable` and the `types` namespace from `@cipherstash/stack/eql/v3`. Each factory in `types` maps 1:1 to a **concrete Postgres domain** named `public.eql_v3_` — the naming rule is: strip the `eql_v3_` prefix and PascalCase each underscore-separated segment. So `types.TextSearch` builds a `public.eql_v3_text_search` column, `types.IntegerOrd` builds `public.eql_v3_integer_ord`. + +There are **no chainable capability methods** — the concrete type fully describes what a column can do. ```typescript -import { encryptedTable, encryptedColumn } from "@cipherstash/stack/schema" +import { encryptedTable, types } from "@cipherstash/stack/eql/v3" const users = encryptedTable("users", { - email: encryptedColumn("email") - .equality() // exact-match queries - .freeTextSearch() // full-text search queries - .orderAndRange(), // sorting and range queries -}) - -const documents = encryptedTable("documents", { - metadata: encryptedColumn("metadata") - .searchableJson(), // encrypted JSONB queries (JSONPath + containment) + email: types.TextSearch("email"), // equality + order/range + free-text search + age: types.IntegerOrd("age"), // equality + order/range + balance: types.Bigint("balance"), // storage only — encrypt/decrypt, no queries + metadata: types.Json("metadata"), // encrypted JSON: containment + JSONPath selectors }) ``` -### Index Types +The returned table is also a column accessor (`users.email`). The JS property name and the DB column name may differ: `createdOn: types.Timestamp("created_at")` reads and writes the `createdOn` property on models but targets the `created_at` column in the database. + +### Capability Suffixes + +The suffix on the type name encodes the query capability: + +| Suffix | Capabilities | Query types | +|---|---|---| +| _(none)_ | Storage only — encrypt/decrypt, no queries | — | +| `Eq` | Equality | `'equality'` | +| `Ord` | Equality + ordering/range (OPE-backed) | `'equality'`, `'orderAndRange'` | +| `OrdOre` | Equality + ordering/range (block-ORE-backed — the ORE operator class is superuser-only and unavailable on managed Postgres such as Supabase) | `'equality'`, `'orderAndRange'` | +| `Match` (text only) | Free-text search only | `'freeTextSearch'` | +| `Search` (text only, as `TextSearch`) | Equality + ordering/range + free-text | all three | +| `Json` | Encrypted JSON containment + JSONPath selector queries | `'searchableJson'` | + +Prefer the plain `Ord` domains unless you know your database supports the ORE operator class. + +### Domain Families and Plaintext Types -| Method | Purpose | Query Type | -|----|-----|------| -| `.equality()` | Exact match lookups | `'equality'` | -| `.freeTextSearch()` | Full-text / fuzzy search | `'freeTextSearch'` | -| `.orderAndRange()` | Sorting, comparison, range queries | `'orderAndRange'` | -| `.searchableJson()` | Encrypted JSONB path and containment queries | `'searchableJson'` | -| `.dataType(cast)` | Set the plaintext data type (`'string'`, `'number'`, `'boolean'`, `'date'`, `'bigint'`, `'json'`) | N/A | +| Family | Factories | Plaintext (TypeScript) type | +|---|---|---| +| `Integer`, `Smallint`, `Numeric`, `Real`, `Double` | base, `Eq`, `Ord`, `OrdOre` | `number` | +| `Bigint` | base, `Eq`, `Ord`, `OrdOre` | `bigint` (native JS bigint, full i64 range) | +| `Date` | base, `Eq`, `Ord`, `OrdOre` | `Date` (calendar date; time-of-day truncated) | +| `Timestamp` | base, `Eq`, `Ord`, `OrdOre` | `Date` (time-of-day preserved) | +| `Text` | base, `Eq`, `Match`, `Ord`, `OrdOre`, `Search` | `string` | +| `Boolean` | base only | `boolean` | +| `Json` | `Json` only | a JSON *document* (object, array, or null — not a top-level scalar) | + +### Database Setup + +Install the EQL v3 SQL into your database with the stash CLI: + +```bash +npx stash eql install --eql-version 3 +# On Supabase, add --supabase to grant the anon/authenticated/service_role +# roles access to the eql_v3 schemas — without it, encrypted queries fail with +# "permission denied for schema eql_v3_internal": +npx stash eql install --eql-version 3 --supabase +``` -Methods are chainable - call as many as you need on a single column. +In migrations, declare each encrypted column as its domain type: + +```sql +CREATE TABLE users ( + id BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY, + email public.eql_v3_text_search, + age public.eql_v3_integer_ord, + balance public.eql_v3_bigint, + metadata public.eql_v3_json +); +``` ## Encryption and Decryption ### Single Values ```typescript -// Encrypt +// Encrypt — plaintext is pinned to the column's domain type const encrypted = await client.encrypt("secret@example.com", { column: users.email, table: users, }) -// Decrypt +// Decrypt (narrow on `.failure` before reading `.data`) +if (encrypted.failure) throw new Error(encrypted.failure.message) const decrypted = await client.decrypt(encrypted.data) ``` ### Model Operations -Encrypt or decrypt an entire object. Only fields matching your schema are encrypted; other fields pass through unchanged. +Encrypt or decrypt an entire object. Only fields matching your schema are encrypted; other fields pass through unchanged. Schema fields are validated against their inferred plaintext type at compile time. -The return type is **schema-aware**: fields matching the table schema are typed as `Encrypted`, while other fields retain their original types. For best results, let TypeScript infer the type parameters from the arguments: +`decryptModel` takes the **table as a second argument** and returns the precise plaintext model: `Date` columns are reconstructed to real `Date` instances, and `bigint` columns round-trip as native `bigint`. ```typescript -type User = { id: string; email: string; createdAt: Date } - const user = { - id: "user_123", - email: "alice@example.com", // defined in schema -> encrypted - createdAt: new Date(), // not in schema -> unchanged + id: "user_123", // not in schema -> passes through + email: "alice@example.com", // TextSearch -> encrypted as string + age: 30, // IntegerOrd -> encrypted as number + balance: 100_000n, // Bigint -> encrypted as bigint } -// Let TypeScript infer the return type from the schema const encryptedResult = await client.encryptModel(user, users) -// encryptedResult.data.email -> Encrypted -// encryptedResult.data.id -> string -// encryptedResult.data.createdAt -> Date +// encryptedResult.data.email -> Encrypted +// encryptedResult.data.id -> string -// Decrypt a model -const decryptedResult = await client.decryptModel(encryptedResult.data) +if (encryptedResult.failure) throw new Error(encryptedResult.failure.message) +const decryptedResult = await client.decryptModel(encryptedResult.data, users) +// decryptedResult.data.email -> string +// decryptedResult.data.balance -> bigint ``` ### Bulk Operations All bulk methods make a single call to ZeroKMS regardless of the number of records, while still using a unique key per value. +#### Bulk Encrypt / Decrypt Models + +```typescript +const userModels = [ + { id: "1", email: "alice@example.com", age: 30, balance: 100_000n }, + { id: "2", email: "bob@example.com", age: 41, balance: 250_000n }, +] + +const encrypted = await client.bulkEncryptModels(userModels, users) +if (encrypted.failure) throw new Error(encrypted.failure.message) +const decrypted = await client.bulkDecryptModels(encrypted.data, users) +``` + #### Bulk Encrypt / Decrypt (raw values) +`bulkEncrypt` / `bulkDecrypt` are untyped passthroughs for raw value arrays: + ```typescript const plaintexts = [ { id: "u1", plaintext: "alice@example.com" }, { id: "u2", plaintext: "bob@example.com" }, - { id: "u3", plaintext: "charlie@example.com" }, ] const encrypted = await client.bulkEncrypt(plaintexts, { @@ -188,7 +246,9 @@ const encrypted = await client.bulkEncrypt(plaintexts, { // encrypted.data = [{ id: "u1", data: EncryptedPayload }, ...] +if (encrypted.failure) throw new Error(encrypted.failure.message) const decrypted = await client.bulkDecrypt(encrypted.data) +if (decrypted.failure) throw new Error(decrypted.failure.message) // Each item has either { data: "plaintext" } or { error: "message" } for (const item of decrypted.data) { @@ -200,175 +260,91 @@ for (const item of decrypted.data) { } ``` -#### Bulk Encrypt / Decrypt Models - -```typescript -const userModels = [ - { id: "1", email: "alice@example.com" }, - { id: "2", email: "bob@example.com" }, -] - -const encrypted = await client.bulkEncryptModels(userModels, users) -const decrypted = await client.bulkDecryptModels(encrypted.data) -``` - ## Searchable Encryption -Encrypt a query term so you can search encrypted data in PostgreSQL. +Encrypt a query term so you can search encrypted data in PostgreSQL. The typed client only accepts queryable columns, and `queryType` is constrained to the column's capabilities — equality and range queries run through the domain's own SQL operators. ```typescript -// Equality query +// Equality const eqQuery = await client.encryptQuery("alice@example.com", { column: users.email, table: users, queryType: "equality", }) -// Free-text search -const matchQuery = await client.encryptQuery("alice", { +// Free-text search — queryType is REQUIRED for a match term (see gotcha below) +const matchQuery = await client.encryptQuery("ali", { column: users.email, table: users, queryType: "freeTextSearch", }) // Order and range -const rangeQuery = await client.encryptQuery("alice@example.com", { - column: users.email, +const rangeQuery = await client.encryptQuery(30, { + column: users.age, table: users, queryType: "orderAndRange", }) ``` -### Searchable JSON - -For columns using `.searchableJson()`, the query type is auto-inferred from the plaintext: +> **Gotcha — `TextSearch` defaults to equality.** A `TextSearch` column carries all three indexes, and `encryptQuery` with **no explicit `queryType` builds an equality term, not a free-text match**. A substring like `"joh"` then matches nothing. Always pass `queryType: 'freeTextSearch'` for substring/token search. -```typescript -// String -> JSONPath selector query -const pathQuery = await client.encryptQuery("$.user.email", { - column: documents.metadata, - table: documents, -}) +Free-text search is fuzzy bloom-filter token matching, surfaced as `matches` in the Drizzle and Supabase adapters — it is order- and multiplicity-insensitive and one-sided (a match may be a false positive, a non-match never is). It is not SQL `LIKE`; don't pass `%` wildcards. -// Object/Array -> containment query -const containsQuery = await client.encryptQuery({ role: "admin" }, { - column: documents.metadata, - table: documents, -}) -``` +### Encrypted JSON -### Batch Query Encryption +A `types.Json` column encrypts a whole JSON document (an object, array, or null — not a top-level scalar) to a `public.eql_v3_json` value. Two query patterns are supported: -Encrypt multiple query terms in one call: +**Exact containment** (jsonb `@>` semantics, no false positives). Pass a sub-object or sub-array needle; array containment is a subset test regardless of element position — `{ roles: ["admin"] }` matches any document whose `roles` array includes `"admin"`: ```typescript -const terms = [ - { value: "alice@example.com", column: users.email, table: users, queryType: "equality" as const }, - { value: "bob", column: users.email, table: users, queryType: "freeTextSearch" as const }, -] - -const results = await client.encryptQuery(terms) -``` +const events = encryptedTable("events", { metadata: types.Json("metadata") }) -### Query Result Formatting (`returnType`) - -By default `encryptQuery` returns an `Encrypted` object (the raw EQL JSON payload). Use `returnType` to change the output format: - -| `returnType` | Output | Use case | -|---|---|---| -| `'eql'` (default) | `Encrypted` object | Parameterized queries, ORMs accepting JSON | -| `'composite-literal'` | `string` | Supabase `.eq()`, string-based APIs | -| `'escaped-composite-literal'` | `string` | Embedding inside another string or JSON value | - -```typescript -// Get a composite literal string for use with Supabase -const term = await client.encryptQuery("alice@example.com", { - column: users.email, - table: users, - queryType: "equality", - returnType: "composite-literal", -}) - -// term.data is a string — use directly with .eq() -await supabase.from("users").select().eq("email", term.data) +const containsQuery = await client.encryptQuery( + { roles: ["admin"] }, + { column: events.metadata, table: events }, // queryType inferred: 'searchableJson' +) ``` -Each term in a batch can have its own `returnType`. +**JSONPath selectors** — equality and ordering at a path (`$.a`, `$.a.b` dot-notation object paths): -### Ordering Encrypted Data - -**`ORDER BY` on encrypted columns requires operator family support in the database.** +- **Drizzle**: `ops.selector(events.metadata, "$.age")` returns comparison methods bound to the path — `eq`, `ne`, `gt`, `gte`, `lt`, `lte` (e.g. `await ops.selector(events.metadata, "$.age").gt(21)`). Its unique power over containment is *ordering* at a path; equality at a path is equivalently `contains(col, { age: 21 })`. +- **Supabase**: `selectorEq(col, path, value)` and `selectorNe(col, path, value)`. -On databases without operator families (e.g. Supabase, or when EQL is installed with `--exclude-operator-family`), sorting on encrypted columns is not currently supported — regardless of the client or ORM used. Sort application-side after decrypting the results as a workaround. +Two semantics to know: -Operator family support for Supabase is being developed in collaboration with the Supabase and CipherStash teams and will be available in a future release. +- **`ne` includes absent paths.** A "not equal at path" query also matches rows where the path does not exist at all. +- **Array-leaf caveat:** a scalar needle does not match an array at the path. `selectorEq("payload", "$.roles", "admin")` does *not* match `{ roles: ["admin", "analyst"] }` — use containment for membership tests. -### PostgreSQL / Drizzle Integration Pattern +`types.Json` carries no equality or ordering on the document itself, so applying `eq` / `gt` / `asc` directly to a `Json` column throws. -Encrypted data is stored as an [EQL](https://github.com/cipherstash/encrypt-query-language) JSON payload. Install the EQL extension in PostgreSQL to enable searchable queries, then store encrypted data in `eql_v2_encrypted` columns. +### Batch Query Encryption -The `@cipherstash/stack-drizzle` module provides `encryptedType` for defining encrypted columns and `createEncryptionOperators` for querying them: +Encrypt multiple query terms in one call: ```typescript -import { pgTable, integer, timestamp } from "drizzle-orm/pg-core" -import { encryptedType, extractEncryptionSchema, createEncryptionOperators } from "@cipherstash/stack-drizzle" -import { Encryption } from "@cipherstash/stack" - -// Define schema with encrypted columns -const usersTable = pgTable("users", { - id: integer("id").primaryKey().generatedAlwaysAsIdentity(), - email: encryptedType("email", { - equality: true, - freeTextSearch: true, - orderAndRange: true, - }), - profile: encryptedType<{ name: string; bio: string }>("profile", { - dataType: "json", - searchableJson: true, - }), -}) - -// Initialize -const usersSchema = extractEncryptionSchema(usersTable) -const client = await Encryption({ schemas: [usersSchema] }) -const ops = createEncryptionOperators(client) - -// Query with auto-encrypting operators -const results = await db.select().from(usersTable) - .where(await ops.eq(usersTable.email, "alice@example.com")) +const terms = [ + { value: "alice@example.com", column: users.email, table: users, queryType: "equality" as const }, + { value: "bob", column: users.email, table: users, queryType: "freeTextSearch" as const }, +] -// JSONB queries on encrypted JSON columns -const jsonResults = await db.select().from(usersTable) - .where(await ops.jsonbPathExists(usersTable.profile, "$.bio")) +const results = await client.encryptQuery(terms) ``` -#### Drizzle `encryptedType` Config Options - -| Option | Type | Description | -|---|---|---| -| `dataType` | `"string"` \| `"number"` \| `"json"` | Plaintext data type (default: `"string"`) | -| `equality` | `boolean` \| `TokenFilter[]` | Enable equality index | -| `freeTextSearch` | `boolean` \| `MatchIndexOpts` | Enable free-text search index | -| `orderAndRange` | `boolean` | Enable ORE index for sorting/range queries | -| `searchableJson` | `boolean` | Enable JSONB path queries (requires `dataType: "json"`) | - -#### Drizzle JSONB Operators +### Ordering Encrypted Data -For columns with `searchableJson: true`, three JSONB operators are available: +`ORDER BY` works on encrypted ordering columns via the domain's order term: -| Operator | Description | -|---|---| -| `jsonbPathExists(col, selector)` | Check if a JSONB path exists (boolean, use in `WHERE`) | -| `jsonbPathQueryFirst(col, selector)` | Extract first value at a JSONB path | -| `jsonbGet(col, selector)` | Get value using the JSONB `->` operator | +- **Drizzle**: `ops.asc(col)` / `ops.desc(col)` emit `ORDER BY eql_v3.ord_term(col)` (or `eql_v3.ord_term_ore` for ORE domains). +- **Supabase**: `.order()` works on OPE-backed ordering columns — every plain `*Ord` domain plus `TextSearch`. -These operators encrypt the JSON path selector using the `steVecSelector` query type and cast it to `eql_v2_encrypted` for use with the EQL PostgreSQL functions. +The one limitation is the ORE-backed `*OrdOre` domains: their ordering term needs the superuser-only ORE operator class, which is unavailable on managed Postgres (e.g. Supabase) — the Supabase adapter rejects `order()` on those columns with a clear error. Prefer the plain `*Ord` (OPE) domains for anything you need to sort in a managed environment. -### EQL v3 Concrete-Type Columns (Drizzle) +### Drizzle Integration -The `@cipherstash/stack-drizzle/v3` module targets EQL v3, where each encrypted column is a **concrete Postgres domain type** (`eql_v3.text_eq`, `eql_v3.integer_ord`, …). Instead of toggling capability flags, you pick a concrete type from the `types` namespace and its query capabilities are fixed by that choice. It is the `/v3` subpath of the same `@cipherstash/stack-drizzle` package whose v2 surface is shown above. +The `@cipherstash/stack-drizzle/v3` subpath (of the separate `@cipherstash/stack-drizzle` package) provides Drizzle-native column factories, schema extraction, and auto-encrypting, capability-checked query operators. -Declare a Drizzle table using the `types` factories. The suffix encodes the capability: `*Eq` (equality), `*Ord` (order + range, which also covers equality), `*Match` / `TextSearch` (free-text search), and the bare name (e.g. `Text`, `Bigint`) is storage-only. +Declare a Drizzle table using the `types` factories — each factory emits its domain as the column's SQL type, so `drizzle-kit generate` produces `ADD COLUMN email public.eql_v3_text_search` etc.: ```ts import { pgTable, integer } from "drizzle-orm/pg-core" @@ -385,12 +361,12 @@ const users = pgTable("users", { id: integer("id").primaryKey().generatedAlwaysAsIdentity(), email: types.TextEq("email"), // equality: eq / ne / inArray age: types.IntegerOrd("age"), // order + range: gt/gte/lt/lte, between, asc/desc - bio: types.TextMatch("bio"), // free-text search: contains + bio: types.TextMatch("bio"), // free-text search: matches balance: types.Bigint("balance"), // storage only (no query capability) }) ``` -Derive the v3 schema from the table, build the typed client, and create the capability-checked operators: +Derive the v3 schema from the table, build the typed client, and create the operators: ```ts const usersSchema = extractEncryptionSchemaV3(users) @@ -419,9 +395,9 @@ const midBand = await db.select().from(users) const listed = await db.select().from(users) .where(await ops.inArray(users.email, ["alice@example.com", "bob@example.com"])) -// Free-text token containment — bio is TextMatch +// Free-text token match — bio is TextMatch const coffee = await db.select().from(users) - .where(await ops.contains(users.bio, "coffee")) + .where(await ops.matches(users.bio, "coffee")) ``` Rows are **pre-encrypted** with `client.bulkEncryptModels(...)` before they reach `db.insert(...).values(...)` — Drizzle never sees plaintext. `Bigint` columns take a native JS `bigint`: @@ -441,41 +417,99 @@ await db.insert(users).values(rows.data) Notes: -- **Free-text search uses `ops.contains`** (token containment on a `match` index), not SQL `like` / `ilike`. It matches whole indexed tokens, so `contains(users.bio, "coffee")` finds rows whose `bio` contains the token `coffee`. -- **The concrete type defines the legal operators.** `TextEq` supports `eq` / `ne` / `inArray` / `notInArray`; `*Ord` types add `gt` / `gte` / `lt` / `lte` / `between` / `notBetween` and `asc` / `desc`; `*Match` and `TextSearch` add `contains`; a bare `Text` / `Integer` / `Bigint` column is storage-only. Using an unsupported operator throws `EncryptionOperatorError`. +- **Free-text search is `ops.matches`** (fuzzy bloom token matching on `TextMatch` / `TextSearch` columns), not SQL `like` / `ilike` — those operators do not exist on the v3 surface. `ops.contains` is a *different* operator: exact encrypted-JSON containment on `types.Json` columns. +- **The concrete type defines the legal operators.** `TextEq` supports `eq` / `ne` / `inArray` / `notInArray`; `*Ord` types add `gt` / `gte` / `lt` / `lte` / `between` / `notBetween` and `asc` / `desc`; `TextMatch` and `TextSearch` add `matches`; `Json` supports `contains` and `selector(col, '$.path').{eq,ne,gt,gte,lt,lte}`; a bare `Text` / `Integer` / `Bigint` column is storage-only. Using an unsupported operator throws `EncryptionOperatorError`. - Combine conditions with `ops.and` / `ops.or`, and do NULL checks with `ops.isNull` / `ops.isNotNull` (the where-clause operators are `async` and must be `await`ed; `ops.asc` / `ops.desc` are synchronous). -## Identity-Aware Encryption +### Supabase Integration -Lock encryption to a specific user by requiring a valid JWT for decryption. +`encryptedSupabaseV3` from the separate `@cipherstash/stack-supabase` package wraps a Supabase client and **introspects the database at connect time** — it detects EQL v3 columns by their Postgres domain and builds the encryption client internally: ```typescript -import { LockContext } from "@cipherstash/stack/identity" +import { encryptedSupabaseV3 } from "@cipherstash/stack-supabase" -// 1. Create a lock context (defaults to the "sub" claim) -const lc = new LockContext() +const es = await encryptedSupabaseV3(supabaseUrl, supabaseKey) -// 2. Identify the user with their JWT -const identifyResult = await lc.identify(userJwt) +await es.from("users").insert({ email: "a@b.com", age: 30 }) +await es.from("users").select("id, email").eq("email", "a@b.com") +await es.from("users").select("id, age").gte("age", 18).order("age") +``` -if (identifyResult.failure) { - throw new Error(identifyResult.failure.message) -} +Encrypted free-text search is `matches()` (fuzzy bloom token search — `contains()` stays native, exact containment for plaintext columns), encrypted-JSON path queries are `selectorEq()` / `selectorNe()`, and `order()` works on OPE-backed ordering columns. Pass optional declared `schemas` for compile-time row types. See the `stash-supabase` skill or the [docs](https://cipherstash.com/docs) for the full guide. + +## Authentication + +The client authenticates to ZeroKMS through `config.authStrategy`. Leave it +unset for the default **auto** strategy: in local development, authenticate +once with `npx stash auth login` (preferred — no credentials in your +environment); in CI/production, set the `CS_*` environment variables. Two +explicit strategies cover the other cases: -const lockContext = identifyResult.data +- **`AccessKeyStrategy`** — service-to-service / CI. Authenticates a *service* + with a CipherStash access key. +- **`OidcFederationStrategy`** — authenticates the client **as the end user** + by federating a third-party OIDC JWT (Clerk, Supabase, Auth0, Okta, ...) + into a CipherStash service token: + +```typescript +import { OidcFederationStrategy } from "@cipherstash/stack" +import { EncryptionV3 } from "@cipherstash/stack/v3" + +// The callback is re-invoked on every (re-)federation and must return the +// CURRENT third-party OIDC JWT. +const strategy = OidcFederationStrategy.create( + process.env.CS_WORKSPACE_CRN!, + () => getUserJwt(), +) +if (strategy.failure) throw new Error(strategy.failure.error.message) + +const client = await EncryptionV3({ + schemas: [users], + config: { authStrategy: strategy.data }, +}) +``` + +Authentication stands on its own — an OIDC-authenticated client encrypts and +decrypts normally. Binding *data* to the authenticated user is a separate, +optional step: the lock context, below. + +## Identity-Aware Encryption (Lock Contexts) + +Bind a data key to a claim from the end user's JWT, so only that user can +decrypt. Chain `.withLockContext({ identityClaim })` on any operation: + +```typescript +// Requires a client authenticated with OidcFederationStrategy (above) — the +// claim's value resolves from the federated JWT. +const IDENTITY = { identityClaim: ["sub"] } -// 3. Encrypt with lock context const encrypted = await client .encrypt("sensitive data", { column: users.email, table: users }) - .withLockContext(lockContext) + .withLockContext(IDENTITY) +if (encrypted.failure) throw new Error(encrypted.failure.message) -// 4. Decrypt with the same lock context const decrypted = await client .decrypt(encrypted.data) - .withLockContext(lockContext) + .withLockContext(IDENTITY) ``` -Lock contexts work with all operations: `encrypt`, `decrypt`, `encryptModel`, `decryptModel`, `bulkEncryptModels`, `bulkDecryptModels`, `bulkEncrypt`, `bulkDecrypt`. +Lock contexts **require** an `OidcFederationStrategy`-authenticated client +(the auto and access-key strategies authenticate no end user, so there is no +JWT to resolve claims from); plain authentication never requires a lock +context. + +`identityClaim` is an array of JWT claim *names* (`["sub"]`), not values, and the +same claim must be supplied to encrypt and decrypt. Lock contexts work with all +operations: `encrypt`, `decrypt`, `encryptModel`, `decryptModel`, +`bulkEncryptModels`, `bulkDecryptModels`, `bulkEncrypt`, `bulkDecrypt`, +`encryptQuery`. `.withLockContext()` also accepts a `LockContext` instance. +On the typed client, `decryptModel` / `bulkDecryptModels` take the lock +context as an optional third argument instead of chaining. + +> **Deprecated: `LockContext.identify()`.** Per-operation CTS tokens were removed +> in `protect-ffi` 0.25; the token `identify()` fetches is no longer used by +> encryption. Authenticate with `OidcFederationStrategy` and pass the claim +> directly, as above. ## CLI Reference @@ -502,22 +536,24 @@ npx stash init --supabase The wizard will: 1. Authenticate with CipherStash (device code flow) -2. Bind your device to the default Keyset +2. Introspect your database and install the EQL v3 SQL 3. Choose your database connection method (Drizzle ORM, Supabase JS, Prisma, or Raw SQL) 4. Build an encryption schema interactively or use a placeholder, then generate the encryption client file 5. Install `stash` as a dev dependency for database tooling -After init, run `npx stash db setup` to configure your database. +`init` installs EQL for you — no separate `eql install` step is needed afterward. | Flag | Description | |------|-------------| -| `--supabase` | Use Supabase-specific setup flow | +| `--supabase` / `--drizzle` / `--prisma-next` | Target a specific integration's setup flow | +| `--proxy` / `--no-proxy` | Opt in/out of the CipherStash Proxy path | +| `--region ` | Workspace region (env `STASH_REGION`); **required for non-interactive init when not already logged in** | ## Configuration ### Local Development -No environment variables or credentials are needed for local development. Run `npx @cipherstash/stack auth login` to authenticate via the device code flow, and the SDK and CLI will use the token saved to `~/.cipherstash/auth.json`. +No environment variables or credentials are needed for local development. Run `npx stash auth login` to authenticate via the device code flow (or `npx stash init` for the agent-assisted end-to-end setup), and the SDK and CLI will use the token saved to `~/.cipherstash/auth.json`. ### Going to Production @@ -537,10 +573,10 @@ See the [Going to Production](https://cipherstash.com/docs/stack/deploy/going-to Pass config directly when initializing the client: ```typescript -import { Encryption } from "@cipherstash/stack" +import { EncryptionV3 } from "@cipherstash/stack/v3" import { users } from "./schema" -const client = await Encryption({ +const client = await EncryptionV3({ schemas: [users], config: { workspaceCrn: "crn:ap-southeast-2.aws:your-workspace-id", @@ -557,7 +593,7 @@ const client = await Encryption({ Isolate encryption keys per tenant using keysets: ```typescript -const client = await Encryption({ +const client = await EncryptionV3({ schemas: [users], config: { keyset: { id: "123e4567-e89b-12d3-a456-426614174000" }, @@ -565,7 +601,7 @@ const client = await Encryption({ }) // or by name -const client2 = await Encryption({ +const client2 = await EncryptionV3({ schemas: [users], config: { keyset: { name: "Company A" }, @@ -622,73 +658,120 @@ if (result.failure) { ## API Reference -### `Encryption(config)` - Initialize the client +### `EncryptionV3(config)` - Initialize the typed client ```typescript -function Encryption(config: EncryptionClientConfig): Promise +function EncryptionV3(config: { + schemas: AnyV3Table[] + config?: ClientConfig +}): Promise ``` -### `EncryptionClient` Methods +The wire format is pinned to EQL v3 — you don't set it yourself. `typedClient(client, ...schemas)` (same subpath) wraps an already-built `EncryptionClient` in the typed surface. + +### `TypedEncryptionClient` Methods + +Method signatures are derived from your schemas: plaintext arguments are pinned to each column's domain type, query methods only accept queryable columns, and `queryType` is constrained to the column's capabilities. | Method | Signature | Returns | |----|------|-----| | `encrypt` | `(plaintext, { column, table })` | `EncryptOperation` (thenable) | | `decrypt` | `(encryptedData)` | `DecryptOperation` (thenable) | | `encryptQuery` | `(plaintext, { column, table, queryType?, returnType? })` | `EncryptQueryOperation` (thenable) | + +`returnType` controls the encrypted query term's shape: `'eql'` (default, the EQL JSON payload for the ORM adapters), `'composite-literal'` (a Postgres composite string for `.eq()`/string-based APIs), or `'escaped-composite-literal'` (the same, escaped for embedding). Most users take the default; the adapters set it as needed. | `encryptQuery` | `(terms: ScalarQueryTerm[])` | `BatchEncryptQueryOperation` (thenable) | -| `encryptModel` | `(model, table)` | `EncryptModelOperation>` (thenable) | -| `decryptModel` | `(encryptedModel)` | `DecryptModelOperation` (thenable) | +| `encryptModel` | `(model, table)` | `EncryptModelOperation` (thenable) | +| `decryptModel` | `(encryptedModel, table, lockContext?)` | `Promise>` | +| `bulkEncryptModels` | `(models, table)` | `BulkEncryptModelsOperation` (thenable) | +| `bulkDecryptModels` | `(encryptedModels, table, lockContext?)` | `Promise>` | | `bulkEncrypt` | `(plaintexts, { column, table })` | `BulkEncryptOperation` (thenable) | | `bulkDecrypt` | `(encryptedPayloads)` | `BulkDecryptOperation` (thenable) | -| `bulkEncryptModels` | `(models, table)` | `BulkEncryptModelsOperation>` (thenable) | -| `bulkDecryptModels` | `(encryptedModels)` | `BulkDecryptModelsOperation` (thenable) | +| `getEncryptConfig` | `()` | The resolved encrypt config | + +The thenable operations support `.withLockContext(lockContext)` for identity-aware encryption. `decryptModel` / `bulkDecryptModels` return a plain `Promise` instead — pass the lock context as the optional third argument. `decrypt` of a single value cannot be strongly typed (a lone ciphertext carries no column identity), and `encryptQuery` rejects storage-only columns at compile time. + +### `LockContext` (legacy) -All operations are thenable (awaitable) and support `.withLockContext(lockContext)` for identity-aware encryption. +Identity-aware encryption is done with `OidcFederationStrategy` + +`.withLockContext({ identityClaim })` (see [Identity-Aware Encryption](#identity-aware-encryption-lock-contexts)). +`LockContext` / `identify()` remain for backwards compatibility only — the +per-operation CTS token `identify()` fetches was removed in `protect-ffi` 0.25 +and is no longer used by encryption. -### `LockContext` +### Schema Builders ```typescript -import { LockContext } from "@cipherstash/stack/identity" +import { encryptedTable, types } from "@cipherstash/stack/eql/v3" -const lc = new LockContext(options?) -const result = await lc.identify(jwtToken) +encryptedTable(tableName, columns) // columns: Record +types.TextSearch("email") // one factory per public.eql_v3_* domain ``` -### Schema Builders +Type inference helpers live on the same subpath: ```typescript -import { encryptedTable, encryptedColumn, csValue } from "@cipherstash/stack/schema" +import type { InferPlaintext, InferEncrypted } from "@cipherstash/stack/eql/v3" -encryptedTable(tableName, columns) -encryptedColumn(columnName) // returns EncryptedColumn -csValue(valueName) // returns ProtectValue (for nested values) +type UserPlaintext = InferPlaintext +// { email: string; age: number; balance: bigint; metadata: JsonDocument } + +type UserEncrypted = InferEncrypted +// { email: Encrypted; age: Encrypted; ... } ``` ## Subpath Exports | Import Path | Provides | |-------|-----| -| `@cipherstash/stack` | `Encryption` function (main entry point) | -| `@cipherstash/stack/schema` | `encryptedTable`, `encryptedColumn`, `csValue`, schema types | +| `@cipherstash/stack/v3` | `EncryptionV3` typed client factory, `typedClient`, plus re-exports of the EQL v3 authoring DSL | +| `@cipherstash/stack/eql/v3` | EQL v3 authoring DSL: `encryptedTable`, the `types` namespace, `buildEncryptConfig`, inference types (`InferPlaintext`, `InferEncrypted`, ...) | +| `@cipherstash/stack` | `Encryption` function (legacy v2 entry point), auth strategies | +| `@cipherstash/stack/schema` | Legacy v2 schema builders (see [Legacy: EQL v2](#legacy-eql-v2)) | | `@cipherstash/stack/identity` | `LockContext` class and identity types | | `@cipherstash/stack/client` | Client-safe exports (schema builders and types only - no native FFI) | | `@cipherstash/stack/types` | All TypeScript types (`Encrypted`, `Decrypted`, `ClientConfig`, `EncryptionClientConfig`, query types, etc.) | -| `@cipherstash/stack/v3` | `EncryptionV3` typed client plus the EQL v3 authoring DSL (`encryptedTable`, `types`, v3 type helpers) | The Drizzle and Supabase integrations are **separate first-party packages** that depend on `@cipherstash/stack` (they are no longer subpaths of it): | Package | Provides | |-------|-----| -| `@cipherstash/stack-drizzle` | Drizzle ORM integration (EQL v2): `encryptedType`, `extractEncryptionSchema`, `createEncryptionOperators` | | `@cipherstash/stack-drizzle/v3` | EQL v3 Drizzle integration: `types` column factories, `createEncryptionOperatorsV3`, `extractEncryptionSchemaV3`, `makeEqlV3Column`, `EncryptionOperatorError` | -| `@cipherstash/stack-supabase` | Supabase integration: `encryptedSupabase` (v2) and `encryptedSupabaseV3` (v3) | - -## Migration from @cipherstash/protect - -If you are migrating from `@cipherstash/protect`, the following table maps the old API to the new one: - -| `@cipherstash/protect` | `@cipherstash/stack` | Import Path | +| `@cipherstash/stack-supabase` | Supabase integration: `encryptedSupabaseV3` (and the legacy v2 `encryptedSupabase`) | +| `@cipherstash/stack-drizzle` | Legacy EQL v2 Drizzle integration (root subpath): `encryptedType`, `extractEncryptionSchema`, `createEncryptionOperators` | + +## Legacy: EQL v2 + +Before the concrete-domain types above, encrypted columns were declared with +chainable capability builders and stored in a single `eql_v2_encrypted` +composite column type. That surface remains fully supported for existing +deployments, but new work should use EQL v3: + +- **Client and schema**: `Encryption` from `@cipherstash/stack` with + `encryptedColumn("email").equality().freeTextSearch().orderAndRange()` and + `.searchableJson()` from `@cipherstash/stack/schema`. v2 and v3 tables cannot + be mixed in one client. +- **Query formatting**: v2 query terms can be rendered as strings with + `returnType: 'composite-literal'` / `'escaped-composite-literal'` for + string-based APIs. +- **Integrations**: the v2 Drizzle surface is the root of + `@cipherstash/stack-drizzle` (`encryptedType`, `extractEncryptionSchema`, + `createEncryptionOperators`); the v2 Supabase surface is `encryptedSupabase`. +- **DynamoDB still requires v2**: `encryptedDynamoDB` from + `@cipherstash/stack/dynamodb` works with the v2 API only — v3 support is + tracked in [#657](https://github.com/cipherstash/stack/issues/657). + +Full v2 documentation lives at [cipherstash.com/docs](https://cipherstash.com/docs). + +### Migrating from @cipherstash/protect + +`@cipherstash/protect` users land on the legacy v2 surface first — the mapping +below is 1:1, and method signatures on the encryption client (`encrypt`, +`decrypt`, `encryptModel`, etc.) and the `Result` pattern (`data` / `failure`) +are unchanged. From there, adopt EQL v3 for new tables: + +| `@cipherstash/protect` | `@cipherstash/stack` (legacy v2) | Import Path | |------------|-----------|-------| | `protect(config)` | `Encryption(config)` | `@cipherstash/stack` | | `csTable(name, cols)` | `encryptedTable(name, cols)` | `@cipherstash/stack/schema` | @@ -696,12 +779,11 @@ If you are migrating from `@cipherstash/protect`, the following table maps the o | `import { LockContext } from "@cipherstash/protect/identify"` | `import { LockContext } from "@cipherstash/stack/identity"` | `@cipherstash/stack/identity` | | N/A | CLI | `npx stash` | -All method signatures on the encryption client (`encrypt`, `decrypt`, `encryptModel`, etc.) remain the same. The `Result` pattern (`data` / `failure`) is unchanged. - ## Requirements -- **Node.js** >= 18 -- The package includes a native FFI module (`@cipherstash/protect-ffi`) written in Rust and embedded via [Neon](https://github.com/neon-bindings/neon). You must opt out of bundling this package in tools like Webpack, esbuild, or Next.js (`serverExternalPackages`). +- **Node.js** >= 22 +- The default entry includes a native FFI module (`@cipherstash/protect-ffi`). On a Node server, externalize it from bundling (e.g. Next.js `serverExternalPackages`). +- For bundled or non-Node runtimes (Deno, Bun, Cloudflare Workers, Supabase Edge Functions), import `@cipherstash/stack/wasm-inline` instead — it inlines the WASM build, so no externalization is needed. See the [bundling guide](https://cipherstash.com/docs/stack/deploy/bundling). ## License diff --git a/skills/stash-cli/SKILL.md b/skills/stash-cli/SKILL.md index 32e0df19..c7103f5f 100644 --- a/skills/stash-cli/SKILL.md +++ b/skills/stash-cli/SKILL.md @@ -336,7 +336,7 @@ Gets a project from zero to installed EQL. It loads an existing `stash.config.ts | `--migration` / `--direct` | Supabase: write a migration file, or run SQL directly | | `--migrations-dir ` | Supabase migrations directory (default `supabase/migrations`) | | `--exclude-operator-family` | Skip operator families (non-superuser roles) | -| `--eql-version <2\|3>` | EQL generation. **Default `2`.** `3` is the native `public.eql_v3_*` domain schema. | +| `--eql-version <2\|3>` | EQL generation. **Default `3`** (the native `public.eql_v3_*` domain schema — the documented approach). `2` is the legacy composite schema. | | `--latest` | Fetch latest EQL from GitHub instead of the bundled copy (**v2 only**) | | `--database-url ` | One-shot install (see below) | @@ -461,7 +461,7 @@ stash encrypt cutover --table users --column email In one transaction it renames `` → `_plaintext` and `_encrypted` → ``, advances the pending config to `encrypting`, activates it, and appends a `cut_over` event. With a Proxy URL configured (`--proxy-url` or `CIPHERSTASH_PROXY_URL`) it then calls `eql_v2.reload_config()` so Proxy picks up the new shape. -> **After cutover, `` holds ciphertext — the read path is not automatic.** Wire reads through the encryption client (`decryptModel(row, table)` for Drizzle, the `encryptedSupabase` wrapper for Supabase, otherwise `decrypt` / `bulkDecryptModels`) before returning values to callers. Skip this and your read paths hand raw EQL payloads to end users. The integration skill has the exact API. **CipherStash Proxy is the one exception** — it decrypts on the wire, so Proxy users need no application change. The cutover plan written by `stash plan` includes this read-path switch as an explicit step. +> **After cutover, `` holds ciphertext — the read path is not automatic.** Wire reads through the encryption client (`decryptModel(row, table)` for Drizzle, the `encryptedSupabaseV3` wrapper for Supabase — `encryptedSupabase` on the legacy v2 surface, otherwise `decrypt` / `bulkDecryptModels`) before returning values to callers. Skip this and your read paths hand raw EQL payloads to end users. The integration skill has the exact API. **CipherStash Proxy is the one exception** — it decrypts on the wire, so Proxy users need no application change. The cutover plan written by `stash plan` includes this read-path switch as an explicit step. > **Known gap.** The pending-configuration precondition is satisfied by `stash db push`. SDK-only users (who otherwise never need `db push`) must therefore run it once before `encrypt cutover`. Decoupling this — under EQL v3 there is no configuration table at all — is tracked in [issue #585](https://github.com/cipherstash/stack/issues/585). @@ -542,7 +542,7 @@ Required: `SUPERUSER`, **or** `CREATE` on the database *and* on the `public` sch **Supabase.** Always pass `--supabase` (or `supabase: true`). It selects a compatible install script and grants `anon`, `authenticated`, and `service_role`. -**`ORDER BY` on encrypted columns doesn't work.** When EQL is installed with `--supabase` or `--exclude-operator-family`, PostgreSQL operator families aren't created, so `ORDER BY` on an encrypted column is unsupported — regardless of client or ORM. Sort application-side after decrypting. Operator-family support for Supabase is in development. +**`ORDER BY` on encrypted columns:** on EQL v3, ordering works on OPE-backed columns — Drizzle emits `ORDER BY eql_v3.ord_term(col)`, and the Supabase adapter's `order()` sorts by the `col->op` term. ORE-flavour (`*OrdOre`) domains need a superuser-only operator class (unavailable on managed Postgres/Supabase) and are rejected; storage-only and equality/match-only columns have no ordering term. For those, order by a plaintext column or sort application-side. (The legacy v2 surface — bare `eql_v2_encrypted` — cannot order encrypted columns without operator families.) **The native binary won't load.** Run `stash doctor`. diff --git a/skills/stash-drizzle/SKILL.md b/skills/stash-drizzle/SKILL.md index 7278e8ae..9269526f 100644 --- a/skills/stash-drizzle/SKILL.md +++ b/skills/stash-drizzle/SKILL.md @@ -1,21 +1,22 @@ --- name: stash-drizzle -description: Integrate CipherStash encryption with Drizzle ORM using @cipherstash/stack-drizzle. Covers the encryptedType column type, encrypted query operators (eq, like, ilike, gt/gte/lt/lte, between, inArray, asc/desc), schema extraction, batched and/or conditions, EQL migration generation, the EQL v3 integration (@cipherstash/stack-drizzle/v3), and the complete Drizzle integration workflow. Use when adding encryption to a Drizzle ORM project, defining encrypted Drizzle schemas, or querying encrypted columns with Drizzle. +description: Integrate CipherStash encryption with Drizzle ORM using @cipherstash/stack-drizzle/v3 (EQL v3). Covers the types.* encrypted column factories (concrete Postgres domains), auto-encrypting query operators (eq, ne, gt/gte/lt/lte, between, inArray, matches, contains, JSON selector, asc/desc), schema extraction, the EncryptionV3 typed client, database setup with stash eql install, and migrating existing plaintext columns to encrypted. Use when adding encryption to a Drizzle ORM project, defining encrypted Drizzle schemas, or querying encrypted columns with Drizzle. --- # CipherStash Stack - Drizzle ORM Integration -Guide for integrating CipherStash field-level encryption with Drizzle ORM using `@cipherstash/stack-drizzle`. Provides a custom column type for encrypted fields and query operators that transparently encrypt search values. +Guide for integrating CipherStash field-level encryption with Drizzle ORM using `@cipherstash/stack-drizzle/v3` (EQL v3). Provides Drizzle-native encrypted column factories and query operators that transparently encrypt search values — Drizzle never sees plaintext in a query. + +In EQL v3 every encrypted column is a **concrete Postgres domain** (`public.eql_v3_text_search`, `public.eql_v3_integer_ord`, ...) whose query capabilities are fixed by the type you pick — there is no capability config object. See the `stash-encryption` skill's "Schema Definition" section (the `types` catalog) for the full catalog and capability suffixes (`Eq`, `Ord`/`OrdOre`, `Match`, `Search`, `Json`). ## When to Use This Skill - Adding field-level encryption to a Drizzle ORM project -- Defining encrypted columns in Drizzle table schemas -- Querying encrypted data with type-safe operators -- Sorting and filtering on encrypted columns -- Generating EQL database migrations +- Defining encrypted columns in Drizzle table schemas with the v3 `types.*` factories +- Querying encrypted data with type-safe, auto-encrypting operators +- Sorting, filtering, and encrypted-JSONB querying on encrypted columns +- Migrating an existing plaintext column to encrypted - Building Express/Hono/Next.js APIs with encrypted Drizzle queries -- Using EQL v3 typed-schema columns in Drizzle (see "EQL v3 Integration" below) ## Installation @@ -25,71 +26,55 @@ npm install @cipherstash/stack @cipherstash/stack-drizzle drizzle-orm The Drizzle integration ships as its own first-party package, `@cipherstash/stack-drizzle`, which depends on `@cipherstash/stack`. Install both. +The v3 surface documented here lives on the `@cipherstash/stack-drizzle/v3` subpath. It is distinct from the older, separate `@cipherstash/drizzle` package (which is `@cipherstash/protect`-based, with different symbol names). ## Database Setup -### Install EQL Extension - -The EQL (Encrypt Query Language) PostgreSQL extension enables searchable encryption functions. Generate a migration: +> **Runner note.** `stash init` adds `stash` to the project as a dev dependency, so `stash ` runs through whichever package manager the project uses (Bun, pnpm, Yarn, or npm) — examples in this skill show this bare form. Before init has run, prefix with your package manager's one-shot runner: `bunx`, `pnpm dlx`, `yarn dlx`, or `npx`. The CLI's behaviour is identical across all of them. -```bash -npx generate-eql-migration -# Options: -# -n, --name Migration name (default: "install-eql") -# -o, --out Output directory (default: "drizzle") -``` +### Install the EQL v3 SQL -Then apply it: +EQL (Encrypt Query Language) provides the PostgreSQL functions and domains that make encrypted columns searchable. Install version 3 directly against the database: ```bash -npx drizzle-kit migrate +stash eql install --eql-version 3 ``` +v3 installs via the direct path only — the v2 `stash eql install --drizzle` Drizzle-migration flow is **not** supported for v3 (`--drizzle`, `--migration`, `--migrations-dir`, and `--latest` are v2-only flags). EQL v3 ships one SQL bundle for every target, including Supabase. + ### Column Storage -Encrypted columns use the `eql_v2_encrypted` PostgreSQL type (installed by EQL). If not using EQL directly, use JSONB: +Each encrypted column is a concrete Postgres domain named `public.eql_v3_`: ```sql CREATE TABLE users ( - id SERIAL PRIMARY KEY, - email eql_v2_encrypted, -- with EQL extension - name jsonb NOT NULL, -- or use jsonb - age INTEGER -- non-encrypted columns are normal types + id INTEGER PRIMARY KEY GENERATED ALWAYS AS IDENTITY, + email public.eql_v3_text_search, -- equality + order/range + free-text + age public.eql_v3_integer_ord, -- equality + order/range + profile public.eql_v3_json, -- encrypted-JSONB queries + role VARCHAR(50) -- non-encrypted columns are normal types ); ``` +You don't usually hand-write this: the `types.*` factories below emit the domain as the column's SQL type, so `drizzle-kit generate` produces the `ADD COLUMN email public.eql_v3_text_search` DDL for you. + ## Schema Definition -Use `encryptedType()` to define encrypted columns in Drizzle table schemas: +Use the `types` namespace from `@cipherstash/stack-drizzle/v3` to define encrypted columns. Each factory maps 1:1 to a Postgres domain, and the column's query capabilities are fixed by the type: ```typescript import { pgTable, integer, timestamp, varchar } from "drizzle-orm/pg-core" -import { encryptedType } from "@cipherstash/stack-drizzle" +import { types } from "@cipherstash/stack-drizzle/v3" const usersTable = pgTable("users", { id: integer("id").primaryKey().generatedAlwaysAsIdentity(), - // Encrypted string with search capabilities - email: encryptedType("email", { - equality: true, // enables: eq, ne, inArray - freeTextSearch: true, // enables: like, ilike - orderAndRange: true, // enables: gt, gte, lt, lte, between, asc, desc - }), - - // Encrypted number - age: encryptedType("age", { - dataType: "number", - equality: true, - orderAndRange: true, - }), - - // Encrypted JSON object with searchable JSONB queries - profile: encryptedType<{ name: string; bio: string }>("profile", { - dataType: "json", - searchableJson: true, - }), + email: types.TextSearch("email"), // equality + order/range + free-text + age: types.IntegerOrd("age"), // equality + order/range + notes: types.Text("notes"), // storage only — encrypt/decrypt, no queries + profile: types.Json("profile"), // encrypted-JSONB containment + selector // Non-encrypted columns role: varchar("role", { length: 50 }), @@ -97,44 +82,52 @@ const usersTable = pgTable("users", { }) ``` -### `encryptedType(name, config?)` +Capability suffixes at a glance (full catalog: `stash-encryption` skill, "Schema Definition" — the `types` namespace): -| Config Option | Type | Description | +| Factory shape | Domain | Enables | |---|---|---| -| `dataType` | `"string"` \| `"number"` \| `"json"` \| `"boolean"` \| `"bigint"` \| `"date"` | Plaintext data type (default: `"string"`) | -| `equality` | `boolean` \| `TokenFilter[]` | Enable equality index | -| `freeTextSearch` | `boolean` \| `MatchIndexOpts` | Enable free-text search index | -| `orderAndRange` | `boolean` | Enable ORE index for sorting and range queries | -| `searchableJson` | `boolean` | Enable JSONB path queries (requires `dataType: "json"`) | +| `types.Text`, `types.Integer`, ... (no suffix) | `eql_v3_text`, ... | Storage only | +| `types.TextEq`, `types.IntegerEq`, ... | `eql_v3_text_eq`, ... | `eq`, `ne`, `inArray`, `notInArray` | +| `types.IntegerOrd`, `types.TimestampOrd`, ... | `eql_v3_integer_ord`, ... | equality + `gt`/`gte`/`lt`/`lte`/`between`/`asc`/`desc` | +| `types.IntegerOrdOre`, ... | `eql_v3_integer_ord_ore`, ... | as `Ord`, with block-ORE ordering (superuser-only install — see Sorting) | +| `types.TextMatch` | `eql_v3_text_match` | `matches` (fuzzy free-text) only | +| `types.TextSearch` | `eql_v3_text_search` | equality + order/range + `matches` | +| `types.Json` | `eql_v3_json` | `contains` + `selector` (encrypted JSONB) | -The generic type parameter `` sets the TypeScript type for the decrypted value. +Value families: `Integer`/`Smallint`/`Numeric`/`Real`/`Double` (`number`), `Bigint` (`bigint`), `Date`/`Timestamp` (`Date`), `Text` (`string`), `Boolean` (`boolean`, storage only), `Json` (a JSON document — object or array, not a top-level scalar). + +`makeEqlV3Column(builder)` wraps a column builder from `@cipherstash/stack/eql/v3` (e.g. `makeEqlV3Column(v3types.TextEq("email"))`) — `types.TextEq("email")` from the Drizzle subpath is shorthand for the same thing. ## Initialization ### 1. Extract Schema from Drizzle Table ```typescript -import { extractEncryptionSchema, createEncryptionOperators } from "@cipherstash/stack-drizzle" -import { Encryption } from "@cipherstash/stack" +import { extractEncryptionSchemaV3, createEncryptionOperatorsV3 } from "@cipherstash/stack-drizzle/v3" +import { EncryptionV3 } from "@cipherstash/stack/v3" -// Convert Drizzle table definition to CipherStash schema -const usersSchema = extractEncryptionSchema(usersTable) +// Convert the Drizzle table definition to a CipherStash v3 schema +const usersSchema = extractEncryptionSchemaV3(usersTable) ``` -### 2. Initialize Encryption Client +### 2. Initialize the Encryption Client ```typescript -const encryptionClient = await Encryption({ +const encryptionClient = await EncryptionV3({ schemas: [usersSchema], }) ``` +`EncryptionV3` returns a strongly-typed client: plaintext types are pinned to each column's domain, and query methods only accept queryable columns. + ### 3. Create Query Operators ```typescript -const encryptionOps = createEncryptionOperators(encryptionClient) +const ops = createEncryptionOperatorsV3(encryptionClient) ``` +`createEncryptionOperatorsV3(client, { lockContext, audit })` optionally sets defaults applied to every operand encryption; the async encrypting operators (`eq`, `ne`, `inArray`, `notInArray`, `gt`/`gte`/`lt`/`lte`, `between`/`notBetween`, `matches`, `contains`, and the comparison methods returned by `selector(...)`) also take an optional trailing `{ lockContext, audit }` argument per call. `asc`/`desc` and the passthrough operators (`isNull`, `isNotNull`, `not`, `and`, `or`, `exists`, `notExists`) encrypt nothing and take no such argument. + ### 4. Create Drizzle Instance ```typescript @@ -146,7 +139,7 @@ const db = drizzle({ client: postgres(process.env.DATABASE_URL!) }) ## Insert Encrypted Data -Encrypt models before inserting: +Rows are pre-encrypted with the client before `db.insert` — Drizzle only ever handles the encrypted EQL envelope: ```typescript // Single insert @@ -173,162 +166,175 @@ if (!encrypted.failure) { ## Query Encrypted Data +Operators auto-encrypt their plaintext operands into EQL v3 query terms — you pass plaintext, the emitted SQL compares encrypted values. Comparison operators are async (they encrypt), so `await` them (or hand them lazily to `ops.and`/`ops.or`, below). + ### Equality ```typescript -// Exact match - await the operator const results = await db .select() .from(usersTable) - .where(await encryptionOps.eq(usersTable.email, "alice@example.com")) + .where(await ops.eq(usersTable.email, "alice@example.com")) ``` -### Text Search +### Free-Text Search (`matches`) -```typescript -// Case-insensitive search -const results = await db - .select() - .from(usersTable) - .where(await encryptionOps.ilike(usersTable.email, "%alice%")) +`matches(col, needle)` is fuzzy bloom-token matching on a `TextMatch`/`TextSearch` column — **not** SQL pattern matching. There are no `like`/`ilike` operators on the v3 surface, by design; don't pass `%` wildcards. -// Case-sensitive search +```typescript const results = await db .select() .from(usersTable) - .where(await encryptionOps.like(usersTable.name, "%Smith%")) + .where(await ops.matches(usersTable.email, "alice")) ``` +Semantics to know: + +- **Fuzzy and one-sided.** The needle's downcased token set is bloom-tested as a subset of the column's — order- and multiplicity-insensitive. A match may be a false positive; a non-match never is. Re-check candidates after decryption if you need exactness. +- **Case-insensitive**, and matches substrings of 3 characters or more. +- **Short needles are rejected.** A needle shorter than the tokenizer's token length (3 by default) produces no tokens and would silently match every row, so the operator throws `EncryptionOperatorError` instead. + ### Range Queries ```typescript const results = await db .select() .from(usersTable) - .where(await encryptionOps.gte(usersTable.age, 18)) + .where(await ops.gte(usersTable.age, 18)) const results = await db .select() .from(usersTable) - .where(await encryptionOps.between(usersTable.age, 18, 65)) + .where(await ops.between(usersTable.age, 18, 65)) ``` -### Array Operators +### Array Membership ```typescript const results = await db .select() .from(usersTable) - .where(await encryptionOps.inArray(usersTable.email, [ + .where(await ops.inArray(usersTable.email, [ "alice@example.com", "bob@example.com", ])) ``` +`inArray`/`notInArray` reject an empty list and encrypt the whole list in a single batch crossing. + ### Sorting ```typescript -// Sort by encrypted column (sync - no await needed) +// Sort by encrypted column (sync — no await needed) const results = await db .select() .from(usersTable) - .orderBy(encryptionOps.asc(usersTable.age)) + .orderBy(ops.asc(usersTable.age)) const results = await db .select() .from(usersTable) - .orderBy(encryptionOps.desc(usersTable.age)) + .orderBy(ops.desc(usersTable.age)) ``` -**Note:** Sorting on encrypted columns requires operator family support in the database. On databases without operator families (e.g. Supabase, or when installed with `--exclude-operator-family`), `ORDER BY` on encrypted columns is not currently supported. Sort application-side after decrypting instead. Operator family support for Supabase is being developed with the Supabase and CipherStash teams. +`ops.asc`/`ops.desc` emit `ORDER BY eql_v3.ord_term(col)` (`ord_term_ore(col)` for the `*OrdOre` domains). The ORE-flavoured domains require a superuser install and are unavailable on managed Postgres (Supabase, RDS, etc.) — prefer the plain `Ord` domains there; ordering works everywhere EQL v3 installs. -## JSONB Queries +### Encrypted-JSONB Containment (`contains`) -Query encrypted JSON columns using JSONB operators. These require `searchableJson: true` and `dataType: "json"` in the column's `encryptedType` config. - -### Check path existence +`contains(col, subDoc)` on a `types.Json` column is **exact** encrypted containment (jsonb `@>` semantics, no false positives). The needle is a ciphertext-free `query_jsonb` term. Array containment is position-independent — `{ roles: ["admin"] }` matches any document whose `roles` array includes `"admin"`: ```typescript -// Check if a JSONB path exists in an encrypted column const results = await db .select() .from(usersTable) - .where(await encryptionOps.jsonbPathExists(usersTable.profile, "$.bio")) + .where(await ops.contains(usersTable.profile, { roles: ["admin"] })) ``` -### Extract value at path +An empty-object needle (`{}`) is rejected — `doc @> '{}'` holds for every document, so it would silently match every row. Omit the predicate if you want all rows. -```typescript -// Extract the first matching value at a JSONB path -const result = await encryptionOps.jsonbPathQueryFirst(usersTable.profile, "$.name") -``` +`types.Json` carries no equality or ordering: `eq`/`gt`/`asc` on a `Json` column throw. -### Get value with `->` operator +### JSONPath Selector-with-Constraint (`selector`) -```typescript -// Get a value using the JSONB -> operator -const result = await encryptionOps.jsonbGet(usersTable.profile, "$.name") -``` - -> **Note:** `jsonbPathExists` returns a boolean and can be used in `WHERE` clauses. `jsonbPathQueryFirst` and `jsonbGet` return encrypted values — use them in `SELECT` expressions. - -### Combine JSONB with other operators +`ops.selector(col, path)` returns comparison methods bound to the encrypted value at a JSONPath inside a `types.Json` column. Its unique power over `contains` is **ordering at a path**: ```typescript +// col->'$.age' > 25 const results = await db .select() .from(usersTable) - .where( - await encryptionOps.and( - encryptionOps.jsonbPathExists(usersTable.profile, "$.name"), - encryptionOps.eq(usersTable.email, "jane@example.com"), - ), - ) + .where(await ops.selector(usersTable.profile, "$.age").gt(25)) + +// col->'$.user' = 'zoe@example.com' +const results = await db + .select() + .from(usersTable) + .where(await ops.selector(usersTable.profile, "$.user").eq("zoe@example.com")) ``` -## Batched Conditions (and / or) +Available methods: `.eq`, `.ne`, `.gt`, `.gte`, `.lt`, `.lte`. Rules: + +- **Paths are dot-notation object keys only** (`"$.a.b"`). Array-index and wildcard syntax (`$.items[0]`) is rejected. +- **Leaves are scalars only**: `string`, `number`, `boolean`, `Date`, or `bigint`. An object or array leaf is rejected — use `contains` for sub-object matching. A `boolean` leaf is rejected under the ordering methods (booleans have no ordering). +- **A scalar needle does not match an array at the path.** `selector(col, "$.tags").eq("a")` will not match `{ tags: ["a"] }` — use `contains(col, { tags: ["a"] })` for that. +- **Absent-path semantics:** `eq` and the ordering methods exclude rows whose document lacks the path; `ne` **includes** them ("not equal to X" covers "has no X"). +- **Interim ciphertext disclosure ([cipherstash/protectjs-ffi#137](https://github.com/cipherstash/protectjs-ffi/issues/137)):** the selector's right-hand value is currently a storage-encrypted needle, so its ciphertext appears in the WHERE clause (and therefore in Postgres logs that capture query text). The comparison itself only reads the needle's index terms. Once #137 lands, the needle becomes ciphertext-free like every other operand. -Use `encryptionOps.and()` and `encryptionOps.or()` to batch multiple encrypted conditions into a single ZeroKMS call. This is more efficient than awaiting each operator individually. +### Batched Conditions (and / or) + +Use `ops.and()` and `ops.or()` to combine encrypted conditions. Pass the operators **lazily** (no `await`) so they resolve concurrently, then `await` the outer call: ```typescript -// Batched AND - all encryptions happen in one call const results = await db .select() .from(usersTable) .where( - await encryptionOps.and( - encryptionOps.gte(usersTable.age, 18), // no await needed - encryptionOps.lte(usersTable.age, 65), // lazy operators - encryptionOps.ilike(usersTable.email, "%example.com"), - eq(usersTable.role, "admin"), // mix with regular Drizzle ops + await ops.and( + ops.gte(usersTable.age, 18), // no await — lazy + ops.lte(usersTable.age, 65), + ops.matches(usersTable.email, "example"), + eq(usersTable.role, "admin"), // mix with regular Drizzle ops ), ) -// Batched OR const results = await db .select() .from(usersTable) .where( - await encryptionOps.or( - encryptionOps.eq(usersTable.email, "alice@example.com"), - encryptionOps.eq(usersTable.email, "bob@example.com"), + await ops.or( + ops.eq(usersTable.email, "alice@example.com"), + ops.eq(usersTable.email, "bob@example.com"), ), ) ``` -**Key pattern:** Pass lazy operators (no `await`) to `and()`/`or()`, then `await` the outer call. This batches all encryption into a single operation. +Both accept `undefined` conditions, which are filtered out — useful for conditional query building: + +```typescript +await ops.and( + maybeEmail ? ops.eq(usersTable.email, maybeEmail) : undefined, + ops.gte(usersTable.age, 18), +) +``` + +### NULLs and Non-Encrypted Columns + +- A `null` operand throws — use `ops.isNull(col)` / `ops.isNotNull(col)` for NULL checks. +- **No plaintext-column fallback.** Every v3 operator requires an encrypted v3 column and throws `EncryptionOperatorError` otherwise. Use regular Drizzle operators (`eq`, `gte`, ...) for non-encrypted columns — mixing the two inside `ops.and`/`ops.or` is fine. ## Decrypt Results +Selected rows hold encrypted envelopes; decrypt with the client. The v3 `decryptModel`/`bulkDecryptModels` take the schema table as the second argument: + ```typescript // Single model -const decrypted = await encryptionClient.decryptModel(results[0]) +const decrypted = await encryptionClient.decryptModel(results[0], usersSchema) if (!decrypted.failure) { console.log(decrypted.data.email) // "alice@example.com" } // Bulk decrypt -const decrypted = await encryptionClient.bulkDecryptModels(results) +const decrypted = await encryptionClient.bulkDecryptModels(results, usersSchema) if (!decrypted.failure) { for (const user of decrypted.data) { console.log(user.email) @@ -336,13 +342,15 @@ if (!decrypted.failure) { } ``` +`Date` columns are reconstructed to real `Date` instances on decrypt; `bigint` columns round-trip as native `bigint`. Non-schema fields pass through unchanged. + ## Migrating an Existing Column to Encrypted The hard case: a Drizzle table that already exists in production with live data in a plaintext column you want to encrypt. You can't just change the column type — that would drop the data and break NOT NULL constraints. CipherStash splits this into two named steps with a hard production-deploy gate between them: an **encryption rollout** (schema-add + dual-write code) and an **encryption cutover** (backfill + rename + drop). (If using CipherStash Proxy, the rollout also includes `stash db push` to register the encryption config with EQL.) The `stash-encryption` skill is the canonical reference for the lifecycle; this section walks the Drizzle-specific shape. -> **Runner note.** `stash init` adds `stash` to the project as a dev dependency, so `stash ` runs through whichever package manager the project uses (Bun, pnpm, Yarn, or npm) — examples below show this bare form. Before init has run, prefix with your package manager's one-shot runner: `bunx`, `pnpm dlx`, `yarn dlx`, or `npx`. The CLI's behaviour is identical across all of them. +> **⚠️ v3 backfill tooling status.** The CLI backfill/cutover tooling (`stash encrypt backfill`, `stash encrypt cutover`, and the underlying `@cipherstash/migrate`) currently targets **EQL v2 columns**. v3 compatibility is tracked in [cipherstash/stack#648](https://github.com/cipherstash/stack/issues/648). The lifecycle below (schema-add → dual-write → deploy gate → backfill → cutover → drop) is the correct shape for v3 either way — until #648 lands, run the backfill/rename steps with your own scripts (encrypt with `bulkEncryptModels`, write in chunks) instead of the `stash encrypt` commands. > **Where am I?** Run `stash status` first (substitute the runner per the note above). It shows you which Drizzle tables/columns are mid-rollout, which are post-deploy, and what the next move is. Re-run after every transition. @@ -370,15 +378,12 @@ Add an `email_encrypted` column **alongside** `email`. Crucially, the encrypted ```typescript // src/db/schema.ts -import { encryptedType } from '@cipherstash/stack-drizzle' +import { types } from '@cipherstash/stack-drizzle/v3' export const users = pgTable('users', { id: integer('id').primaryKey().generatedAlwaysAsIdentity(), - email: text('email').notNull(), // unchanged - email_encrypted: encryptedType('email_encrypted', { // new — nullable - freeTextSearch: true, - equality: true, - }), + email: text('email').notNull(), // unchanged + email_encrypted: types.TextSearch('email_encrypted'), // new — nullable }) ``` @@ -386,27 +391,27 @@ Update the encryption client to harvest the encrypted columns from the table: ```typescript // src/encryption/index.ts -import { Encryption } from '@cipherstash/stack' -import { extractEncryptionSchema } from '@cipherstash/stack-drizzle' +import { EncryptionV3 } from '@cipherstash/stack/v3' +import { extractEncryptionSchemaV3 } from '@cipherstash/stack-drizzle/v3' import { users } from '../db/schema' -const usersEncryptionSchema = extractEncryptionSchema(users) +const usersEncryptionSchema = extractEncryptionSchemaV3(users) -export const encryptionClient = await Encryption({ schemas: [usersEncryptionSchema] }) +export const encryptionClient = await EncryptionV3({ schemas: [usersEncryptionSchema] }) ``` -Generate the migration with `drizzle-kit generate`. The generated SQL should be a single `ALTER TABLE ... ADD COLUMN email_encrypted eql_v2_encrypted;`. Apply with `drizzle-kit migrate`. +Generate the migration with `drizzle-kit generate`. The generated SQL should be a single `ALTER TABLE ... ADD COLUMN email_encrypted public.eql_v3_text_search;`. Apply with `drizzle-kit migrate`. (This requires the EQL v3 SQL to be installed first — see Database Setup.) > **Using CipherStash Proxy?** -> +> > If your app queries encrypted data through CipherStash Proxy, register the new encryption config with EQL: -> +> > ```bash > stash db push > ``` -> +> > If this is the project's first encrypted column, `db push` writes directly to the active EQL config (nothing to rename). If an active config already exists, `db push` writes the new config as `pending` — that's expected. The pending row will be promoted to active by `stash encrypt cutover` in the cutover step. -> +> > SDK-only users can skip this step. #### Dual-writing: write to both columns from app code @@ -452,6 +457,8 @@ Once dual-writes are live in production and `cs_migrations` records `dual_writin #### Backfill: encrypt the historical rows +> Until [#648](https://github.com/cipherstash/stack/issues/648) lands, the commands in this step target v2 columns — for v3 columns, replicate the same shape with a script (chunked `bulkEncryptModels` + UPDATE inside transactions, resumable and idempotent). + ```bash stash encrypt backfill --table users --column email # (Interactive: answer 'yes' to the dual-write confirmation prompt.) @@ -462,16 +469,16 @@ Resumable, idempotent, chunked. The CLI walks the table in keyset-pagination ord If something goes wrong (e.g. you discover the dual-write code wasn't actually live when backfill ran), re-run with `--force` to re-encrypt every row regardless of current state. -> **SDK-only note:** `stash encrypt cutover` currently requires a pending EQL configuration set by `stash db push`. If you're using the SDK without Proxy, you'll hit a "No pending EQL configuration" error from cutover. **Workaround:** run `stash db push` once before `stash encrypt cutover`. [Issue #447](https://github.com/cipherstash/stack/issues/447) tracks decoupling this requirement. +> **SDK-only note:** `stash encrypt cutover` currently requires a pending EQL configuration set by `stash db push`. If you're using the SDK without Proxy, you'll hit a "No pending EQL configuration" error from cutover. **Workaround:** run `stash db push` once before `stash encrypt cutover`. #### Cutover: rename swap and activate -First, update the Drizzle schema to the post-cutover shape — switch `email` to use `encryptedType` and remove the `email_encrypted` column. +First, update the Drizzle schema to the post-cutover shape — switch `email` to the encrypted type and remove the `email_encrypted` column. > **Using CipherStash Proxy?** -> +> > If using Proxy, re-push the encryption config so EQL has a pending row that points at `email` (no `_encrypted` suffix): -> +> > ```bash > stash db push > # → writes the new config as `pending`. Active config (still pointing at @@ -492,10 +499,7 @@ The Drizzle schema you just edited now matches the physical DB shape — `email` // src/db/schema.ts (post-cutover) export const users = pgTable('users', { id: integer('id').primaryKey().generatedAlwaysAsIdentity(), - email: encryptedType('email', { - freeTextSearch: true, - equality: true, - }), + email: types.TextSearch('email'), email_plaintext: text('email_plaintext'), // temporary; dropped next }) ``` @@ -511,12 +515,12 @@ const email = rows[0].email // After const rows = await db.select().from(users).where(eq(users.id, id)) -const decrypted = await encryptionClient.decryptModel(rows[0]) +const decrypted = await encryptionClient.decryptModel(rows[0], usersEncryptionSchema) if (decrypted.failure) throw new Error(decrypted.failure.message) const email = decrypted.data.email ``` -For queries that filter on `email`, switch to the encrypted operators from `createEncryptionOperators` — `eq`, `like`, `gte`, etc. (See `## Query Encrypted Data` above.) +For queries that filter on `email`, switch to the encrypted operators from `createEncryptionOperatorsV3` — `eq`, `matches`, `gte`, etc. (See `## Query Encrypted Data` above.) #### Drop: remove the plaintext column @@ -532,10 +536,7 @@ The CLI emits a Drizzle migration file with `ALTER TABLE users DROP COLUMN email // src/db/schema.ts (final) export const users = pgTable('users', { id: integer('id').primaryKey().generatedAlwaysAsIdentity(), - email: encryptedType('email', { - freeTextSearch: true, - equality: true, - }), + email: types.TextSearch('email'), }) ``` @@ -553,148 +554,58 @@ All three are read-only. ## Complete Operator Reference +All comparison/containment operators auto-encrypt their operands and are async; `asc`/`desc` and the passthroughs are sync. + ### Encrypted Operators (async) -| Operator | Usage | Required Index | +| Operator | Usage | Required column capability (domain suffix) | |---|---|---| -| `eq(col, value)` | Equality | `equality: true` or `orderAndRange: true` | -| `ne(col, value)` | Not equal | `equality: true` or `orderAndRange: true` | -| `gt(col, value)` | Greater than | `orderAndRange: true` | -| `gte(col, value)` | Greater than or equal | `orderAndRange: true` | -| `lt(col, value)` | Less than | `orderAndRange: true` | -| `lte(col, value)` | Less than or equal | `orderAndRange: true` | -| `between(col, min, max)` | Between (inclusive) | `orderAndRange: true` | -| `notBetween(col, min, max)` | Not between | `orderAndRange: true` | -| `like(col, pattern)` | LIKE pattern match | `freeTextSearch: true` | -| `ilike(col, pattern)` | ILIKE case-insensitive | `freeTextSearch: true` | -| `notIlike(col, pattern)` | NOT ILIKE | `freeTextSearch: true` | -| `inArray(col, values)` | IN array | `equality: true` | -| `notInArray(col, values)` | NOT IN array | `equality: true` | -| `jsonbPathQueryFirst(col, selector)` | Extract first value at JSONB path | `searchableJson: true` | -| `jsonbGet(col, selector)` | Get value using JSONB `->` operator | `searchableJson: true` | -| `jsonbPathExists(col, selector)` | Check if JSONB path exists | `searchableJson: true` | +| `eq(col, value)` | Equality | equality (`Eq`, `Ord`, `OrdOre`, `TextSearch`) | +| `ne(col, value)` | Not equal | equality | +| `gt` / `gte` / `lt` / `lte` `(col, value)` | Comparison | order/range (`Ord`, `OrdOre`, `TextSearch`) | +| `between(col, min, max)` | Inclusive range | order/range | +| `notBetween(col, min, max)` | Negated range | order/range | +| `inArray(col, values)` / `notInArray(col, values)` | Membership (single-batch encryption; empty list rejected) | equality | +| `matches(col, needle)` | Fuzzy free-text token match (short needles rejected) | free-text (`TextMatch`, `TextSearch`) | +| `contains(col, subDoc)` | Exact encrypted-JSONB containment (`{}` rejected) | `Json` | +| `selector(col, path).eq/ne/gt/gte/lt/lte(value)` | JSONPath selector-with-constraint (dot-notation paths, scalar leaves) | `Json` | ### Sort Operators (sync) -| Operator | Usage | Required Index | +| Operator | Usage | Required capability | |---|---|---| -| `asc(col)` | Ascending sort | `orderAndRange: true` | -| `desc(col)` | Descending sort | `orderAndRange: true` | +| `asc(col)` | `ORDER BY eql_v3.ord_term(col)` ascending | order/range | +| `desc(col)` | `ORDER BY eql_v3.ord_term(col)` descending | order/range | -### Logical Operators (async, batched) +(`ord_term_ore` for `*OrdOre` domains — superuser-only, unavailable on managed Postgres.) -| Operator | Usage | Description | -|---|---|---| -| `and(...conditions)` | Combine with AND | Batches encryption | -| `or(...conditions)` | Combine with OR | Batches encryption | - -Both `and()` and `or()` accept `undefined` conditions, which are filtered out. This is useful for conditional query building: +### Logical Operators (async, concurrent) -```typescript -const results = await db - .select() - .from(usersTable) - .where( - await encryptionOps.and( - maybeCond ? encryptionOps.eq(usersTable.email, value) : undefined, - encryptionOps.gte(usersTable.age, 18), - ), - ) -``` +| Operator | Description | +|---|---| +| `and(...conditions)` | Conjunction — accepts lazy (un-awaited) operators and `undefined`, resolves concurrently | +| `or(...conditions)` | Disjunction — same | ### Passthrough Operators (sync, no encryption) -`exists`, `notExists`, `isNull`, `isNotNull`, `not`, `arrayContains`, `arrayContained`, `arrayOverlaps` - -These are re-exported from Drizzle and work identically. - -## Non-Encrypted Column Fallback - -All operators automatically detect whether a column is encrypted. If the column is not encrypted (regular Drizzle column), the operator falls back to the standard Drizzle operator: - -```typescript -// This works for both encrypted and non-encrypted columns -await encryptionOps.eq(usersTable.email, "alice@example.com") // encrypted -await encryptionOps.eq(usersTable.role, "admin") // falls back to drizzle eq() -``` - -## Complete Example: Express API - -```typescript -import "dotenv/config" -import express from "express" -import { eq } from "drizzle-orm" -import { drizzle } from "drizzle-orm/postgres-js" -import postgres from "postgres" -import { pgTable, integer, timestamp, varchar } from "drizzle-orm/pg-core" -import { encryptedType, extractEncryptionSchema, createEncryptionOperators, EncryptionOperatorError, EncryptionConfigError } from "@cipherstash/stack-drizzle" -import { Encryption } from "@cipherstash/stack" - -// Schema -const usersTable = pgTable("users", { - id: integer("id").primaryKey().generatedAlwaysAsIdentity(), - email: encryptedType("email", { equality: true, freeTextSearch: true }), - age: encryptedType("age", { dataType: "number", orderAndRange: true }), - role: varchar("role", { length: 50 }), - createdAt: timestamp("created_at").defaultNow(), -}) - -// Init -const usersSchema = extractEncryptionSchema(usersTable) -const encryptionClient = await Encryption({ schemas: [usersSchema] }) -const encryptionOps = createEncryptionOperators(encryptionClient) -const db = drizzle({ client: postgres(process.env.DATABASE_URL!) }) - -const app = express() -app.use(express.json()) - -// Create user -app.post("/users", async (req, res) => { - const encrypted = await encryptionClient.encryptModel(req.body, usersSchema) - if (encrypted.failure) return res.status(500).json({ error: encrypted.failure.message }) - - const [user] = await db.insert(usersTable).values(encrypted.data).returning() - res.json(user) -}) +`isNull`, `isNotNull`, `not`, `exists`, `notExists` — re-exported from Drizzle and work identically. -// Search users -app.get("/users", async (req, res) => { - const conditions = [] +### Other v3 Exports - if (req.query.email) { - conditions.push(encryptionOps.ilike(usersTable.email, `%${req.query.email}%`)) - } - if (req.query.minAge) { - conditions.push(encryptionOps.gte(usersTable.age, Number(req.query.minAge))) - } - if (req.query.role) { - conditions.push(eq(usersTable.role, req.query.role as string)) - } - - let query = db.select().from(usersTable) - if (conditions.length > 0) { - query = query.where(await encryptionOps.and(...conditions)) as typeof query - } - - const results = await query - const decrypted = await encryptionClient.bulkDecryptModels(results) - if (decrypted.failure) return res.status(500).json({ error: decrypted.failure.message }) - - res.json(decrypted.data) -}) - -app.listen(3000) -``` +`types`, `makeEqlV3Column`, `getEqlV3Column`, `isEqlV3Column`, `extractEncryptionSchemaV3`, `createEncryptionOperatorsV3`, `EncryptionOperatorError`, and the codec helpers `v3ToDriver` / `v3FromDriver` / `EqlV3CodecError` — all from `@cipherstash/stack-drizzle/v3`. ## Error Handling -Individual operators (e.g., `eq()`, `gte()`, `like()`) throw errors when invoked with invalid configuration or missing indexes: +Operators throw `EncryptionOperatorError` (exported from `@cipherstash/stack-drizzle/v3`) whenever the query cannot be answered safely: -- **`EncryptionOperatorError`** — thrown for operator-level issues (e.g., invalid arguments, unsupported operations). -- **`EncryptionConfigError`** — thrown for configuration issues (e.g., using `like` on a column without `freeTextSearch: true`). +- the column is not an encrypted v3 column (there is no plaintext fallback); +- the column's domain lacks the operator's capability (e.g. ordering a `TextEq` column, `eq` on a `Json` column); +- the operand is `null` (use `isNull`/`isNotNull`), an empty list (`inArray`), an empty object (`contains`), or a too-short needle (`matches`); +- a `selector` path is malformed / uses array syntax, or its leaf value is a non-scalar; +- operand encryption itself fails. ```typescript -import { createEncryptionOperators, EncryptionOperatorError, EncryptionConfigError } from "@cipherstash/stack-drizzle" +import { EncryptionOperatorError } from "@cipherstash/stack-drizzle/v3" class EncryptionOperatorError extends Error { context?: { @@ -703,95 +614,12 @@ class EncryptionOperatorError extends Error { operator?: string } } - -class EncryptionConfigError extends Error { - context?: { - tableName?: string - columnName?: string - operator?: string - } -} -``` - -Encryption client operations return `Result` objects with `data` or `failure`. - -## EQL v3 Integration (`@cipherstash/stack-drizzle/v3`) - -Everything above covers the v2 integration (`@cipherstash/stack-drizzle`). The **EQL v3** typed schema has its own Drizzle integration on the `@cipherstash/stack-drizzle/v3` subpath. In v3 every encrypted column is a concrete Postgres domain (`public.eql_v3_text_search`, `public.eql_v3_integer_ord`, ...) whose query capabilities are fixed by the type — there is no `equality: true` / `freeTextSearch: true` config object. See the `stash-encryption` skill's "EQL v3 Typed Schema" section for the full `types` catalog and capability suffixes (`Eq`, `Ord`/`OrdOre`, `Match`, `Search`). - -Exports: `types` (Drizzle-native column factories mirroring the `@cipherstash/stack/eql/v3` namespace), `makeEqlV3Column`, `getEqlV3Column`, `isEqlV3Column`, `extractEncryptionSchemaV3`, `createEncryptionOperatorsV3`, `EncryptionOperatorError`, and the codec helpers `v3ToDriver` / `v3FromDriver` / `EqlV3CodecError`. - -### Schema, Client, and Operators - -```typescript -import { pgTable, integer } from "drizzle-orm/pg-core" -import { EncryptionV3 } from "@cipherstash/stack/v3" -import { - types, - extractEncryptionSchemaV3, - createEncryptionOperatorsV3, -} from "@cipherstash/stack-drizzle/v3" - -const users = pgTable("users", { - id: integer("id").primaryKey().generatedAlwaysAsIdentity(), - email: types.TextSearch("email"), // equality + order/range + free-text - age: types.IntegerOrd("age"), // equality + order/range -}) - -const usersSchema = extractEncryptionSchemaV3(users) -const client = await EncryptionV3({ schemas: [usersSchema] }) -const ops = createEncryptionOperatorsV3(client) ``` -Each `types.*` factory emits its domain as the column's SQL type, so `drizzle-kit generate` produces `ADD COLUMN email public.eql_v3_text_search` etc. Install the EQL v3 SQL first with `stash eql install --eql-version 3` (direct install only for v3 — the v2 `generate-eql-migration` Drizzle path is not supported yet). - -`makeEqlV3Column(builder)` wraps a column builder from `@cipherstash/stack/eql/v3` (e.g. `makeEqlV3Column(v3types.TextEq("email"))`) — `types.TextEq("email")` from the drizzle subpath is shorthand for the same thing. +There is no `EncryptionConfigError` on the v3 path — capability problems surface as `EncryptionOperatorError` with the offending column/table/operator in `context`. -### Insert, Query, Decrypt +Encryption client operations (`encryptModel`, `bulkDecryptModels`, ...) don't throw — they return `Result` objects with `data` or `failure`. Check `.failure` before using `.data`. -The Drizzle column stores the encrypted EQL envelope, so encrypt models before insert and decrypt after select — passing the extracted schema table: +## Legacy: EQL v2 -```typescript -// Insert -const enc = await client.bulkEncryptModels( - [{ email: "alice@example.com", age: 30 }], - usersSchema, -) -if (!enc.failure) await db.insert(users).values(enc.data) - -// Query — operators auto-encrypt their plaintext operands -const rows = await db.select().from(users) - .where(await ops.and( - ops.matches(users.email, "alice"), // fuzzy free-text token match - ops.between(users.age, 18, 65), - )) - .orderBy(ops.asc(users.age)) - -// Decrypt — v3 decryptModel/bulkDecryptModels take the schema table -const dec = await client.bulkDecryptModels(rows, usersSchema) -``` - -### Operators - -| Operator | Required capability (domain suffix) | -|---|---| -| `eq`, `ne`, `inArray`, `notInArray` | equality (`Eq`, `Ord`, `OrdOre`, `TextSearch`) | -| `gt`, `gte`, `lt`, `lte`, `between`, `notBetween`, `asc`, `desc` | order/range (`Ord`, `OrdOre`, `TextSearch`) | -| `matches` | fuzzy free-text token match (`TextMatch`, `TextSearch`) | -| `contains` | exact encrypted-JSONB containment (`Json`) | -| `selector(col, '$.path').{eq,ne,gt,gte,lt,lte}` | JSONPath selector-with-constraint on a `Json` column — ordering/equality at a path | -| `and`, `or` | combinators — accept lazy (un-awaited) operators and `undefined`, resolve concurrently | -| `isNull`, `isNotNull`, `not`, `exists`, `notExists` | Drizzle passthroughs, no encryption | - -Differences from the v2 operators to know about: - -- **`like` / `ilike` do not exist — by design.** v3 free-text search is tokenised bloom matching, not SQL pattern matching; `matches(col, needle)` is the free-text operator. It is fuzzy (order- and multiplicity-insensitive) and one-sided (a match may be a false positive, a non-match never is). Don't pass `%` wildcards. -- **`matches` is fuzzy free-text, `contains` is exact JSON containment — two distinct operators (#617).** `matches(col, needle)` requires a `TextMatch` / `TextSearch` column and throws `EncryptionOperatorError` (`requires free-text search`) otherwise. `contains(col, subdoc)` requires a `types.Json` column and throws (`requires JSON containment`) otherwise. -- **`contains` on a `types.Json` column** answers exact encrypted-JSONB containment: `contains(col, { roles: ['admin'] })` matches every row whose document contains that sub-object (jsonb `@>` semantics, no false positives; array containment is position-independent). It emits the `@>` operator with a `query_jsonb` needle. Applying `eq` / `gt` / `asc` **directly** to a `Json` column throws — use `selector(col, '$.path')` for equality/ordering **at a path** (next bullet). -- **`selector(col, '$.path')` — JSONPath selector-with-constraint on a `types.Json` column.** Returns comparison methods bound to a path: `await ops.selector(events.metadata, '$.age').gt(21)`. Methods: `eq`, `ne`, `gt`, `gte`, `lt`, `lte`. Its unique power over `contains` is **ordering at a path** (`gt`/`gte`/`lt`/`lte`); `eq`/`ne` are also provided (equality at a path is equivalently `contains(col, { age: 21 })`). v1 supports dot-notation object paths (`$.a`, `$.a.b`); array-index/wildcard and empty/root paths are rejected with a clear `Error`. **Interim behavior:** the comparison value is currently sent as a storage-encrypted document, so the (encrypted) value appears in the `WHERE` clause; a ciphertext-free form is tracked in `cipherstash/protectjs-ffi#137`. -- **No plaintext-column fallback.** Every v3 operator requires an encrypted v3 column and throws `EncryptionOperatorError` otherwise. Use regular Drizzle operators for non-encrypted columns. -- A `null` operand throws — use `isNull()` / `isNotNull()` for NULL checks. -- `inArray` / `notInArray` reject an empty list, and encrypt the whole list in a single `encryptQuery` batch crossing. -- `matches` rejects a needle shorter than the match tokenizer's token length, and `contains` rejects an empty-object needle (either would otherwise silently match every row). -- Operators gate on the column's capabilities and throw `EncryptionOperatorError` (with `context.columnName` / `tableName` / `operator`) when the domain can't answer the operator. This `EncryptionOperatorError` is exported from `@cipherstash/stack-drizzle/v3` and is deliberately separate from the v2 class of the same name; there is no `EncryptionConfigError` on the v3 path. -- Every operator takes an optional trailing `{ lockContext, audit }` argument; `createEncryptionOperatorsV3(client, { lockContext, audit })` sets defaults applied to every operand encryption. +The original v2 integration — `encryptedType` config-flag columns, `extractEncryptionSchema`, and `createEncryptionOperators` (with `like`/`ilike`) from the `@cipherstash/stack-drizzle` package root, over the `eql_v2_encrypted` column type installed via `stash eql install --drizzle` (the older standalone `@cipherstash/drizzle` package shipped its own `generate-eql-migration` bin for the same purpose) — still exists for existing deployments and is documented at https://cipherstash.com/docs. New projects must use the `/v3` surface documented above. Note: `stash init --drizzle` currently pins EQL v2 because the v2 Drizzle-migration install path has no v3 equivalent yet. diff --git a/skills/stash-encryption/SKILL.md b/skills/stash-encryption/SKILL.md index 85e68b3a..344c6f97 100644 --- a/skills/stash-encryption/SKILL.md +++ b/skills/stash-encryption/SKILL.md @@ -1,23 +1,48 @@ --- name: stash-encryption -description: Implement field-level encryption with @cipherstash/stack. Covers schema definition, encrypt/decrypt operations, searchable encryption (equality, free-text, range, JSON), bulk operations, model operations, identity-aware encryption with LockContext, multi-tenant keysets, the EQL v3 typed schema (concrete Postgres domain columns and the strongly-typed EncryptionV3 client), and the full TypeScript type system. Use when adding encryption to a project, defining encrypted schemas, or working with the CipherStash Encryption API. +description: Implement field-level encryption with @cipherstash/stack using the EQL v3 typed schema. Covers the types.* column catalog (concrete Postgres domains with fixed query capabilities), the strongly-typed EncryptionV3 client, encrypt/decrypt and model operations, searchable encryption (equality, free-text, range), encrypted JSON (containment and JSONPath selectors), bulk operations, identity-aware encryption with lock contexts, multi-tenant keysets, and the rollout/cutover lifecycle. Use when adding encryption to a project, defining encrypted schemas, or working with the CipherStash Encryption API. --- # CipherStash Stack - Encryption Comprehensive guide for implementing field-level encryption with `@cipherstash/stack`. Every value is encrypted with its own unique key via ZeroKMS (backed by AWS KMS). Encryption happens client-side before data leaves the application. +Encrypted columns are **EQL v3 concrete Postgres domains** (`public.eql_v3_text_search`, `public.eql_v3_integer_ord`, ...): each column's query capabilities are fixed by the domain type you pick in the schema, and the `EncryptionV3` client derives precise TypeScript types from that schema — wrong-typed plaintext is a compile error, not a runtime surprise. + +> An older schema surface (EQL v2, with chainable capability builders) still exists for existing deployments — see "Legacy: EQL v2" at the end. Everything else in this skill is the v3 surface. New projects should use v3. + ## When to Use This Skill - Adding field-level encryption to a TypeScript/Node.js project -- Defining encrypted table schemas +- Defining encrypted table schemas with the `types.*` domain catalog - Encrypting and decrypting individual values or entire models -- Implementing searchable encryption (equality, free-text, range, JSON queries) +- Implementing searchable encryption (equality, free-text, range, encrypted JSON) - Bulk encrypting/decrypting large datasets - Implementing identity-aware encryption with JWT-based lock contexts - Setting up multi-tenant encryption with keysets -- Using the EQL v3 typed schema (`@cipherstash/stack/eql/v3`) — concrete Postgres domain columns with a strongly-typed client (see "EQL v3 Typed Schema" below) -- Migrating from `@cipherstash/protect` to `@cipherstash/stack` +- Rolling encryption out to a production table with live plaintext data + +## Quick Start + +```typescript +import { EncryptionV3, encryptedTable, types } from "@cipherstash/stack/v3" + +const users = encryptedTable("users", { + email: types.TextSearch("email"), +}) + +const client = await EncryptionV3({ schemas: [users] }) + +const encrypted = await client.encrypt("secret@example.com", { + table: users, + column: users.email, +}) +if (encrypted.failure) { + throw new Error(encrypted.failure.message) +} + +const decrypted = await client.decrypt(encrypted.data) +``` ## Installation @@ -63,9 +88,26 @@ If you skip this step, you'll see runtime errors like `Cannot find module '@ciph ## Configuration -### Environment Variables +### Local Development (preferred: `stash auth login`) + +No environment variables are needed for local development: + +```bash +npx stash init # agent-assisted setup: auth + schema + database, end to end +# or, if the project is already set up: +npx stash auth login # device code flow; token saved to ~/.cipherstash/auth.json +``` + +`npx stash init` is the assisted flow — it authenticates, builds the encryption +schema with you, generates the client file, and wires the database. For an +already-initialized project, `npx stash auth login` alone authenticates the +machine; the SDK and CLI pick up the saved profile automatically. Sign up at +[cipherstash.com/signup](https://cipherstash.com/signup) first. + +### CI and Production (environment variables) -Set these in `.env` or your hosting platform: +Deployed environments and CI use machine credentials via environment variables +(set in your hosting platform or pipeline secrets — not committed `.env`): ```bash CS_WORKSPACE_CRN=crn:ap-southeast-2.aws:your-workspace-id @@ -74,12 +116,13 @@ CS_CLIENT_KEY=your-client-key CS_CLIENT_ACCESS_KEY=your-access-key ``` -Sign up at [cipherstash.com/signup](https://cipherstash.com/signup) to generate credentials. +When both are present, the `CS_*` variables take precedence over the saved +profile. ### Programmatic Config ```typescript -const client = await Encryption({ +const client = await EncryptionV3({ schemas: [users], config: { workspaceCrn: "crn:ap-southeast-2.aws:your-workspace-id", @@ -91,7 +134,7 @@ const client = await Encryption({ }) ``` -If `config` is omitted, the client reads `CS_*` environment variables automatically. +If `config` is omitted, the client resolves credentials automatically: `CS_*` environment variables when set (CI/production), otherwise the local `stash auth login` profile (development). ### Logging @@ -115,99 +158,138 @@ The SDK never logs plaintext data. | Import Path | Provides | |---|---| -| `@cipherstash/stack` | `Encryption` function, `encryptedTable`, `encryptedColumn`, `encryptedField` (convenience re-exports) | -| `@cipherstash/stack/schema` | `encryptedTable`, `encryptedColumn`, `encryptedField`, schema types | +| `@cipherstash/stack/v3` | `EncryptionV3` factory, `typedClient`, `TypedEncryptionClient` — plus re-exports of everything in `@cipherstash/stack/eql/v3`. The one-stop import for v3. | +| `@cipherstash/stack/eql/v3` | `encryptedTable`, the `types` namespace, `buildEncryptConfig`, inference types (`InferPlaintext`, `InferEncrypted`, `V3ModelInput`, ...) | +| `@cipherstash/stack` | `OidcFederationStrategy`, `AccessKeyStrategy`, the untyped `Encryption` function (also accepts v3 schemas), legacy v2 re-exports | | `@cipherstash/stack/identity` | `LockContext` class and identity types | -| `@cipherstash/stack-drizzle` | `encryptedType`, `extractEncryptionSchema`, `createEncryptionOperators` for Drizzle ORM | -| `@cipherstash/stack-supabase` | `encryptedSupabase` wrapper for Supabase | -| `@cipherstash/stack/dynamodb` | `encryptedDynamoDB` helper for DynamoDB | -| `@cipherstash/stack/encryption` | `EncryptionClient` class, `Encryption` function | | `@cipherstash/stack/errors` | `EncryptionErrorTypes`, `StackError`, error subtypes, `getErrorMessage` | -| `@cipherstash/stack/client` | Client-safe exports: schema builders, schema types, `EncryptionClient` type (no native FFI) | | `@cipherstash/stack/types` | All TypeScript types | -| `@cipherstash/stack/eql/v3` | EQL v3 typed schema: `encryptedTable`, `types` namespace, `buildEncryptConfig`, inference types (see "EQL v3 Typed Schema" below) | -| `@cipherstash/stack/v3` | `EncryptionV3` factory, `typedClient`, `TypedEncryptionClient` — plus re-exports of everything in `@cipherstash/stack/eql/v3` | | `@cipherstash/stack-drizzle/v3` | Drizzle ORM integration for EQL v3 schemas (see the `stash-drizzle` skill) | +| `@cipherstash/stack-supabase` | `encryptedSupabaseV3` wrapper for Supabase (see the `stash-supabase` skill) | +| `@cipherstash/stack/dynamodb` | `encryptedDynamoDB` — **still requires the legacy v2 schema surface**; see "Legacy: EQL v2" below | +| `@cipherstash/stack/schema`, `@cipherstash/stack/client`, `@cipherstash/stack/encryption` | Legacy v2 schema builders and client surface — see "Legacy: EQL v2" below | ## Schema Definition -Define which tables and columns to encrypt using `encryptedTable` and `encryptedColumn`: +Define which tables and columns to encrypt with `encryptedTable` and the `types` namespace. Every encrypted column is a **concrete Postgres domain** whose query capabilities are **fixed by the type** — there is no chainable capability tuner; every domain is fully described by its `types.*` factory. ```typescript -import { encryptedTable, encryptedColumn } from "@cipherstash/stack/schema" +import { encryptedTable, types } from "@cipherstash/stack/eql/v3" +// (also re-exported from "@cipherstash/stack/v3") const users = encryptedTable("users", { - email: encryptedColumn("email") - .equality() // exact-match queries - .freeTextSearch() // full-text / fuzzy search - .orderAndRange(), // sorting and range queries - - age: encryptedColumn("age") - .dataType("number") - .equality() - .orderAndRange(), - - address: encryptedColumn("address"), // encrypt-only, no search indexes + email: types.TextSearch("email"), // equality + order/range + free-text + username: types.TextEq("username"), // equality only + balance: types.BigintOrd("balance"), // equality + order/range + lastLogin: types.TimestampOrd("last_login"), + active: types.Boolean("active"), // storage only — encrypt/decrypt, no queries + notes: types.Text("notes"), // storage only }) -const documents = encryptedTable("documents", { - metadata: encryptedColumn("metadata") - .searchableJson(), // encrypted JSONB queries (JSONPath + containment) +const events = encryptedTable("events", { + metadata: types.Json("metadata"), // encrypted JSONB: containment + selectors }) ``` -### Index Types +The returned table is also a column accessor (`users.email`). The JS property name and the DB column name may differ: `lastLogin: types.TimestampOrd("last_login")` reads/writes the `lastLogin` property on models but targets the `last_login` column in the database. + +### The `types` Namespace + +Each factory in `types` maps 1:1 to a Postgres domain named `public.eql_v3_`. The naming rule: strip the `eql_v3_` prefix and PascalCase each underscore-separated segment. So `types.TextSearch` builds a `public.eql_v3_text_search` column, `types.IntegerOrd` builds `public.eql_v3_integer_ord`, and `types.Timestamp` builds `public.eql_v3_timestamp`. + +**Capability suffixes:** -| Method | Purpose | Query Type | +| Suffix | Capabilities | Query types | |---|---|---| -| `.equality(tokenFilters?)` | Exact match lookups. Accepts an optional array of token filters (e.g., `[{ kind: 'downcase' }]`) for case-insensitive matching. | `'equality'` | -| `.freeTextSearch(opts?)` | Full-text / fuzzy search | `'freeTextSearch'` | -| `.orderAndRange()` | Sorting, comparison, range queries | `'orderAndRange'` | -| `.searchableJson()` | Encrypted JSONB path and containment queries (auto-sets `dataType` to `'json'`) | `'searchableJson'` | -| `.dataType(cast)` | Set plaintext data type | N/A | +| _(none)_ | Storage only — encrypt/decrypt, no queries | — | +| `Eq` | Equality | `'equality'` | +| `Ord` | Equality + ordering/range (OPE-backed) | `'equality'`, `'orderAndRange'` | +| `OrdOre` | Equality + ordering/range (block-ORE flavour — see caveat below) | `'equality'`, `'orderAndRange'` | +| `Match` (text only) | Free-text containment only | `'freeTextSearch'` | +| `Search` (text only) | Equality + ordering/range + free-text | all three | +| `Json` (no suffix) | Encrypted-JSONB containment + JSONPath selector queries | `'searchableJson'` | -**Supported data types:** `'string'` (default), `'text'`, `'number'`, `'boolean'`, `'date'`, `'bigint'`, `'json'` +> **`Ord` vs `OrdOre`:** prefer `Ord`. The `OrdOre` domains are backed by an ORE operator class whose installation requires **superuser** — unavailable on managed Postgres (including Supabase), where the install bundle skips the ORE opclass and disables the `_ord_ore` domains it cannot support. The two ordering flavours produce different, non-cross-comparable terms (`Ord`/`Search` extract via `eql_v3.ord_term`; `OrdOre` via `eql_v3.ord_term_ore`). -Methods are chainable - call as many as you need on a single column. +**Domain families and plaintext types:** -### Free-Text Search Options +| Family | Factories | Plaintext (TypeScript) type | +|---|---|---| +| `Integer`, `Smallint`, `Numeric`, `Real`, `Double` | base, `Eq`, `Ord`, `OrdOre` | `number` | +| `Bigint` | base, `Eq`, `Ord`, `OrdOre` | `bigint` (native JS bigint; full i64 range, out-of-range values rejected client-side before the FFI) | +| `Date` | base, `Eq`, `Ord`, `OrdOre` | `Date` (calendar date; time-of-day is truncated) | +| `Timestamp` | base, `Eq`, `Ord`, `OrdOre` | `Date` (time-of-day preserved) | +| `Text` | base, `Eq`, `Match`, `Ord`, `OrdOre`, `Search` | `string` | +| `Boolean` | base only | `boolean` | +| `Json` | `Json` only | a JSON *document* (`JsonDocument`: object, array, or null — NOT a top-level scalar; nested values are any `JsonValue`) | -```typescript -encryptedColumn("bio").freeTextSearch({ - tokenizer: { kind: "ngram", token_length: 3 }, // or { kind: "standard" } - token_filters: [{ kind: "downcase" }], - k: 6, - m: 2048, - include_original: true, -}) -``` +Examples: `types.Text("notes")` (storage only), `types.TextEq("username")`, `types.BigintOrd("balance")`, `types.TimestampOrdOre("created_at")`, `types.Boolean("active")`, `types.Json("metadata")`. + +The match index on `Match`/`Search` columns is always emitted with the default configuration (trigram tokenizer, downcased) — there is no per-column tuning in v3. Search needles must be at least 3 characters; shorter needles tokenize to nothing and are rejected. ### Type Inference ```typescript -import type { InferPlaintext, InferEncrypted } from "@cipherstash/stack/schema" +import type { InferPlaintext, InferEncrypted } from "@cipherstash/stack/eql/v3" type UserPlaintext = InferPlaintext -// { email: string; age: string; address: string } +// { email: string; lastLogin: Date; balance: bigint; ... } type UserEncrypted = InferEncrypted -// { email: Encrypted; age: Encrypted; address: Encrypted } +// { email: Encrypted; lastLogin: Encrypted; balance: Encrypted; ... } ``` -## Client Initialization +`V3ModelInput`, `V3EncryptedModel`, and `V3DecryptedModel` (same subpath) are the model-shape helpers the typed client uses: schema-column keys are pinned to the column's plaintext type (nullable fields stay nullable), non-schema keys pass through unchanged. + +### Database Setup + +Install the EQL v3 SQL with the stash CLI (v3 is the default): + +```bash +stash eql install --eql-version 3 + +# Supabase targets: add --supabase to apply role grants +stash eql install --eql-version 3 --supabase +``` + +EQL v3 ships **one SQL bundle for every target, including Supabase** — no separate Supabase or no-operator-family variants. For Supabase, though, still pass the `--supabase` flag: it applies the `anon`/`authenticated`/`service_role` grants on the `eql_v3` and `eql_v3_internal` schemas. Without it, encrypted queries fail with `permission denied for schema eql_v3_internal`. v3 installs via the direct path only (`--drizzle`, `--migration`, `--migrations-dir`, and `--latest` are v2-only flags and are not supported for v3). + +In migrations, declare each encrypted column as its domain type: + +```sql +CREATE TABLE users ( + id BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY, + email public.eql_v3_text_search, + last_login public.eql_v3_timestamp_ord, + balance public.eql_v3_bigint_ord +); +``` + +## Client Initialization: `EncryptionV3` + +`EncryptionV3` from `@cipherstash/stack/v3` returns a `TypedEncryptionClient` whose method signatures are derived from your schemas — wrong-typed plaintext is rejected at compile time, and query methods only accept queryable columns with `queryType` constrained to the column's capabilities: ```typescript -import { Encryption } from "@cipherstash/stack" +import { EncryptionV3, encryptedTable, types } from "@cipherstash/stack/v3" + +const users = encryptedTable("users", { + email: types.TextSearch("email"), + lastLogin: types.TimestampOrd("last_login"), + balance: types.BigintEq("balance"), +}) -const client = await Encryption({ schemas: [users, documents] }) +const client = await EncryptionV3({ schemas: [users] }) ``` -The `Encryption()` function returns `Promise` and throws on error (e.g., bad credentials, missing config, invalid keyset UUID). At least one schema is required. +- The wire format is pinned to EQL v3 (`eqlVersion: 3`); you don't set it yourself. +- `EncryptionV3()` throws on init error (bad credentials, missing config, invalid keyset UUID). At least one schema is required. +- `typedClient(client, ...schemas)` (also exported from `@cipherstash/stack/v3`) wraps an already-built untyped `EncryptionClient` in the typed surface. +- The untyped `Encryption({ schemas })` from `@cipherstash/stack` also accepts v3 tables (it auto-detects them and sets the v3 wire format), but you lose the per-column typing — prefer `EncryptionV3`. **v2 and v3 tables cannot be mixed in one client** — a mixed schema set throws at init. Create separate clients if you need both. ```typescript // Error handling try { - const client = await Encryption({ schemas: [users] }) + const client = await EncryptionV3({ schemas: [users] }) } catch (error) { console.error("Init failed:", error.message) } @@ -215,18 +297,22 @@ try { ## Encrypt and Decrypt Single Values +Plaintext is pinned to the column's domain type: + ```typescript -// Encrypt +await client.encrypt("alice@example.com", { table: users, column: users.email }) // string ✓ +await client.encrypt(new Date(), { table: users, column: users.lastLogin }) // Date ✓ +// client.encrypt(42, { table: users, column: users.email }) // ✗ compile error + const encrypted = await client.encrypt("hello@example.com", { - column: users.email, table: users, + column: users.email, }) if (encrypted.failure) { - console.error(encrypted.failure.message) -} else { - console.log(encrypted.data) // Encrypted payload (opaque object) + throw new Error(encrypted.failure.message) } +console.log(encrypted.data) // Encrypted payload (opaque object) // Decrypt const decrypted = await client.decrypt(encrypted.data) @@ -236,62 +322,73 @@ if (!decrypted.failure) { } ``` -All plaintext values must be non-null. Null handling is managed at the model level by `encryptModel` and `decryptModel`. +`decrypt` of a single value cannot be strongly typed — a lone ciphertext carries no column identity. All plaintext values passed to `encrypt` must be non-null; null handling is managed at the model level by `encryptModel` and `decryptModel`. ## Model Operations -Encrypt or decrypt an entire object. Only fields matching your schema are encrypted; other fields pass through unchanged. - -The return type is **schema-aware**: fields matching the table schema are typed as `Encrypted`, while other fields retain their original types. For best results, let TypeScript infer the type parameters from the arguments rather than providing an explicit ``. +Encrypt or decrypt an entire object. Only fields matching your schema are encrypted; other fields pass through unchanged, and schema fields are validated against their inferred plaintext type at compile time. ```typescript -type User = { id: string; email: string; createdAt: Date } - -const user = { - id: "user_123", - email: "alice@example.com", // defined in schema -> encrypted - createdAt: new Date(), // not in schema -> unchanged -} - -// Encrypt model — let TypeScript infer the return type from the schema -const encResult = await client.encryptModel(user, users) -if (!encResult.failure) { - // encResult.data.email is typed as Encrypted - // encResult.data.id is typed as string - // encResult.data.createdAt is typed as Date -} +const enc = await client.encryptModel( + { id: "u1", email: "alice@example.com", lastLogin: new Date(), balance: 100n }, + users, +) +if (!enc.failure) { + // enc.data.email is Encrypted; enc.data.id stays string -// Decrypt model -const decResult = await client.decryptModel(encResult.data) -if (!decResult.failure) { - console.log(decResult.data.email) // "alice@example.com" + const dec = await client.decryptModel(enc.data, users) + if (!dec.failure) { + dec.data.email // string + dec.data.lastLogin // Date — reconstructed on decrypt + dec.data.balance // bigint + dec.data.id // string — non-schema fields pass through + } } ``` -The `Decrypted` type maps encrypted fields back to their plaintext types. +Typed-client model notes: -Passing an explicit type parameter (e.g., `client.encryptModel(...)`) still works for backward compatibility — the return type degrades to `User` in that case. +- `decryptModel` / `bulkDecryptModels` take the **table as a second argument** and return a plain `Promise>` (not a chainable operation) — pass a lock context as the optional third argument instead of chaining `.withLockContext()`. +- `Date` columns are reconstructed to real `Date` instances on decrypt; `bigint` columns round-trip as native `bigint`. +- Nullable schema fields stay nullable through the round trip. ## Bulk Operations All bulk methods make a single call to ZeroKMS regardless of record count, while still using a unique key per value. +### Bulk Encrypt / Decrypt Models + +```typescript +const userModels = [ + { id: "1", email: "alice@example.com", lastLogin: new Date(), balance: 1n }, + { id: "2", email: "bob@example.com", lastLogin: new Date(), balance: 2n }, +] + +const encrypted = await client.bulkEncryptModels(userModels, users) +if (encrypted.failure) throw new Error(encrypted.failure.message) + +const decrypted = await client.bulkDecryptModels(encrypted.data, users) +``` + ### Bulk Encrypt / Decrypt (Raw Values) +`bulkEncrypt` / `bulkDecrypt` are parity passthroughs (not v3-strengthened): + ```typescript const plaintexts = [ { id: "u1", plaintext: "alice@example.com" }, { id: "u2", plaintext: "bob@example.com" }, - { id: "u3", plaintext: "charlie@example.com" }, ] const encrypted = await client.bulkEncrypt(plaintexts, { column: users.email, table: users, }) +if (encrypted.failure) throw new Error(encrypted.failure.message) // encrypted.data = [{ id: "u1", data: EncryptedPayload }, ...] const decrypted = await client.bulkDecrypt(encrypted.data) +if (decrypted.failure) throw new Error(decrypted.failure.message) // Per-item error handling: for (const item of decrypted.data) { if ("data" in item) { @@ -302,23 +399,9 @@ for (const item of decrypted.data) { } ``` -### Bulk Encrypt / Decrypt Models - -```typescript -const userModels = [ - { id: "1", email: "alice@example.com" }, - { id: "2", email: "bob@example.com" }, -] - -const encrypted = await client.bulkEncryptModels(userModels, users) -const decrypted = await client.bulkDecryptModels(encrypted.data) -``` - ## Searchable Encryption -Encrypt query terms so you can search encrypted data in PostgreSQL. - -### Single Query Encryption +Encrypt query terms with `encryptQuery` so you can search encrypted data in PostgreSQL. On the typed client, `encryptQuery` only accepts queryable columns (storage-only columns are rejected at compile time) and constrains `queryType` to the column's capabilities. ```typescript // Equality query @@ -328,7 +411,7 @@ const eqQuery = await client.encryptQuery("alice@example.com", { queryType: "equality", }) -// Free-text search +// Free-text search (substring/token match) const matchQuery = await client.encryptQuery("alice", { column: users.email, table: users, @@ -336,28 +419,37 @@ const matchQuery = await client.encryptQuery("alice", { }) // Order and range -const rangeQuery = await client.encryptQuery(25, { - column: users.age, +const rangeQuery = await client.encryptQuery(new Date("2026-01-01"), { + column: users.lastLogin, table: users, queryType: "orderAndRange", }) +``` -// JSON path query (steVecSelector) -const pathQuery = await client.encryptQuery("$.user.email", { - column: documents.metadata, - table: documents, - queryType: "steVecSelector", -}) +### Query-Type Inference Gotcha on `types.TextSearch` + +A `TextSearch` column carries all three indexes, and `encryptQuery` with **no explicit `queryType` builds an equality term, not a free-text match** (index inference priority: unique > match > ore). A substring like `"joh"` then matches nothing. Pass `queryType: 'freeTextSearch'` explicitly for substring/token search: + +```typescript +// equality (default): exact value only +await client.encryptQuery("john@example.com", { column: users.email, table: users }) -// JSON containment query (steVecTerm) -const containsQuery = await client.encryptQuery({ role: "admin" }, { - column: documents.metadata, - table: documents, - queryType: "steVecTerm", +// free-text match: substring/token search +await client.encryptQuery("joh", { + column: users.email, + table: users, + queryType: "freeTextSearch", }) ``` -If `queryType` is omitted, it's auto-inferred from the column's configured indexes (priority: unique > match > ore > ste_vec). +### Free-Text Search Semantics + +Encrypted free-text search is **fuzzy bloom-filter token matching**, not SQL pattern matching: + +- The needle blooms to its own trigrams; a row matches when the stored value's bloom contains all of them. Case-insensitive; order- and multiplicity-insensitive. +- **One-sided:** a match may be a false positive; a non-match never is. +- Needles must be at least 3 characters (the default trigram length) — shorter needles bloom to nothing and are rejected rather than silently matching every row. +- In the Drizzle and Supabase v3 integrations the operator is **`matches`** — the adapters expose free-text search as `matches`, not `like`/`ilike` (the Supabase adapter keeps a deprecated `like`/`ilike` shim that delegates to `matches` with a warning). Don't pass `%` wildcards. ### Query Result Formatting (`returnType`) @@ -370,7 +462,6 @@ By default `encryptQuery` returns an `Encrypted` object (the raw EQL JSON payloa | `'escaped-composite-literal'` | `string` | Embedding inside another string or JSON value | ```typescript -// Get a composite literal string for use with Supabase const term = await client.encryptQuery("alice@example.com", { column: users.email, table: users, @@ -382,27 +473,9 @@ const term = await client.encryptQuery("alice@example.com", { Each term in a batch can have its own `returnType`. -### Searchable JSON - -For columns using `.searchableJson()`, the query type is auto-inferred from the plaintext: - -```typescript -// String -> JSONPath selector query -const pathQuery = await client.encryptQuery("$.user.email", { - column: documents.metadata, - table: documents, -}) - -// Object/Array -> containment query -const containsQuery = await client.encryptQuery({ role: "admin" }, { - column: documents.metadata, - table: documents, -}) -``` - ### Batch Query Encryption -Encrypt multiple query terms in one ZeroKMS call: +Encrypt multiple query terms in one ZeroKMS call (the per-term columns are heterogeneous, so the batch form takes untyped `ScalarQueryTerm`s): ```typescript const terms = [ @@ -416,59 +489,145 @@ const results = await client.encryptQuery(terms) All values in the array must be non-null. -## Identity-Aware Encryption (Lock Contexts) +### On the Wire: Operators and Ordering -Lock encryption to a specific user by requiring a valid JWT for decryption. +Scalar filters compare through each domain's `eql_v3.*` operators (`col = term`, `col > term`, ...), and `ORDER BY` on an encrypted column goes through the ordering extractors — `eql_v3.ord_term(col)` for OPE-backed (`Ord`/`Search`) domains, `eql_v3.ord_term_ore(col)` for `OrdOre`. The Drizzle v3 integration emits all of this for you (including `asc`/`desc`, which emit `ORDER BY eql_v3.ord_term(col)`). Over Supabase/PostgREST, the adapter's `order()` works on OPE-backed ordering columns (plain `*_ord`, `text_ord`, `text_search`) by sorting on the column's `col->op` term; `OrdOre`-flavour (`*_ord_ore`) domains and columns with no ordering term are rejected. See the `stash-drizzle` and `stash-supabase` skills. + +## Encrypted JSON (`types.Json`) + +A `types.Json("metadata")` column encrypts a whole JSON document to a `public.eql_v3_json` value. The plaintext is a **`JsonDocument`: an object, an array, or `null` — NOT a bare top-level scalar** (protect-ffi rejects a top-level string/number/boolean; a scalar belongs in a scalar domain like `types.TextEq` or `types.IntegerEq`). Nested scalars are fully supported. + +`types.Json` carries no equality or ordering capability — `eq` / `gt` / `asc` on it throw. It supports two query patterns: containment and JSONPath selectors. + +### Containment (exact, `@>`) + +Pass a sub-object or sub-array to `encryptQuery` with `queryType: 'searchableJson'`; it matches documents that contain the needle with jsonb `@>` semantics — **exact** containment, no false positives. Array containment is a subset test regardless of element position: `{ roles: ['admin'] }` matches any document whose `roles` array includes `admin`. ```typescript -import { LockContext } from "@cipherstash/stack/identity" +const events = encryptedTable("events", { metadata: types.Json("metadata") }) -// 1. Create a lock context (defaults to the "sub" claim) -const lc = new LockContext() -// Or with custom claims: new LockContext({ context: { identityClaim: ["sub", "org_id"] } }) -// Or with a pre-fetched CTS token: new LockContext({ ctsToken: { accessToken: "...", expiry: 123456 } }) +// containment: object needle +await client.encryptQuery({ roles: ["admin"] }, { column: events.metadata, table: events }) +``` + +The Drizzle and Supabase adapters reject an empty-object needle (jsonb `{} ⊆ anything` — it would silently match every row); core `client.encryptQuery` does not enforce this, so guard against empty needles yourself when composing raw SQL. + +### JSONPath Selectors (value-at-a-path) -// 2. Identify the user with their JWT -const identifyResult = await lc.identify(userJwt) -if (identifyResult.failure) { - throw new Error(identifyResult.failure.message) +Selector queries constrain the value **at a path** inside the document — `metadata->'$.user.role' = 'admin'`. Their unique power over containment is **ordering at a path** (`gt`/`gte`/`lt`/`lte`), available through the Drizzle v3 integration. Paths are dot-notation object paths (`'$.a.b'`); array/wildcard steps are rejected. + +Semantics to know: + +- **`eq`** at a path excludes rows whose document lacks the path (absent is not equal). +- **`ne`** at a path **includes** rows whose document lacks the path, and — in both adapters — rows whose column is SQL NULL ("not equal to value" covers "has no value"; Drizzle adds `OR IS NULL`, Supabase an `is.null` branch). +- **Array-leaf caveat:** a scalar needle does not match an array at the path — ste_vec encodes array elements under their own selectors, so `{a: [40, 30]}` is NOT matched by a selector-eq of `30` at `$.a`. To match an array-valued path, pass the full array through containment. +- **Ciphertext in the WHERE clause (interim):** pending a ciphertext-free ordering needle from protect-ffi ([cipherstash/protectjs-ffi#137](https://github.com/cipherstash/protectjs-ffi/issues/137)), the selector's right-hand operand is a storage encryption of `{path: value}` whose ciphertext appears in the emitted SQL statement — it can surface in SQL logs and statement views. On Supabase, the selector/containment needle additionally ships as a full storage envelope because of a PostgREST operand-casting gap ([cipherstash/stack#654](https://github.com/cipherstash/stack/issues/654)) — #137 landing does not remove that exposure. + +### Adapter Matrix + +| Capability | Drizzle v3 (`ops` from `@cipherstash/stack-drizzle/v3`) | Supabase v3 (`encryptedSupabaseV3`) | +|---|---|---| +| Containment (`@>`) | `ops.contains(col, subdoc)` | `contains(col, subdoc)` / `filter(col, 'cs', ...)` | +| Selector equality | `ops.selector(col, '$.path').eq(value)` / `.ne(value)` | `selectorEq(col, '$.path', value)` / `selectorNe(...)` | +| Selector ordering | `ops.selector(col, '$.path').gt/gte/lt/lte(value)` | Not available — needs [cipherstash/encrypt-query-language#407](https://github.com/cipherstash/encrypt-query-language/issues/407) | + +See the `stash-drizzle` and `stash-supabase` skills for the full integration guides. + +## Authentication + +The client authenticates to ZeroKMS through `config.authStrategy`. Leave it unset for the default **auto** strategy: in local development, authenticate once with `npx stash auth login` (preferred — no credentials in your environment; `npx stash init` is the agent-assisted flow that also sets up schema and database); in CI/production, set the `CS_*` environment variables. Two explicit strategies cover the other cases: + +- **`AccessKeyStrategy`** — service-to-service / CI. Authenticates a *service* with a CipherStash access key. +- **`OidcFederationStrategy`** — authenticates the client **as the end user** by federating a third-party OIDC JWT (Clerk, Supabase, Auth0, Okta, ...) into a CipherStash service token: + +```typescript +import { OidcFederationStrategy } from "@cipherstash/stack" +import { EncryptionV3, encryptedTable, types } from "@cipherstash/stack/v3" + +const users = encryptedTable("users", { email: types.TextSearch("email") }) + +// `getJwt` is re-invoked on every (re-)federation and must return the +// *current* third-party OIDC JWT. +const strategy = OidcFederationStrategy.create( + process.env.CS_WORKSPACE_CRN!, + () => getUserJwt(), +) +if (strategy.failure) { + throw new Error(`[auth] ${strategy.failure.type}: ${strategy.failure.error.message}`) } -const lockContext = identifyResult.data -// 3. Encrypt with lock context +const client = await EncryptionV3({ + schemas: [users], + config: { authStrategy: strategy.data }, +}) +``` + +`OidcFederationStrategy.create()` returns a `Result` — **unwrap it**. Passing the envelope straight to `authStrategy` gives the FFI an object with no `getToken()` at all. + +Authentication stands on its own — an OIDC-authenticated client encrypts and decrypts normally. Binding *data* to the authenticated user is a separate, optional step: the lock context, below. + +## Identity-Aware Encryption (Lock Contexts) + +Bind a data key to a claim from the end user's JWT, so only that user can decrypt. Chain `.withLockContext({ identityClaim })` on any operation: + +```typescript +// Requires a client authenticated with OidcFederationStrategy (see +// "Authentication" above) — the claim's value resolves from the federated JWT. +const IDENTITY = { identityClaim: ["sub"] } + const encrypted = await client .encrypt("sensitive data", { column: users.email, table: users }) - .withLockContext(lockContext) + .withLockContext(IDENTITY) +if (encrypted.failure) { + throw new Error(`[encryption] ${encrypted.failure.type}: ${encrypted.failure.message}`) +} -// 4. Decrypt with the same lock context +// Decrypt with the SAME claim. Anything else cannot reproduce the key. const decrypted = await client .decrypt(encrypted.data) - .withLockContext(lockContext) + .withLockContext(IDENTITY) +if (decrypted.failure) { + throw new Error(`[encryption] ${decrypted.failure.type}: ${decrypted.failure.message}`) +} ``` -Lock contexts work with ALL operations: `encrypt`, `decrypt`, `encryptModel`, `decryptModel`, `bulkEncrypt`, `bulkDecrypt`, `bulkEncryptModels`, `bulkDecryptModels`, `encryptQuery`. +Lock contexts **require** an `OidcFederationStrategy`-authenticated client: the claim's value resolves from the JWT the strategy federated. The auto and access-key strategies authenticate no end user, so there is no JWT to resolve claims from — `AccessKeyStrategy` in particular authenticates a *service* and cannot be used with a lock context. Plain authentication never requires a lock context. -### CTS Token Service +Every operation returns a `Result`. Narrow on `.failure` before touching `.data`: the `Failure` branch has no `data` property, so skipping the check is a type error, not merely a runtime risk. -The lock context exchanges the JWT for a CTS (CipherStash Token Service) token. Set the endpoint: +`identityClaim` is an array of JWT claim *names*, not values: `["sub"]` (the default) or `["sub", "org_id"]`. ZeroKMS resolves each claim's value from the JWT the strategy federated. **The same claim must be supplied to encrypt and decrypt** — it is baked into the data key's tag, so decrypting without it fails with `Failed to retrieve key`. -```bash -CS_CTS_ENDPOINT=https://ap-southeast-2.aws.auth.viturhosted.net +Lock contexts work with every operation: `encrypt`, `decrypt`, `encryptModel`, `decryptModel`, `bulkEncrypt`, `bulkDecrypt`, `bulkEncryptModels`, `bulkDecryptModels`, `encryptQuery`. On the typed client, `decryptModel` and `bulkDecryptModels` take the lock context as their optional **third argument** (they return a plain `Promise>`, so there is no `.withLockContext()` to chain): + +```typescript +const dec = await client.decryptModel(enc.data, users, IDENTITY) ``` +### Deprecated: `LockContext.identify()` + +Older code fetched a per-operation CTS token: + +```typescript +const lc = new LockContext() +const identified = await lc.identify(userJwt) // deprecated +await client.encrypt(...).withLockContext(identified.data) +``` + +**Per-operation CTS tokens were removed in `protect-ffi` 0.25.** `LockContext`, `identify()` and `getLockContext()` still exist for backwards compatibility, but the token `identify()` fetches is no longer used by encryption — and `CS_CTS_ENDPOINT` is only read on that dead path. Authenticate with `OidcFederationStrategy` instead and pass the claim directly. `.withLockContext()` accepts either a `LockContext` instance or a plain `{ identityClaim }`. + ## Multi-Tenant Encryption (Keysets) Isolate encryption keys per tenant: ```typescript // By name -const client = await Encryption({ +const client = await EncryptionV3({ schemas: [users], config: { keyset: { name: "Company A" } }, }) // By UUID -const client = await Encryption({ +const client = await EncryptionV3({ schemas: [users], config: { keyset: { id: "123e4567-e89b-12d3-a456-426614174000" } }, }) @@ -478,7 +637,7 @@ Each keyset provides full cryptographic isolation between tenants. ## Operation Chaining -All operations return thenable objects that support chaining: +Encrypt operations return thenable objects that support chaining: ```typescript const result = await client @@ -487,6 +646,8 @@ const result = await client .audit({ metadata: { action: "create" } }) // optional: audit trail ``` +(The typed client's `decryptModel` / `bulkDecryptModels` are the exception: they return a plain `Promise>` and take the lock context as an argument — see "Model Operations".) + ## Error Handling All async methods return a `Result` object - a discriminated union with either `data` (success) or `failure` (error), never both. @@ -552,245 +713,16 @@ try { ### Validation Rules - NaN and Infinity are rejected for numeric values -- `freeTextSearch` index only supports string values +- `bigint` values outside the signed 64-bit range are rejected client-side before the FFI +- Free-text search only applies to string values (`Match`/`Search` text domains); needles shorter than the trigram length (3) are rejected +- A `types.Json` column rejects a bare top-level scalar — the document must be an object, array, or null - At least one `encryptedTable` schema must be provided - Keyset UUIDs must be valid format -## Ordering Encrypted Data - -**`ORDER BY` on encrypted columns requires operator family support in the database.** - -On databases without operator families (e.g. Supabase, or when EQL is installed with `--exclude-operator-family`), sorting on encrypted columns is not currently supported — regardless of the client or ORM used. This applies to Drizzle, the Supabase JS SDK, raw SQL, and any other database client. - -**Workaround:** Sort application-side after decrypting the results. - -Operator family support for Supabase is being developed in collaboration with the Supabase and CipherStash teams and will be available in a future release. - -## PostgreSQL Storage - -Encrypted data is stored as EQL (Encrypt Query Language) JSON payloads. Install the EQL extension in PostgreSQL: - -```sql -CREATE EXTENSION IF NOT EXISTS eql_v2; - -CREATE TABLE users ( - id BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY, - email eql_v2_encrypted -); -``` - -Or store as JSONB if not using the EQL extension directly: - -```sql -CREATE TABLE users ( - id SERIAL PRIMARY KEY, - email jsonb NOT NULL -); -``` - -## EQL v3 Typed Schema - -EQL v3 is a newer schema surface where every encrypted column is a **concrete Postgres domain** (`public.eql_v3_`) and its query capabilities are fixed by the column type you pick. There is **no chainable capability tuner** — no `.equality()`, `.freeTextSearch()`, or `.orderAndRange()` — every domain is fully described by its type. The v2 surface documented above (`encryptedColumn` from `@cipherstash/stack/schema`) continues to work unchanged; v3 lives on its own subpaths. - -### Quick Start - -```typescript -import { Encryption } from "@cipherstash/stack" -import { encryptedTable, types } from "@cipherstash/stack/eql/v3" - -const users = encryptedTable("users", { - email: types.TextSearch("email"), -}) - -const client = await Encryption({ schemas: [users] }) - -const encryptResult = await client.encrypt("secret@example.com", { - column: users.email, - table: users, -}) -if (encryptResult.failure) { - // handle encryptResult.failure.message -} - -const decryptResult = await client.decrypt(encryptResult.data) -``` - -`Encryption({ schemas })` auto-detects v3 tables and sets the EQL v3 wire format. **v2 and v3 tables cannot be mixed in one client** — a mixed schema set throws at init. Create separate clients if you need both. - -The v3 `encryptedTable` intentionally shares its name with the v2 builder — the import path picks the model. The returned table is also a column accessor (`users.email`). The JS property name and the DB column name may differ: `createdOn: types.Timestamp("created_at")` reads/writes the `createdOn` property on models but targets the `created_at` column in the database. - -### The `types` Namespace - -Each factory in `types` maps 1:1 to a Postgres domain named `public.eql_v3_`. The naming rule: strip the `eql_v3_` prefix and PascalCase each underscore-separated segment. So `types.TextSearch` builds a `public.eql_v3_text_search` column, `types.IntegerOrd` builds `public.eql_v3_integer_ord`, and `types.Timestamp` builds `public.eql_v3_timestamp`. - -**Capability suffixes:** - -| Suffix | Capabilities | Query types | -|---|---|---| -| _(none)_ | Storage only — encrypt/decrypt, no queries | — | -| `Eq` | Equality | `'equality'` | -| `Ord` / `OrdOre` | Equality + ordering/range | `'equality'`, `'orderAndRange'` | -| `Match` (text only) | Free-text containment only | `'freeTextSearch'` | -| `Search` (text only) | Equality + ordering/range + free-text | all three | -| `Json` (no suffix) | Encrypted-JSONB containment + JSONPath selector-with-constraint queries (selector RHS is interim: encrypted value currently appears in the `WHERE` clause — see below) | `'searchableJson'` | - -**Domain families and plaintext types:** - -| Family | Factories | Plaintext (TypeScript) type | -|---|---|---| -| `Integer`, `Smallint`, `Numeric`, `Real`, `Double` | base, `Eq`, `Ord`, `OrdOre` | `number` | -| `Bigint` | base, `Eq`, `Ord`, `OrdOre` | `bigint` (native JS bigint; full i64 range, out-of-range values rejected client-side before the FFI) | -| `Date` | base, `Eq`, `Ord`, `OrdOre` | `Date` (calendar date; time-of-day is truncated) | -| `Timestamp` | base, `Eq`, `Ord`, `OrdOre` | `Date` (time-of-day preserved) | -| `Text` | base, `Eq`, `Match`, `Ord`, `OrdOre`, `Search` | `string` | -| `Boolean` | base only | `boolean` | -| `Json` | `Json` only | a JSON *document* (`JsonDocument`: object, array, or null — NOT a top-level scalar; nested values are any `JsonValue`) | - -Examples: `types.Text("notes")` (storage only), `types.TextEq("username")`, `types.BigintOrd("balance")`, `types.TimestampOrdOre("created_at")`, `types.Boolean("active")`, `types.Json("metadata")`. - -The match index on `Match`/`Search` columns is always emitted with the default configuration — there is no per-column tuning in v3. - -### Free-Text Queries on `types.TextSearch` - -A `TextSearch` column carries all three indexes, and `encryptQuery` with **no explicit `queryType` builds an equality term, not a free-text match** (index inference priority: unique > match > ore). A substring like `"joh"` then matches nothing. Pass `queryType: 'freeTextSearch'` explicitly for substring/token search: - -```typescript -// equality (default): exact value only -await client.encryptQuery("john@example.com", { column: users.email, table: users }) - -// free-text match: substring/token search -await client.encryptQuery("joh", { - column: users.email, - table: users, - queryType: "freeTextSearch", -}) -``` - -### Encrypted-JSONB Queries on `types.Json` - -A `types.Json("metadata")` column encrypts a whole JSON document (a -`JsonDocument`: object, array, or null — not a top-level scalar) to a -`public.eql_v3_json` value. - -**Containment** is the supported query today. Pass a sub-object or sub-array to -`encryptQuery` with `queryType: 'searchableJson'`; it matches documents that -contain the needle (jsonb `@>` semantics). Array containment is a subset test -regardless of element position — `{ roles: ['admin'] }` matches any document -whose `roles` array includes `admin`. - -```typescript -const events = encryptedTable("events", { metadata: types.Json("metadata") }) - -// containment: object needle -await client.encryptQuery({ roles: ["admin"] }, { column: events.metadata, table: events }) -``` - -Through the Drizzle v3 integration this is `ops.contains(col, subObject)` (on Supabase: `contains(col, subObject)` — see the `stash-supabase` skill) — see -the `stash-drizzle` skill. - -**JSONPath selector-with-constraint** is the second query pattern: it compares -the encrypted value at a JSONPath, e.g. `metadata->'age' > 21`. Through the -Drizzle v3 integration this is `ops.selector(col, '$.path').{eq,ne,gt,gte,lt,lte}(value)`; -on Supabase it is `selectorEq(col, path, value)` / `selectorNe(col, path, value)` -— equality/inequality only, since selector ORDERING over PostgREST needs an -EQL-bundle change (cipherstash/encrypt-query-language#407); see the -`stash-supabase` skill. Drizzle's unique power over containment is **ordering at a path** -(`ops.selector(events.metadata, '$.age').gt(21)`); equality at a path is also -expressible as containment (`contains(col, { age: 21 })`). v1 supports -dot-notation object paths (`$.a`, `$.a.b`); array-index/wildcard and empty/root -paths are rejected with a clear error. Applying `eq` / `gt` / `asc` **directly** -to a `types.Json` column (rather than through `ops.selector`) still throws — the -column declares no scalar capabilities of its own. - -**Interim behavior:** the selector's comparison value is currently sent as a -storage-encrypted document, so the (encrypted) value appears in the `WHERE` -clause; a ciphertext-free form is tracked in `cipherstash/protectjs-ffi#137`. - -### Strongly-Typed Client: `EncryptionV3` - -`EncryptionV3` from `@cipherstash/stack/v3` returns a `TypedEncryptionClient` whose method signatures are derived from your schemas — wrong-typed plaintext is rejected at compile time, and query methods only accept queryable columns with `queryType` constrained to the column's capabilities: - -```typescript -import { EncryptionV3, encryptedTable, types } from "@cipherstash/stack/v3" - -const users = encryptedTable("users", { - email: types.TextSearch("email"), - lastLogin: types.TimestampOrd("last_login"), - balance: types.BigintEq("balance"), -}) - -const client = await EncryptionV3({ schemas: [users] }) - -// Plaintext is pinned to the column's domain type: -await client.encrypt("alice@example.com", { table: users, column: users.email }) // string ✓ -await client.encrypt(new Date(), { table: users, column: users.lastLogin }) // Date ✓ -// client.encrypt(42, { table: users, column: users.email }) // ✗ compile error - -const enc = await client.encryptModel( - { id: "u1", email: "alice@example.com", lastLogin: new Date(), balance: 100n }, - users, -) -if (!enc.failure) { - const dec = await client.decryptModel(enc.data, users) - if (!dec.failure) { - dec.data.email // string - dec.data.lastLogin // Date — reconstructed on decrypt - dec.data.balance // bigint - dec.data.id // string — non-schema fields pass through - } -} -``` - -Typed-client notes: - -- The wire format is pinned to EQL v3 (`eqlVersion: 3`); you don't set it yourself. -- Methods: `encrypt`, `encryptQuery`, `encryptModel`, `bulkEncryptModels`, `decrypt`, `decryptModel`, `bulkDecryptModels`, plus `bulkEncrypt`/`bulkDecrypt` passthroughs and `getEncryptConfig`. -- `decryptModel` / `bulkDecryptModels` take the **table as a second argument** and return a plain `Promise>` (not a chainable operation) — pass a lock context as the optional third argument instead of chaining `.withLockContext()`. `Date` columns are reconstructed to real `Date` instances on decrypt; `bigint` columns round-trip as native `bigint`. -- `decrypt` of a single value cannot be strongly typed — a lone ciphertext carries no column identity. -- `encryptQuery` rejects storage-only columns outright at compile time. -- `typedClient(client, ...schemas)` (also exported from `@cipherstash/stack/v3`) wraps an already-built `EncryptionClient` in the typed surface. - -### Database Setup - -Install the EQL v3 SQL with the stash CLI: - -```bash -stash eql install --eql-version 3 -``` - -EQL v3 ships one SQL bundle for every target, including Supabase — no separate Supabase or no-operator-family variants. v3 currently installs via the direct path only (`--drizzle`, `--migration`, `--migrations-dir`, and `--latest` are not supported for v3 yet). - -In migrations, declare each encrypted column as its domain type: - -```sql -CREATE TABLE users ( - id BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY, - email public.eql_v3_text_search, - last_login public.eql_v3_timestamp_ord, - balance public.eql_v3_bigint_eq -); -``` - -### Type Inference - -```typescript -import type { InferPlaintext, InferEncrypted } from "@cipherstash/stack/eql/v3" - -type UserPlaintext = InferPlaintext -// { email: string; lastLogin: Date; balance: bigint } - -type UserEncrypted = InferEncrypted -// { email: Encrypted; lastLogin: Encrypted; balance: Encrypted } -``` - -`V3ModelInput`, `V3EncryptedModel`, and `V3DecryptedModel` (same subpath) are the model-shape helpers the typed client uses: schema-column keys are pinned to the column's plaintext type (nullable fields stay nullable), non-schema keys pass through unchanged. - -### Drizzle ORM - -`@cipherstash/stack-drizzle/v3` provides Drizzle-native v3 column factories, schema extraction, and auto-encrypting query operators. See the `stash-drizzle` skill for the full guide. - ## Rolling Encryption Out to Production +> **EQL v2 note ([#648](https://github.com/cipherstash/stack/issues/648)).** The backfill/cutover tooling in this section (`stash encrypt *`, `stash db push`, `@cipherstash/migrate`) currently targets **EQL v2 columns** — the `eql_v2_configuration` registry, `eql_v2_encrypted` payload columns, and v2 schemas. v3 support for the rollout lifecycle is tracked in [cipherstash/stack#648](https://github.com/cipherstash/stack/issues/648). The *process* below — rollout, deploy gate, cutover — is the model to follow either way; every `eql_v2`-named artifact it mentions is part of that v2-targeting tooling. + Adding a fresh encrypted column to a table you don't yet write to is the easy case — declare it in the schema, run the migration, start writing. The harder case is taking an **existing plaintext column with live data** and turning it into an encrypted one without dropping a write or returning the wrong value mid-cutover. CipherStash splits that into two named steps with a hard production-deploy gate between them: @@ -841,7 +773,7 @@ Once dual-writes are recorded as live in `cs_migrations`: | `stash encrypt backfill` | Walks the table in keyset-pagination order, encrypts each chunk, writes a single transactional `UPDATE` per chunk plus a `cs_migrations` checkpoint. SIGINT-safe; idempotent re-runs converge. | | Schema rename | Update the schema file: drop the `_encrypted` suffix; switch the original column declaration onto the encrypted type. | | `stash encrypt cutover` | One transaction: renames `` → `_plaintext`, `_encrypted` → ``, and promotes `pending` → `active`. Application reads of `` now return decrypted ciphertext transparently. | -| Wire reads through the encryption client | Read paths must decrypt before returning the value to callers (`decryptModel(row, table)` for Drizzle; `encryptedSupabase` wrapper for Supabase; `decrypt`/`bulkDecryptModels` otherwise). Without this step, reads return raw `eql_v2_encrypted` payloads to end users. | +| Wire reads through the encryption client | Read paths must decrypt before returning the value to callers (`decryptModel(row, table)` for Drizzle; the Supabase wrapper for Supabase; `decrypt`/`bulkDecryptModels` otherwise). Without this step, reads return raw `eql_v2_encrypted` payloads to end users. | | Remove dual-write code | The plaintext column is now `_plaintext` and is no longer authoritative. Delete the dual-write logic. | | `stash encrypt drop` | Emits a migration that removes `_plaintext`. Apply with the project's normal migration tooling. | @@ -861,7 +793,7 @@ Three sources of truth, kept separate on purpose: ### CLI sequence for a single column -> **Known limitation:** `stash encrypt cutover` currently requires a pending EQL configuration registered via `stash db push`. SDK-only users may hit a "No pending EQL configuration" error. **Workaround:** Run `stash db push` once before `stash encrypt cutover`, even if you don't use CipherStash Proxy. Decoupling cutover from EQL config for SDK users is tracked in issue [#447](https://github.com/cipherstash/stack/issues/447) follow-up work. +> **Known limitation:** `stash encrypt cutover` currently requires a pending EQL configuration registered via `stash db push`. SDK-only users may hit a "No pending EQL configuration" error. **Workaround:** Run `stash db push` once before `stash encrypt cutover`, even if you don't use CipherStash Proxy. Decoupling cutover from EQL config for SDK users is tracked separately. ```bash # Run this often — it's the canonical "where am I?" command. @@ -973,7 +905,7 @@ await runBackfill({ }) ``` -Useful when the backfill needs to run in a worker, on a schedule, or alongside an existing job runner. +Useful when the backfill needs to run in a worker, on a schedule, or alongside an existing job runner. (Like the CLI, `runBackfill` currently targets EQL v2 columns — see the note at the top of this section.) ### Invariants the rollout preserves @@ -983,40 +915,60 @@ Useful when the backfill needs to run in a worker, on a schedule, or alongside a - **Re-runs are safe.** Backfill is idempotent (` IS NOT NULL AND _encrypted IS NULL` guards every chunk). `cs_migrations` is append-only. - **Rollback is possible up to cutover.** Until the rename happens, the plaintext column is authoritative; aborting just leaves the encrypted twin partially populated. After cutover, rollback is a manual restore — treat cutover as the one-way door for data. -## Migration from @cipherstash/protect +## Integrations -| `@cipherstash/protect` | `@cipherstash/stack` | Import Path | +| Target | Package / entry point | Skill | |---|---|---| -| `protect(config)` | `Encryption(config)` | `@cipherstash/stack` | -| `csTable(name, cols)` | `encryptedTable(name, cols)` | `@cipherstash/stack/schema` | -| `csColumn(name)` | `encryptedColumn(name)` | `@cipherstash/stack/schema` | -| `LockContext` from `/identify` | `LockContext` from `/identity` | `@cipherstash/stack/identity` | - -All method signatures on the encryption client remain the same. The `Result` pattern is unchanged. +| Drizzle ORM | `@cipherstash/stack-drizzle/v3` — v3 column factories (each `types.*` factory emits its domain as the column's SQL type for `drizzle-kit generate`), schema extraction, auto-encrypting operators (`ops.eq`, `ops.matches`, `ops.contains`, `ops.selector`, `ops.asc`, ...) | `stash-drizzle` | +| Supabase | `encryptedSupabaseV3` from `@cipherstash/stack-supabase` — schema-aware query builder (`eq`, `matches`, `contains`, `selectorEq`/`selectorNe`, ...) that works through PostgREST, including as `anon` | `stash-supabase` | +| Prisma | `@cipherstash/prisma-next` — searchable field-level encryption for Postgres | — | +| DynamoDB | `encryptedDynamoDB` from `@cipherstash/stack/dynamodb` — **v2 schema surface only**, see the exception note under "Legacy: EQL v2" | `stash-dynamodb` | ## Complete API Reference -### EncryptionClient Methods +### TypedEncryptionClient Methods | Method | Signature | Returns | |---|---|---| -| `encrypt` | `(plaintext, { column, table })` | `EncryptOperation` | -| `decrypt` | `(encryptedData)` | `DecryptOperation` | -| `encryptQuery` | `(plaintext, { column, table, queryType?, returnType? })` | `EncryptQueryOperation` | -| `encryptQuery` | `(terms: readonly ScalarQueryTerm[])` | `BatchEncryptQueryOperation` | -| `encryptModel` | `(model, table)` | `EncryptModelOperation>` | -| `decryptModel` | `(encryptedModel)` | `DecryptModelOperation` — resolves to `Decrypted` | -| `bulkEncrypt` | `(plaintexts, { column, table })` | `BulkEncryptOperation` | -| `bulkDecrypt` | `(encryptedPayloads)` | `BulkDecryptOperation` | -| `bulkEncryptModels` | `(models, table)` | `BulkEncryptModelsOperation>` | -| `bulkDecryptModels` | `(encryptedModels)` | `BulkDecryptModelsOperation` — resolves to `Decrypted[]` | - -All operations are thenable (awaitable) and support `.withLockContext()` and `.audit()` chaining. +| `encrypt` | `(plaintext, { table, column })` — plaintext pinned to the column's domain type | `EncryptOperation` | +| `decrypt` | `(encryptedData)` — untyped (no column identity) | `DecryptOperation` | +| `encryptQuery` | `(plaintext, { table, column, queryType?, returnType? })` — queryable columns only; `queryType` constrained to the column's capabilities | `EncryptQueryOperation` | +| `encryptQuery` | `(terms: readonly ScalarQueryTerm[])` — batch form | `BatchEncryptQueryOperation` | +| `encryptModel` | `(model, table)` — schema fields validated against inferred plaintext types | `EncryptModelOperation>` | +| `decryptModel` | `(model, table, lockContext?)` | `Promise, EncryptionError>>` | +| `bulkEncryptModels` | `(models, table)` | `BulkEncryptModelsOperation>` | +| `bulkDecryptModels` | `(models, table, lockContext?)` | `Promise[], EncryptionError>>` | +| `bulkEncrypt` | `(plaintexts, { column, table })` — parity passthrough | `BulkEncryptOperation` | +| `bulkDecrypt` | `(encryptedPayloads)` — parity passthrough | `BulkDecryptOperation` | +| `getEncryptConfig` | `()` | The client's encrypt config | + +The encrypt-side operations are thenable (awaitable) and support `.withLockContext()` and `.audit()` chaining; `decryptModel`/`bulkDecryptModels` return plain promises and take the lock context as an argument. ### Schema Builders ```typescript -encryptedTable(tableName: string, columns: Record) -encryptedColumn(columnName: string) // chainable: .equality(), .freeTextSearch(), .orderAndRange(), .searchableJson(), .dataType() -encryptedField(valueName: string) // for nested encrypted fields (not searchable), chainable: .dataType() +encryptedTable(tableName: string, columns: Record) +types.(dbColumnName: string) +// e.g. types.TextSearch("email"), types.BigintOrd("balance"), types.Json("metadata") +``` + +## Legacy: EQL v2 + +Before the v3 typed schema, encrypted columns were a single composite type (`eql_v2_encrypted`) and query capabilities were enabled per column with chainable builders: + +```typescript +// Legacy v2 surface — for existing deployments only +import { Encryption } from "@cipherstash/stack" +import { encryptedTable, encryptedColumn } from "@cipherstash/stack/schema" + +const users = encryptedTable("users", { + email: encryptedColumn("email").equality().freeTextSearch().orderAndRange(), + metadata: encryptedColumn("metadata").searchableJson(), +}) + +const client = await Encryption({ schemas: [users] }) ``` + +The v2 API — `Encryption` plus the `@cipherstash/stack/schema` builders, the v2 Drizzle/Supabase integrations (`@cipherstash/stack-drizzle`, `encryptedSupabase`), and `stash eql install --eql-version 2` — **still exists and is supported for existing deployments**. Full v2 documentation lives at [cipherstash.com/docs](https://cipherstash.com/docs). Remember: v2 and v3 tables cannot be mixed in one client. (If you are migrating code from the old `@cipherstash/protect` package, its `protect`/`csTable`/`csColumn` names map onto this v2 surface.) + +> **Exception — DynamoDB.** The DynamoDB integration (`encryptedDynamoDB` from `@cipherstash/stack/dynamodb`) **still requires the v2 schema surface** — `encryptedTable`/`encryptedColumn`/`encryptedField` from `@cipherstash/stack/schema`. Do not use v3 `types.*` schemas with it. See the `stash-dynamodb` skill; v3 support is tracked in [cipherstash/stack#657](https://github.com/cipherstash/stack/issues/657). diff --git a/skills/stash-supabase/SKILL.md b/skills/stash-supabase/SKILL.md index ae3c33e9..1ed8a440 100644 --- a/skills/stash-supabase/SKILL.md +++ b/skills/stash-supabase/SKILL.md @@ -1,16 +1,26 @@ --- name: stash-supabase -description: Integrate CipherStash encryption with Supabase using @cipherstash/stack-supabase. Covers the encryptedSupabase wrapper, transparent encryption/decryption on insert/update/select, encrypted query filters (eq, like, ilike, gt/gte/lt/lte, in, or, match), identity-aware encryption, and the complete query builder API. Use when adding encryption to a Supabase project, querying encrypted columns, or building secure Supabase applications. +description: Integrate CipherStash encryption with Supabase using @cipherstash/stack-supabase. Covers the encryptedSupabaseV3 wrapper over native EQL v3 column domains, transparent encryption/decryption on insert/update/select, encrypted query filters (eq, matches, gt/gte/lt/lte, in, or, match), encrypted JSON querying (contains, selectorEq/selectorNe), ordering on encrypted columns, identity-aware encryption, and the complete query builder API. Use when adding encryption to a Supabase project, querying encrypted columns, or building secure Supabase applications. --- # CipherStash Stack - Supabase Integration -Guide for integrating CipherStash field-level encryption with Supabase using the `encryptedSupabase` wrapper. The wrapper provides transparent encryption on mutations and decryption on selects, with full support for querying encrypted columns. +Guide for integrating CipherStash field-level encryption with Supabase using +the `encryptedSupabaseV3` wrapper over native EQL v3 column domains. The +wrapper provides transparent encryption on mutations and decryption on +selects, with full support for querying encrypted columns — equality, range, +fuzzy free-text search, encrypted JSON containment and selectors, and +ordering. + +A legacy EQL v2 wrapper (`encryptedSupabase`) still ships for existing +deployments — see "Legacy: EQL v2" at the end. New projects should use +`encryptedSupabaseV3`. ## When to Use This Skill - Adding field-level encryption to a Supabase project -- Querying encrypted data with Supabase's query builder (eq, like, gt, in, or, etc.) +- Querying encrypted data with Supabase's query builder (eq, matches, gt, in, or, etc.) +- Querying encrypted JSON documents (containment, path selectors) - Inserting, updating, or upserting encrypted data - Using identity-aware encryption (lock contexts) with Supabase - Building applications where sensitive columns need encryption at rest and in transit @@ -24,101 +34,141 @@ npm install @cipherstash/stack @cipherstash/stack-supabase @supabase/supabase-js The Supabase integration ships as its own first-party package, `@cipherstash/stack-supabase`, which depends on `@cipherstash/stack`. Install both. -## Database Schema +## Setup + +**Credentials first:** for local development run `npx stash init` (the +agent-assisted flow — auth, schema, and database end to end) or +`npx stash auth login` (device code flow; no environment variables needed). +CI and production use the `CS_*` machine-credential environment variables — +see the `stash-encryption` skill's Configuration section. + +### 1. Install EQL v3 on the database + +```bash +stash eql install --eql-version 3 --supabase +``` + +Since eql-3.0.0 there is **one** v3 SQL artifact for every target — there is +no separate Supabase variant. The bundle's only superuser-requiring +statements (the ORE operator class/family) skip themselves when the install +role lacks the privilege, and the bundle then disables the ORE-opclass-backed +domains it cannot support. `--supabase` changes one thing: it additionally +applies the role grants for `anon` / `authenticated` / `service_role` to the +two schemas the bundle creates — `eql_v3` (the operator-backing functions) +and `eql_v3_internal` (SEM internals). Without the grants, encrypted queries +fail loudly with a permission error (e.g. `permission denied for schema +eql_v3_internal`). + +No **Exposed schemas** change is needed: the column domains and their +operators live in `public`, so bare `col = term` filters resolve under +Supabase's default PostgREST configuration. Do not expose `eql_v3_internal`. + +### 2. Database schema (per-domain columns) -Encrypted columns must be stored as JSONB in your Supabase database: +Each encrypted column is declared with a concrete `public.eql_v3_*` domain — +the domain encodes both the plaintext type and the column's query +capabilities. There is no extension to enable and no generic `jsonb` column: ```sql CREATE TABLE users ( id SERIAL PRIMARY KEY, - email jsonb NOT NULL, -- encrypted column - name jsonb NOT NULL, -- encrypted column - age jsonb, -- encrypted column (numeric) - role VARCHAR(50), -- regular column (not encrypted) + email public.eql_v3_text_search, -- eq + range + free-text search + amount public.eql_v3_integer_ord, -- eq + range + joined_at public.eql_v3_timestamp_ord, -- eq + range, decrypts to Date + payload public.eql_v3_json, -- encrypted JSON document + role VARCHAR(50), -- regular plaintext column created_at TIMESTAMPTZ DEFAULT NOW() ); ``` -For searchable encryption (equality, range, text search), install the EQL extension: +The `types.*` member name (see declared schemas below) maps to the flat +`public.eql_v3_` domain — strip the `eql_v3_` prefix and PascalCase +each `_`-separated segment: `types.TextEq` → `public.eql_v3_text_eq`, +`types.IntegerOrd` → `public.eql_v3_integer_ord`. The domains use +SQL-standard type names (`integer`, `smallint`, `real`, `double`, `boolean`, +`timestamp`). -```sql -CREATE EXTENSION IF NOT EXISTS eql_v2; +### 3. Initialize the wrapper + +```typescript +import { encryptedSupabaseV3 } from "@cipherstash/stack-supabase" + +// Introspects the database via options.databaseUrl or DATABASE_URL +const es = await encryptedSupabaseV3(supabaseUrl, supabaseKey) +// or wrap an existing client: await encryptedSupabaseV3(supabaseClient, options) + +await es.from("users").insert({ email: "a@b.com", amount: 30 }) +await es.from("users").select("id, email, amount").eq("email", "a@b.com") ``` -## Setup +`encryptedSupabaseV3` **introspects the database at connect time**: it +detects EQL v3 columns by their Postgres domain, derives each column's +encryption config from the domain, and builds the encryption client +internally — there is no client-side schema to hand-maintain. Introspection +needs a direct Postgres connection (`options.databaseUrl`, defaulting to +`DATABASE_URL`), so the factory cannot run in a Worker or the browser. -### 1. Define Encrypted Schema +Options: `{ schemas?, databaseUrl?, config? }` — `config` is the encryption +client config (e.g. `config.authStrategy`, see Authentication below). -```typescript -import { encryptedTable, encryptedColumn } from "@cipherstash/stack/schema" +`from(tableName)` takes only the table name — no schema argument. Column +capabilities come from the introspected domains. -const users = encryptedTable("users", { - email: encryptedColumn("email") - .equality() // eq, neq, in - .freeTextSearch(), // like, ilike - - name: encryptedColumn("name") - .equality() - .freeTextSearch(), - - age: encryptedColumn("age") - .dataType("number") - .equality() - .orderAndRange(), // gt, gte, lt, lte -}) -``` +### 4. Optional declared schemas (compile-time types) -### 2. Initialize Clients +Declaring tables is optional. Passing `schemas` — a record whose keys must +equal each table's name — adds compile-time types and verifies the declared +tables against the database at construction: ```typescript -import { createClient } from "@supabase/supabase-js" -import { Encryption } from "@cipherstash/stack" -import { encryptedSupabase } from "@cipherstash/stack-supabase" - -const supabase = createClient( - process.env.SUPABASE_URL!, - process.env.SUPABASE_ANON_KEY!, -) +import { encryptedTable, types } from "@cipherstash/stack/eql/v3" +import { encryptedSupabaseV3 } from "@cipherstash/stack-supabase" -const encryptionClient = await Encryption({ schemas: [users] }) +const users = encryptedTable("users", { + email: types.TextSearch("email"), // public.eql_v3_text_search — eq + range + free-text + amount: types.IntegerOrd("amount"), // public.eql_v3_integer_ord — eq + range + joined: types.TimestampOrd("joined_at") // public.eql_v3_timestamp_ord — eq + range, decrypts to Date +}) -const eSupabase = encryptedSupabase({ - encryptionClient, - supabaseClient: supabase, +const es = await encryptedSupabaseV3(supabaseUrl, supabaseKey, { + schemas: { users }, }) -``` -### 3. Use the Wrapper +const { data } = await es.from("users").select("id, email, joined").eq("email", "a@b.com") +``` -All queries go through `eSupabase.from(tableName, schema)`: +A declared table gets a typed builder: rows infer each column's plaintext +type (`types.IntegerOrd` → `number`, `types.TimestampOrd` → `Date`), +storage-only columns are excluded from every filter method, `matches()` is +narrowed to match-indexed columns, and `order()` to orderable columns. +Undeclared tables behave exactly as with no `schemas` at all. Every v3 column +is fully described by its `types.*` factory — there are no capability or +tuning chains on v3 columns. -```typescript -const { data, error } = await eSupabase - .from("users", users) - .select("id, email, name") - .eq("email", "alice@example.com") -``` +A JS property may map to a different DB column name +(`joined: types.TimestampOrd("joined_at")`) — filters, selects, and results +are translated automatically, and `date`/`timestamp` columns decrypt to real +`Date` objects. ## Insert (Encrypted Automatically) ```typescript // Single insert -const { data, error } = await eSupabase - .from("users", users) +const { data, error } = await es + .from("users") .insert({ email: "alice@example.com", // encrypted automatically - name: "Alice Smith", // encrypted automatically - age: 30, // encrypted automatically - role: "admin", // not in schema, passed through + amount: 30, // encrypted automatically + role: "admin", // plaintext column, passed through }) .select("id") // Bulk insert -const { data, error } = await eSupabase - .from("users", users) +const { data, error } = await es + .from("users") .insert([ - { email: "alice@example.com", name: "Alice", age: 30, role: "admin" }, - { email: "bob@example.com", name: "Bob", age: 25, role: "user" }, + { email: "alice@example.com", amount: 30, role: "admin" }, + { email: "bob@example.com", amount: 25, role: "user" }, ]) .select("id") ``` @@ -126,105 +176,109 @@ const { data, error } = await eSupabase ## Update (Encrypted Automatically) ```typescript -const { data, error } = await eSupabase - .from("users", users) - .update({ name: "Alice Johnson" }) // encrypted automatically +const { data, error } = await es + .from("users") + .update({ email: "alice@new.example.com" }) // encrypted automatically .eq("id", 1) - .select("id, name") + .select("id, email") ``` ## Upsert ```typescript -const { data, error } = await eSupabase - .from("users", users) +const { data, error } = await es + .from("users") .upsert( - { id: 1, email: "alice@example.com", name: "Alice", role: "admin" }, + { id: 1, email: "alice@example.com", role: "admin" }, { onConflict: "id" }, ) - .select("id, email, name") + .select("id, email") ``` ## Select (Decrypted Automatically) ```typescript -// List query - returns decrypted array -const { data, error } = await eSupabase - .from("users", users) - .select("id, email, name, role") -// data: [{ id: 1, email: "alice@example.com", name: "Alice Smith", role: "admin" }] +// All columns — select('*') (and bare select()) expands to the +// introspected column list +const { data, error } = await es.from("users").select("*") + +// Explicit columns +const { data, error } = await es + .from("users") + .select("id, email, amount, role") +// data: [{ id: 1, email: "alice@example.com", amount: 30, role: "admin" }] // Single result -const { data, error } = await eSupabase - .from("users", users) - .select("id, email, name") +const { data, error } = await es + .from("users") + .select("id, email") .eq("id", 1) .single() -// data: { id: 1, email: "alice@example.com", name: "Alice Smith" } // Maybe single (returns null if no match) -const { data, error } = await eSupabase - .from("users", users) +const { data, error } = await es + .from("users") .select("id, email") .eq("email", "nobody@example.com") .maybeSingle() // data: null ``` -**Important:** You must list columns explicitly in `select()` — using `select('*')` will throw an error. The wrapper automatically adds `::jsonb` casts to encrypted columns so PostgreSQL parses them correctly. - `select()` also accepts an optional second parameter: `select(columns, { head?: boolean, count?: 'exact' | 'planned' | 'estimated' })`. ## Query Filters -All filter values for encrypted columns are automatically encrypted before the query executes. Multiple filters are batch-encrypted in a single ZeroKMS call for efficiency. +All filter values for encrypted columns are automatically encrypted before +the query executes. Filter operands are grouped by column and each column +group takes one `bulkEncrypt` crossing — a query filtering N distinct +encrypted columns makes N ZeroKMS calls, run in parallel. ### Equality Filters ```typescript -// Exact match (requires .equality() on column) +// Exact match (requires an equality-capable domain) .eq("email", "alice@example.com") // Not equal .neq("email", "alice@example.com") -// IN array (requires .equality()) +// IN array .in("email", ["alice@example.com", "bob@example.com"]) -// NULL check (no encryption needed) +// NULL check (no encryption needed). Use this for genuine null checks — a +// null operand passed to eq/neq is not rejected; it is forwarded unencrypted. .is("email", null) ``` -### Text Search Filters +### Free-Text Search (`matches`) ```typescript -// LIKE - case sensitive (requires .freeTextSearch()) -.like("name", "%alice%") - -// ILIKE - case insensitive (requires .freeTextSearch()) -.ilike("name", "%alice%") +// Fuzzy bloom token search (requires a match-indexed domain, +// e.g. text_search / text_match). Matches substrings of >= 3 characters. +.matches("email", "alice") ``` +`like`/`ilike` on an encrypted column are a **deprecated approximate shim** +that delegates to `matches()` — see "Query behaviour on encrypted columns" +below. On plaintext columns they stay real SQL LIKE. The shim is +**runtime-only**: neither the typed nor the untyped v3 builder interface +declares `like`/`ilike`, so in TypeScript the call is a compile error (use +`matches()`) — the shim is reachable only from plain JavaScript or via a cast. + ### Range/Comparison Filters ```typescript -// Greater than (requires .orderAndRange()) -.gt("age", 21) - -// Greater than or equal -.gte("age", 18) - -// Less than -.lt("age", 65) - -// Less than or equal -.lte("age", 100) +// Requires a range-capable domain (e.g. *_ord, text_search) +.gt("amount", 21) +.gte("amount", 18) +.lt("amount", 65) +.lte("amount", 100) ``` ### Match (Multi-Column Equality) ```typescript -.match({ email: "alice@example.com", name: "Alice" }) +.match({ email: "alice@example.com", amount: 30 }) ``` ### OR Conditions @@ -257,8 +311,8 @@ Both forms encrypt values for encrypted columns automatically. ## Delete ```typescript -const { data, error } = await eSupabase - .from("users", users) +const { data, error } = await es + .from("users") .delete() .eq("id", 1) ``` @@ -268,7 +322,7 @@ const { data, error } = await eSupabase These are passed through to Supabase directly: ```typescript -.order("name", { ascending: true }) +.order("email", { ascending: true }) // encrypted columns: see behaviour below .limit(10) .range(0, 9) .csv() @@ -277,251 +331,16 @@ These are passed through to Supabase directly: .returns() ``` -### Ordering by Encrypted Columns - -**`ORDER BY` on encrypted columns is not currently supported** on databases without operator family support (including Supabase). - -Without operator families installed in PostgreSQL, the database cannot sort on `eql_v2_encrypted` columns. This affects all clients — the Supabase JS SDK, Drizzle, raw SQL, and any other ORM. - -**Workaround:** Sort application-side after decrypting the results. - -Operator family support is currently being developed in collaboration with the Supabase and CipherStash teams and will be available in a future release. - -`.order()` on non-encrypted columns works normally. - -## Identity-Aware Encryption - -Chain `.withLockContext()` to tie encryption to a specific user's JWT: - -```typescript -import { LockContext } from "@cipherstash/stack/identity" - -const lc = new LockContext() -const identified = await lc.identify(userJwt) -if (identified.failure) throw new Error(identified.failure.message) -const lockContext = identified.data - -const { data, error } = await eSupabase - .from("users", users) - .insert({ email: "alice@example.com", name: "Alice" }) - .withLockContext(lockContext) - .select("id") -``` - -## Audit Logging - -Chain `.audit()` to attach metadata for ZeroKMS audit logging: - -```typescript -const { data, error } = await eSupabase - .from("users", users) - .select("id, email, name") - .eq("email", "alice@example.com") - .audit({ metadata: { action: "user-lookup", requestId: "abc-123" } }) -``` - -## Complete Example - -```typescript -import { createClient } from "@supabase/supabase-js" -import { Encryption } from "@cipherstash/stack" -import { encryptedSupabase } from "@cipherstash/stack-supabase" -import { encryptedTable, encryptedColumn } from "@cipherstash/stack/schema" - -// Schema -const users = encryptedTable("users", { - email: encryptedColumn("email").equality().freeTextSearch(), - name: encryptedColumn("name").equality().freeTextSearch(), - age: encryptedColumn("age").dataType("number").equality().orderAndRange(), -}) - -// Clients -const supabase = createClient(process.env.SUPABASE_URL!, process.env.SUPABASE_ANON_KEY!) -const encryptionClient = await Encryption({ schemas: [users] }) -const eSupabase = encryptedSupabase({ encryptionClient, supabaseClient: supabase }) - -// Insert -await eSupabase - .from("users", users) - .insert([ - { email: "alice@example.com", name: "Alice", age: 30 }, - { email: "bob@example.com", name: "Bob", age: 25 }, - ]) - -// Query with multiple filters -const { data } = await eSupabase - .from("users", users) - .select("id, email, name, age") - .gte("age", 18) - .lte("age", 35) - .ilike("name", "%ali%") - -// data is fully decrypted: -// [{ id: 1, email: "alice@example.com", name: "Alice", age: 30 }] -``` - -## Response Type - -```typescript -type EncryptedSupabaseResponse = { - data: T | null // Decrypted rows - error: EncryptedSupabaseError | null - count: number | null - status: number - statusText: string -} -``` - -Errors can come from Supabase (API errors) or from encryption operations. Check `error.encryptionError` for encryption-specific failures. - -The full `EncryptedSupabaseError` type: - -```typescript -type EncryptedSupabaseError = { - message: string - details?: string // Supabase error details - hint?: string // Supabase error hint - code?: string // Supabase/PostgreSQL error code - encryptionError?: EncryptionError // CipherStash encryption-specific error -} -``` - -## Filter to Index Mapping - -| Filter Method | Required Index | Query Type | -|---|---|---| -| `eq`, `neq`, `in` | `.equality()` | `'equality'` | -| `like`, `ilike` | `.freeTextSearch()` | `'freeTextSearch'` | -| `gt`, `gte`, `lt`, `lte` | `.orderAndRange()` | `'orderAndRange'` | -| `is` | None | No encryption (NULL/boolean check) | - -## Exported Types - -`@cipherstash/stack-supabase` also exports the following types: - -- `EncryptedSupabaseConfig` -- `EncryptedSupabaseInstance` -- `EncryptedQueryBuilder` -- `PendingOrCondition` -- `SupabaseClientLike` -- `EncryptedSupabaseV3Options`, `EncryptedSupabaseV3Instance`, `TypedEncryptedSupabaseV3Instance`, `EncryptedQueryBuilderV3`, `EncryptedQueryBuilderV3Untyped`, `V3Schemas` (EQL v3) - -## EQL v3 (native `public.eql_v3_*` domains) - -`encryptedSupabaseV3` is the EQL v3 counterpart of `encryptedSupabase`. -Instead of taking a schema, it **introspects the database at connect time**: -it detects EQL v3 columns by their Postgres domain, derives each column's -encryption config from the domain, and builds the encryption client -internally. Columns are stored in their native `public.eql_v3_*` domain (a -`DOMAIN … AS jsonb` with a CHECK) instead of the v2 composite -`eql_v2_encrypted`. - -The query surface matches v2 — same filter methods, `withLockContext`, -`audit` — with one deliberate fork: encrypted free-text search is `matches()` -(fuzzy bloom token search); `contains()` stays native (exact) containment for -plaintext columns; and `like`/`ilike` on an encrypted column are an approximate -shim that delegates to `matches()` (see below). - -### Setup - -```typescript -import { encryptedSupabaseV3 } from "@cipherstash/stack-supabase" - -// Introspects the database via options.databaseUrl or DATABASE_URL -const es = await encryptedSupabaseV3(supabaseUrl, supabaseKey) -// or wrap an existing client: await encryptedSupabaseV3(supabaseClient, options) - -await es.from("users").insert({ email: "a@b.com", amount: 30 }) -await es.from("users").select("id, email, amount").eq("email", "a@b.com") -await es.from("users").select("id, amount").gte("amount", 10).lte("amount", 100) -``` - -`from(tableName)` takes only the table name — no schema argument. Column -capabilities come from the introspected domains. Introspection needs a direct -Postgres connection (`options.databaseUrl`, defaulting to `DATABASE_URL`), so -the factory cannot run in a Worker or the browser. +`order()` works on plaintext columns and on OPE-backed encrypted ordering +columns — see the `order()` bullet in the next section for exactly which +domains qualify. -### Optional declared schemas (compile-time types) - -Declaring tables is optional. Passing `schemas` — a record whose keys must -equal each table's name — adds compile-time types and verifies the declared -tables against the database at construction: - -```typescript -import { encryptedTable, types } from "@cipherstash/stack/eql/v3" -import { encryptedSupabaseV3 } from "@cipherstash/stack-supabase" - -const users = encryptedTable("users", { - email: types.TextSearch("email"), // public.eql_v3_text_search — eq + range + free-text - amount: types.IntegerOrd("amount"), // public.eql_v3_integer_ord — eq + range - joined: types.TimestampOrd("joined_at") // public.eql_v3_timestamp_ord — eq + range, decrypts to Date -}) - -const es = await encryptedSupabaseV3(supabaseUrl, supabaseKey, { - schemas: { users }, -}) - -const { data } = await es.from("users").select("id, email, joined").eq("email", "a@b.com") -``` - -A declared table gets a typed builder: rows infer each column's plaintext -type (`types.IntegerOrd` → `number`, `types.TimestampOrd` → `Date`), -storage-only columns are excluded from every filter method, `matches()` is -narrowed to match-indexed columns, and `order()` to plaintext columns. -Undeclared tables behave exactly as with no `schemas` at all. Every v3 column -is fully described by its `types.*` factory — there are no capability or -tuning chains on v3 columns. - -A JS property may map to a different DB column name -(`joined: types.TimestampOrd("joined_at")`) — filters, selects, and results -are translated automatically, and `date`/`timestamp` columns decrypt to real -`Date` objects. - -### Database schema (per-domain columns) - -Each column is declared with its native domain — the `types.*` member name -maps to the flat `public.eql_v3_` domain (strip the `eql_v3_` prefix and -PascalCase each `_`-separated segment: `types.TextEq` → `public.eql_v3_text_eq`, -`types.IntegerOrd` → `public.eql_v3_integer_ord`). The domains use SQL-standard -type names (`integer`, `smallint`, `real`, `double`, `boolean`, `timestamp`): - -```sql -CREATE TABLE users ( - id SERIAL PRIMARY KEY, - email public.eql_v3_text_search, - amount public.eql_v3_integer_ord, - joined_at public.eql_v3_timestamp_ord -); -``` - -### Install EQL v3 on Supabase - -```bash -stash eql install --eql-version 3 --supabase -``` - -Since eql-3.0.0 there is **one** v3 SQL artifact for every target — there is -no separate Supabase variant. The bundle's only superuser-requiring -statements (the ORE operator class/family) skip themselves when the install -role lacks the privilege, and the bundle then disables the ORE-opclass-backed -domains it cannot support. `--supabase` changes one thing: it additionally -applies the role grants for `anon` / `authenticated` / `service_role` to the -two schemas the bundle creates — `eql_v3` (the operator-backing functions) -and `eql_v3_internal` (SEM internals). Without the grants, encrypted queries -fail loudly with a permission error (e.g. `permission denied for schema -eql_v3_internal`). - -No **Exposed schemas** change is needed for v3: the column domains and their -operators live in `public`, so bare `col = term` filters resolve under -Supabase's default PostgREST configuration. Do not expose `eql_v3_internal`. - -### v3-specific behaviour +## Query behaviour on encrypted columns All envelopes (stored payloads and filter operands) are versioned `v: 3`. -- **`select('*')` (and bare `select()`) works on v3** — it expands to the - introspected column list. (v2 has no column list to expand, so it still - requires explicit columns.) +- **`select('*')` (and bare `select()`) works** — it expands to the + introspected column list. - **Encrypted free-text search is `matches()`, not `contains()`/`like`/`ilike`.** The v3 domains define no LIKE operator — encrypted free-text search is fuzzy bloom-filter token matching (PostgREST `cs` / SQL `@>`): one-sided (a match @@ -537,7 +356,10 @@ All envelopes (stored payloads and filter operands) are versioned `v: 3`. Results are APPROXIMATE — case-insensitive, one-sided, and anchoring is not honored. A pattern with an internal `%` or any `_` cannot be approximated and throws; call `matches()` directly to make the fuzzy intent explicit. On - plaintext columns `like`/`ilike` stay real SQL LIKE. + plaintext columns `like`/`ilike` stay real SQL LIKE. Note the shim is + reachable only at runtime: `like`/`ilike` are absent from the v3 builder + interfaces (typed and untyped alike), so on the TypeScript surface the call + is a compile error — only untyped JavaScript or a cast reaches the shim. - **`matches()` matches substrings.** The search term blooms to its own trigrams, and a row matches when the stored value's bloom contains all of them — so any substring of at least 3 characters (the tokenizer's @@ -573,10 +395,13 @@ All envelopes (stored payloads and filter operands) are versioned `v: 3`. `types.Text`): a filter (including `.match()`) on one is a type error on a declared table, and always a clear runtime error. `.is(column, null)` remains available. -- **Null filter values are rejected** with a pointer to `.is(column, null)` — - a null cannot be encrypted into an operand. +- **Null filter operands are forwarded unencrypted, not rejected.** A null + cannot be encrypted into an operand, so the builder skips encryption and + passes the null through to PostgREST as-is (e.g. a `col=eq.null` filter), + which is rarely what you want. Use `.is(column, null)` for genuine null + checks — the builder does not throw. -### Encrypted JSON querying (`types.Json`) +## Encrypted JSON querying (`types.Json`) A `types.Json("payload")` column (`public.eql_v3_json`) stores an encrypted JSONB document and supports two query forms: @@ -595,8 +420,11 @@ es.from("events").select("id").selectorNe("payload", "$.user.role", "admin") `path` — it compiles to containment of the path-shaped needle (`{user: {role: "admin"}}`), which is the same operation. Paths are dot-notation object keys (`"$.a.b"` or `"a.b"`); array/wildcard steps are - rejected, and values must be scalars (an object operand belongs to - `contains()`). + rejected, and values must be JSON scalars — string, number, or boolean (an + object operand belongs to `contains()`). On Supabase a `Date` or `bigint` + leaf is rejected with a serialization steer: pass a Date as + `date.toISOString()` and a bigint as its string/number form. (The Drizzle + selector accepts Date/bigint directly.) - `selectorNe` matches rows that do NOT carry the value — **including rows where the path is absent entirely, and rows whose document column is SQL NULL** (it compiles to `payload.is.null OR payload.not.cs.`, matching @@ -627,18 +455,217 @@ es.from("events").select("id").selectorNe("payload", "$.user.role", "admin") document, so an accidentally-empty filter would silently return the whole table (the Drizzle adapter rejects the same needle). +## Authentication + +The encryption client authenticates to ZeroKMS through `config.authStrategy`. +Unset, it uses the default **auto** strategy — the `npx stash auth login` +profile in local development (preferred), `CS_*` environment variables in +CI/production — which is fine for service-level encryption. To authenticate **as the end user**, federate their +third-party OIDC JWT (Clerk, Supabase, Auth0, ...) with +`OidcFederationStrategy`: + +```typescript +import { OidcFederationStrategy } from "@cipherstash/stack" +import { encryptedSupabaseV3 } from "@cipherstash/stack-supabase" + +const strategy = OidcFederationStrategy.create( + process.env.CS_WORKSPACE_CRN!, + () => getUserJwt(), // re-invoked on every (re-)federation +) +if (strategy.failure) throw new Error(strategy.failure.error.message) + +const es = await encryptedSupabaseV3(supabaseUrl, supabaseKey, { + config: { authStrategy: strategy.data }, +}) +``` + +Authentication stands on its own — an OIDC-authenticated client runs every +query normally. Binding *data* to the authenticated user is the optional next +step: the lock context. + +## Identity-Aware Encryption (Lock Contexts) + +Bind the data key to a claim from the end user's JWT by chaining +`.withLockContext({ identityClaim })` on a query. This **requires** an +`OidcFederationStrategy`-authenticated client (above) — the claim's value +resolves from the federated JWT; auto/access-key auth has no user JWT to +resolve claims from. Plain authentication never requires a lock context. + +```typescript +const { data, error } = await es + .from("users") + .insert({ email: "alice@example.com" }) + .withLockContext({ identityClaim: ["sub"] }) + .select("id") +``` + +`identityClaim` is an array of JWT claim *names* (`["sub"]`), not values; the same +claim must be used to encrypt and decrypt. `.withLockContext()` also accepts a +`LockContext` instance. + +> **Deprecated: `LockContext.identify()`.** Older code did +> `new LockContext().identify(userJwt)` to fetch a per-operation CTS token. Those +> tokens were removed in `protect-ffi` 0.25 and the fetched token is no longer +> used by encryption. Authenticate with `OidcFederationStrategy` and pass the +> claim directly, as above. + +## Audit Logging + +Chain `.audit()` to attach metadata for ZeroKMS audit logging: + +```typescript +const { data, error } = await es + .from("users") + .select("id, email") + .eq("email", "alice@example.com") + .audit({ metadata: { action: "user-lookup", requestId: "abc-123" } }) +``` + +## Complete Example + +```typescript +import { encryptedTable, types } from "@cipherstash/stack/eql/v3" +import { encryptedSupabaseV3 } from "@cipherstash/stack-supabase" + +// Optional declared schema — compile-time types. Introspection alone +// (no `schemas`) also works. +const users = encryptedTable("users", { + email: types.TextSearch("email"), + amount: types.IntegerOrd("amount"), +}) + +const es = await encryptedSupabaseV3( + process.env.SUPABASE_URL!, + process.env.SUPABASE_ANON_KEY!, + { schemas: { users } }, // databaseUrl defaults to DATABASE_URL +) + +// Insert — values encrypted automatically +await es.from("users").insert([ + { email: "alice@example.com", amount: 30 }, + { email: "bob@example.com", amount: 25 }, +]) + +// Query with multiple filters — operands encrypted automatically +const { data } = await es + .from("users") + .select("id, email, amount") + .gte("amount", 18) + .lte("amount", 35) + .matches("email", "alice") + +// data is fully decrypted: +// [{ id: 1, email: "alice@example.com", amount: 30 }] +``` + +## Response Type + +```typescript +type EncryptedSupabaseResponse = { + data: T | null // Decrypted rows + error: EncryptedSupabaseError | null + count: number | null + status: number + statusText: string +} +``` + +Errors can come from Supabase (API errors) or from encryption operations. Check `error.encryptionError` for encryption-specific failures. + +The full `EncryptedSupabaseError` type: + +```typescript +type EncryptedSupabaseError = { + message: string + details?: string // Supabase error details + hint?: string // Supabase error hint + code?: string // Supabase/PostgreSQL error code + encryptionError?: EncryptionError // CipherStash encryption-specific error +} +``` + +## Filter to Domain Capability Mapping + +The column's `public.eql_v3_*` domain determines which filters it accepts: + +| Filter Method | Works On | +|---|---| +| `eq`, `neq`, `in`, `match()` | Equality-capable domains (`*_eq`, `*_ord`, `text_search`) | +| `matches()` (and the runtime-only `like`/`ilike` shim — absent from the TS interfaces) | Match-indexed domains (`text_match`, `text_search`) | +| `gt`, `gte`, `lt`, `lte` | Range-capable domains (`*_ord`, `text_search`) | +| `contains()`, `selectorEq()`, `selectorNe()` | `eql_v3_json` (encrypted); `contains()` is also native containment on plaintext jsonb/array columns | +| `order()` | OPE-backed ordering domains (plain `*_ord`, `text_ord`, `text_search`) — never `*_ord_ore` | +| `is` | Any column (no encryption; NULL check) | + +Storage-only domains (e.g. `eql_v3_text`, `eql_v3_boolean`) accept no filters +at all — only `.is(column, null)`. + +## Exported Types + +`@cipherstash/stack-supabase` also exports the following types: + +- `EncryptedSupabaseV3Options`, `EncryptedSupabaseV3Instance`, `TypedEncryptedSupabaseV3Instance`, `EncryptedQueryBuilderV3`, `EncryptedQueryBuilderV3Untyped`, `V3Schemas` +- `SupabaseClientLike` +- `EncryptedSupabaseConfig`, `EncryptedSupabaseInstance`, `EncryptedQueryBuilder`, `PendingOrCondition` (legacy EQL v2) + ## Migrating an Existing Column to Encrypted The hard case: a Supabase table that already exists with live data in a plaintext column you want to encrypt. You can't just change the column type — that would drop the data. CipherStash splits this into two named steps with a hard production-deploy gate between them: an **encryption rollout** (schema-add + dual-write code) and an **encryption cutover** (backfill + rename + drop). The `stash-encryption` skill is the canonical reference for the lifecycle; this section walks the Supabase-specific shape. +> **Known gap (EQL v3 backfill).** The `stash encrypt backfill` / `stash encrypt cutover` tooling currently targets EQL v2 (`eql_v2_encrypted`) columns — EQL v3 domain columns are not yet supported end-to-end. Tracked in [cipherstash/stack#648](https://github.com/cipherstash/stack/issues/648). The lifecycle below (rollout → deploy gate → cutover) is the correct shape either way; until #648 lands, run the CLI backfill/cutover commands against a v2 encrypted twin (see "Interim path until #648" just below), or backfill v3 columns with your own script. + > **Using CipherStash Proxy?** If you query encrypted data through [CipherStash Proxy](https://github.com/cipherstash/proxy) instead of the SDK, also run `stash db push` after schema-add and again before cutover to register the encrypted column shape with EQL. > **Runner note.** `stash init` adds `stash` to the project as a dev dependency, so `stash ` runs through whichever package manager the project uses (Bun, pnpm, Yarn, or npm) — examples below show this bare form. Before init has run, prefix with your package manager's one-shot runner: `bunx`, `pnpm dlx`, `yarn dlx`, or `npx`. The CLI's behaviour is identical across all of them. > **Where am I?** Run `stash status` first (substitute the runner per the note above). It shows you which tables/columns are mid-rollout, which are post-deploy, and what the next move is. Re-run after every transition. +### Interim path until #648: the v2 encrypted twin + +**This is the interim EQL v2 path, distinct from the v3 surface the rest of +this skill documents.** `stash encrypt backfill` / `stash encrypt cutover` +only operate on `eql_v2_encrypted` columns today, so to get the CLI-managed +backfill/cutover, make the twin a **v2** column and dual-write it through the +legacy `encryptedSupabase` (v2) wrapper. The lifecycle below (rollout → deploy +gate → cutover) is unchanged — only the twin's column type and the dual-write +client differ: + +```sql +-- schema-add: v2 twin instead of the v3 domain (still nullable) +ALTER TABLE users + ADD COLUMN email_encrypted eql_v2_encrypted; +``` + +```typescript +// Dual-write through the legacy v2 wrapper: hand-written client-side schema, +// two-argument from(table, schema). +import { Encryption } from '@cipherstash/stack' +import { encryptedColumn, encryptedTable } from '@cipherstash/stack/schema' +import { encryptedSupabase } from '@cipherstash/stack-supabase' + +const users = encryptedTable('users', { + email_encrypted: encryptedColumn('email_encrypted').equality().freeTextSearch(), +}) + +const encryptionClient = await Encryption({ schemas: [users] }) +const esV2 = encryptedSupabase({ supabaseClient, encryptionClient }) + +// Dual-write: BOTH the plaintext column and the encrypted twin. +export async function insertUser(email: string) { + return esV2.from('users', users).insert({ + email, // plaintext — keep writing until drop + email_encrypted: email, // v2 twin — encrypted automatically + }) +} +``` + +With the v2 twin in place, the backfill and cutover commands in Step 2 work +as written. Alternatively, keep the twin v3 (as shown in the steps below) and +backfill it with your own script — once #648 lands, the v3 twin becomes the +CLI-supported path end to end. + ### Starting state You have: @@ -671,28 +698,22 @@ Edit the generated file to add an `email_encrypted` column **alongside** `email` ```sql -- supabase/migrations/_add_users_email_encrypted.sql ALTER TABLE users - ADD COLUMN email_encrypted eql_v2_encrypted; -- nullable + ADD COLUMN email_encrypted public.eql_v3_text_search; -- nullable ``` Apply with `supabase db reset` locally or `supabase migration up` against the remote project. -Update the encryption schema to declare the new encrypted column: +No client-side schema change is required — `encryptedSupabaseV3` introspects +the new column's domain at the next client startup. If you use declared +`schemas`, add the column so it is typed: ```typescript -// src/encryption/schema.ts -import { encryptedTable, encryptedColumn } from '@cipherstash/stack/schema' +// src/encryption/schema.ts (optional — compile-time types) +import { encryptedTable, types } from '@cipherstash/stack/eql/v3' export const users = encryptedTable('users', { - email_encrypted: encryptedColumn('email_encrypted') - .freeTextSearch() - .equality(), + email_encrypted: types.TextSearch('email_encrypted'), }) - -// src/encryption/index.ts -import { Encryption } from '@cipherstash/stack' -import { users } from './schema' - -export const encryptionClient = await Encryption({ schemas: [users] }) ``` > **Using CipherStash Proxy?** Register the new encryption config with EQL: @@ -707,31 +728,21 @@ export const encryptionClient = await Encryption({ schemas: [users] }) #### Dual-writing: write to both columns from app code -Find **every** code path that writes to `users.email` and update it to encrypt and also write to `email_encrypted`. The cleanest pattern is to keep the raw `supabase` client for the plaintext write and use the `encryptedSupabase` wrapper for the encrypted write — wrapped in a single function so callers can't forget one half: +Find **every** code path that writes to `users.email` and update it to also write the encrypted twin. With the v3 wrapper this is a single insert: `email` is a plaintext column and passes through unchanged, while `email_encrypted` is a v3 domain column the wrapper encrypts automatically. Wrap it in one function so callers can't forget one half: ```typescript // src/db/users.ts -import { supabase, encrypted } from './clients' -import { users } from '../encryption/schema' +import { es } from './clients' // encryptedSupabaseV3 instance export async function insertUser(email: string) { - // The encryptedSupabase wrapper handles the encryption call for you; - // the plaintext write is a separate `supabase` call so the rollout - // does not change read behaviour for `email` yet. - const ciphertext = await encrypted.encryptValue(email, { - table: users, - column: 'email_encrypted', - }) - if (ciphertext.failure) throw new Error(ciphertext.failure.message) - - return supabase.from('users').insert({ - email, // plaintext — keep writing - email_encrypted: ciphertext.data, // encrypted twin — new + return es.from('users').insert({ + email, // plaintext — keep writing + email_encrypted: email, // encrypted twin — new, encrypted automatically }) } ``` -Same shape for UPDATE: every site that updates `email` must also re-encrypt and update `email_encrypted` in the same statement. +Same shape for UPDATE: every site that updates `email` must also update `email_encrypted` in the same statement. **The dual-write rule.** Every persistence path that mutates this row writes both columns, in the same transaction, on every code branch. Insert sites, update sites, upserts, ON CONFLICT clauses, seeders, fixtures, edge functions, RPC functions, admin actions, background jobs, third-party webhooks — all of them. A single missed branch means rows inserted in production after deploy land in plaintext only, and backfill won't catch them. Grep for every site that touches `users.email` before declaring this step done. @@ -762,22 +773,24 @@ stash encrypt backfill --table users --column email # (CI: pass --confirm-dual-writes-deployed instead.) ``` -Resumable, idempotent, chunked. The CLI walks the table in keyset-pagination order, encrypts each chunk via the encryption client, and writes the ciphertext into `email_encrypted` inside transactions that also checkpoint to `cs_migrations`. SIGINT-safe. +Resumable, idempotent, chunked. The CLI walks the table in keyset-pagination order, encrypts each chunk via the encryption client, and writes the ciphertext into `email_encrypted` inside transactions that also checkpoint to `cs_migrations`. SIGINT-safe. (Remember the known-gap callout above: today this command targets EQL v2 columns — see #648.) If something goes wrong (e.g. you discover the dual-write code wasn't actually live when backfill ran), re-run with `--force` to re-encrypt every row regardless of current state. #### Cutover: rename swap and activate -First, update the encryption schema to the post-cutover shape — the encrypted column will live under the original column name: +First, if you use declared `schemas`, update them to the post-cutover shape — the encrypted column will live under the original column name: ```typescript // src/encryption/schema.ts (post-cutover) export const users = encryptedTable('users', { - email: encryptedColumn('email').freeTextSearch().equality(), + email: types.TextSearch('email'), }) ``` -> **Known gap (SDK-only users):** `stash encrypt cutover` currently requires a pending EQL configuration, which is set by `stash db push`. If you're using the SDK without Proxy, you'll hit a "No pending EQL configuration" error from cutover. **Workaround:** run `stash db push` once before `stash encrypt cutover`. This will be decoupled in a future release — see [issue #447](https://github.com/cipherstash/stack/issues/447). +(Without declared schemas, introspection picks up the renamed column at the next client startup.) + +> **Known gap (SDK-only users):** `stash encrypt cutover` currently requires a pending EQL configuration, which is set by `stash db push`. If you're using the SDK without Proxy, you'll hit a "No pending EQL configuration" error from cutover. **Workaround:** run `stash db push` once before `stash encrypt cutover`. This will be decoupled in a future release (tracked separately). > > **Using CipherStash Proxy?** Re-push the encryption config so EQL has a pending row that points at `email` (no `_encrypted` suffix): > @@ -795,29 +808,29 @@ stash encrypt cutover --table users --column email Inside one transaction it: (1) renames `email` → `email_plaintext` and `email_encrypted` → `email`, (2) promotes the pending EQL config to `active` (and the prior active to `inactive`), (3) records a `cut_over` event in `cs_migrations`. -App code that does `select('email')` now returns ciphertext that must be decrypted via the `encryptedSupabase` wrapper. **This is the moment that breaks read paths if they aren't going through the wrapper.** +App code that does `select('email')` now returns ciphertext that must be decrypted via the `encryptedSupabaseV3` wrapper. **This is the moment that breaks read paths if they aren't going through the wrapper.** -Update read paths to use `encryptedSupabase`: +Update read paths to use the wrapper: ```typescript // Before const { data } = await supabase.from('users').select('email').eq('id', id).single() -// After — encryptedSupabase decrypts transparently -const { data } = await encrypted.from('users').select('email').eq('id', id).single() +// After — the wrapper decrypts transparently +const { data } = await es.from('users').select('email').eq('id', id).single() ``` -For queries that filter on `email`, the `encryptedSupabase` wrapper handles the encrypted operators internally — the call site is the same shape as before (`.eq()`, `.like()`, `.ilike()`, `.gte()`, etc.), but the values are encrypted before reaching the database. See `## Query Filters` above. +For queries that filter on `email`, the wrapper handles the encrypted operators internally — the call site is the same shape as before (`.eq()`, `.matches()`, `.gte()`, etc.), but the values are encrypted before reaching the database. See `## Query Filters` above. #### Drop: remove the plaintext column -Once read paths are routing through `encryptedSupabase` and you're confident reads are decrypting correctly: +Once read paths are routing through the wrapper and you're confident reads are decrypting correctly: ```bash stash encrypt drop --table users --column email ``` -The CLI emits a Supabase migration file with `ALTER TABLE users DROP COLUMN email_plaintext;`. Review and apply with `supabase migration up` (or `supabase db reset` locally). Then remove the dual-write code from app paths — `email_plaintext` is gone; only `email` (encrypted) is written now via `encryptedSupabase`. +The CLI emits a Supabase migration file with `ALTER TABLE users DROP COLUMN email_plaintext;`. Review and apply with `supabase migration up` (or `supabase db reset` locally). Then remove the dual-write code from app paths — `email_plaintext` is gone; only `email` (encrypted) is written now, through the wrapper. ### Inspecting progress at any time @@ -828,3 +841,19 @@ stash encrypt plan # diffs your migrations.json intent vs observed state ``` All three are read-only. + +## Legacy: EQL v2 + +Earlier versions of this integration stored ciphertext in `jsonb` / +composite `eql_v2_encrypted` columns (enabled via `CREATE EXTENSION eql_v2` +or the v2 EQL bundle) and queried them through the `encryptedSupabase({ +supabaseClient, encryptionClient })` factory, which takes a hand-written +client-side schema and a two-argument `from(tableName, schema)`. That surface +still ships in `@cipherstash/stack-supabase` and is unchanged — keep using it +for existing v2 deployments — but it is not the recommended path for new +projects: use `encryptedSupabaseV3`. One current exception: the v2 wrapper is +also the interim dual-write path for CLI-managed `stash encrypt backfill` / +`stash encrypt cutover` until [cipherstash/stack#648](https://github.com/cipherstash/stack/issues/648) +lands — see "Interim path until #648: the v2 encrypted twin" in the migration +section above for a minimal recipe. For the v2 wrapper's full API and +semantics, see the docs at https://cipherstash.com/docs.