Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,7 @@ Each entry that ships in a published release links to the PR that introduced it.

### Changed

- **Query-operand domains renamed to a `query_<name>` prefix AND moved into the `eql_v3` schema (CIP-3442).** Every scalar query twin introduced by the query-operand surface (above) is now `eql_v3.query_<name>` — `query_integer_eq`, `query_text_ord`, `query_timestamp_ord_ope`, … — and the encrypted-JSONB containment needle follows the same convention: `public.jsonb_query` is now `eql_v3.query_jsonb`. Predicates cast accordingly (`WHERE col = $1::eql_v3.query_integer_eq`; `WHERE doc @> $1::eql_v3.query_jsonb`), the `eql-bindings` `DomainType::sql_domain` strings, `QueryPayload::parse` domain names, and the exported JSON Schema file names (`schema/v3/query_<name>.json`) all carry the new names, and `from_v2_query` / `from_v2_query_typed` target them. This supersedes the `<name>_query` naming in the earlier `[Unreleased]` entries; the old names shipped only in 3.0.0 pre-releases. **Why the prefix:** alphabetical type listings interleaved never-a-column-type query operands with the actual column types (`integer_eq` next to `integer_eq_query`); the shared `query_` prefix sorts every query operand together. **Why the schema move:** query operands are never valid column types, so they don't belong in `public` — the column-type namespace whose survive-schema-drop rationale (dropping EQL-owned schemas must not drop application columns) doesn't apply to them. In `eql_v3` they are versioned with the rest of the public API surface, are uninstalled with it (a column misusing a query domain is dropped by the uninstaller's CASCADE — pinned by the uninstall suite), and casting a query operand requires the same `USAGE ON SCHEMA eql_v3` a caller already needs for the extractors and comparison wrappers. See [U-002](docs/upgrading/v3.0.md#u-002-query-operand-domains-are-eql_v3query_name) in the 3.0 upgrade guide. ([CIP-3442](https://linear.app/cipherstash/issue/CIP-3442))
- **The `eql_v3` tier's JSON envelope version is now `v: 3` (was `v: 2`).** Every `eql_v3` domain CHECK — the generated scalar families and the hand-written `eql_v3.json` SteVec document domain — now pins `VALUE->>'v' = '3'`, and the canonical payload bindings (`SchemaVersion` in `eql-bindings`, the emitted TypeScript alias, and the JSON Schema `const`) accept exactly `3`, rejecting the legacy `2` at the type boundary. The v3 tier previously carried the v2 wire version for continuity; with the tier now diverging from the legacy wire (the new `op` term), the envelope version matches the schema generation. The legacy `eql_v2` wire documentation (`docs/reference/schema/eql-payload-v2.*.schema.json` and its validation tests) is unchanged and stays `v: 2`. **Compatibility:** payloads produced for the v3 tier must now carry `v: 3` — a cipherstash-client that emits `v: 2` cannot insert into `eql_v3` domain columns until it is updated to emit the v3 envelope. See [U-001](docs/upgrading/v3.0.md#u-001-eql_v3-payloads-carry-v-3) in the 3.0 upgrade guide. ([#340](https://github.com/cipherstash/encrypt-query-language/pull/340))
- **Scalar encrypted-domain types are now defined in a Rust catalog, not TOML manifests; the Python codegen toolchain is removed.** Adding a scalar encrypted-domain type (`int4`, `int8`, …) is now one row in `eql-scalars::CATALOG` (`crates/eql-scalars/src/lib.rs`) instead of authoring `tasks/codegen/types/<T>.toml`. `mise run build` regenerates the gitignored SQL surface via `cargo run -p eql-codegen` (Rust, std-only) rather than the Python generator. The catalog row's `Fixture` list is the single source of truth for that type's plaintext fixtures: the SQLx test matrix reads it directly as a compile-time-materialised const (`eql_scalars::INT4_VALUES` / `INT2_VALUES`, `ScalarType::FIXTURE_VALUES`), so there is no longer a generated, committed `tests/sqlx/src/fixtures/<T>_values.rs` — a Rust source of truth no longer round-trips through generated Rust. The shipped SQL is unchanged — `release/*.sql` is byte-identical across the cutover — so there is no change for callers installing EQL; this only affects contributors who extend the scalar domain families. The `python` mise tool, the `pytest`-based `test:codegen` (now `cargo test -p eql-scalars -p eql-codegen`), the per-type `mise run codegen:domain` tasks, and the per-type `tests/sqlx/snapshots/<T>_matrix_tests.txt` baselines (collapsed into one catalog-reconciled `tests/sqlx/snapshots/matrix_tests.txt`) are gone. Why: a single compiler-validated source of truth shared by the generator and the SQLx test harness, and one fewer toolchain in the build/test path — building and testing EQL no longer needs Python (Python remains only for the separate docs-markdown tooling). ([#252](https://github.com/cipherstash/encrypt-query-language/pull/252))
- **Scalar encrypted-domain types are now defined in a Rust catalog, not TOML manifests; the Python codegen toolchain is removed.** Adding a scalar encrypted-domain type (`integer`, `bigint`, …) is now one row in `eql-scalars::CATALOG` (`crates/eql-scalars/src/lib.rs`) instead of authoring `tasks/codegen/types/<T>.toml`. `mise run build` regenerates the gitignored SQL surface via `cargo run -p eql-codegen` (Rust, std-only) rather than the Python generator. The catalog row's `Fixture` list is the single source of truth for that type's plaintext fixtures: the SQLx test matrix reads it directly as a compile-time-materialised const (`eql_scalars::INT4_VALUES` / `INT2_VALUES`, `ScalarType::FIXTURE_VALUES`), so there is no longer a generated, committed `tests/sqlx/src/fixtures/<T>_values.rs` — a Rust source of truth no longer round-trips through generated Rust. The shipped SQL is unchanged — `release/*.sql` is byte-identical across the cutover — so there is no change for callers installing EQL; this only affects contributors who extend the scalar domain families. The `python` mise tool, the `pytest`-based `test:codegen` (now `cargo test -p eql-scalars -p eql-codegen`), the per-type `mise run codegen:domain` tasks, and the per-type `tests/sqlx/snapshots/<T>_matrix_tests.txt` baselines (collapsed into one catalog-reconciled `tests/sqlx/snapshots/matrix_tests.txt`) are gone. Why: a single compiler-validated source of truth shared by the generator and the SQLx test harness, and one fewer toolchain in the build/test path — building and testing EQL no longer needs Python (Python remains only for the separate docs-markdown tooling). ([#252](https://github.com/cipherstash/encrypt-query-language/pull/252))
Expand Down
4 changes: 2 additions & 2 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ This is the **Encrypt Query Language (EQL)** - a PostgreSQL extension for search
### Directory Structure
- `src/` - contains only the self-contained `v3` surface (the modular `eql_v2` component directories were removed in 3.0.0)
- `crates/` - Rust workspace: `eql-domains` (the catalog), `eql-codegen` (SQL/bindings generator), `eql-bindings` (payload bindings), `eql-tests-macros`
- `src/v3/` - Self-contained `eql_v3` / `eql_v3_internal` surface: `src/v3/schema.sql` (creates both schemas), forked `src/v3/crypto.sql` / `src/v3/common.sql`, hand-written SEM index-term types under `src/v3/sem/` (`hmac_256`, `ore_block_256`) — now created in `eql_v3_internal` — the generated scalar encrypted-domain families under `src/v3/scalars/<T>/` (user-column domains in `public`; extractors **and the supported comparison wrappers** in `eql_v3`; only the blockers and aggregate state functions in `eql_v3_internal`; plus the shared blocker `src/v3/scalars/functions.sql`), and the hand-written encrypted-JSONB (SteVec) surface under `src/v3/jsonb/` (`types.sql`, `functions.sql`, `operators.sql`, `aggregates.sql`, `blockers.sql` — the `public.json` / `public.jsonb_entry` / `public.jsonb_query` domains and CHECK validators live in `public`; typed operators, `jsonb_entry` comparison wrappers, the containment engine (`ste_vec_contains`), and raw-jsonb GIN helpers (`jsonb_array` / `jsonb_contains` / `jsonb_contained_by`) live in `eql_v3`; only the `is_ste_vec_array` helper and aggregate state functions live in `eql_v3_internal`)
- `src/v3/` - Self-contained `eql_v3` / `eql_v3_internal` surface: `src/v3/schema.sql` (creates both schemas), forked `src/v3/crypto.sql` / `src/v3/common.sql`, hand-written SEM index-term types under `src/v3/sem/` (`hmac_256`, `ore_block_256`) — now created in `eql_v3_internal` — the generated scalar encrypted-domain families under `src/v3/scalars/<T>/` (user-column domains in `public`; extractors **and the supported comparison wrappers** in `eql_v3`; only the blockers and aggregate state functions in `eql_v3_internal`; plus the shared blocker `src/v3/scalars/functions.sql`), and the hand-written encrypted-JSONB (SteVec) surface under `src/v3/jsonb/` (`types.sql`, `functions.sql`, `operators.sql`, `aggregates.sql`, `blockers.sql` — the `public.json` / `public.jsonb_entry` column domains and their CHECK validators live in `public`, while the containment needle `eql_v3.query_jsonb` — a query operand, never a column type — lives in `eql_v3` (CIP-3442); typed operators, `jsonb_entry` comparison wrappers, the containment engine (`ste_vec_contains`), and raw-jsonb GIN helpers (`jsonb_array` / `jsonb_contains` / `jsonb_contained_by`) live in `eql_v3`; only the `is_ste_vec_array` helper and aggregate state functions live in `eql_v3_internal`)
- `tasks/` - mise task scripts
- `tests/sqlx/` - Rust/SQLx test framework (PostgreSQL 14-17 support)
- `release/` - Generated SQL installation files
Expand All @@ -63,7 +63,7 @@ This is the **Encrypt Query Language (EQL)** - a PostgreSQL extension for search

### Encrypted-Domain Types

`src/v3/scalars/` holds the generated **encrypted-domain type families** — jsonb-backed PostgreSQL domains in the **`public` schema**, one domain per operator/index capability (`public.<T>` storage-only, `public.<T>_eq`, `public.<T>_ord`). The domains are `public.integer`, `public.integer_eq`, `public.integer_ord`, `public.integer_ord_ore`; their extractors (`eql_v3.eq_term`, `eql_v3.ord_term`), aggregates (`eql_v3.min`/`max`), **and the supported comparison wrappers** (`eq`/`neq`/`lt`/`lte`/`gt`/`gte`/`contains`/`contained_by`) all live in **`eql_v3`** — the wrappers are public so every operator has a callable function equivalent (Supabase/PostgREST). Only the **blockers** (for unsupported operators — they just raise), the **aggregate state functions**, and the SEM index-term types the extractors/wrappers return and construct (`eql_v3_internal.hmac_256`, `eql_v3_internal.ore_block_256`) live in **`eql_v3_internal`** — hand-written under `src/v3/sem/`, schema-qualified via the codegen's `INTERNAL_SCHEMA` constant for the generated surface (the codegen's `SCHEMA` constant qualifies the public wrappers; the `operator_entry` renderer picks the backing function's schema by whether the operator is supported) — so the whole v3 surface (both schemas together) is self-contained (no `eql_v2.<symbol>` appears anywhere in v3 SQL; CI gates this via `mise run test:self_contained_v3` and the self-contained `release/cipherstash-encrypt.sql` installer). `public.integer` (PR #239, supersedes #225) is the reference scalar implementation; the catalog now generates a full surface for `integer`, `smallint`, `bigint`, `date`, `timestamp`, `numeric`, `text`, `boolean`, `real`, and `double`, all following this materializer pattern. `jsonb` is a `CATALOG` family too, but a permanently **hand-written**, not generated, one: its three domains (`public.json` document, `public.jsonb_entry`, `public.jsonb_query`) carry `Shape::SteVecDocument` / `SteVecEntry` / `SteVecQuery` (the `Shape` enum in `crates/eql-domains/src/lib.rs`) instead of `Shape::Scalar`, and their SQL lives under `src/v3/jsonb/` rather than `src/v3/scalars/<T>/` — `eql-codegen` renders SQL only for `scalar_families()` (the `Shape::Scalar` rows), so the fixed-envelope ordered-scalar materializer described below never touches `jsonb`. This is a deliberate, permanent split, not a gap awaiting a future generator.
`src/v3/scalars/` holds the generated **encrypted-domain type families** — jsonb-backed PostgreSQL domains in the **`public` schema**, one domain per operator/index capability (`public.<T>` storage-only, `public.<T>_eq`, `public.<T>_ord`), plus an `eql_v3.query_<T>_<cap>` **query-operand twin** per term-bearing domain (index-terms-only, no ciphertext `c`; in `eql_v3`, not `public`, because a query operand is never a column type — CIP-3442). The domains are `public.integer`, `public.integer_eq`, `public.integer_ord`, `public.integer_ord_ore`; their extractors (`eql_v3.eq_term`, `eql_v3.ord_term`), aggregates (`eql_v3.min`/`max`), **and the supported comparison wrappers** (`eq`/`neq`/`lt`/`lte`/`gt`/`gte`/`contains`/`contained_by`) all live in **`eql_v3`** — the wrappers are public so every operator has a callable function equivalent (Supabase/PostgREST). Only the **blockers** (for unsupported operators — they just raise), the **aggregate state functions**, and the SEM index-term types the extractors/wrappers return and construct (`eql_v3_internal.hmac_256`, `eql_v3_internal.ore_block_256`) live in **`eql_v3_internal`** — hand-written under `src/v3/sem/`, schema-qualified via the codegen's `INTERNAL_SCHEMA` constant for the generated surface (the codegen's `SCHEMA` constant qualifies the public wrappers; the `operator_entry` renderer picks the backing function's schema by whether the operator is supported) — so the whole v3 surface (both schemas together) is self-contained (no `eql_v2.<symbol>` appears anywhere in v3 SQL; CI gates this via `mise run test:self_contained_v3` and the self-contained `release/cipherstash-encrypt.sql` installer). `public.integer` (PR #239, supersedes #225) is the reference scalar implementation; the catalog now generates a full surface for `integer`, `smallint`, `bigint`, `date`, `timestamp`, `numeric`, `text`, `boolean`, `real`, and `double`, all following this materializer pattern. `jsonb` is a `CATALOG` family too, but a permanently **hand-written**, not generated, one: its three domains (`public.json` document, `public.jsonb_entry`, `eql_v3.query_jsonb`) carry `Shape::SteVecDocument` / `SteVecEntry` / `SteVecQuery` (the `Shape` enum in `crates/eql-domains/src/lib.rs`) instead of `Shape::Scalar`, and their SQL lives under `src/v3/jsonb/` rather than `src/v3/scalars/<T>/` — `eql-codegen` renders SQL only for `scalar_families()` (the `Shape::Scalar` rows), so the fixed-envelope ordered-scalar materializer described below never touches `jsonb`. This is a deliberate, permanent split, not a gap awaiting a future generator.

Adding a scalar encrypted-domain type is one row in the Rust catalog `eql-domains::CATALOG` (`crates/eql-domains/src/lib.rs`): a `DomainFamily` giving the type `name` (e.g. `bigint`), its `ScalarKind` (the `kind` field), the `Domain`s mapping each generated (bare) domain name to its fixed index `Term`s (`eq => [Hm]`, `ord`/`ord_ore => [Ore]`), and the `Fixture` value list. Term capabilities are fixed in the `Term` enum's `impl` methods (with unit tests): `Hm` provides equality, and `Ore` provides equality plus ordering. There is no TOML manifest and no Python — the catalog is the source of truth, validated by the compiler (an undefined term or unknown scalar is a compile error) plus catalog `#[test]`s. `mise run build` runs `cargo run -p eql-codegen`, which regenerates the scalar SQL surface into `src/v3/scalars/<T>/` from `CATALOG` at the start of every build; that surface includes supported comparison wrappers plus blockers for native `jsonb` operators that would otherwise be reachable through domain fallback. `cargo run -p eql-codegen` regenerates every type at once (the same call `mise run build` uses; there is no per-type codegen task). The generated `*_types.sql` / `*_functions.sql` / `*_operators.sql` / `*_aggregates.sql` files are **committed in place** under `src/v3/scalars/<T>/` and drift-gated by `mise run codegen:parity` (regenerate in place + `git diff` + untracked check — the same regenerate-and-diff pattern `types:check` uses for the committed bindings). They are still machine-generated: change the catalog and rebuild, never hand-edit (CI fails on drift). The per-type plaintext fixture lists the SQLx matrix consumes are **not** a generated file — they are materialised from each `CATALOG` row at compile time as `eql_domains::INT4_VALUES` / `INT2_VALUES` (the `int_values!` macro) and read directly by `ScalarType::FIXTURE_VALUES`; a Rust source of truth no longer round-trips through a committed generated `.rs`. Generated SQL carries a `-- AUTOMATICALLY GENERATED FILE` header (the project-wide marker `docs:validate` greps on); change the catalog and rebuild, never hand-edit. Hand-written SQL beyond the fixed surface goes in `src/v3/scalars/<T>/<T>_extensions.sql` with no auto-generated header and explicit `-- REQUIRE:` edges — that file IS committed. `jsonb` (see above) never enters this pipeline: every consumer here (`generate_all`, `list-types`, the SQLx matrix) iterates `eql_domains::scalar_families()`, which excludes any non-`Shape::Scalar` row, so `jsonb` is invisible to the materializer and the matrix by construction, not by a per-type exception.

Expand Down
2 changes: 1 addition & 1 deletion SUPABASE.md
Original file line number Diff line number Diff line change
Expand Up @@ -181,7 +181,7 @@ the `eql_v3.jsonb_path_*` helper functions, all without operator classes:

```sql
-- Document containment (GIN-indexable on Supabase)
SELECT * FROM orders WHERE data_encrypted @> $1::public.jsonb_query;
SELECT * FROM orders WHERE data_encrypted @> $1::eql_v3.query_jsonb;

-- Field access (selector is the deterministic selector hash, typed as text)
SELECT data_encrypted -> '<selector>'::text FROM orders;
Expand Down
Loading
Loading