Mount retention routes so a host can offer forget and purge (CL-6288)

tombstoneDocument, hardDeleteDocument and setRetentionClass were implemented, tested, exported — and reachable by no host over HTTP. A memory UI could show a user everything the system remembered about them and offer no way to take any of it back.

Three routes, each guarded on the same grant model as add/search and usable by browser and machine callers alike:
  POST .../memory/documents/:documentId/forget           memory:forget
  POST .../memory/documents/:documentId/purge            memory:purge
  POST .../memory/versions/:versionId/retention-class    memory:forget

forget and purge stay distinct routes with distinct grant actions, never a boolean flag, so a host can wire "forget this" broadly while restricting the irreversible one. Ownership is enforced by resolving the document's true creator before any destructive call: a share grant that lets a peer read a document never lets them destroy it, and grants only ever carry action "search".

Honest about what forget does: it stops the document appearing in search and overwrites chunk text with [redacted]. The content is destroyed, not archived, and there is no un-tombstone verb — earlier wording claiming reversibility was wrong and is corrected here.

Grant metadata renamed from source to installHint, because nothing reads it at request time. ARCHITECTURE.md now states plainly that this package has two authorization mechanisms — grant tags for capability and visibility, and the imperative ownership check as the sole source of truth for whose document it is.

sweepEphemeral is deliberately not routed: it is tenant-wide maintenance with no per-caller ownership, so a host schedules it against the in-process plane.

Known gap, filed: CL-6294 — purge deletes the version rows carrying deprecatedAt/reason, so it destroys its own audit trail.

Author: TheGreatAxios

AGENTS.md

@@ -22,7 +22,8 @@ CI runs `typecheck` + `test` — both must pass before any push.
 - `src/index.ts` — public surface: `createMemory` (optional `app` registers HTTP), `registerMemoryRoutes`
 
 - `src/mount-config.ts` / `src/config.ts` — mount config + engine config
-- `src/routes/` — Hono routes (`add`, `search`, `list`)
+- `src/routes/` — Hono routes (`add`, `search`, `list`, `feed`, retention
+  `forget`/`purge`/`retention-class`)
 - `src/tools/` — Interchange `defineTool` factories (`@corbits/memory/tools`);
   HTTP clients for mounted routes (env credentials; no in-process plane)
 - `src/services/` — capture / search / transform internals (not public verbs)

ARCHITECTURE.md

@@ -56,6 +56,20 @@ helpers are optional multi-writer / backfill — not the primary path.
   anything from the request body, is what `grantGuard` authorizes.
 - **Grants delegate to the host.** Pass `grantStore` + `conditionRegistry`;
   routes use `createRequireGrant("memory", action)`.
+- **Two authorization mechanisms, not one — know which is source of truth
+  for what.** (1) Grant tags decide *capability* (may this principal call
+  `add`/`search`/`forget`/`purge` at all — `requireGrant`) and *visibility*
+  (which documents a principal may see — `accessTags` + `canAccessDocument`
+  in `grant-tags.ts`, where a share grant legitimately widens who can find a
+  document). (2) A separate, imperative **ownership** check — the creator
+  lookup in `services/retention-ownership.ts`, called from `memory.ts` —
+  decides who may *forget or purge* a specific document, and is the sole
+  source of truth for "whose document is this": it is never derived from
+  grant tags and a share grant never satisfies it. `MemoryGrantRequirement.
+  installHint` (`grant-requirements.ts`) looks adjacent to this but is not:
+  it is advisory metadata for install tooling sizing a capability grant,
+  read by nothing at request time. Do not extend mechanism (1) expecting it
+  to cover ownership — extend `retention-ownership.ts` instead.
 - **Dependencies**: `@intx/hub-api`, `@intx/authz`, `@intx/log`, Hono, Drizzle,
   arktype, `postgres`, `hono-openapi`. LGPL-2.1 — see `LICENSE`.
 
@@ -85,9 +99,23 @@ exposes the same three verbs.
   authority/recency → MMR); optional live `SourceProvider` merge (fail-soft).
 - `GET /api/tenants/:tenantId/memory/list` — recent documents, same grant-tag filter as local
   search.
-
-Returns an in-process `Memory` (`add`, `search`, `list`, `close`) for host
-workers and ingestion modules that already resolved identity.
+- `POST /api/tenants/:tenantId/memory/documents/:documentId/forget` — tombstone
+  (grant `memory:forget`; creator-only, see below).
+- `POST /api/tenants/:tenantId/memory/documents/:documentId/purge` — hard
+  delete (grant `memory:purge`; creator-only; irreversible).
+- `POST /api/tenants/:tenantId/memory/versions/:versionId/retention-class` —
+  set retention class (grant `memory:forget`; creator-only).
+
+Forget and purge are deliberately separate routes and separate grant actions
+(never one route with a boolean flag) — a host wiring a "forget this" button
+cannot accidentally wire up permanent deletion. `sweepEphemeral` (TTL
+auto-deprecation) is **not** HTTP-routed: it is a maintenance sweep a host
+schedules on its own cron, not a user action; call it in-process against the
+returned `Memory`. See docs/RETENTION.md.
+
+Returns an in-process `Memory` (`add`, `search`, `list`, `close`, plus the
+optional retention writes) for host workers and ingestion modules that
+already resolved identity.
 
 **Agent tools live in this package** as thin HTTP clients
 (`@corbits/memory/tools` / `interchange.tools`): `defineTool` factories that

CHANGELOG.md

@@ -9,6 +9,18 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Added
 
+- Retention HTTP routes (CL-6288): `POST …/memory/documents/:documentId/forget`
+  (tombstone, grant `memory:forget`), `POST …/memory/documents/:documentId/purge`
+  (hard delete, grant `memory:purge`), and
+  `POST …/memory/versions/:versionId/retention-class` (grant `memory:forget`).
+  Forget and purge are separate routes with separate grant actions — never one
+  route with a boolean flag — and both are refused with 403 unless the caller
+  is the document/version's creator, independent of any share grant that lets
+  them merely see it. `sweepEphemeral` stays off the HTTP surface (maintenance
+  sweep, not a user action); a host schedules it on its own cron against the
+  in-process `Memory`. New `memory:forget` / `memory:purge` grant requirements
+  (`source: "creator"`) and `capabilityIdsForSurface()` so distiller/tools
+  installs no longer pick up routes-only capabilities by accident.
 - `RouteDeps.callerResolver` / `createMemory({ callerResolver })` — an
   optional host-supplied resolver from a request to a `{ tenantId,
   principalId }` scope, for a caller that never goes through the host's

IMPLEMENTATION.md

@@ -633,8 +633,13 @@ surface, or a migrating host silently loses them.
 | `POST /api/tenants/:tenantId/memory/add` | `add` | `{ title, text, access_tags?, share? }` | `200 { documentId, versionId }`; `400` on validation |
 | `POST /api/tenants/:tenantId/memory/search` | `search` | `{ query, limit?, kinds?, entity_ids?, sources?, includeEvidence? }` (limit 1–50; `kinds`/`entity_ids`/`sources` narrow retrieval before fusion; unset or `[]` = unfiltered; `includeEvidence` adds a short evidence string when true) | `200 { items[], evidence?, degraded? }`; `400` on bad input |
 | `GET /api/tenants/:tenantId/memory/list` | `search` | query `?limit=` (1–100, string on the wire) | `200 { events: [{ at, title, source, tenantId, principalId }] }` — durable recent documents for the caller's scope, filtered with grant-tag access (`canAccessDocument`). One event per document (active live version). |
+| `GET /api/tenants/:tenantId/memory/feed` | `search` | query `?after=&limit=&exclude_generator=` | `200 { entries[], nextCursor }` — cursor pull of new live versions. See `docs/FEED.md`. |
+| `POST /api/tenants/:tenantId/memory/documents/:documentId/forget` | `forget` | `{ reason? }` | `200 { documentId, versions }`; `403` unless caller is the document's creator; `404` unknown document. Tombstones — content is redacted, not archived; see docs/RETENTION.md. |
+| `POST /api/tenants/:tenantId/memory/documents/:documentId/purge` | `purge` | none | `200 { documentId, deleted, reason? }`; `403` unless caller is the document's creator; `404` unknown document. Hard-deletes the row — irreversible; refused while a `durable` version is untombstoned. |
+| `POST /api/tenants/:tenantId/memory/versions/:versionId/retention-class` | `forget` | `{ retention_class }` | `200 { versionId, documentId, status }`; `400` invalid class; `403` unless caller is the version's creator; `404` unknown version. |
 
-`registerMemoryRoutes` and `createMemory({ app })` register the three HTTP routes.
+`registerMemoryRoutes` and `createMemory({ app })` register these seven HTTP
+routes (add, search, list, feed, forget, purge, retention-class).
 Agent tools ship in this package as Interchange `defineTool` factories
 (`@corbits/memory/tools` / `interchange.tools`): thin HTTP clients that call the
 mounted routes with install env (`memoryBaseUrl`, `memoryTenantId`,
@@ -645,7 +650,10 @@ protection — the client has no default timeout. OpenAPI→MCP remains an optio
 host bridge. The plane surface is `add` / `search` / `list` / `close`, plus
 optional transform methods when backed by the engine DocumentStore
 (`createTransformConfig`, `listTransformConfigs`, `runTransform`,
-`promoteGeneration`, `demoteGeneration`). Inference stays on the host.
+`promoteGeneration`, `demoteGeneration`) and optional retention methods
+(`tombstoneDocument`, `hardDeleteDocument`, `setRetentionClass`,
+`sweepEphemeral`, `deprecateVersion`) — see docs/RETENTION.md. Inference stays
+on the host.
 
 ### Share materialization (CL-5873)
 

docs/RETENTION.md

@@ -7,7 +7,7 @@ Versions carry a **retention class** orthogonal to temporal ranking class
 | --- | --- |
 | `durable` | Long-lived claims; hard-delete blocked until tombstoned |
 | `standard` | Default working memory |
-| `ephemeral` | Short TTL; sweeper hard-deletes past `valid_until` (or 7d from `ingested_at`) |
+| `ephemeral` | Short TTL; sweeper deprecates past `valid_until` (or 7d from `ingested_at`) — hard delete is a separate explicit step |
 | `source_only` | Keep raw capture; derived versions may be dropped by host policy |
 
 Schema: `memory.version.retention_class` (migration `0007_retention.sql`).
@@ -21,7 +21,7 @@ CHECK constraint `version_retention_class_check` stays lockstep with
 | Deprecate | `memory.deprecateVersion` | `status=deprecated`, `deprecated_at` / reason |
 | Tombstone | `memory.tombstoneDocument` | All active/deprecated/superseded versions → `tombstoned`; chunk text redacted to `[redacted]` |
 | Hard delete | `memory.hardDeleteDocument` | Deletes document row (cascade); **refuses** if any non-tombstoned version is `durable` |
-| Sweep | `memory.sweepEphemeral` | Auto-deprecates ephemeral versions past `valid_until` (or 7d from `ingested_at`); host schedules, core is cron-free |
+| Sweep | `memory.sweepEphemeral` | Auto-**deprecates** (never deletes) ephemeral versions past `valid_until` (or 7d from `ingested_at`); host schedules, core is cron-free |
 | Set class | `memory.setRetentionClass` | Update `retention_class` on a version |
 
 Search and feed exclude non-active (and non-superseded for feed) rows by
@@ -30,3 +30,46 @@ versions intentionally (ops / audit). Hard-delete is a separate explicit
 verb — TTL never hard-deletes.
 
 Service module: `src/services/retention.ts`.
+
+## HTTP surface (CL-6288)
+
+| Route | Grant action | Plane verb |
+| --- | --- | --- |
+| `POST …/memory/documents/:documentId/forget` | `memory:forget` | `tombstoneDocument` |
+| `POST …/memory/documents/:documentId/purge` | `memory:purge` | `hardDeleteDocument` |
+| `POST …/memory/versions/:versionId/retention-class` | `memory:forget` | `setRetentionClass` |
+
+`deprecateVersion` and `sweepEphemeral` have no route (see below).
+
+**Tombstone vs. hard delete stay distinct verbs, distinct grants.** A UI
+offering "forget this" must never be one flag away from "shred this" by
+accident. `forget` (tombstone) is **not** an undo-able action: the document
+stops appearing in search/feed and its chunk text is overwritten with
+`[redacted]` — the original content does not survive, there is no
+un-tombstone/restore verb, and only version metadata (status, timestamps,
+retention class) remains for audit. `purge` (hard delete) goes further and
+removes the document row itself; it has its own grant action and is refused
+outright while a `durable`-class version on the document is untombstoned. The
+distinction that matters is *what's still queryable*: after `forget` a
+document row and its metadata still exist (for audit) but its content is
+gone; after `purge` nothing does. A host can grant `forget` broadly (every
+user gets a "forget this" button) while keeping `purge` to an operator role —
+but should not describe `forget` to end users as reversible.
+
+**Ownership, not just visibility.** `memory:search`/a document's `accessTags`
+say who can *see* a document — never who may forget or purge it. Every
+retention route additionally checks that the caller is the document's
+creator (`created_by_principal_id` — the document's first version for
+`forget`/`purge`, the specific version's own creator for `retention-class`),
+independent of any share grant. A peer who can search a shared document gets
+403 on `forget`/`purge`/`retention-class` for it. See
+`src/services/retention-ownership.ts` and the ownership tests in
+`src/memory.test.ts` / `src/routes/routes.test.ts`.
+
+**`sweepEphemeral` stays off the HTTP surface.** It is a maintenance sweep —
+"deprecate every ephemeral version past its TTL for this tenant" — not
+something a single user requests about their own data, and it has no natural
+per-caller grant (it does not take a `principalId` and touches every
+matching row tenant-wide). A host that wants it schedules a cron job calling
+`memory.sweepEphemeral({ tenantId })` in-process (the returned `Memory`
+already exposes it); the engine stays cron-free per `ARCHITECTURE.md`.

package.json

@@ -15,14 +15,26 @@
       {
         "resource": "memory",
         "action": "add",
-        "source": "tenant",
+        "installHint": "tenant",
         "surfaces": ["tools", "distiller", "routes"]
       },
       {
         "resource": "memory",
         "action": "search",
-        "source": "tenant",
+        "installHint": "tenant",
         "surfaces": ["tools", "distiller", "routes"]
+      },
+      {
+        "resource": "memory",
+        "action": "forget",
+        "installHint": "creator",
+        "surfaces": ["routes"]
+      },
+      {
+        "resource": "memory",
+        "action": "purge",
+        "installHint": "creator",
+        "surfaces": ["routes"]
       }
     ]
   },

src/core/schemas/non-blank-id.ts

@@ -0,0 +1,15 @@
+import { type } from "arktype";
+
+/**
+ * `"string >= 1"` is a LENGTH constraint, not a content one — `" "` has
+ * length 1 and would pass it, seating/accepting a whitespace-only id exactly
+ * like the empty-string case such a schema exists to reject. Require at
+ * least one non-whitespace character instead.
+ *
+ * Shared by the resolved-caller trust boundary (`routes/deps.ts`, CL-6286)
+ * and the retention path-param schemas (`http-bodies.ts`, CL-6288) so the
+ * fix lives in one schema instead of a comment repeated at each call site.
+ */
+export const NonBlankId = type("string").narrow(
+  (s, ctx) => s.trim().length > 0 || ctx.mustBe("non-blank (not just whitespace)"),
+);

src/distiller/workflow.ts

@@ -23,7 +23,7 @@ import { defineWorkflow, type WorkflowDefinition } from "@intx/workflow";
 import { memoryAdd } from "../tools/add.ts";
 import { memoryFeed } from "../tools/feed.ts";
 import { memorySearch } from "../tools/search.ts";
-import { MEMORY_CAPABILITY_IDS } from "../grant-requirements.ts";
+import { capabilityIdsForSurface } from "../grant-requirements.ts";
 import {
   RESIDENT_DISTILLER_AGENT_ID,
   RESIDENT_DISTILLER_CRON_DEFAULT,
@@ -97,7 +97,7 @@ export function createResidentDistiller(
       "Resident memory distiller — feed → classify → claim write",
     systemPrompt: opts.systemPrompt ?? DEFAULT_SYSTEM_PROMPT(generatorAgentId),
     tools,
-    capabilities: [...MEMORY_CAPABILITY_IDS],
+    capabilities: capabilityIdsForSurface("distiller"),
 
     inference: opts.inference,
     tags: {

src/grant-requirements.test.ts

@@ -3,27 +3,79 @@ import { readFileSync } from "node:fs";
 import { join } from "node:path";
 
 import {
+  capabilityIdsForSurface,
   MEMORY_CAPABILITY_IDS,
   MEMORY_GRANT_REQUIREMENTS,
 } from "./grant-requirements.ts";
 
 describe("MEMORY_GRANT_REQUIREMENTS", () => {
-  test("covers add + search on memory resource", () => {
+  test("covers add, search, forget, and purge on memory resource", () => {
     expect(MEMORY_GRANT_REQUIREMENTS.map((r) => r.action).sort()).toEqual([
       "add",
+      "forget",
+      "purge",
       "search",
     ]);
     for (const r of MEMORY_GRANT_REQUIREMENTS) {
       expect(r.resource).toBe("memory");
-      expect(r.source).toBe("tenant");
+    }
+  });
+
+  test("add/search hint tenant-wide and reach tools + distiller + routes", () => {
+    for (const action of ["add", "search"]) {
+      const r = MEMORY_GRANT_REQUIREMENTS.find((x) => x.action === action)!;
+      expect(r.installHint).toBe("tenant");
       expect(r.surfaces).toContain("tools");
       expect(r.surfaces).toContain("distiller");
+      expect(r.surfaces).toContain("routes");
+    }
+  });
+
+  test("forget/purge hint creator-scoped and are routes-only", () => {
+    for (const action of ["forget", "purge"]) {
+      const r = MEMORY_GRANT_REQUIREMENTS.find((x) => x.action === action)!;
+      expect(r.installHint).toBe("creator");
+      expect(r.surfaces).toEqual(["routes"]);
+    }
+  });
+
+  test("installHint is advisory only — the requirement shape carries no enforcement field", () => {
+    // The actual ownership check lives in services/retention-ownership.ts,
+    // wired imperatively into memory.ts, entirely independent of this hint.
+    // This test exists so a future reader who tightens grant-requirements.ts
+    // notices this comment rather than assuming installHint is load-bearing.
+    for (const r of MEMORY_GRANT_REQUIREMENTS) {
+      expect(Object.keys(r).sort()).toEqual([
+        "action",
+        "installHint",
+        "resource",
+        "surfaces",
+      ]);
     }
   });
 
   test("capability ids are resource:action", () => {
     expect([...MEMORY_CAPABILITY_IDS].sort()).toEqual([
       "memory:add",
+      "memory:forget",
+      "memory:purge",
+      "memory:search",
+    ]);
+  });
+
+  test("capabilityIdsForSurface excludes routes-only actions from distiller/tools", () => {
+    expect(capabilityIdsForSurface("distiller").sort()).toEqual([
+      "memory:add",
+      "memory:search",
+    ]);
+    expect(capabilityIdsForSurface("tools").sort()).toEqual([
+      "memory:add",
+      "memory:search",
+    ]);
+    expect(capabilityIdsForSurface("routes").sort()).toEqual([
+      "memory:add",
+      "memory:forget",
+      "memory:purge",
       "memory:search",
     ]);
   });
@@ -36,7 +88,7 @@ describe("MEMORY_GRANT_REQUIREMENTS", () => {
         grantRequirements?: Array<{
           resource: string;
           action: string;
-          source: string;
+          installHint: string;
           surfaces: string[];
         }>;
       };
@@ -46,7 +98,7 @@ describe("MEMORY_GRANT_REQUIREMENTS", () => {
       MEMORY_GRANT_REQUIREMENTS.map((r) => ({
         resource: r.resource,
         action: r.action,
-        source: r.source,
+        installHint: r.installHint,
         surfaces: [...r.surfaces],
       })),
     );