Retain bounded live liveness materializations

CHANGELOG.md

@@ -66,13 +66,21 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   operation-local instead of shared across materializations. This bounds
   mutated-page residency only; removal of whole-state projection remains
   tracked by #738.
+- Changed cold live-handle resolution to stream only node and edge liveness
+  into bounded git-cas pages. Partial materialization handles carry
+  `stateHash: null`, cannot masquerade as whole-state snapshots, and avoid
+  constructing `WarpState`, adjacency, properties, receipts, diffs,
+  provenance, or a state-cache snapshot.
 
 ### Removed
 
 - Removed the `browser` and `legacy` package entrypoints, the graph-first
   compatibility barrel, and their compatibility-only examples and tests.
 - Removed the v18 graph-first package exports rather than carrying a deprecated
   second application API through v19.
+- Removed migration and cleanup support for unreleased materialization
+  descriptor schemas v2 through v4. Schema v5 is the first release contract;
+  stale derived-cache entries miss and rebuild from authoritative WARP history.
 
 ### Fixed
 

docs/topics/cas-first-memoized-materialization.md

@@ -98,14 +98,17 @@ Until cache payloads carry provenance indexes, that derived snapshot retains a
 degraded provenance posture rather than claiming support for the cached prefix.
 
 Retained materializations can now satisfy the same two resume cases without a
-separate state-cache hit. Descriptor schema v4 retains a canonical replay basis
-beside the node, edge, and property roots. An exact retained hit validates and
-loads that basis, reuses the retained roots, and performs no patch replay. When
-there is no exact hit, the adapter inspects at most 1,024 current-schema cache
-entries in pages of 100. It validates their descriptors and coordinates, checks
-causal ancestry, and resumes the newest compatible predecessor by replaying only
-the suffix. Receipt-producing reads still use the ordinary replay path, as do
-diff-producing predecessor reads.
+separate state-cache hit. A complete descriptor retains a canonical replay
+basis beside the node, edge, and property roots. An exact retained hit validates
+and loads that basis, reuses the retained roots, and performs no patch replay.
+Descriptor schema v5 also admits explicitly partial handles: `stateHash: null`
+means the roots can answer only the reads whose root status is retained or
+empty, and the handle cannot resume a whole-state snapshot. When there is no
+exact hit, the adapter inspects at most 1,024 current-schema cache entries in
+pages of 100. It validates their descriptors and coordinates, checks causal
+ancestry, and resumes the newest compatible complete predecessor by replaying
+only the suffix. Receipt-producing reads still use the ordinary replay path, as
+do diff-producing predecessor reads.
 
 ### 3. Fall back to replay and publish
 
@@ -140,23 +143,23 @@ Both exact paths release their operation borrow without hydrating `WarpState`,
 building adjacency, publishing a state snapshot, or populating `_cachedState`.
 The runtime storage adapter keeps one git-cas acquisition for the current
 coordinate, retires it after in-flight readers finish when the coordinate
-changes, and releases it from `RuntimeHost.close()`. A cold handle miss still
-performs the current materialization path before retaining and reading the new
-roots. Once assembled, a newly built property root joins the operation's
-expiring git-cas workspace before state hashing and final promotion, so the
-completed root remains reachable throughout promotion. A custom
-state-session opener owns its root storage and
-encoding, so git-warp does not pair it with the default reader and instead
-preserves the compatibility fallback.
-
-The replay-basis contract advances the retained-materialization descriptor and
-coordinate cache key to schema v4. A v4 exact miss leaves corresponding legacy
-entries anchored until replacement succeeds. Successful v4 retention then
-removes incompatible v2 and v3 anchors through the git-cas cache API. Legacy
-descriptors may still be structurally valid, but they cannot satisfy the v4 root
-profile because their property or replay-basis root may be unavailable. New v4
-descriptors reject either root as unavailable; an empty graph still records the
-property root as explicitly empty.
+changes, and releases it from `RuntimeHost.close()`. On a cold handle miss, the
+built-in session now streams only node and edge OR-Set operations into bounded
+git-cas pages and retains those roots as a partial handle. It does not construct
+`WarpState`, adjacency, property registers, receipts, diffs, provenance, or a
+state-cache snapshot. A later compatibility or property operation can replace
+that partial entry with a complete descriptor.
+
+Once assembled, a newly built property root joins the operation's expiring
+git-cas workspace before state hashing and final promotion, so the completed
+root remains reachable throughout promotion. A custom state-session opener owns
+its root storage and encoding, so git-warp does not pair it with the default
+reader and instead preserves the compatibility fallback.
+
+Schema v5 is the first release contract for retained-materialization
+descriptors. Earlier v2, v3, and v4 profiles were unreleased derived-cache
+formats; v19 does not decode or migrate them. A current-key miss rebuilds from
+authoritative WARP history.
 
 ## `git-cas` Encapsulation
 
@@ -221,20 +224,23 @@ lost payload bytes, or run Git garbage collection.
 
 - RuntimeHost exact node-liveness and node-property reads consume the
   handle-first result when the built-in trie session and reader pair is active.
-  Custom session openers, neighborhoods, list reads, checkpoint creation, and
-  other compatibility operations still own process-resident whole state.
+  Cold node liveness now produces a partial retained handle through bounded
+  node/edge replay. Cold properties, custom session openers, neighborhoods,
+  list reads, checkpoint creation, and other compatibility operations still own
+  process-resident whole state.
 - Exact state-cache hits bypass replay, but full materialization still hydrates
   a full `WarpState`, scans retained node/edge tries, and builds full adjacency.
 - Retained exact and predecessor resume load a complete canonical `WarpState`
   replay basis before they reuse roots or replay a suffix. This eliminates
   redundant prefix replay but is still a whole-state compatibility bridge, not
   the bounded-memory observer representation.
-- Retained materialization descriptors currently carry node/edge trie roots and
-  a per-node property-shard root plus the full-state replay basis. Frontier,
-  edge-birth, adjacency,
-  provenance-support, and roaring roots remain explicitly unavailable until
-  their paged representations land. Cold property-root construction still
-  projects a complete `WarpState`; only exact retained reads avoid that state.
+- Complete retained materialization descriptors carry node/edge trie roots, a
+  per-node property-shard root, and the full-state replay basis. Partial cold
+  handles carry node/edge roots and mark the remaining roots unavailable.
+  Frontier, edge-birth, adjacency, provenance-support, and roaring roots remain
+  explicitly unavailable until their paged representations land. Cold
+  property-root construction still projects a complete `WarpState`; only
+  exact retained property reads avoid that state.
 - One node's complete encoded property bag must currently fit within the 16 MiB
   shard limit. Property-key pagination or a property trie is not yet available.
 - The first property-root profile stores one bundle member per property-bearing

src/domain/materialization/MaterializationHandle.ts

@@ -10,15 +10,22 @@ export default class MaterializationHandle {
   readonly bundle: BundleHandle;
   readonly coordinate: MaterializationCoordinate;
   readonly roots: MaterializationRoots;
-  readonly stateHash: string;
+  /**
+   * Semantic hash of a complete WarpState replay basis.
+   *
+   * Partial materializations intentionally carry `null`: their retained roots
+   * can answer bounded reads, but must never masquerade as a whole-state
+   * snapshot.
+   */
+  readonly stateHash: string | null;
   readonly retention: StorageRetentionWitness;
 
   constructor(options: {
     readonly laneName: string;
     readonly bundle: BundleHandle;
     readonly coordinate: MaterializationCoordinate;
     readonly roots: MaterializationRoots;
-    readonly stateHash: string;
+    readonly stateHash: string | null;
     readonly retention: StorageRetentionWitness;
   }) {
     requireOptions(options);
@@ -30,7 +37,7 @@ export default class MaterializationHandle {
       'coordinate',
     );
     this.roots = requireInstance(options.roots, MaterializationRoots, 'roots');
-    this.stateHash = requireNonEmpty(options.stateHash, 'stateHash');
+    this.stateHash = requireOptionalStateHash(options.stateHash);
     this.retention = requireInstance(
       options.retention,
       StorageRetentionWitness,
@@ -59,6 +66,10 @@ function requireNonEmpty(value: string, field: string): string {
   return value;
 }
 
+function requireOptionalStateHash(value: string | null): string | null {
+  return value === null ? null : requireNonEmpty(value, 'stateHash');
+}
+
 function requireOptions(options: object): void {
   if (options === null || typeof options !== 'object' || Array.isArray(options)) {
     throw handleError('options must be an object');

src/domain/services/JoinReducerSession.ts

@@ -131,6 +131,38 @@ export async function applyWithReceiptInSession(
   return result.receipt;
 }
 
+/**
+ * Applies only the page-backed OR-Set portion of a patch.
+ *
+ * This is the bounded cold-read reducer: it deliberately does not allocate
+ * property registers, edge-birth metadata, receipts, diffs, or a frontier
+ * projection.
+ */
+export async function applyLivenessInSession(
+  session: StateSession,
+  patch: PatchLike, // nosemgrep: ts-no-like-types -- 0025C
+): Promise<void> {
+  for (const rawOp of patch.ops) {
+    const op = normalizeRawOp(rawOp);
+    if (!(op instanceof Op)) {
+      continue;
+    }
+    op.validate();
+    if (op instanceof NodeAdd) {
+      await session.addNode(op.node, op.dot);
+    } else if (op instanceof NodeRemove) {
+      await session.removeNode(op.node, new Set(op.observedDots));
+    } else if (op instanceof EdgeAdd) {
+      await session.addEdge(encodeEdgeKey(op.from, op.to, op.label), op.dot);
+    } else if (op instanceof EdgeRemove) {
+      await session.removeEdge(
+        encodeEdgeKey(op.from, op.to, op.label),
+        new Set(op.observedDots),
+      );
+    }
+  }
+}
+
 export function reducePatchesInSession(
   patches: SessionPatchSource,
   frame: ReducerSessionFrame,

src/domain/services/controllers/BoundedLiveMaterialization.ts

@@ -0,0 +1,179 @@
+import type MaterializationCoordinate from '../../materialization/MaterializationCoordinate.ts';
+import LiveMaterializationResolution from '../../materialization/LiveMaterializationResolution.ts';
+import type MaterializationHandle from '../../materialization/MaterializationHandle.ts';
+import MaterializationRoot from '../../materialization/MaterializationRoot.ts';
+import MaterializationRoots from '../../materialization/MaterializationRoots.ts';
+import BundleHandle from '../../storage/BundleHandle.ts';
+import WarpError from '../../errors/WarpError.ts';
+import type StateSession from '../../orset/session/StateSession.ts';
+import type MaterializationWorkspacePort from '../../../ports/MaterializationWorkspacePort.ts';
+import { applyLivenessInSession } from '../JoinReducerSession.ts';
+import type { MaterializeDeps } from './MaterializeDeps.ts';
+import type { MaterializeSessionOpener } from './MaterializeSessionBridge.ts';
+import {
+  releaseAcquisitionAfterFailure,
+  releaseWorkspaceAfterFailure,
+} from './MaterializationWorkspaceCleanup.ts';
+
+export type BoundedLiveMaterializationResult = Readonly<{
+  materialization: Awaited<
+    ReturnType<MaterializeDeps['materializations']['retain']>
+  >;
+  patchCount: number;
+}>;
+
+export async function resolveBoundedLiveMaterialization(args: {
+  readonly deps: MaterializeDeps;
+  readonly coordinate: MaterializationCoordinate;
+}): Promise<LiveMaterializationResolution | null> {
+  const bounded = await retainBoundedLiveMaterialization(args);
+  if (bounded === null) {
+    return null;
+  }
+  const acquired = await args.deps.materializations.acquireExact(args.coordinate);
+  if (acquired === null) {
+    throw resolutionError('newly retained bounded handle could not be acquired');
+  }
+  try {
+    requireSameHandle(bounded.materialization, acquired.materialization);
+    return new LiveMaterializationResolution({
+      materialization: acquired.materialization,
+      source: 'materialized',
+      replayedPatchCount: bounded.patchCount,
+      release: async () => await acquired.release(),
+    });
+  } catch (raw) {
+    await releaseAcquisitionAfterFailure(acquired, args.deps.logger);
+    throw raw;
+  }
+}
+
+/**
+ * Replays only node/edge OR-Set state into bounded CAS pages and retains those
+ * roots as a partial materialization. No WarpState or whole-graph adjacency is
+ * constructed.
+ */
+export async function retainBoundedLiveMaterialization(args: {
+  readonly deps: MaterializeDeps;
+  readonly coordinate: MaterializationCoordinate;
+}): Promise<BoundedLiveMaterializationResult | null> {
+  const { deps, coordinate } = args;
+  const { openStateSession } = deps;
+  if (openStateSession === undefined) {
+    return null;
+  }
+  const workspace = await deps.materializations.openWorkspace(coordinate);
+  try {
+    return await buildBoundedMaterialization({
+      deps,
+      coordinate,
+      workspace,
+      openStateSession,
+    });
+  } catch (raw) {
+    await releaseWorkspaceAfterFailure(workspace, deps.logger);
+    throw raw;
+  }
+}
+
+async function buildBoundedMaterialization(args: {
+  readonly deps: MaterializeDeps;
+  readonly coordinate: MaterializationCoordinate;
+  readonly workspace: MaterializationWorkspacePort;
+  readonly openStateSession: MaterializeSessionOpener;
+}): Promise<BoundedLiveMaterializationResult | null> {
+  const { deps, coordinate, workspace, openStateSession } = args;
+  const session = await openStateSession(
+    { nodeAliveRootOid: null, edgeAliveRootOid: null },
+    { workspace },
+  );
+  const patchCount = await replayLiveness(session, deps, coordinate);
+  if (patchCount === 0) {
+    await session.close();
+    await workspace.release();
+    return null;
+  }
+  return await retainPreparedLiveness({
+    session,
+    coordinate,
+    workspace,
+    patchCount,
+  });
+}
+
+async function replayLiveness(
+  session: StateSession,
+  deps: MaterializeDeps,
+  coordinate: MaterializationCoordinate,
+): Promise<number> {
+  let patchCount = 0;
+  for await (const entry of deps.patches.streamForFrontier(
+    coordinate.frontier(),
+    coordinate.ceiling,
+  )) {
+    patchCount += 1;
+    await applyLivenessInSession(session, entry.patch);
+  }
+  return patchCount;
+}
+
+async function retainPreparedLiveness(args: {
+  readonly session: StateSession;
+  readonly coordinate: MaterializationCoordinate;
+  readonly workspace: MaterializationWorkspacePort;
+  readonly patchCount: number;
+}): Promise<BoundedLiveMaterializationResult> {
+  const { session, coordinate, workspace, patchCount } = args;
+  const prepared = await session.prepareClose();
+  const materialization = await workspace.promote({
+    coordinate,
+    roots: livenessRoots(prepared.roots),
+    stateHash: null,
+  });
+  prepared.accept(materialization.retention);
+  await workspace.release();
+  return Object.freeze({ materialization, patchCount });
+}
+
+function livenessRoots(roots: {
+  readonly nodeAliveRootOid: string | null;
+  readonly edgeAliveRootOid: string | null;
+}): MaterializationRoots {
+  return new MaterializationRoots({
+    adjacency: MaterializationRoot.unavailable(),
+    edgeAlive: sessionRoot(roots.edgeAliveRootOid),
+    edgeBirths: MaterializationRoot.unavailable(),
+    frontier: MaterializationRoot.unavailable(),
+    nodeAlive: sessionRoot(roots.nodeAliveRootOid),
+    properties: MaterializationRoot.unavailable(),
+    provenanceSupport: MaterializationRoot.unavailable(),
+    replayBasis: MaterializationRoot.unavailable(),
+    roaringIndexes: MaterializationRoot.unavailable(),
+  });
+}
+
+function sessionRoot(token: string | null): MaterializationRoot {
+  return token === null
+    ? MaterializationRoot.empty()
+    : MaterializationRoot.retained(new BundleHandle(token));
+}
+
+function requireSameHandle(
+  expected: MaterializationHandle,
+  acquired: MaterializationHandle,
+): void {
+  if (
+    !acquired.coordinate.equals(expected.coordinate)
+    || acquired.stateHash !== expected.stateHash
+    || !acquired.bundle.equals(expected.bundle)
+  ) {
+    throw resolutionError('newly retained handle changed before it could be acquired');
+  }
+}
+
+function resolutionError(message: string): WarpError {
+  return new WarpError(
+    `Materialization resolution ${message}`,
+    'E_MATERIALIZATION_RESUME',
+  );
+}