Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

### Fixed

- Exact state-cache hits now retain and reopen coordinate-keyed materialization
descriptors through the `git-cas` cache API. Repeated materialization,
including through a fresh runtime adapter, resumes the retained node/edge
trie roots without replaying writer patches already covered by the snapshot.
- Failed state-session reductions no longer flush partial trie roots into
unretained CAS objects; roots are flushed only after successful projection.
- Git-backed state-cache payload trees are now anchored through a graph-scoped
`git-cas` RootSet before their index record is published, then reconciled
after publication so live cache entries remain reachable across Git garbage
Expand Down
49 changes: 33 additions & 16 deletions docs/topics/cas-first-memoized-materialization.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,11 +4,12 @@ Use this page when you need to understand how `git-warp` skips redundant
materialization replay by memoizing WARP-owned state snapshots in
`@git-stunts/git-cas`.

`git-cas` provides byte storage and generic Git-reachability primitives. It does
not know about WARP frontiers, optics, checkpoints, graph state, or
materialization rules. `git-warp` owns those semantics through
`WarpStateCachePort`; the Git-backed adapter stores snapshot payloads in
`git-cas` and declares the live payload trees through a `RootSet`.
`git-cas` provides byte storage, retained cache entries, and generic
Git-reachability primitives. It does not know about WARP frontiers, optics,
checkpoints, graph state, or materialization rules. `git-warp` owns those
semantics through `WarpStateCachePort` and `MaterializationStorePort`; the
Git-backed adapters store snapshot payloads and coordinate-keyed retained roots
through `git-cas`.

## The Live Materialization Lifecycle

Expand All @@ -19,7 +20,7 @@ coordinate-first lifecycle:
[current frontier]
|
v
[state-cache exact hit?] ---- yes ---> [return cached state]
[state-cache exact hit?] ---- yes ---> [reopen retained roots; zero patch replay]
|
no
v
Expand All @@ -44,8 +45,13 @@ This coordinate belongs to `git-warp`; it is not a `git-cas` concept.
### 2. Check the WARP state cache

The runtime asks `WarpStateCachePort` for an exact snapshot at that coordinate.
On a hit, it returns the cached state without replaying writer patch streams and
without republishing the same snapshot.
On a hit, it asks `MaterializationStorePort` for the matching retained-root
descriptor. A descriptor hit reopens the node/edge trie roots and projects the
result without replaying writer patch streams or republishing the same snapshot.
The descriptor records every named materialization root as `retained`, `empty`,
or `unavailable`; only retained roots become bundle members. On the first exact
snapshot hit without a descriptor, the runtime seeds the trie roots from the
snapshot and retains the resulting descriptor for later runtime instances.

The current payload records state but not the provenance index. A runtime may
retain its resident provenance index when the cached state has the same hash and
Expand All @@ -67,20 +73,24 @@ equivalent read can hit the cache.

## Memory Boundaries

State-cache hits avoid redundant CRDT replay and can remove repeated startup
costs for graph-sized materializations. They do not make legacy full
materialization an `O(1)` memory API: a caller that asks for a full
`SnapshotWarpState` still receives a full in-memory state object.
State-cache hits with retained roots avoid redundant CRDT patch replay across
runtime instances. They do not make legacy full materialization an `O(1)` time
or memory API: the current result contract still loads a full snapshot and scans
the retained node/edge tries to produce a full `WarpState` and adjacency map.

The bounded-memory read path is optic/worldline/query work over a sharded or
streamed basis. The state cache is the replay-skipping compatibility bridge for
legacy materialization and checkpoint flows.

## `git-cas` Encapsulation

All state-cache payload storage routes through the formal `@git-stunts/git-cas`
library API. Raw Git plumbing remains an adapter concern for WARP refs and Git
object access; WARP state-cache payloads should not hand-roll a parallel CAS.
Materialization-root retention routes through the formal
`@git-stunts/git-cas` `CacheSet` API. The legacy state-cache adapter also routes
payload bytes through `git-cas`, but still owns its snapshot index and RootSet
reconciliation. Removing that compatibility cache lifecycle is required before
the one-cache boundary is complete. Raw Git plumbing remains an adapter concern
for WARP refs and Git object access; WARP code must not hand-roll a parallel
CAS.

Routing state snapshots through `git-cas` allows content-addressed storage and
chunk-level reuse where the underlying CAS representation can identify unchanged
Expand Down Expand Up @@ -134,7 +144,14 @@ lost payload bytes, or run Git garbage collection.
## Current Limitations

- Exact state-cache hits bypass replay, but full materialization still hydrates
a full `WarpState`.
a full `WarpState`, scans retained node/edge tries, and builds full adjacency.
- Retained materialization descriptors currently carry node/edge trie roots;
property, frontier, edge-birth, adjacency, provenance-support, and roaring
roots are explicitly marked unavailable until their paged representations
land.
- `WarpStateCachePort` remains a legacy full-snapshot compatibility cache with
a WARP-owned index. Ordinary bounded observers cannot rely on it as their
final storage contract.
- The Git-backed state-cache adapter stores full-state snapshots today. A future
sharded basis format should make optic reads avoid full-state hydration.
- Cache coordinates must stay schema/version aware. A snapshot is reusable only
Expand Down
2 changes: 2 additions & 0 deletions src/domain/RuntimeHost.ts
Original file line number Diff line number Diff line change
Expand Up @@ -288,6 +288,7 @@ export default class RuntimeHost {
checkpointStore,
indexStore,
intentStore,
materializations,
viewService,
stateHashService,
auditService,
Expand Down Expand Up @@ -400,6 +401,7 @@ export default class RuntimeHost {
crypto: this._crypto,
persistence: this._persistence,
checkpointStore,
materializations,
getStateCache: () => this._stateCache ?? null,
...(openStateSession === undefined ? {} : { openStateSession }),
patches: new RuntimePatchCollector(this),
Expand Down
35 changes: 35 additions & 0 deletions src/domain/materialization/MaterializationRoot.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
import WarpError from '../errors/WarpError.ts';
import BundleHandle from '../storage/BundleHandle.ts';

export type MaterializationRootStatus = 'retained' | 'empty' | 'unavailable';

/** Availability and optional retained handle for one materialization root. */
export default class MaterializationRoot {
readonly status: MaterializationRootStatus;
readonly handle: BundleHandle | null;

private constructor(status: MaterializationRootStatus, handle: BundleHandle | null) {
this.status = status;
this.handle = handle;
Object.freeze(this);
}

static retained(handle: BundleHandle): MaterializationRoot {
if (!(handle instanceof BundleHandle)) {
throw rootError('retained root must carry a BundleHandle');
}
return new MaterializationRoot('retained', handle);
}

static empty(): MaterializationRoot {
return new MaterializationRoot('empty', null);
}

static unavailable(): MaterializationRoot {
return new MaterializationRoot('unavailable', null);
}
}

function rootError(message: string): WarpError {
return new WarpError(`Materialization root ${message}`, 'E_MATERIALIZATION_ROOT');
}
90 changes: 45 additions & 45 deletions src/domain/materialization/MaterializationRoots.ts
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
import WarpError from '../errors/WarpError.ts';
import BundleHandle from '../storage/BundleHandle.ts';
import MaterializationRoot from './MaterializationRoot.ts';

export const MATERIALIZATION_ROOT_NAMES = defineRootNames(
'adjacency',
Expand All @@ -15,75 +15,75 @@ export const MATERIALIZATION_ROOT_NAMES = defineRootNames(
export type MaterializationRootName = (typeof MATERIALIZATION_ROOT_NAMES)[number];

export type MaterializationRootsOptions = Readonly<{
adjacency: BundleHandle;
edgeAlive: BundleHandle;
edgeBirths: BundleHandle;
frontier: BundleHandle;
nodeAlive: BundleHandle;
properties: BundleHandle;
provenanceSupport: BundleHandle;
roaringIndexes: BundleHandle;
adjacency: MaterializationRoot;
edgeAlive: MaterializationRoot;
edgeBirths: MaterializationRoot;
frontier: MaterializationRoot;
nodeAlive: MaterializationRoot;
properties: MaterializationRoot;
provenanceSupport: MaterializationRoot;
roaringIndexes: MaterializationRoot;
}>;

/** Independently addressable retained roots for one materialized causal chart. */
export default class MaterializationRoots {
private readonly handles: Readonly<Record<MaterializationRootName, BundleHandle>>;
readonly adjacency: BundleHandle;
readonly edgeAlive: BundleHandle;
readonly edgeBirths: BundleHandle;
readonly frontier: BundleHandle;
readonly nodeAlive: BundleHandle;
readonly properties: BundleHandle;
readonly provenanceSupport: BundleHandle;
readonly roaringIndexes: BundleHandle;
private readonly roots: Readonly<Record<MaterializationRootName, MaterializationRoot>>;
readonly adjacency: MaterializationRoot;
readonly edgeAlive: MaterializationRoot;
readonly edgeBirths: MaterializationRoot;
readonly frontier: MaterializationRoot;
readonly nodeAlive: MaterializationRoot;
readonly properties: MaterializationRoot;
readonly provenanceSupport: MaterializationRoot;
readonly roaringIndexes: MaterializationRoot;

constructor(options: MaterializationRootsOptions) {
requireOptions(options);
this.handles = Object.freeze({
adjacency: requireBundle(options.adjacency, 'adjacency'),
'edge-alive': requireBundle(options.edgeAlive, 'edgeAlive'),
'edge-births': requireBundle(options.edgeBirths, 'edgeBirths'),
frontier: requireBundle(options.frontier, 'frontier'),
'node-alive': requireBundle(options.nodeAlive, 'nodeAlive'),
properties: requireBundle(options.properties, 'properties'),
'provenance-support': requireBundle(options.provenanceSupport, 'provenanceSupport'),
'roaring-indexes': requireBundle(options.roaringIndexes, 'roaringIndexes'),
} satisfies Record<MaterializationRootName, BundleHandle>);
this.adjacency = this.handles.adjacency;
this.edgeAlive = this.handles['edge-alive'];
this.edgeBirths = this.handles['edge-births'];
this.frontier = this.handles.frontier;
this.nodeAlive = this.handles['node-alive'];
this.properties = this.handles.properties;
this.provenanceSupport = this.handles['provenance-support'];
this.roaringIndexes = this.handles['roaring-indexes'];
this.roots = Object.freeze({
adjacency: requireRoot(options.adjacency, 'adjacency'),
'edge-alive': requireRoot(options.edgeAlive, 'edgeAlive'),
'edge-births': requireRoot(options.edgeBirths, 'edgeBirths'),
frontier: requireRoot(options.frontier, 'frontier'),
'node-alive': requireRoot(options.nodeAlive, 'nodeAlive'),
properties: requireRoot(options.properties, 'properties'),
'provenance-support': requireRoot(options.provenanceSupport, 'provenanceSupport'),
'roaring-indexes': requireRoot(options.roaringIndexes, 'roaringIndexes'),
} satisfies Record<MaterializationRootName, MaterializationRoot>);
this.adjacency = this.roots.adjacency;
this.edgeAlive = this.roots['edge-alive'];
this.edgeBirths = this.roots['edge-births'];
this.frontier = this.roots.frontier;
this.nodeAlive = this.roots['node-alive'];
this.properties = this.roots.properties;
this.provenanceSupport = this.roots['provenance-support'];
this.roaringIndexes = this.roots['roaring-indexes'];
Object.freeze(this);
}

entries(): readonly (readonly [MaterializationRootName, BundleHandle])[] {
entries(): readonly (readonly [MaterializationRootName, MaterializationRoot])[] {
return Object.freeze(
MATERIALIZATION_ROOT_NAMES.map((name) => rootEntry(name, this.handles[name])),
MATERIALIZATION_ROOT_NAMES.map((name) => rootEntry(name, this.roots[name])),
);
}
}

function rootEntry(
name: MaterializationRootName,
handle: BundleHandle,
): readonly [MaterializationRootName, BundleHandle] {
return Object.freeze([name, handle]);
root: MaterializationRoot,
): readonly [MaterializationRootName, MaterializationRoot] {
return Object.freeze([name, root]);
}

function defineRootNames<const Names extends readonly string[]>(...names: Names): Names {
Object.freeze(names);
return names;
}

function requireBundle(handle: BundleHandle, field: string): BundleHandle {
if (!(handle instanceof BundleHandle)) {
throw rootsError(`${field} must be a BundleHandle`);
function requireRoot(root: MaterializationRoot, field: string): MaterializationRoot {
if (!(root instanceof MaterializationRoot)) {
throw rootsError(`${field} must be a MaterializationRoot`);
}
return handle;
return root;
}

function requireOptions(options: object): void {
Expand Down
Loading
Loading