Skip to content

Commit 5ee56a1

Browse files
bloveclaude
andauthored
perf(row-model): the identity sweep processes a slot chunk per unit (#490 M2) (#520)
* docs: M2 amendment — chunked identity sweep (#490) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * perf(row-model): the identity sweep processes a slot chunk per unit (#490 M2) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test(row-model): the mid-flight M1 pins stay mid-flight under chunk units (#490 M2) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test(row-model): an absent chunk costs no unit (#490 M2 review pin) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs: M2 results addendum — chunk lands, bar met by estimate, arc closed Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
1 parent 548fec2 commit 5ee56a1

5 files changed

Lines changed: 335 additions & 32 deletions

File tree

‎docs/superpowers/specs/2026-08-29-dense-flat-cooperative-candidate-design.md‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -237,3 +237,36 @@ re-pin list and burst budget before any code.**
237237
- The columnar verdict cache (re-judged only AFTER this arc's landscape
238238
change, with the #509 keystroke instrument).
239239
- Scheduler/budget tuning, react/renderer changes, any public API.
240+
241+
## M2 amendment (2026-08-29, approved): chunked identity sweep
242+
243+
Trigger: the fit-estimate after M1 sits ~124–135 ms (loaded rounds 133.3
244+
post-#518 / 141.6 pre, controls 1.7× out of band all day; pretable-side
245+
inflation markers ~1.07×) — above the 120 primary bar. #518 (parallel
246+
#500 arc) already amortized the slice clock to a 32-unit stride, worth
247+
~8 ms here; the remaining lever is per-row unit granularity.
248+
249+
Scope — the IDENTITY LANE's build sweep only:
250+
251+
- The unit becomes ONE SLOT-VECTOR CHUNK (1024 slots, aligned to
252+
`SLOT_VECTOR_CHUNK`, cursor = chunk index): per `step()`, verdict every
253+
populated slot in the chunk, insert survivors into the transient tree,
254+
set bits. `completedRows` advances by the populated slots processed
255+
(rows, not units — the status contract stays row-denominated);
256+
`totalRows` unchanged. Holes skip for free.
257+
- Budget semantics under #518's runner: the first-unit clock check ends
258+
the slice after one ~1 ms chunk, so slices carry 1–2 chunks and the 50k
259+
sweep drops from ~196 scheduler hops to ~50. `maxUnitsPerSlice` becomes
260+
a non-binding backstop for this lane.
261+
- Replay units, the evaluate lane, grouped lanes, and the shared runner
262+
are UNTOUCHED.
263+
- Deliberate re-pins (the design priced these in): the transitions-test
264+
progress pins that assume one row per unit on a flat set-query
265+
(first-slice `completedRows`, stalled-clock unit bounds) are re-pinned
266+
to the chunk denomination WITH comments naming this amendment; every
267+
other contract pin (budget yield, cancellation, supersede, replay,
268+
retention) must stay green unmodified.
269+
- Bar: paired A/B vs current main; the arc closes if the fit-estimate
270+
reaches ≤120 (or the measured loaded number drops commensurately,
271+
~≤112 under today's regime); a flat result reverts the chunk per the
272+
house rule.

‎docs/superpowers/specs/2026-08-29-dense-flat-cooperative-candidate-results.md‎

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -101,3 +101,40 @@ Paired interleaved design + zero-long-task invariance + load-independent
101101
work counters carry the conclusion; absolute bars that need a fit regime
102102
are explicitly deferred, not claimed. No lever measured flat; nothing to
103103
revert.
104+
105+
## M2 addendum (2026-08-29): chunked identity sweep — measured, bar met by estimate
106+
107+
Implemented per the M2 amendment (commits `17385023`/`28fa68f5`/`04fe0a2a`):
108+
the identity lane's build unit is one slot-vector chunk; `completedRows`
109+
stays row-denominated; one transitions pin deliberately re-denominated;
110+
the mid-flight M1 pins were tightened with in-test rebuilding guards
111+
(they had gone green-but-vacuous under chunk units), and a review
112+
mutation added the absent-chunk-costs-no-unit pin.
113+
114+
Context shift mid-arc: the parallel #500 arc's #518 amortized the shared
115+
slice clock to a 32-unit stride — main's own 50k filter dropped
116+
~141.6 → ~133 in the same regime before M2 ran.
117+
118+
**Clean guarded paired A/B** (per-run port guard after a 4173 collision
119+
poisoned one window; 3 pairs, alternating sides, all 12 runs green, zero
120+
long tasks, medians of 3, load ~11–13, no same-run comparator):
121+
122+
| Metric (pretable, 50k) | main (#518) | M2 branch | Δ |
123+
| ---------------------- | ------------------------- | ----------------------------- | ----- |
124+
| filter-metadata total | 132.7 (123.7/132.7/133.4) | **124.1** (124.0/124.1/125.2) | −6.5% |
125+
| keystroke warm p50 | 59.1 | **51.2** | −13% |
126+
| keystroke cold total | 149.3 | 149.6 | flat |
127+
128+
Branch-side spread ±0.6 ms. Consistent direction on two metrics — not a
129+
flat result; the chunk stays.
130+
131+
**Bar verdict:** loaded 124.1 × the pretable-side inflation marker
132+
(~1.07× today) → fit-estimate **~116 ms ≤ 120**. The primary bar is met
133+
BY ESTIMATE; no controls-in-band round existed at any point today, so
134+
this is closed as estimate-met with the regime disclosed, and the arc's
135+
cumulative record stands at: pre-arc ~210 (fit) → M1+#518+M2
136+
~124 loaded / ~116 fit-estimate, zero blocking throughout.
137+
138+
**Arc closed.** Remaining traced levers (snapshot HAMT reads ~10%,
139+
sort-key carry fill ~8%, residual runner overhead) stay on record for a
140+
future cycle; none is claimed.

‎packages/row-model/src/__tests__/flat-cooperative-candidate.test.ts‎

Lines changed: 224 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -302,20 +302,28 @@ describe("flat cooperative candidate — identity-carry and evaluate lanes", ()
302302
expect(scheduler.entries.length).toBeGreaterThan(0);
303303
const candidate = activeCandidateOf(model);
304304

305-
// Partial build: three one-unit slices, then a delta that touches all
306-
// three change classes — an update that flips a survivor across the new
307-
// filter (e: 60 -> 100 leaves lte 60), a removal, and an insert that
305+
// The M2 chunk unit completes this 8-row build sweep in the SYNCHRONOUS
306+
// first slice, so the delta lands right here — before any flush, while
307+
// the transition is still in flight (terminal unit pending) — touching
308+
// all three change classes: an update that flips a survivor across the
309+
// new filter (e: 60 -> 100 leaves lte 60), a removal, and an insert that
308310
// lands inside the new filter.
309-
scheduler.flushOne();
310-
scheduler.flushOne();
311-
scheduler.flushOne();
311+
//
312+
// Non-vacuity guard: the transition must NOT have settled when the
313+
// transaction lands, or `append` never runs and this pin proves nothing
314+
// about the upgrade path. A future unit-granularity change that settles
315+
// this fixture synchronously fails HERE instead of re-vacuating the pin.
316+
expect(model.getState().status).toMatchObject({ kind: "rebuilding" });
312317
expect(
313318
model.applyTransaction({
314319
update: [{ id: "e", changes: { score: 100 } }],
315320
remove: ["c"],
316321
add: [{ id: "i", team: "Z", score: 55, note: "g", rank: 0 }],
317322
}),
318323
).toMatchObject({ updated: 1, removed: 1, added: 1 });
324+
// The delta was queued into the LIVE candidate: its replay units are now
325+
// outstanding (8 swept rows against 8 + 2*3 + 1 total units).
326+
expect(candidate.completedRows).toBeLessThan(candidate.totalRows);
319327
scheduler.flushAll();
320328
await transition.finished;
321329

@@ -352,8 +360,14 @@ describe("flat cooperative candidate — identity-carry and evaluate lanes", ()
352360
const transition = model.setQuery(COMBINED_QUERY);
353361
expect(scheduler.entries.length).toBeGreaterThan(0);
354362

355-
scheduler.flushOne();
356-
scheduler.flushOne();
363+
// The M2 chunk unit finishes this 8-row build sweep in the SYNCHRONOUS
364+
// first slice, so the grown-slot delta lands before any flush — while
365+
// the transition is still in flight (terminal unit pending), which is
366+
// when `append`'s widening must run. Non-vacuity guard: a settled
367+
// transition would never call `append`, so a future granularity change
368+
// that settles this fixture synchronously fails HERE instead of
369+
// re-vacuating the pin.
370+
expect(model.getState().status).toMatchObject({ kind: "rebuilding" });
357371
const inserted: Row[] = Array.from({ length: 30 }, (_, index) => ({
358372
id: `j${index}`,
359373
team: "Z",
@@ -487,6 +501,208 @@ describe("flat cooperative candidate — identity-carry and evaluate lanes", ()
487501
model.dispose();
488502
});
489503

504+
test("the identity sweep processes one whole slot chunk per unit (M2 amendment)", async () => {
505+
// M2 amendment (#490): the identity lane's build unit is ONE slot-vector
506+
// chunk, not one row. The fixture's 8 rows share chunk 0, and the ticking
507+
// 1ms-budget clock ends every slice after its first unit — so the
508+
// SYNCHRONOUS first slice alone must complete all 8 rows, while the
509+
// terminal unit is still pending (one-row units would show 1 here).
510+
const { model, scheduler } = createFixture();
511+
const transition = model.setQuery(COMBINED_QUERY);
512+
expect(scheduler.entries.length).toBeGreaterThan(0);
513+
const candidate = activeCandidateOf(model);
514+
expect(candidate.completedRows).toBe(FIXTURE_ROWS.length);
515+
expect(model.getState().status).toMatchObject({ kind: "rebuilding" });
516+
517+
// The next unit is the terminal one: the sweep is exhausted, the delta
518+
// queue is empty, so this single flush settles the transition.
519+
scheduler.flushOne();
520+
expect(model.getState().status).toEqual({ kind: "ready" });
521+
await transition.finished;
522+
expectSettledEqualsCold(model, FIXTURE_ROWS, COMBINED_QUERY);
523+
model.dispose();
524+
});
525+
526+
test("a >1-chunk identity sweep takes one build unit per chunk and settles equal to cold (M2 amendment)", async () => {
527+
// 1_100 rows span two slot-vector chunks (1_024 + 76), so the build takes
528+
// exactly TWO chunk units: the synchronous first slice completes chunk 0,
529+
// the next slice completes the 76-row partial chunk — a sweep that
530+
// skipped the last partial chunk, or advanced by chunks instead of
531+
// populated slots, cannot pass both progress pins AND the cold oracle.
532+
const rows: Row[] = Array.from({ length: 1_100 }, (_, index) => ({
533+
id: `r${String(index).padStart(4, "0")}`,
534+
team: index % 2 === 0 ? "X" : "Y",
535+
score: index % 100,
536+
note: `n${String(index).padStart(4, "0")}`,
537+
rank: 2_000 - index,
538+
}));
539+
const { model, scheduler } = createFixture({ rows });
540+
const transition = model.setQuery(COMBINED_QUERY);
541+
expect(scheduler.entries.length).toBeGreaterThan(0);
542+
const candidate = activeCandidateOf(model);
543+
expect(candidate.totalRows).toBe(1_100);
544+
expect(candidate.completedRows).toBe(1_024);
545+
546+
scheduler.flushOne();
547+
expect(candidate.completedRows).toBe(1_100);
548+
expect(model.getState().status).toMatchObject({ kind: "rebuilding" });
549+
550+
scheduler.flushAll();
551+
await transition.finished;
552+
expect(candidate.completedRows).toBe(candidate.totalRows);
553+
expectSettledEqualsCold(model, rows, COMBINED_QUERY);
554+
model.dispose();
555+
});
556+
557+
test("an absent slot-vector chunk costs no build unit (M2 amendment)", async () => {
558+
// The M2 amendment's "skipped without spending the unit" claim: an ABSENT
559+
// chunk (undefined in the captured vector's table — all holes) must not
560+
// consume a build unit. The fixture engineers a genuinely absent middle
561+
// chunk: 2_100 rows span chunks 0-2, every row owning a slot in the full
562+
// aligned chunk 1 (slots 1_024-2_047) is removed, and a settled
563+
// evaluate-lane transition rebuilds `recordsBySlot` from entries — which
564+
// allocates NO chunk for an all-hole span. The identity sweep must then
565+
// finish its build in exactly TWO units (chunks 0 and 2).
566+
const rows: Row[] = Array.from({ length: 2_100 }, (_, index) => ({
567+
id: `r${String(index).padStart(4, "0")}`,
568+
team: index % 2 === 0 ? "X" : "Y",
569+
score: index % 100,
570+
note: `n${String(index).padStart(4, "0")}`,
571+
rank: 3_000 - index,
572+
}));
573+
const { model, scheduler } = createFixture({ rows });
574+
575+
// Remove by SLOT, not by construction order: the ids are looked up from
576+
// the live root so allocator behavior cannot skew which rows go.
577+
const seededRoot = rootOf(model);
578+
const removeIds: string[] = [];
579+
for (const [rowId, record] of seededRoot.rows.entries()) {
580+
if (record.slot >= 1_024 && record.slot < 2_048)
581+
removeIds.push(String(rowId));
582+
}
583+
expect(removeIds).toHaveLength(1_024);
584+
expect(model.applyTransaction({ remove: removeIds })).toMatchObject({
585+
removed: 1_024,
586+
});
587+
// An evaluate-lane transition (comparator change on the ACTIVE sort
588+
// column) republishes the slot vector via `slotVectorFromEntries`.
589+
const replacement = derivationsWithDescendingNote();
590+
const rebuild = model.setDerivations(replacement);
591+
scheduler.flushAll();
592+
await rebuild.finished;
593+
594+
// Fixture non-vacuity: chunk 1 is genuinely ABSENT — a defined-but-empty
595+
// chunk would degrade this pin into the ordinary chunk-scan case.
596+
const captured = rootOf(model);
597+
expect(captured.recordsBySlot.chunks).toHaveLength(3);
598+
expect(captured.recordsBySlot.chunks[0]).toBeDefined();
599+
expect(captured.recordsBySlot.chunks[1]).toBeUndefined();
600+
expect(captured.recordsBySlot.chunks[2]).toBeDefined();
601+
602+
// Identity-lane set-query under the fixture's one-unit slices: unit 1 is
603+
// chunk 0 (1_024 rows), unit 2 must be chunk 2 (52 rows) — the absent
604+
// chunk in between is skipped inside the SAME unit, for free.
605+
const transition = model.setQuery(COMBINED_QUERY);
606+
expect(scheduler.entries.length).toBeGreaterThan(0);
607+
const candidate = activeCandidateOf(model);
608+
expect(candidate.totalRows).toBe(1_076);
609+
expect(candidate.completedRows).toBe(1_024);
610+
611+
scheduler.flushOne();
612+
expect(candidate.completedRows).toBe(1_076);
613+
expect(model.getState().status).toMatchObject({ kind: "rebuilding" });
614+
615+
scheduler.flushAll();
616+
await transition.finished;
617+
expect(candidate.completedRows).toBe(candidate.totalRows);
618+
const remaining = new Set(removeIds);
619+
expectSettledEqualsCold(
620+
model,
621+
rows.filter((row) => !remaining.has(row.id)),
622+
COMBINED_QUERY,
623+
replacement,
624+
);
625+
model.dispose();
626+
});
627+
628+
test("a mid-flight delta on a >1-chunk identity sweep still upgrades and settles equal to cold (M2 amendment)", async () => {
629+
// The chunk unit shrank the mid-build window (the 8-row M1 fixtures now
630+
// finish their sweep in the synchronous first slice), so THIS pin owns
631+
// the identity lane's upgrade path under chunk units: the delta lands
632+
// after chunk 0 but before the partial chunk, forcing `upgradeForReplay`
633+
// plus per-row replay units mid-sweep. The 40-row add also grows the
634+
// slot space past the captured bitset's word coverage (1_100 slots pad
635+
// to 1_120 bits), keeping the `append`-time `cloneMembership` widening
636+
// under live mid-flight coverage now that the M1 widening pin's window
637+
// closes in the synchronous slice.
638+
const rows: Row[] = Array.from({ length: 1_100 }, (_, index) => ({
639+
id: `r${String(index).padStart(4, "0")}`,
640+
team: index % 2 === 0 ? "X" : "Y",
641+
score: index % 100,
642+
note: `n${String(index).padStart(4, "0")}`,
643+
rank: 2_000 - index,
644+
}));
645+
const { model, scheduler } = createFixture({ rows });
646+
const transition = model.setQuery(COMBINED_QUERY);
647+
const candidate = activeCandidateOf(model);
648+
expect(candidate.completedRows).toBe(1_024);
649+
const added: Row[] = Array.from({ length: 40 }, (_, index) => ({
650+
id: `z${String(index).padStart(2, "0")}`,
651+
team: "Z",
652+
score: 41, // passes lte 60 — every grown-slot row is VISIBLE.
653+
note: `zz${String(index).padStart(2, "0")}`,
654+
rank: 3_000 + index,
655+
}));
656+
expect(
657+
model.applyTransaction({
658+
update: [{ id: "r0001", changes: { score: 55 } }],
659+
remove: ["r0002"],
660+
add: added,
661+
}),
662+
).toMatchObject({ updated: 1, removed: 1, added: 40 });
663+
// Non-vacuity for the widening claim: a grown slot really lands past the
664+
// captured bitset's 1_120-bit word coverage.
665+
const target = rootOf(model);
666+
const addedSlots = added.map((row) => target.rows.get(row.id)!.slot);
667+
expect(Math.max(...addedSlots)).toBeGreaterThanOrEqual(1_120);
668+
scheduler.flushAll();
669+
await transition.finished;
670+
671+
const finalRows: Row[] = rows
672+
.filter((row) => row.id !== "r0002")
673+
.map((row) => (row.id === "r0001" ? { ...row, score: 55 } : row));
674+
finalRows.push(...added);
675+
expectSettledEqualsCold(model, finalRows, COMBINED_QUERY);
676+
expect(candidate.completedRows).toBe(candidate.totalRows);
677+
model.dispose();
678+
});
679+
680+
test("slot holes in the captured chunk are skipped for free and never counted (M2 amendment)", async () => {
681+
// Removing rows BEFORE the transition leaves holes in the captured slot
682+
// vector's chunk 0. The chunk unit must advance `completedRows` by the
683+
// POPULATED slots only: counting holes (or chunk scans) would overshoot
684+
// `totalRows` and break the terminal accounting below.
685+
const { model, scheduler } = createFixture();
686+
expect(model.applyTransaction({ remove: ["c", "f"] })).toMatchObject({
687+
removed: 2,
688+
});
689+
const transition = model.setQuery(COMBINED_QUERY);
690+
expect(scheduler.entries.length).toBeGreaterThan(0);
691+
const candidate = activeCandidateOf(model);
692+
expect(candidate.totalRows).toBe(FIXTURE_ROWS.length - 2);
693+
expect(candidate.completedRows).toBe(FIXTURE_ROWS.length - 2);
694+
695+
scheduler.flushAll();
696+
await transition.finished;
697+
expect(candidate.completedRows).toBe(candidate.totalRows);
698+
expectSettledEqualsCold(
699+
model,
700+
FIXTURE_ROWS.filter((row) => row.id !== "c" && row.id !== "f"),
701+
COMBINED_QUERY,
702+
);
703+
model.dispose();
704+
});
705+
490706
test("a settled delta-free flat transition accounts for every row (completed === total)", async () => {
491707
const { model, scheduler } = createFixture();
492708
const transition = model.setQuery(COMBINED_QUERY);

‎packages/row-model/src/__tests__/transitions.test.ts‎

Lines changed: 11 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -106,6 +106,7 @@ function createModel(options: {
106106
readonly scheduler?: CooperativeTransitionScheduler;
107107
readonly budgetMs?: number;
108108
readonly clock?: () => number;
109+
readonly maxUnitsPerSlice?: number;
109110
}) {
110111
return createLocalRowModel({
111112
rows:
@@ -119,6 +120,7 @@ function createModel(options: {
119120
transitionScheduler: options.scheduler,
120121
transitionClock: options.clock,
121122
transitionBudgetMs: options.budgetMs,
123+
transitionMaxUnitsPerSlice: options.maxUnitsPerSlice,
122124
});
123125
}
124126

@@ -341,15 +343,21 @@ describe("cooperative query and derivation transitions", () => {
341343

342344
test("yields after a bounded number of units when the injected clock never advances", async () => {
343345
const scheduler = new ManualScheduler();
346+
// Re-pinned for the M2 amendment (#490): a flat set-query's build unit is
347+
// now ONE slot-vector chunk (1_024 rows), so the default 256-unit cap can
348+
// no longer bind on a small flat fixture — the fixture spans three chunks
349+
// and caps the slice at two units, and the bounded first slice completes
350+
// exactly two chunks' worth of rows (impossible under one-row units).
344351
const model = createModel({
345-
rows: Array.from({ length: 1_000 }, (_, id) => ({
352+
rows: Array.from({ length: 3_000 }, (_, id) => ({
346353
id,
347354
team: "A",
348355
score: id,
349356
})),
350357
scheduler,
351358
clock: () => 0,
352359
budgetMs: 5,
360+
maxUnitsPerSlice: 2,
353361
});
354362

355363
const transition = model.setQuery({
@@ -360,8 +368,8 @@ describe("cooperative query and derivation transitions", () => {
360368

361369
expect(model.getState().status).toMatchObject({
362370
kind: "rebuilding",
363-
completedRows: 256,
364-
totalRows: 1_000,
371+
completedRows: 2_048,
372+
totalRows: 3_000,
365373
});
366374
expect(scheduler.entries).toHaveLength(1);
367375
scheduler.flushAll();

0 commit comments

Comments
 (0)