Skip to content

Commit 28acf8e

Browse files
bloveclaude
andauthored
feat(bench): H13 phase 2 — comparative streaming hypothesis (#23)
Promote H13 from absolute-only (PR #15) to a comparative claim. All four bench adapters now wire their idiomatic streaming pattern: - Pretable: stream-adapter batcher → grid.applyTransaction (RAF-batched) - AG Grid: gridApi.applyTransaction({ update }) (native batching) - MUI X Community: apiRef.updateRows([row]) — looped per patch since Community caps updateRows to one row per call (Pro/Premium-only batch feature). Faithful to the idiomatic Community API. - TanStack Virtual: setRows merge — rows lifted to useState, every batch triggers a React reconciliation. Inherent cost of "primitive virtualization + React state" updates, which is the wedge story. The H13 evaluator now mirrors H1's structure: 1. Pretable must clear absolute thresholds (frame p95 ≤ 16ms, 0 long tasks) 2. Frame p95 must be within 10% of the best measured comparator 3. At least one comparator must fail the absolute thresholds (uniqueness) Without comparator data → directional. All comparators clear absolutes → directional. Pretable beats parity + uniqueness → satisfied. bench-runner gating updated: `updates` script now allowed for all four adapters on S5 (was pretable-only). bench-app waits up to ~1s for the adapter's update API to publish (AG Grid fires onGridReady several RAFs after mount, was racing past it). ## H13 — comparative result on Chromium, hypothesis scale, 3 repeats | Adapter | frame p95 | long tasks | clears absolutes | |---------|-----------|------------|------------------| | Pretable | 8.9 ms | 0 | yes | | AG Grid | 9.0 ms | 0 | yes | | TanStack | 8.8 ms | 0 | yes | | MUI X (Community) | **100 ms** | **59** | no | Pretable parity vs best comparator (TanStack 8.8ms): 9/8.8 = 1.01, within the 10% band. MUI fails the absolute thresholds → uniqueness claim supported → H13 SATISFIED with comparative evidence. The honest read: Pretable's idiomatic streaming pattern matches the strong comparators (AG Grid, TanStack) and decisively beats the weak one (MUI Community). The streaming wedge is real but narrower than a universal "we're 4× faster than every grid" framing — it's specifically "we don't fall apart under streaming the way some grids do." bench-matrix tests: 36 → 38 (added comparative pass + parity-failing cases, replaced absolute-only satisfied test with directional-without- comparator). bench-runner test updated to expect all 4 adapters on S5/updates. Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
1 parent 2cb7bd2 commit 28acf8e

10 files changed

Lines changed: 464 additions & 59 deletions

File tree

‎apps/bench/src/ag-grid-adapter.tsx‎

Lines changed: 28 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,25 +1,50 @@
1-
import { useMemo } from "react";
1+
import { useCallback, useMemo, useRef } from "react";
22

33
import type { ScenarioDataset } from "@pretable-internal/scenario-data";
44
import { AgGridReact } from "ag-grid-react";
55
import {
66
AllCommunityModule,
77
ModuleRegistry,
88
type ColDef,
9+
type GridApi,
10+
type GridReadyEvent,
911
} from "ag-grid-community";
1012

1113
import "ag-grid-community/styles/ag-grid.css";
1214
import "ag-grid-community/styles/ag-theme-quartz.css";
1315

16+
import type { ApplyBenchUpdates } from "./bench-runtime";
17+
1418
ModuleRegistry.registerModules([AllCommunityModule]);
1519

1620
export interface AGGridAdapterProps {
1721
dataset: ScenarioDataset;
1822
runKey: number;
23+
/**
24+
* AG Grid's idiomatic streaming pattern: gridApi.applyTransaction({
25+
* update }). Native batching, no wrapper layer.
26+
*/
27+
onUpdateApiReady?: (apply: ApplyBenchUpdates) => void;
1928
}
2029

21-
export function AGGridAdapter({ dataset, runKey }: AGGridAdapterProps) {
30+
export function AGGridAdapter({
31+
dataset,
32+
runKey,
33+
onUpdateApiReady,
34+
}: AGGridAdapterProps) {
2235
const rowData = useMemo(() => [...dataset.rows], [dataset.rows]);
36+
37+
const onUpdateApiReadyRef = useRef(onUpdateApiReady);
38+
// eslint-disable-next-line react-hooks/refs -- sync ref to latest prop for use in callbacks
39+
onUpdateApiReadyRef.current = onUpdateApiReady;
40+
41+
const handleGridReady = useCallback((event: GridReadyEvent) => {
42+
const api: GridApi = event.api;
43+
const apply: ApplyBenchUpdates = (patches) => {
44+
api.applyTransaction({ update: patches });
45+
};
46+
onUpdateApiReadyRef.current?.(apply);
47+
}, []);
2348
const columnDefs = useMemo<ColDef[]>(
2449
() =>
2550
dataset.columns.map((column) => ({
@@ -81,6 +106,7 @@ export function AGGridAdapter({ dataset, runKey }: AGGridAdapterProps) {
81106
defaultColDef={defaultColDef}
82107
getRowId={(params) => String(params.data.id)}
83108
rowData={rowData}
109+
onGridReady={handleGridReady}
84110
/>
85111
</div>
86112
</section>

‎apps/bench/src/bench-app.tsx‎

Lines changed: 42 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,11 @@
1-
import { useEffect, useEffectEvent, useMemo, useRef, useState } from "react";
1+
import {
2+
useCallback,
3+
useEffect,
4+
useEffectEvent,
5+
useMemo,
6+
useRef,
7+
useState,
8+
} from "react";
29
import type { PretableGrid } from "@pretable/react";
310
import type { PretableTelemetry } from "@pretable/react/internal";
411

@@ -17,6 +24,7 @@ import {
1724

1825
import type { BenchQueryState } from "./bench-types";
1926
import {
27+
type ApplyBenchUpdates,
2028
createBenchInteractionStateFromTelemetry,
2129
createPretableTelemetryNotes,
2230
createBenchRequest,
@@ -91,6 +99,16 @@ export function BenchApp({ search, browserVersion }: BenchAppProps) {
9199
const autorunRef = useRef(false);
92100
const pretableTelemetryRef = useRef<PretableTelemetry | null>(null);
93101
const pretableGridRef = useRef<PretableGrid<ScenarioRow> | null>(null);
102+
/**
103+
* Adapter-agnostic update API ref. Each adapter wires its idiomatic
104+
* streaming pattern (Pretable: stream-adapter batcher → applyTransaction;
105+
* AG Grid: gridApi.applyTransaction; MUI: apiRef.updateRows; TanStack:
106+
* setRows merge) and exposes a uniform `apply(patches)` callback here.
107+
*/
108+
const updateApiRef = useRef<ApplyBenchUpdates | null>(null);
109+
const handleUpdateApiReady = useCallback((apply: ApplyBenchUpdates) => {
110+
updateApiRef.current = apply;
111+
}, []);
94112
const interactionPlan =
95113
interactionPlanOverride?.search === search
96114
? interactionPlanOverride.plan
@@ -187,11 +205,25 @@ export function BenchApp({ search, browserVersion }: BenchAppProps) {
187205
: null
188206
: null;
189207

208+
// Wait up to ~1s for the current adapter to publish its update API.
209+
// AG Grid in particular fires onGridReady asynchronously a few RAFs
210+
// after mount, so kicking off the updates script in the very next
211+
// frame after setRunKey would race past it and leave updateApiRef
212+
// null (no metrics get collected).
213+
let updatesApi = updateApiRef.current;
214+
if (scriptName === "updates" && !updatesApi) {
215+
for (let i = 0; i < 60 && !updateApiRef.current; i++) {
216+
await waitForNextAnimationFrame();
217+
}
218+
updatesApi = updateApiRef.current;
219+
}
220+
190221
const updatesRun =
191-
scriptName === "updates" && pretableGridRef.current
222+
scriptName === "updates" && updatesApi
192223
? await measureBenchUpdatesRun(
193224
viewportRef.current ?? document.body,
194-
pretableGridRef.current,
225+
query.adapterId,
226+
updatesApi,
195227
dataset,
196228
)
197229
: null;
@@ -371,10 +403,16 @@ export function BenchApp({ search, browserVersion }: BenchAppProps) {
371403
onTelemetryChange={(telemetry) => {
372404
pretableTelemetryRef.current = telemetry;
373405
}}
406+
onUpdateApiReady={handleUpdateApiReady}
374407
runKey={runKey}
375408
/>
376409
) : (
377-
<AdapterSurface dataset={dataset} key={runKey} runKey={runKey} />
410+
<AdapterSurface
411+
dataset={dataset}
412+
key={runKey}
413+
onUpdateApiReady={handleUpdateApiReady}
414+
runKey={runKey}
415+
/>
378416
)}
379417
</div>
380418

‎apps/bench/src/bench-runtime.ts‎

Lines changed: 25 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -568,17 +568,27 @@ export interface UpdatesBenchRunResult {
568568
notes: string[];
569569
}
570570

571+
/**
572+
* Caller-supplied function that applies a batch of update patches to the
573+
* adapter's grid. Each adapter wires this to its idiomatic streaming
574+
* pattern (Pretable: stream-adapter batcher → applyTransaction; AG Grid:
575+
* gridApi.applyTransaction directly; MUI: apiRef.updateRows directly;
576+
* TanStack: setRows merge).
577+
*/
578+
export type ApplyBenchUpdates = (
579+
patches: Record<string, unknown>[],
580+
) => void | Promise<void>;
581+
571582
export async function measureBenchUpdatesRun(
572583
root: HTMLElement,
573-
grid: {
574-
applyTransaction(transaction: { update?: Record<string, unknown>[] }): void;
575-
},
584+
adapterId: BenchQueryState["adapterId"],
585+
apply: ApplyBenchUpdates,
576586
dataset: {
577587
rows: readonly Record<string, unknown>[];
578588
columns: readonly { id: string }[];
579589
},
580590
): Promise<UpdatesBenchRunResult> {
581-
const profile = scrollRuntimeProfiles.pretable;
591+
const profile = scrollRuntimeProfiles[adapterId];
582592
const viewport = await waitForScrollViewport(root, profile.viewportSelector);
583593
const viewportPolicyNotes = viewport
584594
? detectViewportPolicyNotes(viewport)
@@ -597,9 +607,6 @@ export async function measureBenchUpdatesRun(
597607
const UPDATES_PER_TICK = 50;
598608
const columnIds = dataset.columns.map((c) => c.id);
599609

600-
const { createBatcher } = await import("@pretable-internal/stream-adapter");
601-
const batcher = createBatcher(grid);
602-
603610
let totalUpdates = 0;
604611
const longTaskDurations: number[] = [];
605612
const observer = createLongTaskObserver(longTaskDurations);
@@ -641,12 +648,21 @@ export async function measureBenchUpdatesRun(
641648
patches.push({ id, [columnId]: `upd-${totalUpdates + i}` });
642649
}
643650

644-
batcher.update(patches);
651+
const applyResult = apply(patches);
652+
if (applyResult && typeof applyResult.then === "function") {
653+
// The caller can return a Promise (e.g., flush before resolve);
654+
// we don't await within the interval to keep the cadence honest,
655+
// but we do swallow rejections so they surface via the outer
656+
// try/catch above.
657+
applyResult.catch((err) => {
658+
clearInterval(interval);
659+
reject(err);
660+
});
661+
}
645662
totalUpdates += UPDATES_PER_TICK;
646663

647664
if (elapsed >= DURATION_MS) {
648665
clearInterval(interval);
649-
batcher.flush();
650666
resolve();
651667
}
652668
} catch (err) {
@@ -656,7 +672,6 @@ export async function measureBenchUpdatesRun(
656672
}, BATCH_INTERVAL_MS);
657673
});
658674
} finally {
659-
batcher.dispose();
660675
rafHandle.running = false;
661676
cancelAnimationFrame(rafHandle.id);
662677
observer?.disconnect();

‎apps/bench/src/mui-adapter.tsx‎

Lines changed: 34 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,14 +1,44 @@
1-
import { useMemo } from "react";
1+
import { useEffect, useMemo, useRef } from "react";
22

33
import type { ScenarioDataset } from "@pretable-internal/scenario-data";
4-
import { DataGrid, type GridColDef } from "@mui/x-data-grid";
4+
import { DataGrid, useGridApiRef, type GridColDef } from "@mui/x-data-grid";
5+
6+
import type { ApplyBenchUpdates } from "./bench-runtime";
57

68
export interface MuiAdapterProps {
79
dataset: ScenarioDataset;
810
runKey: number;
11+
/**
12+
* MUI X DataGrid's idiomatic streaming pattern:
13+
* apiRef.current.updateRows([{ id, ...patch }]). Native batching.
14+
*/
15+
onUpdateApiReady?: (apply: ApplyBenchUpdates) => void;
916
}
1017

11-
export function MuiAdapter({ dataset, runKey }: MuiAdapterProps) {
18+
export function MuiAdapter({
19+
dataset,
20+
runKey,
21+
onUpdateApiReady,
22+
}: MuiAdapterProps) {
23+
const apiRef = useGridApiRef();
24+
const onUpdateApiReadyRef = useRef(onUpdateApiReady);
25+
// eslint-disable-next-line react-hooks/refs -- sync ref to latest prop for use in callbacks
26+
onUpdateApiReadyRef.current = onUpdateApiReady;
27+
28+
// Wire updates after mount, when apiRef.current is populated.
29+
useEffect(() => {
30+
const api = apiRef.current;
31+
if (!api) return;
32+
const apply: ApplyBenchUpdates = (patches) => {
33+
// MUI X Community caps updateRows to a single row per call (batched
34+
// updates are a Pro/Premium feature). To exercise the idiomatic
35+
// Community API faithfully, loop per patch.
36+
for (const patch of patches) {
37+
api.updateRows([patch]);
38+
}
39+
};
40+
onUpdateApiReadyRef.current?.(apply);
41+
}, [apiRef, runKey]);
1242
const rows = useMemo(
1343
() =>
1444
dataset.rows.map((row, index) => ({
@@ -76,6 +106,7 @@ export function MuiAdapter({ dataset, runKey }: MuiAdapterProps) {
76106
>
77107
<DataGrid
78108
key={runKey}
109+
apiRef={apiRef}
79110
rows={rows}
80111
columns={columns}
81112
getRowHeight={() => "auto" as const}

‎apps/bench/src/pretable-adapter.tsx‎

Lines changed: 30 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,20 +1,29 @@
1-
import { useCallback, useMemo, useRef } from "react";
1+
import { useCallback, useEffect, useMemo, useRef } from "react";
22

33
import type { PretableGrid } from "@pretable/react";
44
import type { PretableTelemetry } from "@pretable/react/internal";
55
import { PretableSurface } from "@pretable/react/internal";
6+
import { createBatcher } from "@pretable-internal/stream-adapter";
67
import type {
78
ScenarioDataset,
89
ScenarioRow,
910
} from "@pretable-internal/scenario-data";
1011

12+
import type { ApplyBenchUpdates } from "./bench-runtime";
1113
import type { BenchInteractionPlan } from "./interaction-plan";
1214

1315
export interface PretableAdapterProps {
1416
dataset: ScenarioDataset;
1517
interactionPlan?: BenchInteractionPlan | null;
1618
onGridReady?: (grid: PretableGrid<ScenarioRow>) => void;
1719
onTelemetryChange?: (telemetry: PretableTelemetry) => void;
20+
/**
21+
* Called once the adapter has wired up its update mechanism. Pretable
22+
* routes batches through @pretable-internal/stream-adapter's RAF-based
23+
* batcher → grid.applyTransaction. This is the wedge's idiomatic
24+
* streaming pattern.
25+
*/
26+
onUpdateApiReady?: (apply: ApplyBenchUpdates) => void;
1827
runKey: number;
1928
}
2029

@@ -36,6 +45,7 @@ export function PretableAdapter({
3645
interactionPlan,
3746
onGridReady,
3847
onTelemetryChange,
48+
onUpdateApiReady,
3949
runKey,
4050
}: PretableAdapterProps) {
4151
const adapterRef = useRef<HTMLElement>(null);
@@ -47,12 +57,31 @@ export function PretableAdapter({
4757
const onGridReadyRef = useRef(onGridReady);
4858
// eslint-disable-next-line react-hooks/refs -- sync ref to latest prop for use in callbacks
4959
onGridReadyRef.current = onGridReady;
60+
const onUpdateApiReadyRef = useRef(onUpdateApiReady);
61+
// eslint-disable-next-line react-hooks/refs -- sync ref to latest prop for use in callbacks
62+
onUpdateApiReadyRef.current = onUpdateApiReady;
5063

5164
const handleGridReady = useCallback((grid: PretableGrid<ScenarioRow>) => {
5265
gridRef.current = grid;
5366
onGridReadyRef.current?.(grid);
5467
}, []);
5568

69+
// Wire updates through the stream-adapter batcher (RAF-aligned), the
70+
// same path real consumers use for LLM-rate streaming. The batcher is
71+
// recreated on each runKey change so a re-run starts with empty buffers.
72+
useEffect(() => {
73+
const grid = gridRef.current;
74+
if (!grid) return;
75+
const batcher = createBatcher<ScenarioRow>(grid);
76+
const apply: ApplyBenchUpdates = (patches) => {
77+
batcher.update(patches as Partial<ScenarioRow>[]);
78+
};
79+
onUpdateApiReadyRef.current?.(apply);
80+
return () => {
81+
batcher.dispose();
82+
};
83+
}, [runKey]);
84+
5685
const onTelemetryChangeRef = useRef(onTelemetryChange);
5786
// eslint-disable-next-line react-hooks/refs -- sync ref to latest prop for use in callbacks
5887
onTelemetryChangeRef.current = onTelemetryChange;

0 commit comments

Comments
 (0)