feat(bench): H13 — streaming update frame budget hypothesis by blove · Pull Request #15 · cacheplane/pretable

blove · 2026-04-26T00:06:11Z

Summary

Adds H13 — Streaming Update Frame Budget to the bench hypothesis report. Pretable's updates script on the S5 streaming-updates scenario must keep frame p95 within one 60Hz frame (≤ 16 ms) with zero long tasks (>50 ms blocking) across the 3-second test.

This is Phase 1: absolute thresholds against pretable only. Phase 2 will promote H13 to a comparative claim once the comparator update paths are wired (Grid Alpha via applyTransaction, GridGamma X via apiRef.updateRows, GridBeta via setRows).

Initial evidence (local, 3 repeats, hypothesis scale)

H13: SATISFIED
  frame p95:        9 ms median (8.8–9.1 ms across 3 repeats)
  long tasks:       0 / 0 / 0
  updates total:    3,000 over 3 seconds (50/tick × 60 ticks)
  threshold p95:    ≤ 16 ms (44% headroom)
  threshold tasks:  0 long tasks

This is the first measurable claim about Pretable's streaming behavior, closing the wedge story we shipped in PRs #9 (json-stream), #10 (stream-adapter), and #14 (streaming-demo app).

Test plan

33 bench-matrix tests pass (29 baseline + 4 new: array-length bump, insufficient-without-data, satisfied-at-12ms, failing-at-24ms, failing-with-long-tasks)
Local matrix run confirms H13 SATISFIED on real data
CI: workspace tests, typecheck, lint, format, build all green

Notes for reviewers

H13 returns "insufficient" until someone runs the matrix with explicit --scenarios=S5 --scripts=updates --adapters=pretable. The default scenario set (S1, S2, S3, S7) doesn't include S5.
Default scenarios/scripts are intentionally not changed in this PR. Adding S5/updates to defaults would significantly slow standard bench runs; that's a separate decision.
"scroll_frame_p95_ms" metric name is reused for the updates script (poor naming inherited from the runtime). Tightening the metric naming is out of scope for this PR.

Phase 1 of the streaming hypothesis: absolute thresholds against pretable only on the S5 streaming-updates scenario. The `updates` script must keep frame p95 within one 60Hz frame budget (≤ 16ms) with zero long tasks (>50ms blocking) across the 3-second test. Phase 2 (deferred) will promote H13 to a comparative claim once the comparator update paths are wired (AG Grid via applyTransaction, MUI X via apiRef.updateRows, TanStack via setRows-with-batching). For now, H13 returns "insufficient" until someone runs the matrix with explicit `--scenarios=S5 --scripts=updates --adapters=pretable` since the default scenario set doesn't include S5. Adds 4 tests: array-length bump (9 → 10 hypotheses), insufficient without data, satisfied at 12ms p95 / 0 long tasks, failing at 24ms p95, failing with 2 long tasks. All 33 bench-matrix tests pass. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Two stale gates removed from apps/bench/tests/bench.spec.ts: 1. interactionSupported only allowed S2 — but bench-runner has supported interaction scripts on both S2 and S7 since the pinned-inspection work landed. Running the matrix with S7/sort failed because the test expected status: "unsupported" while the runner returned status: "completed". 2. The unsupported-bail branch only fired for interaction scripts. `updates` on a non-pretable adapter or a non-S5 scenario now also returns status: "unsupported", which the test failed to handle. Now the test simply trusts the runtime: if the result is unsupported, make the appropriate assertion and bail; otherwise run the completed-run assertions. This matches whatever support matrix the bench-runner declares without re-encoding it in the test. Discovered while running pnpm bench:matrix to refresh hypothesis state post-streaming work; needed to populate H13. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

#16) The pitch landing now surfaces the streaming wedge between the inspection grid and the receipts band. Section copy frames the wedge ("stream tokens, render 500 rows in 30s") and shows three H13-derived metrics — frame p95 9ms, 0 long tasks, 1k updates/sec sustained. CTA links to /streaming/, which is where the standalone apps/streaming-demo app is expected to deploy. Numbers come from the H13 hypothesis report (S5 streaming-updates, hypothesis scale, 3 repeats) merged in PR #15. They are not yet wired to a live source; treating them as a placeholder is consistent with how ReceiptsBand currently handles its own metrics. Order of sections in the pitch landing is now: PitchHero → PitchGrid → StreamingProof → ReceiptsBand Tests: 3 new for the section + 1 updated app.test.tsx ordering check. Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

Promote H13 from absolute-only (PR #15) to a comparative claim. All four bench adapters now wire their idiomatic streaming pattern: - Pretable: stream-adapter batcher → grid.applyTransaction (RAF-batched) - AG Grid: gridApi.applyTransaction({ update }) (native batching) - MUI X Community: apiRef.updateRows([row]) — looped per patch since Community caps updateRows to one row per call (Pro/Premium-only batch feature). Faithful to the idiomatic Community API. - TanStack Virtual: setRows merge — rows lifted to useState, every batch triggers a React reconciliation. Inherent cost of "primitive virtualization + React state" updates, which is the wedge story. The H13 evaluator now mirrors H1's structure: 1. Pretable must clear absolute thresholds (frame p95 ≤ 16ms, 0 long tasks) 2. Frame p95 must be within 10% of the best measured comparator 3. At least one comparator must fail the absolute thresholds (uniqueness) Without comparator data → directional. All comparators clear absolutes → directional. Pretable beats parity + uniqueness → satisfied. bench-runner gating updated: `updates` script now allowed for all four adapters on S5 (was pretable-only). bench-app waits up to ~1s for the adapter's update API to publish (AG Grid fires onGridReady several RAFs after mount, was racing past it). ## H13 — comparative result on Chromium, hypothesis scale, 3 repeats | Adapter | frame p95 | long tasks | clears absolutes | |---------|-----------|------------|------------------| | Pretable | 8.9 ms | 0 | yes | | AG Grid | 9.0 ms | 0 | yes | | TanStack | 8.8 ms | 0 | yes | | MUI X (Community) | **100 ms** | **59** | no | Pretable parity vs best comparator (TanStack 8.8ms): 9/8.8 = 1.01, within the 10% band. MUI fails the absolute thresholds → uniqueness claim supported → H13 SATISFIED with comparative evidence. The honest read: Pretable's idiomatic streaming pattern matches the strong comparators (AG Grid, TanStack) and decisively beats the weak one (MUI Community). The streaming wedge is real but narrower than a universal "we're 4× faster than every grid" framing — it's specifically "we don't fall apart under streaming the way some grids do." bench-matrix tests: 36 → 38 (added comparative pass + parity-failing cases, replaced absolute-only satisfied test with directional-without- comparator). bench-runner test updated to expect all 4 adapters on S5/updates. Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

The pitch landing now surfaces the streaming wedge between the inspection grid and the receipts band. Section copy frames the wedge ("stream tokens, render 500 rows in 30s") and shows three H13-derived metrics — frame p95 9ms, 0 long tasks, 1k updates/sec sustained. CTA links to /streaming/, which is where the standalone apps/streaming-demo app is expected to deploy. Numbers come from the H13 hypothesis report (S5 streaming-updates, hypothesis scale, 3 repeats) merged in PR #15. They are not yet wired to a live source; treating them as a placeholder is consistent with how ReceiptsBand currently handles its own metrics. Order of sections in the pitch landing is now: PitchHero → PitchGrid → StreamingProof → ReceiptsBand Tests: 3 new for the section + 1 updated app.test.tsx ordering check. Co-Authored-By: Assistant Opus 4.6 <noreply@anthropic.com>

Promote H13 from absolute-only (PR #15) to a comparative claim. All four bench adapters now wire their idiomatic streaming pattern: - Pretable: stream-adapter batcher → grid.applyTransaction (RAF-batched) - Grid Alpha: gridApi.applyTransaction({ update }) (native batching) - GridGamma X Community: apiRef.updateRows([row]) — looped per patch since Community caps updateRows to one row per call (Pro/Premium-only batch feature). Faithful to the idiomatic Community API. - GridBeta Virtual: setRows merge — rows lifted to useState, every batch triggers a React reconciliation. Inherent cost of "primitive virtualization + React state" updates, which is the wedge story. The H13 evaluator now mirrors H1's structure: 1. Pretable must clear absolute thresholds (frame p95 ≤ 16ms, 0 long tasks) 2. Frame p95 must be within 10% of the best measured comparator 3. At least one comparator must fail the absolute thresholds (uniqueness) Without comparator data → directional. All comparators clear absolutes → directional. Pretable beats parity + uniqueness → satisfied. bench-runner gating updated: `updates` script now allowed for all four adapters on S5 (was pretable-only). bench-app waits up to ~1s for the adapter's update API to publish (Grid Alpha fires onGridReady several RAFs after mount, was racing past it). ## H13 — comparative result on Chromium, hypothesis scale, 3 repeats | Adapter | frame p95 | long tasks | clears absolutes | |---------|-----------|------------|------------------| | Pretable | 8.9 ms | 0 | yes | | Grid Alpha | 9.0 ms | 0 | yes | | GridBeta | 8.8 ms | 0 | yes | | GridGamma X (Community) | **100 ms** | **59** | no | Pretable parity vs best comparator (GridBeta 8.8ms): 9/8.8 = 1.01, within the 10% band. GridGamma fails the absolute thresholds → uniqueness claim supported → H13 SATISFIED with comparative evidence. The honest read: Pretable's idiomatic streaming pattern matches the strong comparators (Grid Alpha, GridBeta) and decisively beats the weak one (GridGamma Community). The streaming wedge is real but narrower than a universal "we're 4× faster than every grid" framing — it's specifically "we don't fall apart under streaming the way some grids do." bench-matrix tests: 36 → 38 (added comparative pass + parity-failing cases, replaced absolute-only satisfied test with directional-without- comparator). bench-runner test updated to expect all 4 adapters on S5/updates. Co-Authored-By: Assistant Opus 4.6 <noreply@anthropic.com>

blove and others added 2 commits April 25, 2026 20:04

blove merged commit 322fade into main Apr 26, 2026
5 checks passed

blove mentioned this pull request Apr 27, 2026

feat(playground): add StreamingProof section to pitch landing #16

Merged

3 of 4 tasks

blove mentioned this pull request May 1, 2026

feat(bench): H13 phase 2 — comparative streaming hypothesis #23

Merged

4 of 5 tasks

Navigation Menu

Repository navigation

feat(bench): H13 — streaming update frame budget hypothesis - #15

feat(bench): H13 — streaming update frame budget hypothesis#15
blove merged 2 commits into
mainfrom
feat/h13-streaming-hypothesis

blove commented Apr 26, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

blove commented Apr 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Initial evidence (local, 3 repeats, hypothesis scale)

Test plan

Notes for reviewers

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

blove commented Apr 26, 2026 •

edited

Loading