Repository navigation
feat(bench): H13 phase 2 — comparative streaming hypothesis - #23
Merged
Merged
Conversation
Promote H13 from absolute-only (PR #15) to a comparative claim. All four bench adapters now wire their idiomatic streaming pattern: - Pretable: stream-adapter batcher → grid.applyTransaction (RAF-batched) - AG Grid: gridApi.applyTransaction({ update }) (native batching) - MUI X Community: apiRef.updateRows([row]) — looped per patch since Community caps updateRows to one row per call (Pro/Premium-only batch feature). Faithful to the idiomatic Community API. - TanStack Virtual: setRows merge — rows lifted to useState, every batch triggers a React reconciliation. Inherent cost of "primitive virtualization + React state" updates, which is the wedge story. The H13 evaluator now mirrors H1's structure: 1. Pretable must clear absolute thresholds (frame p95 ≤ 16ms, 0 long tasks) 2. Frame p95 must be within 10% of the best measured comparator 3. At least one comparator must fail the absolute thresholds (uniqueness) Without comparator data → directional. All comparators clear absolutes → directional. Pretable beats parity + uniqueness → satisfied. bench-runner gating updated: `updates` script now allowed for all four adapters on S5 (was pretable-only). bench-app waits up to ~1s for the adapter's update API to publish (AG Grid fires onGridReady several RAFs after mount, was racing past it). ## H13 — comparative result on Chromium, hypothesis scale, 3 repeats | Adapter | frame p95 | long tasks | clears absolutes | |---------|-----------|------------|------------------| | Pretable | 8.9 ms | 0 | yes | | AG Grid | 9.0 ms | 0 | yes | | TanStack | 8.8 ms | 0 | yes | | MUI X (Community) | **100 ms** | **59** | no | Pretable parity vs best comparator (TanStack 8.8ms): 9/8.8 = 1.01, within the 10% band. MUI fails the absolute thresholds → uniqueness claim supported → H13 SATISFIED with comparative evidence. The honest read: Pretable's idiomatic streaming pattern matches the strong comparators (AG Grid, TanStack) and decisively beats the weak one (MUI Community). The streaming wedge is real but narrower than a universal "we're 4× faster than every grid" framing — it's specifically "we don't fall apart under streaming the way some grids do." bench-matrix tests: 36 → 38 (added comparative pass + parity-failing cases, replaced absolute-only satisfied test with directional-without- comparator). bench-runner test updated to expect all 4 adapters on S5/updates. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Promotes H13 from absolute-only (PR #15) to a comparative claim, mirroring H1's structure. All four bench adapters wire their idiomatic streaming pattern:
H13 evaluator structure
Without comparator data → `directional`. All comparators clear absolutes → `directional`. Pretable hits parity + uniqueness → `satisfied`.
Real comparative result (Chromium, hypothesis scale, 3 repeats)
Pretable parity vs best (GridBeta 8.8 ms): 9 / 8.8 = 1.01 — within 10% band. GridGamma fails absolutes → uniqueness supported → H13 SATISFIED with comparative evidence.
The honest read: Pretable matches the strong comparators (Grid Alpha, GridBeta) and decisively beats the weak one (GridGamma Community). The streaming wedge is real but narrower than a universal "we're 4× faster than every grid" framing — it's specifically "we don't fall apart under streaming the way some grids do."
Side fixes
Test plan