docs(bench): re-derive the in-process prose from the re-measured leg - #200
Merged
Conversation
…sured on mains Re-measure in-process and Redis on the pinned matrix and regenerate the charts. Both passes ran 2026-09-11 on the document's host and toolchain, one at a time, with Low Power Mode off; the other four legs are untouched. in-process: the published leg predated the 32 GiB drain byte cap, so its 64 KiB drain corpus was 49 152 and all eleven 64 KiB drain rows were setup_bound and withheld. At 524 288 all twelve publish with windows of 3.4 to 9.1 s, the 1 KiB leg sits on the pinned 6 000 000 corpus, and the one failure (consume_batch 64 KiB 8c consumed before assembly) is gone. 167 rows, no failures. The twelve 64 B fills run 9.8 to 10.7 s in execution order. redis: the published leg predated the 1 s acked-entry trim cadence and the direct consumer honouring concurrent_processing. Its nine failures, all 64 KiB offered-load rungs refused with XADD OOM, do not recur, and the 64 KiB ladder carries 24 rows against 15. The consume_parallel drains rise 10 to 25x at 64 B and 1 KiB and now sit level with consumer_group on the same primitive; every other cell, publish included, rises a near-uniform 26 to 45 percent while the 64 B fill drops from 89 s to 64 s, the host speed signature the runbook records for Low Power Mode. The twelve 64 B fills run 63.8 to 68.9 s in execution order. 156 rows, no failures; the consume_batch 64 KiB 8c drain is newly setup_bound at a 0.9 s window over the byte-capped 49 152 corpus.
One table under Performance: the batch flow's best published drain cell per backend and payload, its consumer count, and the ratio to the parallel consumer at the same cell, all read off the committed document.
…pport
The 64 B floor was written as 3x, but the published two-consumer cell
measures 2.76x, so a bar the charts plot sits under its own stated range.
The 1 KiB floor had the same inward rounding (1.96x written as 2x). Inward
rounding on a range is the direction that makes a claim stronger than its
data, and it is falsifiable by one row a reader can see.
Both ranges are now the measured values to one decimal, each bound
re-derivable from the drain rows in benches/results/bench-results.json:
64 B batch/parallel 2.8x - 10.5x (2.76 / 3.77 / 9.83 / 10.47 at 1-8c;
all four handler_cost=framework)
1 KiB batch/parallel 2.0x - 6.9x
64 KiB batch/parallel 0.27x - 0.50x, i.e. 2.0x - 3.7x slower
The 64 KiB figure was already rounded outward and is unchanged in
substance. Remaining diff lines are paragraph reflow.
Prose only; no results-document, chart or code change.
Re-derived every claim on the performance page against benches/results/bench-results.json. Three did not survive. The n/s marker. The page said a backend that cannot do a flow "appears as an explicit 'not supported' marker". The marker is spelled `n/s`, it covers five distinct causes, and chartgen is explicit that it is "never 'not supported' for a mere gap". Every n/s in the committed set is in-process on a flow it fully supports - four cells, all in the parallel-vs-sequenced family: sequenced at 64 B and 1 KiB, parallel and batch at 64 KiB - each withheld because every window ran under one second - so a reader following the old sentence would conclude in-process cannot consume in parallel at 64 KiB, which is the fastest cell in the document. The convention is now described as it is: an explicit marker instead of a zero or a silent gap, with the caption naming which cause applies. The ratio bounds. Restated to one decimal in the previous commit, but rounded to nearest rather than outward, so three of six bounds still excluded a row: the 64 B floor 2.8 against a measured 2.762, the 1 KiB floor 2.0 against 1.964 and ceiling 6.9 against 6.902, and the 64 KiB slower ceiling 3.7 against 3.745. Now 2.7-10.5x, 1.9-7.0x and 2.0-3.8x slower, each bound rounded away from the rows it covers. The reproduction recipe. The methodology section gave a hand-assembled command as what generated the published charts. It omits --concurrent, both drain knobs and the whole offered-load ladder, so it produces neither the 6.0M/3.1M/49k drain corpora nor any of the 502 offered_load rows the dispatch-latency chart renders from, and it defaults the byte cap to 2 GiB against the pinned 3 GiB. It also restates a matrix the runbook pins in one place on purpose. Replaced with scripts/bench.sh per backend, plus the --fresh a different host needs, and the two measurement methods named. README's matching "reproduce with cargo run --example <backend>_stress" sentence and its pointer to "what is still pending" (nothing is) follow. Prose only: cargo nextest run --no-default-features -E 'binary(chartgen)' is 131/131, including the byte-compare against the committed SVGs.
The in-process leg in the results document was re-measured under the 32 GiB byte cap, which moves every number the prose quoted about it. `performance.mdx`: the batch-vs-parallel drain ratios are recomputed cell by cell against the new rows - 3.6-9.7x at 64 B, 0.9-3.1x at 1 KiB, 0.5-1.2x at 64 KiB, each bound rounded outward from the widest row. The previous "1.9-7.0x faster at 1 KiB" and "2.0-3.8x slower at 64 KiB" are both falsified by rows in the document. The "charts withhold every 64 KiB drain" caveat is gone because all twelve now publish; the one withheld cell behind the ranges is the 64 B two-consumer batch drain, so it is named instead. The off-matrix batch-size probe is dropped: it was anchored to a 356k parallel row measured under the 3 GiB cap, and the leg that row came from is no longer published. `benches/README.md`: the byte cap section described what a future pass would produce and said outright that none of it was in the document. That pass has run, so the section states measured rows - 524 288 at 64 KiB, all twelve drains `framework` in 3.4-9.1 s, the 1 KiB leg back on the pinned 6 000 000 with no `setup_bound` drain row at either payload - and records that the forecast rate did not hold at the larger corpus. The corpus-differs caption is updated to the one the charts now render, and the RabbitMQ "consumed before assembly" cell is restated as carried over rather than re-measured. Also: the "Reading a run" list said three items and has six.
Deploying with
|
| Status | Name | Latest Commit | Preview URL | Updated (UTC) |
|---|---|---|---|---|
| ✅ Deployment successful! View logs |
shove-docs | 8620725 | Commit Preview URL Branch Preview URL |
Sep 11 2026, 12:02 PM |
This was referenced Sep 11, 2026
main squash-merged #199, so this branch's copies of its two commits are now duplicates. The only conflict is the batching paragraph in performance.mdx, where main carries the pre-re-derivation text and this branch carries the version derived from the re-measured leg; the merge keeps the latter.
Owner
Author
The body above is out of date — #197 and #199 both merged while this was being written
What changed. The body says this depends on #199 and supersedes #197. Both merged in the meantime (
The diff is unaffected. Against Everything else in the body still holds. The document it derives from is byte-identical to the one on |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Re-derives the in-process prose from the leg #199 re-measured. #197 wrote the batch-vs-parallel paragraph the new leg falsifies, which is why this re-derivation exists.
#197 and #199 have both merged to
mainsince this was opened, andmainhas been merged into this branch, so the diff againstmainis now only the re-derivation itself:benches/README.mdanddocs/pages/ops/performance.mdx.What the re-measured leg changed
Every figure below was recomputed from
benches/results/bench-results.jsonas it stands on #199, not carried over.docs/pages/ops/performance.mdxThe batch-vs-parallel ratios are the in-process batch drain over the plain parallel drain at the same cell, every bound rounded outward from the widest row:
Two of the three previous claims are falsified by rows in the new document: at 1 KiB the single-consumer cell is below parity (0.948), and at 64 KiB the eight-consumer cell is above it (1.135), so "1.9-7.0x faster" and "2.0-3.8x slower" both have counterexamples. The 1 KiB leg moved from the byte-bound 3 145 728 to the pinned 6 000 000 and the 64 KiB leg from 49 152 to 524 288, which is where the movement comes from.
Two more sentences went stale rather than wrong:
frameworkin 3.4-9.1 s. The one cell behind the ranges that is withheld is the 64 B two-consumer batch drain (0.776 s), so the page names that one instead.benches/README.md— "The in-process byte cap deviation"The section was written before the cap had ever been measured, and said so: "None of that is in the published document. ... The two paragraphs above are what the next in-process pass produces, not a description of what is plotted today." That pass has now run, so the section states rows:
framework, windows 3.4-9.1 s, leg publishes whole where it was withheld whole.consumer_group2c) at 514k msg/s in ~0.9 s; measured, that cell is 138.9k in 3.39 s, and the leg spans 52k-139k against 95k-514k at the smaller corpus. A rate measured over a 0.1 s window does not predict the rate over a 5 s one.setup_bounddrain row at either payload (the cell that previously drained in 0.95 s cleared).inmemory 524k / 6.0M; kafka, nats, rabbitmq, redis 49k / 3.1M / 6.0M; sqs 49k / 60k.failures[]entry is byte-identical between the old and new documents, so "expected to fail again" is now "was not re-run".peak_rss_mbis 0.0 on every row. The 32 GiB figure that remains is the cap's own arithmetic (524 288 x 64 KiB).Also: the "Reading a run" list announced three items and has six.
Verification
cargo nextest run --no-default-features -E 'binary(chartgen)'— 131 run, 131 passed, 0 skipped, includingthe_committed_results_document_renders_every_family.Every ratio and window in this body and in the prose was recomputed from the committed document; none is quoted from a log or a previous revision.