Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
77 changes: 40 additions & 37 deletions benches/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -313,10 +313,11 @@ This is the mirror image of the SQS deviation. SQS drains a smaller corpus
because it is the only backend slow enough that the pinned *count* means
something else there; in-process is allowed a larger one because it is the only
backend fast enough that the pinned *byte cap* does. At 3 GiB the 64 KiB leg is
49 152 messages, and the 2026-09-08 run drained every one of the in-process
49 152 messages, and the last pre-cap in-process leg drained every one of its
64 KiB cells in 0.09-0.46 s — under the 1 s floor, so the harness marked all
eleven rows `setup_bound` and the charts withheld all eleven. The fastest
backend in the set published no 64 KiB consume rate at all.
backend in the set published no 64 KiB consume rate at all. That is the hole
the cap was added to close.

Why only in-process gets it: 3 GiB is not a cautious number for the others, it
is the containerised limit. Every other backend stages its corpus inside the
Expand All @@ -326,42 +327,44 @@ backlog has already taken Redis down mid-pass (see the backlog cap). In-process
stages its corpus in the harness process on the 64 GB host and never starts a
container, so it is the one backend whose cap can rise without touching the VM.

At 32 GiB the 64 KiB leg drains 524 288 messages, which puts the fastest cell
(`consumer_group` at two consumers, 514 k msg/s) at about 0.9 s and the other
ten at 1.0-5.0 s: the leg would publish instead of being withheld whole.
Resident cost is ~34 GB. Clearing that last cell as well would need ~40 GiB, and
`MIN_FRAMEWORK_CORPUS_MESSAGES`'s own doc declines to chase a window that
hardware speed keeps moving — a cell landing under the floor is withheld and
captioned exactly as before, which costs one bar rather than the pass.

It moves the 1 KiB leg too, by design: at 3 GiB that leg is byte-bound at
3 145 728, and 32 GiB would put it back on the pinned count of 6 000 000 where
the 64 B leg already sits. In-process would then run two of its three legs on
the matrix's own corpus rather than one, and the single 1 KiB cell that drained
in 0.97 s would clear the floor as well. It costs about forty seconds and 6 GiB
rather than 3 GiB resident.

**None of that is in the published document.** The cap landed after the
committed in-process leg was measured, and a leg changes only by being
re-measured — so `benches/results/bench-results.json` still carries in-process
at the pinned 3 GiB: `drain.corpus` 49 152 at 64 KiB with all eleven rows
`setup_bound` and withheld, and 3 145 728 at 1 KiB. That is what the chart
captions name. The two paragraphs above are what the next in-process pass
produces, not a description of what is plotted today.
The published in-process leg carries the cap, so the paragraphs below are
measured rows rather than a forecast. At 32 GiB the 64 KiB leg drains 524 288
messages — 32 GiB of payload staged in the harness process, which is the cap
doing exactly what it says. All twelve 64 KiB drain rows land in 3.4-9.1 s,
every one of them clears the floor as `framework`, and the leg publishes whole
where it was withheld whole.

Rates at that corpus are 52 k-139 k msg/s, well below the 95 k-514 k the same
cells reported at 49 152. A rate measured over a 0.1 s window does not predict
the rate over a 5 s one, so size the corpus from the window you want, not from
the throughput a sub-floor cell happened to report. `MIN_FRAMEWORK_CORPUS_MESSAGES`'s
own doc declines to chase a window that hardware speed keeps moving — a cell
landing under the floor is withheld and captioned, which costs one bar rather
than the pass.

It moves the 1 KiB leg too, by design: at 3 GiB that leg was byte-bound at
3 145 728, and 32 GiB puts it back on the pinned count of 6 000 000 where the
64 B leg already sits. In-process now runs two of its three legs on the matrix's
own corpus rather than one, in 4.9-15.1 s per cell, and the 1 KiB cell that
previously drained in 0.95 s clears the floor with the rest: the leg has no
`setup_bound` drain row at either payload.

As with the SQS deviation it is recorded on the rows, not only here: every
drain row carries `drain.corpus`, and where a slice's backends disagree the
charts name them — today "corpus differs by backend: inmemory, kafka, nats,
rabbitmq, redis 49k / 3.1M / 6.0M; sqs 49k / 60k" — rather than listing sizes
unattributed.

One thing it does **not** fix, so a rerun is not read as fixing it: of the two
cells in the published document that failed "consumed before assembly" at
64 KiB with eight consumers, in-process is one and this clears it; the other is
RabbitMQ, whose eight-consumer group assembly ran through 46 467 of 49 152
messages. Putting that well under half a corpus needs ~196 k messages, or
12 GiB inside an 8 GB VM. That one is a group-assembly cost rather than a
corpus size, and it is expected to fail again.
charts name them — today "corpus differs by backend: inmemory 524k / 6.0M;
kafka, nats, rabbitmq, redis 49k / 3.1M / 6.0M; sqs 49k / 60k" — rather than
listing sizes unattributed.

One thing it does **not** fix, so the rerun is not read as fixing it: two cells
failed "consumed before assembly" at 64 KiB with eight consumers, and the cap
clears only one. In-process cleared — its batch drain assembled at 30 500 of
524 288, comfortably inside the first half — and it has no failures left. The
other is RabbitMQ, whose eight-consumer drain ran through 46 467 of 49 152
messages before the last worker was assigned. That leg was not re-measured and
the cap cannot reach it anyway: RabbitMQ stages inside the 8 GB VM at the pinned
3 GiB, and putting the assembly well under half a corpus needs ~196 k messages,
or 12 GiB inside that VM. That one is a worker-assembly cost rather than a
corpus size, and it is still the document's one drain failure.

## Results document and provenance

Expand Down Expand Up @@ -420,8 +423,8 @@ ones. Keep them in a separate results file.

## Reading a run

The harness prints a table per scenario and a summary at the end. Three
things to check before trusting a document:
The harness prints a table per scenario and a summary at the end. What to
check before trusting a document:

- `Failed scenarios:` at the end of the log. A failed scenario is recorded in
`failures[]` and produces no row, so a chart drawn from it is missing a
Expand Down
26 changes: 12 additions & 14 deletions docs/pages/ops/performance.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -67,20 +67,18 @@ In order of impact:

**Batching and payload size.** `BatchConsumer` amortises one handler call over
`max_batch_size` messages, and that pays when the per-message cost is the
handler call itself. It does not pay for large payloads. In the committed
document, the in-process batch flow drains 2.7-10.5x faster than the plain
parallel consumer at 64 B and 1.9-7.0x faster at 1 KiB, and 2.0-3.8x slower at
64 KiB — so what turns the comparison around is payload size. Every bound is
rounded outward, so no drain row falls outside a range stated here. Those 64 KiB
rows are read from the document rather than off the charts: every in-process
64 KiB drain in the published run finished under the harness's reliability
floor, so the charts withhold all of them. Batch size is not the lever either — an
off-matrix probe (the pinned matrix has no batch-size axis) varied the flush at
64 KiB and found the deficit flat: a 500-message batch (a 32 MB flush) at
55k msg/s, a 50-message batch at 80k, a 10-message batch at 79k, against the
document's about 356k for the single-consumer parallel drain. The extra cost is
per message, not per flush. Batch small messages; consume large ones one at a
time.
handler call itself. It stops paying as the payload grows. Comparing the
in-process batch drain against the plain parallel drain at the same cell, the
committed document has the batch flow at **3.6-9.7x** the parallel rate at 64 B,
**0.9-3.1x** at 1 KiB and **0.5-1.2x** at 64 KiB — so what turns the comparison
around is payload size, and by 64 KiB batching is a wash at best: only the
eight-consumer cell is above parity there, and the single-consumer one runs at
about half the parallel rate. Every bound is rounded outward, so no drain row
falls outside a range stated here. One cell behind these ranges is in the
document but not on the charts — the 64 B two-consumer batch drain finished
under the harness's reliability floor and is withheld with an `n/s` marker;
every 64 KiB drain row publishes. Batch small messages; consume large ones one
at a time.

## prefetch_count deep-dive

Expand Down
Loading