From cc8923c023f83319868e5c4ccfc9a03d98c15612 Mon Sep 17 00:00:00 2001 From: Zannis Kalampoukis Date: Wed, 9 Sep 2026 09:40:12 +0000 Subject: [PATCH] fix(bench): give in-process its own drain byte cap At the matrix's 3 GiB the 64 KiB leg is 49 152 messages, and the 2026-09-08 run drained every one of the in-process 64 KiB cells in 0.09-0.46 s. That is under MIN_FRAMEWORK_WINDOW_SECS, so the harness marked all eleven rows setup_bound and the charts withheld all eleven and captioned why: the fastest backend in the set publishes no 64 KiB consume rate at all, and 99 of the document's 822 rows are setup_bound. 3 GiB is not a cautious number for the other five, it is the containerised limit -- they stage the corpus inside the 8 GB Docker VM, where 3 GiB is already over a third of it and where a 3.2 GB backlog has already taken Redis down mid-pass. Raising the cap globally would buy the 64 KiB leg at the price of the VM. In-process stages its corpus in the harness process on the 64 GB host and never starts a container, so it is the one backend whose cap can rise without touching the VM. Substitute --drain-max-bytes for the inmemory target only, at 32 GiB, the mirror of the SQS corpus deviation already here: SQS drains a smaller corpus because it is the only backend slow enough that the pinned count means something else, in-process a larger one because it is the only one fast enough that the pinned byte cap does. Factor both substitutions through substitute_knob so a deviation still dies when its knob leaves the matrix. 32 GiB is 524 288 messages at 64 KiB. Projecting the measured rates onto that corpus puts ten of the eleven cells at 1.0-5.0 s and the eleventh (consumer_group at two consumers, 514 k msg/s) at ~0.9 s, so the leg publishes instead of being withheld whole; a cell under the floor is still withheld and captioned. It also puts the 1 KiB leg back on the pinned 6 000 000 from 3 145 728, clearing the one cell that drained in 0.97 s. Rendering control: projecting that corpus onto the committed six-backend document and running chartgen drops the "inmemory / parallel, batch: measured, but every window under 1 s" note and adds a two-line "corpus differs by backend: inmemory 524k; kafka, nats, rabbitmq, redis, sqs 49k" group. The 64 KiB parallel-vs-sequenced canvas goes 672 -> 688 of the 768 that MAX_CAPTION_GROWTH_LINES allows, so three of eight growth lines are spent and five stay spare. Script and runbook only; no document or SVG churn until the rerun. --- benches/README.md | 55 ++++++++++++++++++++++++++- scripts/bench.sh | 94 +++++++++++++++++++++++++++++++++++++++-------- 2 files changed, 133 insertions(+), 16 deletions(-) diff --git a/benches/README.md b/benches/README.md index 794e530f..76196361 100644 --- a/benches/README.md +++ b/benches/README.md @@ -26,7 +26,7 @@ Every published row is measured with the same sampled core matrix: | `--consumers 1,2,4,8` | four points on the scaling curve | the sampling lever for the published matrix | | `--concurrent` | concurrent processing within each consumer | without it every consumer handles one message at a time, and the parallel flows measure ack round trips instead of throughput | | `--drain-messages 6000000` | the drain corpus for the consume flows | see below; sized so the group assembles well inside the first half of the corpus on the fastest cell and the window still lasts several seconds. **SQS is the one exception, at 60 000** — see "The SQS corpus deviation" | -| `--drain-max-bytes 3221225472` | cap on `corpus × payload_bytes` | 3 GiB: 6 M messages at 64 B, 3 M at 1 KiB, about 49 k at 64 KiB — the in-process backend holds the corpus resident and Kafka writes it to disk before every cell | +| `--drain-max-bytes 3221225472` | cap on `corpus × payload_bytes` | 3 GiB: 6 M messages at 64 B, 3 M at 1 KiB, about 49 k at 64 KiB — what the 8 GB Docker VM holds, and every containerised backend stages its corpus inside it. **In-process is the one exception, at 32 GiB** — see "The in-process byte cap deviation" | | `--load-rates …` | the offered-load ladder for the consume flows | see below; three rungs, because the ladder measures latency at a sustained rate and the ceiling comes from the drain | | `--load-window-secs 10` | how long each rung holds its rate | long enough for a rate | | `--load-producers 8` | paced producer tasks sharing each rung's rate, and fill tasks for the drain | one sequential publisher tops out near 70k msg/s on Kafka; on Kafka each task gets its own connection, because publishers cloned from one client share one producer instance and cap near 260k msg/s together | @@ -265,6 +265,59 @@ for scheduling, not published numbers: Ratios move with the host; the ordering does not. Time one cell before committing an evening to a backend nobody has measured here. +### The in-process byte cap deviation + +In-process is the one backend that does not cap its drain corpus at the +matrix's 3 GiB. It runs **32 GiB**, set as `INMEMORY_DRAIN_MAX_BYTES` in +`scripts/bench.sh` and substituted into the matrix for that target only; every +other knob, including `--drain-messages`, is shared. The script prints a +`deviation:` line at the top of the in-process log. + +This is the mirror image of the SQS deviation. SQS drains a smaller corpus +because it is the only backend slow enough that the pinned *count* means +something else there; in-process is allowed a larger one because it is the only +backend fast enough that the pinned *byte cap* does. At 3 GiB the 64 KiB leg is +49 152 messages, and the 2026-09-08 run drained every one of the in-process +64 KiB cells in 0.09-0.46 s — under the 1 s floor, so the harness marked all +eleven rows `setup_bound` and the charts withheld all eleven. The fastest +backend in the set published no 64 KiB consume rate at all. + +Why only in-process gets it: 3 GiB is not a cautious number for the others, it +is the containerised limit. Every other backend stages its corpus inside the +8 GB Docker VM — Redis and NATS in container memory, Kafka through its page +cache — where 3 GiB is already over a third of the VM, and where a 3.2 GB +backlog has already taken Redis down mid-pass (see the backlog cap). In-process +stages its corpus in the harness process on the 64 GB host and never starts a +container, so it is the one backend whose cap can rise without touching the VM. + +At 32 GiB the 64 KiB leg drains 524 288 messages, which puts the fastest cell +(`consumer_group` at two consumers, 514 k msg/s) at about 0.9 s and the other +ten at 1.0-5.0 s: the leg publishes instead of being withheld whole. Resident +cost is ~34 GB. Clearing that last cell as well would need ~40 GiB, and +`MIN_FRAMEWORK_CORPUS_MESSAGES`'s own doc declines to chase a window that +hardware speed keeps moving — a cell landing under the floor is withheld and +captioned exactly as before, which costs one bar rather than the pass. + +It moves the 1 KiB leg too, by design: at 3 GiB that leg is byte-bound at +3 145 728, and 32 GiB puts it back on the pinned count of 6 000 000 where the +64 B leg already sits. In-process then runs two of its three legs on the +matrix's own corpus rather than one, and the single 1 KiB cell that drained in +0.97 s clears the floor as well. It costs about forty seconds and 6 GiB rather +than 3 GiB resident. + +As with the SQS deviation it is recorded on the rows, not only here: every +drain row carries `drain.corpus`, and where a slice's backends disagree the +charts name them ("corpus differs by backend: inmemory 524k; …") rather than +listing sizes unattributed. + +One thing it does **not** fix, so a rerun is not read as fixing it: of the four +cells that failed "consumed before assembly" at 64 KiB with eight consumers, +in-process was one and this clears it, but the other three are RabbitMQ, whose +eight-consumer group assembly ran through 37 000-46 300 messages. Putting that +well under half a corpus needs ~196 k messages, or 12 GiB inside an 8 GB VM. +Those three are a group-assembly cost rather than a corpus size, and they are +expected to fail again. + ## Results document and provenance The harness merges each backend's run into the results document by backend diff --git a/scripts/bench.sh b/scripts/bench.sh index 0036af62..28207ba7 100755 --- a/scripts/bench.sh +++ b/scripts/bench.sh @@ -25,8 +25,9 @@ LOG_DIR="target/bench-logs" # the harness process can bound it. `--drain-messages` sizes the corpus so # that on the fastest cell the group assembles well inside the first half # of it and the window still lasts several seconds; `--drain-max-bytes` -# caps the 64 KiB corpus at 3 GiB (about 49 k messages), which the -# in-process backend holds resident and Kafka writes to disk per cell. +# caps the 64 KiB corpus at 3 GiB (about 49 k messages), which fits the +# 8 GB Docker VM every containerised backend stages its corpus in — see +# "The in-process byte cap deviation" for the one backend that does not. # - The offered-load ladder is the latency measurement: a paced producer # holds each rate for the window while the consumers run, and only a rung # the consumers kept up with has dispatch percentiles that are latency @@ -47,8 +48,10 @@ MATRIX=( --load-producers 8 ) -# The one documented deviation from the matrix above, and the only backend -# that gets one. +# The first of the two documented deviations from the matrix above. Both are +# per-backend substitutions of a single knob, both exist because one number +# means two different things at opposite ends of the backend set, and both +# are recorded on every row they touch — see the two runbook sections. # # The corpus is a message *count*, and it was sized for the fastest cell — # in-process at ~4.7 M msg/s, where six million messages buy a window of @@ -78,6 +81,55 @@ SQS_DRAIN_MESSAGES=60000 # (`messages`). SQS_FIFO_MESSAGES=100 +# The second deviation, and the mirror image of the first: SQS drains a +# smaller corpus because it is the only backend slow enough that the pinned +# count means something else there, and in-process is allowed a larger one +# because it is the only backend fast enough that the pinned *byte cap* does. +# +# At the matrix's 3 GiB the 64 KiB leg is 49 152 messages, and the 2026-09-08 +# run drained every one of the in-process 64 KiB cells in 0.09-0.46 s. That is +# under `MIN_FRAMEWORK_WINDOW_SECS`, so the harness marked all eleven rows +# `setup_bound` and the charts withheld all eleven and captioned why. The +# fastest backend in the set published no 64 KiB consume rate at all. +# +# Why this is in-process's deviation alone: 3 GiB is not a conservative number +# for the others, it is the containerised limit. Every other backend stages its +# corpus inside the 8 GB Docker VM (Redis and NATS in container memory, Kafka +# through its page cache), where 3 GiB is already over a third of the VM and +# where a 3.2 GB backlog has already taken Redis down mid-pass — see the +# backlog cap. In-process stages its corpus in the harness process on the +# 64 GB host and never starts a container, so it is the one backend where the +# cap can rise without touching the VM. +# +# Sized from that run's fastest 64 KiB drain (514 k msg/s, `consumer_group` at +# two consumers): 32 GiB is 524 288 messages, which puts that cell at ~0.9 s +# and the other ten at 1.0-5.0 s, so the leg publishes instead of being +# withheld whole. Resident cost is ~34 GB of the 64 GB host. Clearing the last +# cell too would need ~40 GiB, and the harness's own floor doc declines to +# chase a window that hardware speed keeps moving; a cell that lands under the +# floor is withheld and captioned exactly as today, which loses one bar rather +# than the pass. +# +# It moves the 1 KiB leg as well, and that is intended rather than incidental: +# at 3 GiB that leg is byte-bound at 3 145 728, and 32 GiB puts it back on the +# pinned count of 6 000 000, where the 64 B leg already sits. So in-process +# runs two of its three legs on the matrix's own corpus instead of one, and the +# single 1 KiB cell that drained in 0.97 s clears the floor too. It costs about +# forty seconds of extra wall clock and 6 GiB rather than 3 GiB resident. +# +# `QUEUE_CAPACITY` in examples/inmemory/stress.rs is 8 000 000 and is a bound +# rather than a preallocation, so it already covers 524 288; the harness's +# `refused_drain_capacity` gate is what would catch it otherwise. +# +# What it does not fix, so the run is not read as fixing it: the four cells +# that failed "consumed before assembly" at 64 KiB / 8 consumers. In-process +# was one of them and this clears it, but the other three are RabbitMQ, whose +# eight-consumer group assembly ran through 37 000-46 300 messages. Putting +# that well under half a corpus needs ~196 k messages, which is 12 GiB in an +# 8 GB VM. RabbitMQ's three failures are a group-assembly cost, not a corpus +# size, and they are expected to fail again. +INMEMORY_DRAIN_MAX_BYTES=34359738368 + usage() { sed -n '2,7p' "$0" | sed 's/^# \{0,1\}//' exit "${1:-2}" @@ -88,6 +140,21 @@ die() { exit 1 } +# Replace a knob's value in MATRIX in place, rather than appending a second +# copy: the harness rejects a knob given twice, so a per-backend deviation has +# to substitute. Dies when the knob is absent, so a deviation can never +# silently stop deviating if the matrix above is edited. +substitute_knob() { + local knob="$1" value="$2" i found=0 + for i in "${!MATRIX[@]}"; do + if [ "${MATRIX[$i]}" = "$knob" ]; then + MATRIX[$((i + 1))]="$value" + found=1 + fi + done + [ "$found" = 1 ] || die "the matrix has no $knob for the $target deviation to replace" +} + [ $# -ge 1 ] || usage target="$1" shift @@ -131,22 +198,16 @@ if [ "$target" = sqs ]; then [ -n "${LOCALSTACK_AUTH_TOKEN:-}" ] \ || die "LOCALSTACK_AUTH_TOKEN is not set; run through 'dotenvx run -- scripts/bench.sh sqs'" - # Substituted into the matrix rather than appended after it: the harness - # rejects a knob given twice, so a deviation has to replace the value. - deviated=0 - for i in "${!MATRIX[@]}"; do - if [ "${MATRIX[$i]}" = --drain-messages ]; then - MATRIX[$((i + 1))]="$SQS_DRAIN_MESSAGES" - deviated=1 - fi - done - [ "$deviated" = 1 ] \ - || die "the matrix has no --drain-messages for the sqs corpus deviation to replace" + substitute_knob --drain-messages "$SQS_DRAIN_MESSAGES" # Appended rather than substituted: the matrix carries no --fifo-messages, # because every other backend runs the tier's FIFO corpus. MATRIX+=(--fifo-messages "$SQS_FIFO_MESSAGES") fi +if [ "$target" = inmemory ]; then + substitute_knob --drain-max-bytes "$INMEMORY_DRAIN_MAX_BYTES" +fi + if [ "$fresh" = 1 ] && [ -f "$RESULTS_FILE" ]; then backup="$RESULTS_FILE.$(date -u +%Y%m%dT%H%M%SZ).bak" mv "$RESULTS_FILE" "$backup" @@ -161,6 +222,9 @@ echo "matrix: ${MATRIX[*]} ${extra[*]:-}" if [ "$target" = sqs ]; then echo "deviation: drain corpus $SQS_DRAIN_MESSAGES, not the pinned 6000000, and FIFO corpus $SQS_FIFO_MESSAGES per FIFO worker — see the comments in this script" fi +if [ "$target" = inmemory ]; then + echo "deviation: drain byte cap $INMEMORY_DRAIN_MAX_BYTES, not the pinned 3221225472, so the 64 KiB leg drains 524288 messages rather than 49152 and the 1 KiB leg the pinned 6000000 rather than 3145728 — see the comments in this script" +fi echo "results: $RESULTS_FILE" echo "log: $log"