diff --git a/benches/README.md b/benches/README.md index 794e530f..76196361 100644 --- a/benches/README.md +++ b/benches/README.md @@ -26,7 +26,7 @@ Every published row is measured with the same sampled core matrix: | `--consumers 1,2,4,8` | four points on the scaling curve | the sampling lever for the published matrix | | `--concurrent` | concurrent processing within each consumer | without it every consumer handles one message at a time, and the parallel flows measure ack round trips instead of throughput | | `--drain-messages 6000000` | the drain corpus for the consume flows | see below; sized so the group assembles well inside the first half of the corpus on the fastest cell and the window still lasts several seconds. **SQS is the one exception, at 60 000** — see "The SQS corpus deviation" | -| `--drain-max-bytes 3221225472` | cap on `corpus × payload_bytes` | 3 GiB: 6 M messages at 64 B, 3 M at 1 KiB, about 49 k at 64 KiB — the in-process backend holds the corpus resident and Kafka writes it to disk before every cell | +| `--drain-max-bytes 3221225472` | cap on `corpus × payload_bytes` | 3 GiB: 6 M messages at 64 B, 3 M at 1 KiB, about 49 k at 64 KiB — what the 8 GB Docker VM holds, and every containerised backend stages its corpus inside it. **In-process is the one exception, at 32 GiB** — see "The in-process byte cap deviation" | | `--load-rates …` | the offered-load ladder for the consume flows | see below; three rungs, because the ladder measures latency at a sustained rate and the ceiling comes from the drain | | `--load-window-secs 10` | how long each rung holds its rate | long enough for a rate | | `--load-producers 8` | paced producer tasks sharing each rung's rate, and fill tasks for the drain | one sequential publisher tops out near 70k msg/s on Kafka; on Kafka each task gets its own connection, because publishers cloned from one client share one producer instance and cap near 260k msg/s together | @@ -265,6 +265,59 @@ for scheduling, not published numbers: Ratios move with the host; the ordering does not. Time one cell before committing an evening to a backend nobody has measured here. +### The in-process byte cap deviation + +In-process is the one backend that does not cap its drain corpus at the +matrix's 3 GiB. It runs **32 GiB**, set as `INMEMORY_DRAIN_MAX_BYTES` in +`scripts/bench.sh` and substituted into the matrix for that target only; every +other knob, including `--drain-messages`, is shared. The script prints a +`deviation:` line at the top of the in-process log. + +This is the mirror image of the SQS deviation. SQS drains a smaller corpus +because it is the only backend slow enough that the pinned *count* means +something else there; in-process is allowed a larger one because it is the only +backend fast enough that the pinned *byte cap* does. At 3 GiB the 64 KiB leg is +49 152 messages, and the 2026-09-08 run drained every one of the in-process +64 KiB cells in 0.09-0.46 s — under the 1 s floor, so the harness marked all +eleven rows `setup_bound` and the charts withheld all eleven. The fastest +backend in the set published no 64 KiB consume rate at all. + +Why only in-process gets it: 3 GiB is not a cautious number for the others, it +is the containerised limit. Every other backend stages its corpus inside the +8 GB Docker VM — Redis and NATS in container memory, Kafka through its page +cache — where 3 GiB is already over a third of the VM, and where a 3.2 GB +backlog has already taken Redis down mid-pass (see the backlog cap). In-process +stages its corpus in the harness process on the 64 GB host and never starts a +container, so it is the one backend whose cap can rise without touching the VM. + +At 32 GiB the 64 KiB leg drains 524 288 messages, which puts the fastest cell +(`consumer_group` at two consumers, 514 k msg/s) at about 0.9 s and the other +ten at 1.0-5.0 s: the leg publishes instead of being withheld whole. Resident +cost is ~34 GB. Clearing that last cell as well would need ~40 GiB, and +`MIN_FRAMEWORK_CORPUS_MESSAGES`'s own doc declines to chase a window that +hardware speed keeps moving — a cell landing under the floor is withheld and +captioned exactly as before, which costs one bar rather than the pass. + +It moves the 1 KiB leg too, by design: at 3 GiB that leg is byte-bound at +3 145 728, and 32 GiB puts it back on the pinned count of 6 000 000 where the +64 B leg already sits. In-process then runs two of its three legs on the +matrix's own corpus rather than one, and the single 1 KiB cell that drained in +0.97 s clears the floor as well. It costs about forty seconds and 6 GiB rather +than 3 GiB resident. + +As with the SQS deviation it is recorded on the rows, not only here: every +drain row carries `drain.corpus`, and where a slice's backends disagree the +charts name them ("corpus differs by backend: inmemory 524k; …") rather than +listing sizes unattributed. + +One thing it does **not** fix, so a rerun is not read as fixing it: of the four +cells that failed "consumed before assembly" at 64 KiB with eight consumers, +in-process was one and this clears it, but the other three are RabbitMQ, whose +eight-consumer group assembly ran through 37 000-46 300 messages. Putting that +well under half a corpus needs ~196 k messages, or 12 GiB inside an 8 GB VM. +Those three are a group-assembly cost rather than a corpus size, and they are +expected to fail again. + ## Results document and provenance The harness merges each backend's run into the results document by backend diff --git a/scripts/bench.sh b/scripts/bench.sh index 0036af62..28207ba7 100755 --- a/scripts/bench.sh +++ b/scripts/bench.sh @@ -25,8 +25,9 @@ LOG_DIR="target/bench-logs" # the harness process can bound it. `--drain-messages` sizes the corpus so # that on the fastest cell the group assembles well inside the first half # of it and the window still lasts several seconds; `--drain-max-bytes` -# caps the 64 KiB corpus at 3 GiB (about 49 k messages), which the -# in-process backend holds resident and Kafka writes to disk per cell. +# caps the 64 KiB corpus at 3 GiB (about 49 k messages), which fits the +# 8 GB Docker VM every containerised backend stages its corpus in — see +# "The in-process byte cap deviation" for the one backend that does not. # - The offered-load ladder is the latency measurement: a paced producer # holds each rate for the window while the consumers run, and only a rung # the consumers kept up with has dispatch percentiles that are latency @@ -47,8 +48,10 @@ MATRIX=( --load-producers 8 ) -# The one documented deviation from the matrix above, and the only backend -# that gets one. +# The first of the two documented deviations from the matrix above. Both are +# per-backend substitutions of a single knob, both exist because one number +# means two different things at opposite ends of the backend set, and both +# are recorded on every row they touch — see the two runbook sections. # # The corpus is a message *count*, and it was sized for the fastest cell — # in-process at ~4.7 M msg/s, where six million messages buy a window of @@ -78,6 +81,55 @@ SQS_DRAIN_MESSAGES=60000 # (`messages`). SQS_FIFO_MESSAGES=100 +# The second deviation, and the mirror image of the first: SQS drains a +# smaller corpus because it is the only backend slow enough that the pinned +# count means something else there, and in-process is allowed a larger one +# because it is the only backend fast enough that the pinned *byte cap* does. +# +# At the matrix's 3 GiB the 64 KiB leg is 49 152 messages, and the 2026-09-08 +# run drained every one of the in-process 64 KiB cells in 0.09-0.46 s. That is +# under `MIN_FRAMEWORK_WINDOW_SECS`, so the harness marked all eleven rows +# `setup_bound` and the charts withheld all eleven and captioned why. The +# fastest backend in the set published no 64 KiB consume rate at all. +# +# Why this is in-process's deviation alone: 3 GiB is not a conservative number +# for the others, it is the containerised limit. Every other backend stages its +# corpus inside the 8 GB Docker VM (Redis and NATS in container memory, Kafka +# through its page cache), where 3 GiB is already over a third of the VM and +# where a 3.2 GB backlog has already taken Redis down mid-pass — see the +# backlog cap. In-process stages its corpus in the harness process on the +# 64 GB host and never starts a container, so it is the one backend where the +# cap can rise without touching the VM. +# +# Sized from that run's fastest 64 KiB drain (514 k msg/s, `consumer_group` at +# two consumers): 32 GiB is 524 288 messages, which puts that cell at ~0.9 s +# and the other ten at 1.0-5.0 s, so the leg publishes instead of being +# withheld whole. Resident cost is ~34 GB of the 64 GB host. Clearing the last +# cell too would need ~40 GiB, and the harness's own floor doc declines to +# chase a window that hardware speed keeps moving; a cell that lands under the +# floor is withheld and captioned exactly as today, which loses one bar rather +# than the pass. +# +# It moves the 1 KiB leg as well, and that is intended rather than incidental: +# at 3 GiB that leg is byte-bound at 3 145 728, and 32 GiB puts it back on the +# pinned count of 6 000 000, where the 64 B leg already sits. So in-process +# runs two of its three legs on the matrix's own corpus instead of one, and the +# single 1 KiB cell that drained in 0.97 s clears the floor too. It costs about +# forty seconds of extra wall clock and 6 GiB rather than 3 GiB resident. +# +# `QUEUE_CAPACITY` in examples/inmemory/stress.rs is 8 000 000 and is a bound +# rather than a preallocation, so it already covers 524 288; the harness's +# `refused_drain_capacity` gate is what would catch it otherwise. +# +# What it does not fix, so the run is not read as fixing it: the four cells +# that failed "consumed before assembly" at 64 KiB / 8 consumers. In-process +# was one of them and this clears it, but the other three are RabbitMQ, whose +# eight-consumer group assembly ran through 37 000-46 300 messages. Putting +# that well under half a corpus needs ~196 k messages, which is 12 GiB in an +# 8 GB VM. RabbitMQ's three failures are a group-assembly cost, not a corpus +# size, and they are expected to fail again. +INMEMORY_DRAIN_MAX_BYTES=34359738368 + usage() { sed -n '2,7p' "$0" | sed 's/^# \{0,1\}//' exit "${1:-2}" @@ -88,6 +140,21 @@ die() { exit 1 } +# Replace a knob's value in MATRIX in place, rather than appending a second +# copy: the harness rejects a knob given twice, so a per-backend deviation has +# to substitute. Dies when the knob is absent, so a deviation can never +# silently stop deviating if the matrix above is edited. +substitute_knob() { + local knob="$1" value="$2" i found=0 + for i in "${!MATRIX[@]}"; do + if [ "${MATRIX[$i]}" = "$knob" ]; then + MATRIX[$((i + 1))]="$value" + found=1 + fi + done + [ "$found" = 1 ] || die "the matrix has no $knob for the $target deviation to replace" +} + [ $# -ge 1 ] || usage target="$1" shift @@ -131,22 +198,16 @@ if [ "$target" = sqs ]; then [ -n "${LOCALSTACK_AUTH_TOKEN:-}" ] \ || die "LOCALSTACK_AUTH_TOKEN is not set; run through 'dotenvx run -- scripts/bench.sh sqs'" - # Substituted into the matrix rather than appended after it: the harness - # rejects a knob given twice, so a deviation has to replace the value. - deviated=0 - for i in "${!MATRIX[@]}"; do - if [ "${MATRIX[$i]}" = --drain-messages ]; then - MATRIX[$((i + 1))]="$SQS_DRAIN_MESSAGES" - deviated=1 - fi - done - [ "$deviated" = 1 ] \ - || die "the matrix has no --drain-messages for the sqs corpus deviation to replace" + substitute_knob --drain-messages "$SQS_DRAIN_MESSAGES" # Appended rather than substituted: the matrix carries no --fifo-messages, # because every other backend runs the tier's FIFO corpus. MATRIX+=(--fifo-messages "$SQS_FIFO_MESSAGES") fi +if [ "$target" = inmemory ]; then + substitute_knob --drain-max-bytes "$INMEMORY_DRAIN_MAX_BYTES" +fi + if [ "$fresh" = 1 ] && [ -f "$RESULTS_FILE" ]; then backup="$RESULTS_FILE.$(date -u +%Y%m%dT%H%M%SZ).bak" mv "$RESULTS_FILE" "$backup" @@ -161,6 +222,9 @@ echo "matrix: ${MATRIX[*]} ${extra[*]:-}" if [ "$target" = sqs ]; then echo "deviation: drain corpus $SQS_DRAIN_MESSAGES, not the pinned 6000000, and FIFO corpus $SQS_FIFO_MESSAGES per FIFO worker — see the comments in this script" fi +if [ "$target" = inmemory ]; then + echo "deviation: drain byte cap $INMEMORY_DRAIN_MAX_BYTES, not the pinned 3221225472, so the 64 KiB leg drains 524288 messages rather than 49152 and the 1 KiB leg the pinned 6000000 rather than 3145728 — see the comments in this script" +fi echo "results: $RESULTS_FILE" echo "log: $log"