fix(bench): give in-process its own drain byte cap

benches/README.md

@@ -26,7 +26,7 @@ Every published row is measured with the same sampled core matrix:
 | `--consumers 1,2,4,8` | four points on the scaling curve | the sampling lever for the published matrix |
 | `--concurrent` | concurrent processing within each consumer | without it every consumer handles one message at a time, and the parallel flows measure ack round trips instead of throughput |
 | `--drain-messages 6000000` | the drain corpus for the consume flows | see below; sized so the group assembles well inside the first half of the corpus on the fastest cell and the window still lasts several seconds. **SQS is the one exception, at 60 000** — see "The SQS corpus deviation" |
-| `--drain-max-bytes 3221225472` | cap on `corpus × payload_bytes` | 3 GiB: 6 M messages at 64 B, 3 M at 1 KiB, about 49 k at 64 KiB — the in-process backend holds the corpus resident and Kafka writes it to disk before every cell |
+| `--drain-max-bytes 3221225472` | cap on `corpus × payload_bytes` | 3 GiB: 6 M messages at 64 B, 3 M at 1 KiB, about 49 k at 64 KiB — what the 8 GB Docker VM holds, and every containerised backend stages its corpus inside it. **In-process is the one exception, at 32 GiB** — see "The in-process byte cap deviation" |
 | `--load-rates …` | the offered-load ladder for the consume flows | see below; three rungs, because the ladder measures latency at a sustained rate and the ceiling comes from the drain |
 | `--load-window-secs 10` | how long each rung holds its rate | long enough for a rate |
 | `--load-producers 8` | paced producer tasks sharing each rung's rate, and fill tasks for the drain | one sequential publisher tops out near 70k msg/s on Kafka; on Kafka each task gets its own connection, because publishers cloned from one client share one producer instance and cap near 260k msg/s together |
@@ -265,6 +265,59 @@ for scheduling, not published numbers:
 Ratios move with the host; the ordering does not. Time one cell before
 committing an evening to a backend nobody has measured here.
 
+### The in-process byte cap deviation
+
+In-process is the one backend that does not cap its drain corpus at the
+matrix's 3 GiB. It runs **32 GiB**, set as `INMEMORY_DRAIN_MAX_BYTES` in
+`scripts/bench.sh` and substituted into the matrix for that target only; every
+other knob, including `--drain-messages`, is shared. The script prints a
+`deviation:` line at the top of the in-process log.
+
+This is the mirror image of the SQS deviation. SQS drains a smaller corpus
+because it is the only backend slow enough that the pinned *count* means
+something else there; in-process is allowed a larger one because it is the only
+backend fast enough that the pinned *byte cap* does. At 3 GiB the 64 KiB leg is
+49 152 messages, and the 2026-09-08 run drained every one of the in-process
+64 KiB cells in 0.09-0.46 s — under the 1 s floor, so the harness marked all
+eleven rows `setup_bound` and the charts withheld all eleven. The fastest
+backend in the set published no 64 KiB consume rate at all.
+
+Why only in-process gets it: 3 GiB is not a cautious number for the others, it
+is the containerised limit. Every other backend stages its corpus inside the
+8 GB Docker VM — Redis and NATS in container memory, Kafka through its page
+cache — where 3 GiB is already over a third of the VM, and where a 3.2 GB
+backlog has already taken Redis down mid-pass (see the backlog cap). In-process
+stages its corpus in the harness process on the 64 GB host and never starts a
+container, so it is the one backend whose cap can rise without touching the VM.
+
+At 32 GiB the 64 KiB leg drains 524 288 messages, which puts the fastest cell
+(`consumer_group` at two consumers, 514 k msg/s) at about 0.9 s and the other
+ten at 1.0-5.0 s: the leg publishes instead of being withheld whole. Resident
+cost is ~34 GB. Clearing that last cell as well would need ~40 GiB, and
+`MIN_FRAMEWORK_CORPUS_MESSAGES`'s own doc declines to chase a window that
+hardware speed keeps moving — a cell landing under the floor is withheld and
+captioned exactly as before, which costs one bar rather than the pass.
+
+It moves the 1 KiB leg too, by design: at 3 GiB that leg is byte-bound at
+3 145 728, and 32 GiB puts it back on the pinned count of 6 000 000 where the
+64 B leg already sits. In-process then runs two of its three legs on the
+matrix's own corpus rather than one, and the single 1 KiB cell that drained in
+0.97 s clears the floor as well. It costs about forty seconds and 6 GiB rather
+than 3 GiB resident.
+
+As with the SQS deviation it is recorded on the rows, not only here: every
+drain row carries `drain.corpus`, and where a slice's backends disagree the
+charts name them ("corpus differs by backend: inmemory 524k; …") rather than
+listing sizes unattributed.
+
+One thing it does **not** fix, so a rerun is not read as fixing it: of the four
+cells that failed "consumed before assembly" at 64 KiB with eight consumers,
+in-process was one and this clears it, but the other three are RabbitMQ, whose
+eight-consumer group assembly ran through 37 000-46 300 messages. Putting that
+well under half a corpus needs ~196 k messages, or 12 GiB inside an 8 GB VM.
+Those three are a group-assembly cost rather than a corpus size, and they are
+expected to fail again.
+
 ## Results document and provenance
 
 The harness merges each backend's run into the results document by backend

scripts/bench.sh

@@ -25,8 +25,9 @@ LOG_DIR="target/bench-logs"
 #   the harness process can bound it. `--drain-messages` sizes the corpus so
 #   that on the fastest cell the group assembles well inside the first half
 #   of it and the window still lasts several seconds; `--drain-max-bytes`
-#   caps the 64 KiB corpus at 3 GiB (about 49 k messages), which the
-#   in-process backend holds resident and Kafka writes to disk per cell.
+#   caps the 64 KiB corpus at 3 GiB (about 49 k messages), which fits the
+#   8 GB Docker VM every containerised backend stages its corpus in — see
+#   "The in-process byte cap deviation" for the one backend that does not.
 # - The offered-load ladder is the latency measurement: a paced producer
 #   holds each rate for the window while the consumers run, and only a rung
 #   the consumers kept up with has dispatch percentiles that are latency
@@ -47,8 +48,10 @@ MATRIX=(
   --load-producers 8
 )
 
-# The one documented deviation from the matrix above, and the only backend
-# that gets one.
+# The first of the two documented deviations from the matrix above. Both are
+# per-backend substitutions of a single knob, both exist because one number
+# means two different things at opposite ends of the backend set, and both
+# are recorded on every row they touch — see the two runbook sections.
 #
 # The corpus is a message *count*, and it was sized for the fastest cell —
 # in-process at ~4.7 M msg/s, where six million messages buy a window of
@@ -78,6 +81,55 @@ SQS_DRAIN_MESSAGES=60000
 # (`messages`).
 SQS_FIFO_MESSAGES=100
 
+# The second deviation, and the mirror image of the first: SQS drains a
+# smaller corpus because it is the only backend slow enough that the pinned
+# count means something else there, and in-process is allowed a larger one
+# because it is the only backend fast enough that the pinned *byte cap* does.
+#
+# At the matrix's 3 GiB the 64 KiB leg is 49 152 messages, and the 2026-09-08
+# run drained every one of the in-process 64 KiB cells in 0.09-0.46 s. That is
+# under `MIN_FRAMEWORK_WINDOW_SECS`, so the harness marked all eleven rows
+# `setup_bound` and the charts withheld all eleven and captioned why. The
+# fastest backend in the set published no 64 KiB consume rate at all.
+#
+# Why this is in-process's deviation alone: 3 GiB is not a conservative number
+# for the others, it is the containerised limit. Every other backend stages its
+# corpus inside the 8 GB Docker VM (Redis and NATS in container memory, Kafka
+# through its page cache), where 3 GiB is already over a third of the VM and
+# where a 3.2 GB backlog has already taken Redis down mid-pass — see the
+# backlog cap. In-process stages its corpus in the harness process on the
+# 64 GB host and never starts a container, so it is the one backend where the
+# cap can rise without touching the VM.
+#
+# Sized from that run's fastest 64 KiB drain (514 k msg/s, `consumer_group` at
+# two consumers): 32 GiB is 524 288 messages, which puts that cell at ~0.9 s
+# and the other ten at 1.0-5.0 s, so the leg publishes instead of being
+# withheld whole. Resident cost is ~34 GB of the 64 GB host. Clearing the last
+# cell too would need ~40 GiB, and the harness's own floor doc declines to
+# chase a window that hardware speed keeps moving; a cell that lands under the
+# floor is withheld and captioned exactly as today, which loses one bar rather
+# than the pass.
+#
+# It moves the 1 KiB leg as well, and that is intended rather than incidental:
+# at 3 GiB that leg is byte-bound at 3 145 728, and 32 GiB puts it back on the
+# pinned count of 6 000 000, where the 64 B leg already sits. So in-process
+# runs two of its three legs on the matrix's own corpus instead of one, and the
+# single 1 KiB cell that drained in 0.97 s clears the floor too. It costs about
+# forty seconds of extra wall clock and 6 GiB rather than 3 GiB resident.
+#
+# `QUEUE_CAPACITY` in examples/inmemory/stress.rs is 8 000 000 and is a bound
+# rather than a preallocation, so it already covers 524 288; the harness's
+# `refused_drain_capacity` gate is what would catch it otherwise.
+#
+# What it does not fix, so the run is not read as fixing it: the four cells
+# that failed "consumed before assembly" at 64 KiB / 8 consumers. In-process
+# was one of them and this clears it, but the other three are RabbitMQ, whose
+# eight-consumer group assembly ran through 37 000-46 300 messages. Putting
+# that well under half a corpus needs ~196 k messages, which is 12 GiB in an
+# 8 GB VM. RabbitMQ's three failures are a group-assembly cost, not a corpus
+# size, and they are expected to fail again.
+INMEMORY_DRAIN_MAX_BYTES=34359738368
+
 usage() {
   sed -n '2,7p' "$0" | sed 's/^# \{0,1\}//'
   exit "${1:-2}"
@@ -88,6 +140,21 @@ die() {
   exit 1
 }
 
+# Replace a knob's value in MATRIX in place, rather than appending a second
+# copy: the harness rejects a knob given twice, so a per-backend deviation has
+# to substitute. Dies when the knob is absent, so a deviation can never
+# silently stop deviating if the matrix above is edited.
+substitute_knob() {
+  local knob="$1" value="$2" i found=0
+  for i in "${!MATRIX[@]}"; do
+    if [ "${MATRIX[$i]}" = "$knob" ]; then
+      MATRIX[$((i + 1))]="$value"
+      found=1
+    fi
+  done
+  [ "$found" = 1 ] || die "the matrix has no $knob for the $target deviation to replace"
+}
+
 [ $# -ge 1 ] || usage
 target="$1"
 shift
@@ -131,22 +198,16 @@ if [ "$target" = sqs ]; then
   [ -n "${LOCALSTACK_AUTH_TOKEN:-}" ] \
     || die "LOCALSTACK_AUTH_TOKEN is not set; run through 'dotenvx run -- scripts/bench.sh sqs'"
 
-  # Substituted into the matrix rather than appended after it: the harness
-  # rejects a knob given twice, so a deviation has to replace the value.
-  deviated=0
-  for i in "${!MATRIX[@]}"; do
-    if [ "${MATRIX[$i]}" = --drain-messages ]; then
-      MATRIX[$((i + 1))]="$SQS_DRAIN_MESSAGES"
-      deviated=1
-    fi
-  done
-  [ "$deviated" = 1 ] \
-    || die "the matrix has no --drain-messages for the sqs corpus deviation to replace"
+  substitute_knob --drain-messages "$SQS_DRAIN_MESSAGES"
   # Appended rather than substituted: the matrix carries no --fifo-messages,
   # because every other backend runs the tier's FIFO corpus.
   MATRIX+=(--fifo-messages "$SQS_FIFO_MESSAGES")
 fi
 
+if [ "$target" = inmemory ]; then
+  substitute_knob --drain-max-bytes "$INMEMORY_DRAIN_MAX_BYTES"
+fi
+
 if [ "$fresh" = 1 ] && [ -f "$RESULTS_FILE" ]; then
   backup="$RESULTS_FILE.$(date -u +%Y%m%dT%H%M%SZ).bak"
   mv "$RESULTS_FILE" "$backup"
@@ -161,6 +222,9 @@ echo "matrix:   ${MATRIX[*]} ${extra[*]:-}"
 if [ "$target" = sqs ]; then
   echo "deviation: drain corpus $SQS_DRAIN_MESSAGES, not the pinned 6000000, and FIFO corpus $SQS_FIFO_MESSAGES per FIFO worker — see the comments in this script"
 fi
+if [ "$target" = inmemory ]; then
+  echo "deviation: drain byte cap $INMEMORY_DRAIN_MAX_BYTES, not the pinned 3221225472, so the 64 KiB leg drains 524288 messages rather than 49152 and the 1 KiB leg the pinned 6000000 rather than 3145728 — see the comments in this script"
+fi
 echo "results:  $RESULTS_FILE"
 echo "log:      $log"