diff --git a/benches/README.md b/benches/README.md index 35ee02cb..dbb368eb 100644 --- a/benches/README.md +++ b/benches/README.md @@ -25,7 +25,7 @@ Every published row is measured with the same sampled core matrix: | `--handler zero` | no-op handler | measures shove and the broker, not simulated work | | `--consumers 1,2,4,8` | four points on the scaling curve | the sampling lever for the published matrix | | `--concurrent` | concurrent processing within each consumer | without it every consumer handles one message at a time, and the parallel flows measure ack round trips instead of throughput | -| `--drain-messages 6000000` | the drain corpus for the consume flows | see below; sized so the group assembles well inside the first half of the corpus on the fastest cell and the window still lasts several seconds | +| `--drain-messages 6000000` | the drain corpus for the consume flows | see below; sized so the group assembles well inside the first half of the corpus on the fastest cell and the window still lasts several seconds. **SQS is the one exception, at 60 000** — see "The SQS corpus deviation" | | `--drain-max-bytes 3221225472` | cap on `corpus × payload_bytes` | 3 GiB: 6 M messages at 64 B, 3 M at 1 KiB, about 49 k at 64 KiB — the in-process backend holds the corpus resident and Kafka writes it to disk before every cell | | `--load-rates …` | the offered-load ladder for the consume flows | see below; three rungs, because the ladder measures latency at a sustained rate and the ceiling comes from the drain | | `--load-window-secs 10` | how long each rung holds its rate | long enough for a rate | @@ -181,15 +181,61 @@ than clamping it, and the matrix's size exceeds that. The harness clamps to 10 when it builds the scenario and the row records 10, so an SQS batch bar is a 10-message batch and is not like-for-like against a backend that ran 500. +### The SQS corpus deviation + +SQS is also the one backend that does not drain the matrix's corpus. It runs +**60 000** messages, set as `SQS_DRAIN_MESSAGES` in `scripts/bench.sh` and +substituted into the matrix for that target only; every other knob is shared. +The script prints a `deviation:` line at the top of the SQS log. + +The corpus is a message *count*, and six million was sized for the fastest +cell — in-process at ~4.7 M msg/s, where it buys a window of about a second. +Measured against LocalStack, SQS drains at ~900 msg/s and fills at ~660, so +the same count buys a 1.7-hour window to measure what a minute measures, and +the pass as a whole runs about 51 hours. The deviation exists because one +count means two different things on backends three orders of magnitude apart, +not to spare SQS a fair measurement. + +Everything that makes a drain row a rate still holds at 60 000: the window +runs tens of seconds against a 1 s floor, the group assembles in the first +handful of messages, and `--drain-max-bytes` still binds the 64 KiB leg at +49 152, so that leg is not deviated at all. + +It is recorded in two places rather than only here. Every drain row carries +`drain.corpus`, so the document says what each cell drained; and when a +slice's backends disagree the charts name them ("corpus differs by backend: +… sqs 60k") instead of listing the sizes unattributed. Nothing about reading +an SQS row changes — a rate is a rate — but a shorter window is a noisier +estimate, which is why the size is on the chart and not just in the script. + Kafka is the reference backend for the batched-consume flow. Its batch and parallel scenarios declare one partition per consumer so every group member gets work. Its `publish_single` row is bounded by librdkafka's default 5 ms `linger.ms`, because the publisher awaits each delivery report. That is shove's default configuration and is what the row is meant to show. -Budget about an hour per Docker backend for the full matrix: per consume -cell a fill of up to six million messages, a drain of several seconds, three -rungs of ten seconds each, and group setup around each. +Budget per consume cell a fill of up to six million messages, a drain of +several seconds, three rungs of ten seconds each, and group setup around +each. A backend's wall clock is set by its drain rate, because the corpus is +a count: the slower the backend, the longer the same corpus takes to fill and +drain while measuring exactly the same thing. + +An hour is the right order for the fast backends and roughly half what the +slow ones want. One 64 B, two-consumer `consume_batch` drain per backend, run +back to back on one idle 8-core Linux host, measured these rates — a guide +for scheduling, not published numbers: + +| backend | msg/s | full matrix | +|---|---|---| +| `inmemory` | ~1 100 000 | minutes | +| `kafka` | ~240 000 | about an hour | +| `redis` | ~166 000 | about an hour | +| `nats` | ~48 000 | about an hour | +| `rabbitmq` | ~18 000 | closer to two | +| `sqs` | ~900 | about an hour, and only because of the corpus deviation above | + +Ratios move with the host; the ordering does not. Time one cell before +committing an evening to a backend nobody has measured here. ## Results document and provenance diff --git a/scripts/bench.sh b/scripts/bench.sh index 7908ab44..8cabb8e7 100755 --- a/scripts/bench.sh +++ b/scripts/bench.sh @@ -47,6 +47,28 @@ MATRIX=( --load-producers 8 ) +# The one documented deviation from the matrix above, and the only backend +# that gets one. +# +# The corpus is a message *count*, and it was sized for the fastest cell — +# in-process at ~4.7 M msg/s, where six million messages buy a window of +# about a second. SQS measures LocalStack at ~900 msg/s and fills at ~660, +# so the same count buys a 1.7-hour window to measure what a minute would, +# and the pass as a whole runs about 51 hours. That is the count meaning two +# different things on two backends, not SQS being slow in a way worth +# publishing 51 hours to show. +# +# Everything that makes a drain row a rate still holds at this size: the +# window runs tens of seconds (the floor chartgen publishes at is 1 s), the +# group assembles in the first handful of messages, and the byte cap still +# binds the 64 KiB leg at 49 152, so that leg is unchanged. The row records +# `drain.corpus`, so the deviation is in the document rather than only here, +# and the charts name the backend whose corpus differs. +# +# This is a per-backend corpus, not a second matrix: every other knob is +# shared, so the flows and the axes stay comparable. +SQS_DRAIN_MESSAGES=60000 + usage() { sed -n '2,7p' "$0" | sed 's/^# \{0,1\}//' exit "${1:-2}" @@ -99,6 +121,18 @@ fi if [ "$target" = sqs ]; then [ -n "${LOCALSTACK_AUTH_TOKEN:-}" ] \ || die "LOCALSTACK_AUTH_TOKEN is not set; run through 'dotenvx run -- scripts/bench.sh sqs'" + + # Substituted into the matrix rather than appended after it: the harness + # rejects a knob given twice, so a deviation has to replace the value. + deviated=0 + for i in "${!MATRIX[@]}"; do + if [ "${MATRIX[$i]}" = --drain-messages ]; then + MATRIX[$((i + 1))]="$SQS_DRAIN_MESSAGES" + deviated=1 + fi + done + [ "$deviated" = 1 ] \ + || die "the matrix has no --drain-messages for the sqs corpus deviation to replace" fi if [ "$fresh" = 1 ] && [ -f "$RESULTS_FILE" ]; then @@ -112,6 +146,9 @@ log="$LOG_DIR/$target-$(date -u +%Y%m%dT%H%M%SZ).log" echo "backend: $target ($example, --features $features)" echo "matrix: ${MATRIX[*]} ${extra[*]:-}" +if [ "$target" = sqs ]; then + echo "deviation: drain corpus $SQS_DRAIN_MESSAGES, not the pinned 6000000 — see the comment in this script" +fi echo "results: $RESULTS_FILE" echo "log: $log"