Merge pull request #534 from bernardladenthin/local/bench-b11558

Isolate upstream #30270: correct fix, no measurable effect here, and a guard proven red

Author: bernardladenthin

TODO.md

@@ -274,12 +274,12 @@ Kept as a list because each entry names the machine it needs -- the harness is
 | what | needs | why it cannot wait for CI |
 |---|---|---|
 | ~~the cost of `GGML_OPENMP=OFF`~~ | ~~16-vCPU Linux~~ | **done 2026-10-11: no measurable cost** |
-| **re-baseline the module tables at the current pin** | the same instances | the tables are b11538, and **#30270 in the b11556 range fixes F16C on AVX2**, i.e. the `haswell` class every "against haswell" column is computed against -- so nothing measured from b11556 on may be compared with that page (noted there too) |
-| Does a self-contained jar load on a **slim/distroless** image? | any Linux instance, a container | CI runs on full runners, so it cannot see this class at all |
-| Does **`75-real 80-real 90-real`** make a T4 work? | a rebuilt CUDA module + a `g4dn` instance | the module has to be built first, so this follows a CI run |
-| **Windows extraction reuse**: second start copies nothing, a running JVM's locked DLLs survive another build's cleanup | a rented Windows instance, two JVMs | file locking is Windows-only and needs two processes |
-| the **`sapphirerapids` vs `haswell`** gap at high core counts | a large Intel instance | one 4-vCPU table exists; generation ranking can reverse with threads |
-| **OpenVINO on an Intel GPU/NPU** | hardware nobody rents | the 172x figure is the CPU-plugin fallback only, so nothing is known about the hardware it is for |
+| ~~re-baseline the module tables at the current pin~~ | — | **done 2026-10-12 at b11558 on 64 cores**, and the suspicion behind it was **refuted**: #30270 changes nothing measurable ([`docs/benchmarks/upstream-f16c-30270.md`](docs/benchmarks/upstream-f16c-30270.md)), so the baseline moved with the machine size, not with the pin |
+| ~~Does a self-contained jar load on a slim/distroless image?~~ | — | **done 2026-10-12**: distroless and `temurin:21-jre` load, `temurin:21-jre-alpine` is refused by name (`os.name=Linux-Musl`) -- see [`docs/benchmarks/native-loading.md`](docs/benchmarks/native-loading.md) |
+| ~~Does `75-real 80-real 90-real` make a T4 work?~~ | — | **done 2026-10-12: yes.** Three `exit 134` aborts became three clean runs on the same AMI, driver and GPU |
+| ~~Windows extraction reuse~~ | — | **done 2026-10-12 locally**, no rented machine needed: ~810 ms saved per start, and a sweep leaves exactly the five files a running JVM still holds |
+| ~~the `sapphirerapids` vs `haswell` gap at high core counts~~ | — | **done 2026-10-12: the ranking reverses.** At 64 threads `haswell` leads token generation at 192.6 while `sapphirerapids` is last at 181.8 -- and ggml picks the latter |
+| **OpenVINO on an Intel GPU/NPU** | hardware nobody rents | the 21.7x/336x figures are the CPU-plugin fallback only, so nothing is known about the hardware the jar is actually for |
 | the two **SME** aarch64 modules | not rentable anywhere | — |
 
 ## Open — jllama-specific
@@ -860,6 +860,45 @@ three different subsystems and a combined report gets triaged as one.
 | 3 | `ggml-rpc: the client answers every supports_op with true` | the RPC section of `CLAUDE.md`: a served device that lacks an op **aborts the server**. Reproduced on the macOS CI runners' paravirtual Metal GPU, which has no `MUL_MAT` and took the first run down; it is why every test here serves `CPU` by name | upstream's own TODO comment admits it. Should the client forward the query to the server, or should `rpc-server -d` refuse a device whose op set it cannot report? |
 | 4 | `windows-rocm: 83,362 compiler warnings, against 9 on linux-rocm` | the table below in this section (714,345 log lines against 2,455; the per-class counts; all of them in `ggml/src/ggml-cuda/*.cu`) | the cause is the language, not the arch count: HIP sources are compiled as plain C++ on Windows (`CXX_IS_HIPCC` on `WIN32`) and as HIP on Linux. Fix the warnings, or the C++-on-Windows choice? Their own `windows-rocm` job pays the identical cost |
 
+#### A fifth, and this one is a small PR rather than an issue: the #30270 follow-up
+
+Unlike the four above, here there **is** something to offer -- and the measurement is already done
+([`docs/benchmarks/upstream-f16c-30270.md`](docs/benchmarks/upstream-f16c-30270.md)).
+
+[#30270](https://github.com/ggml-org/llama.cpp/pull/30270) (commit `c28e174141dd`, merged 2026-10-11)
+fixes a real defect: the `#elif defined(__F16C__)` branch of `simd-mappings.h` defined only
+`GGML_CPU_COMPUTE_FP16_TO_FP32`, so the selected macro fell through to the 64K lookup table and an
+F16C CPU never used the hardware instruction. NEON, POWER9 and RISC-V each define **both** macros in
+their branch; only F16C defined half of its own decision.
+
+Two things to offer, both independent of throughput:
+
+1. **Move the two `#define`s into the branch**, where the COMPUTE macros are defined, as every other
+   branch does. They currently sit **88 lines** further down and outside the chain, which is why the
+   new code has to explain the ordering in its own comment ("checks below don't test for `__F16C__`").
+   A future reordering of the file breaks it **silently**, back to the lookup table -- the exact defect
+   that went unnoticed until now.
+2. **Add a guard so the omission cannot recur**, after the branch chain and before the fallback:
+
+   ```c
+   #if defined(GGML_CPU_COMPUTE_FP16_TO_FP32) && !defined(GGML_CPU_FP16_TO_FP32)
+   #error "a branch defined GGML_CPU_COMPUTE_FP16_TO_FP32 but not GGML_CPU_FP16_TO_FP32"
+   #endif
+   ```
+
+   **Proven both ways** with clang `-fsyntax-only` against the real b11574 header: it passes on the
+   fixed tree, fails with the `#error` on the reverted one, does not fire without `__F16C__`, and both
+   trees compile when the guard is absent -- so the guard is the only thing that changes the outcome.
+
+And one thing to **say** in the same comment, because it is honest and useful rather than an
+objection: a clean isolation on Granite Rapids (one tree built twice, upstream's own `llama-bench`,
+5 repetitions, two non-F16C control modules) finds **no measurable effect** for Q4_K_M at 8 and 16
+threads -- every difference inside ±2 % with random signs. The author's own description (a FreeBSD box
+with "clock-mismatched RAM", "gains like these may be greater than reasoned") fits that: the fix
+removes a ~256 KB table from the path, which costs on a bandwidth-starved machine and nothing on a
+server CPU. **The change stays right; the number is not general.** Offering the measurement with the
+PR is the point, not disputing it.
+
 **Two rules for the day.** State the measurement and ask the question; do not propose the fix for 1-3,
 because each is a design decision in code this project does not own. And say plainly what was done
 downstream instead (the opt-in, the non-aborting registration, the `-w` on `ggml-hip`), so a

docs/benchmarks/README.md

@@ -15,6 +15,7 @@ themselves live in [`../../TODO.md`](../../TODO.md).
 |---|---|
 | [`cpu-backend-modules.md`](cpu-backend-modules.md) | What each of the 22 shipped CPU backend modules is worth, per microarchitecture — and whether `ggml_backend_load_best` picks the fastest one |
 | [`gpu-module-jars.md`](gpu-module-jars.md) | What happens when a GPU module jar is actually *used* -- two findings that contradict the "a module that cannot help is skipped" premise |
+| [`upstream-f16c-30270.md`](upstream-f16c-30270.md) | An isolation of one upstream commit: what it does, that it changes nothing measurable here, and a compile-time guard proven red against the defect |
 | [`native-loading.md`](native-loading.md) | What extracting the natives costs and what the reuse saves, how a running JVM's locked files behave, and which container base images can load the Linux jar |
 | [`ec2-harness.md`](ec2-harness.md) | How a measurement is run on rented hardware, and the invariants that came out of getting it wrong |
 

docs/benchmarks/cpu-backend-modules.md

@@ -153,6 +153,88 @@ issue with these tables (`TODO.md`, "Upstream submissions"), and worth knowing f
 about generation latency — `net.ladenthin.llama.backend` cannot select a *CPU* module, so the only
 lever today is shipping a directory with one module in it.
 
+## Re-measured at b11558 on 64 cores: the baseline moved, and the ranking reverses
+
+Measured **2026-10-12** against the **published** jars (`5.2.0-20261012.112736-13`, llama.cpp
+**b11558**) on an `m8i.16xlarge` -- the **same CPU model** as the table above (Xeon 6975P-C, Granite
+Rapids) but **64 vCPU instead of 4**. Same method: one module staged at a time, same model, 35 runs in
+one boot.
+
+This exists because the section below predicted it: the b11538 tables are relative to `haswell`, and
+**upstream #30270 (merged 2026-10-11, in the b11538..b11556 range) makes AVX2+F16C CPUs use the
+hardware conversion instead of a lookup table** -- i.e. it moves exactly that baseline.
+
+### All 14 modules at 16 threads
+
+| module | pp512 | tg128 | against `haswell` (pp) |
+|---|---:|---:|---:|
+| **`sapphirerapids`** (AMX) ← ggml's choice | **2775.5** | **160.0** | **+96 %** |
+| `cooperlake` | 1949.1 | 152.5 | +37 % |
+| `skylakex` | 1801.4 | 153.0 | +27 % |
+| `cannonlake` | 1787.7 | 151.9 | +26 % |
+| `icelake` | 1565.7 | 152.9 | +10 % |
+| `cascadelake` | 1563.1 | 152.8 | +10 % |
+| `zen4` | 1553.2 | 153.9 | +10 % |
+| `haswell` | 1417.6 | 151.8 | — |
+| `alderlake` | 1410.0 | 150.4 | −1 % |
+| `piledriver` | 892.6 | 140.6 | −37 % |
+| `ivybridge` | 861.2 | 138.3 | −39 % |
+| `sandybridge` | 841.2 | 76.2 | −41 % |
+| `sse42` | 68.9 | 36.4 | −95 % |
+| `x64` | 63.4 | 33.2 | −96 % |
+
+Full set on the same machine: ggml chose `sapphirerapids`, 2763.4 / 159.6 -- within noise of the
+staged run, so the staging still does not distort anything. **All 14 load and run**, as at b11538.
+
+**AMX is worth +96 % on prompt processing here, against +148 % in the 4-vCPU table.** Two things
+changed at once (the pin and the machine size), so see the isolation attempt below before reading that
+as the effect of #30270 alone.
+
+### The thread sweep: at 64 threads `haswell` beats the module ggml picks
+
+The open question was whether the generation ranking reverses with thread count, as the Windows
+OpenMP measurements showed it can. **It does.** Token generation, four modules, same machine:
+
+| threads | 1st | 2nd | 3rd | 4th |
+|---:|---|---|---|---|
+| 4 | **`sapphirerapids`** 59.7 | `zen4` 55.2 | `cascadelake` 54.1 | `haswell` 53.8 |
+| 8 | **`sapphirerapids`** 105.6 | `cascadelake` 102.0 | `zen4` 101.5 | `haswell` 99.4 |
+| 16 | **`sapphirerapids`** 159.1 | `zen4` 153.1 | `cascadelake` 151.7 | `haswell` 150.7 |
+| 32 | **`sapphirerapids`** 188.2 | `cascadelake` 186.8 | `haswell` 185.1 | `zen4` 183.8 |
+| **64** | **`haswell` 192.6** | `zen4` 190.0 | `cascadelake` 188.6 | **`sapphirerapids` 181.8** |
+
+At 64 threads the richest module is **last**, and the one ggml would never pick on this CPU is first.
+Prompt processing turns over there too: between 32 and 64 threads `sapphirerapids` falls 4119.7 → 3880.2
+and `zen4` collapses 2069.0 → **1143.3**, while `haswell` keeps rising 1752.8 → 1960.3. The machine has
+32 physical cores with SMT, so 64 threads means hyperthreads -- which is a realistic default, since
+`std::thread::hardware_concurrency()` and most "use all cores" settings report 64 here.
+
+**This sharpens the upstream finding below rather than adding a new one**: `ggml_backend_load_best`
+scores by feature level, which costs 10-11 % of generation at 4 vCPU and **loses outright at 64
+threads**.
+
+### #30270 is NOT the reason the baseline moved -- measured, and it refutes what this page first said
+
+Same CPU model and the **same thread count** (4), the only pair comparable across the two tables:
+
+| | b11538, 4 vCPU | b11558, 64 vCPU at t=4 | change |
+|---|---:|---:|---:|
+| `haswell` pp512 | 221.8 | 422.7 | **+91 %** |
+| `sapphirerapids` pp512 | 550.0 | 847.9 | +54 % |
+| ratio | 2.48x | **2.01x** | |
+
+This page first read that asymmetry as *consistent with #30270*, since `haswell` gained far more than
+the AMX module and #30270 is an AVX2-specific fix. **That was wrong, and the clean isolation says so**
+([`upstream-f16c-30270.md`](upstream-f16c-30270.md)): one source tree built twice on one machine with
+those eight lines removed shows **no measurable effect at all** -- every difference inside ±2 %, with
+two non-F16C control modules moving just as much.
+
+So the +91 % is **the instance size**: 4 threads on a 64-vCPU machine have far more memory bandwidth
+to themselves than a 4-vCPU machine does, and `haswell` is the more bandwidth-bound of the two. The
+asymmetry between two rows of two different machines is not an isolation, and reading it as one was
+the mistake. It is kept here rather than quietly removed, because the next person comparing two tables
+across a pin **and** a machine size will be tempted the same way.
+
 ## These numbers are bound to llama.cpp b11538, and b11556 already moved one of them
 
 Every table above was measured against the natives jars built at **b11538**. That is not boilerplate: