fix(llama.cpp): fall back to an even layer split when the split weights sum to zero

[bernardladenthin]

Summary

• Adds patches/0012, which fixes the macOS-arm64 model-load failure red since the b10792→b10797 step: a GPU reporting zero free memory made load_tensors divide by zero, putting NaN in every layer split point and throwing std::out_of_range — whose libc++ what() is the bare string "vector", printed as the unactionable error loading model: vector.
• The same line is reachable without any GPU memory pressure and on any backend: --tensor-split values are never range-checked, so -ts 1,-1 cancels out and lands on it too. Verified against the unfixed library, not reasoned about.
• Ships both an upstream test (tests/test-model-split.cpp, 5 cases) and a runnable project-side guard (src/test/cpp/test_model_split.cpp, 7 cases), because a FetchContent subproject builds with LLAMA_BUILD_TESTS=OFF and would never compile the upstream one.

Root cause

llama_model_base::load_tensors weights the per-device layer split by ggml_backend_dev_memory()'s free, then normalises with splits[i] /= split_sum. When the weights sum to zero that is 0/0:

1. every split point becomes NaN;
2. NaN compares false against everything, so std::upper_bound returns the end iterator;
3. layer_gpu == n_devices(), and devices.at(layer_gpu) throws std::out_of_range;
4. libc++ renders that as what() == "vector", which the model-load catch prints verbatim.

Why it started at b10618..b10797. Upstream 8c0b9cd04 ("metal : fix memory query under low-memory conditions") changed the Metal query to *free = *total > cur ? *total - cur : 0. Before that clamp, an over-committed device (currentAllocatedSize > recommendedMaxWorkingSetSize) underflowed to a huge size_t, which normalised harmlessly. The free == 0 && total == 0 host-memory fallback does not fire, because total is recommendedMaxWorkingSetSize and is non-zero.

Why only macOS went red. act_gpu_layers is devices.empty() ? 0 : …, so with no GPU backend every layer returns early on cpu_dev and the .at() line is unreachable. That is the Metal / no-Metal discriminator seen in run 33994480652.

The second trigger, which is what makes this a general defect. common/arg.cpp parses --tensor-split with std::stof and never range-checks the values, so -ts 1,-1 cancels: split_sum is 0 again, the split points become [inf, -nan], and every layer maps one past the last device — on CUDA, Vulkan or ROCm as much as on Metal, in a fresh process. Reproduced here against the unfixed library.

The change

Both blocks move out of load_tensors into free functions declared in src/llama-model.h, purely so they can be tested — the failing state needs a real over-committed GPU and cannot be arranged through any public API.

Function	Role
llama_model_splits_normalize()	the fix — on split_sum == 0, warn and split evenly; the only neutral choice when no device can be preferred, and exactly right for a single device
llama_model_splits_select_device()	the diagnostic — bounds-check the lookup and throw a message naming the function, the layer, the device index, the split count and the split values

A recurrence now reads

llama_model_splits_select_device: offloaded layer 0 of 32 mapped to device index 1,
but there are only 1 split point(s) [-nan] -- the split points are malformed

instead of vector. The [-nan] names the cause outright — which is precisely what was missing when this had to be diagnosed by reading source.

Test plan

• Affected unit / integration tests pass locally
• CI is green on this branch
• Docs / CHANGELOG updated where applicable

Run locally on a fresh rm -rf llama/build, so the applier re-applied all 9 patches from a pristine fetch:

Gate	Result
Patch applier	9/9 clean, stamp written
ctest	527/527 (the 7 new tests included)
mvn clean verify	1742 tests, 0 failures, 0 errors
Bytecode gate	616 classes / 5 jars, 0 over major 52
Shipped libjllama.so	both new strings present
Upstream test, standalone-linked	5 cases / 22 assertions, green

Both guards were falsified, not just run: with the fix removed and llama rebuilt, the upstream test fails 12 of 23 assertions and the gtest guard 2 of 6, each printing the NaN split points through the new message. The -ts 1,-1 shape was falsified the same way and yields [inf, -nan].

What this does not prove

The failing path still needs a GPU backend — on Linux devices is empty and the line is unreachable — so these runs prove the arithmetic, which is what broke, not the end-to-end load. The macOS CI job on this branch is the end-to-end gate. This is stated in CLAUDE.md too, so a later reader does not over-trust the local green.

Related issues / PRs

• Refs the investigation rows in docs/history/llama-cpp-breaking-changes.md (2026-09-05 / -06), whose open question "which commit, and has upstream fixed it?" this closes: 8c0b9cd04, and no — the bare splits[i] /= split_sum is still there at b10850.
• Upstream-submittable; not yet filed with ggml-org/llama.cpp.

Checklist

• I have read CONTRIBUTING.md and CODE_OF_CONDUCT.md
• My commits follow Conventional Commits
• No security-sensitive changes

🤖 Generated with Claude Code

https://claude.ai/code/session_01AnNYn8W1xuVxVJtyL34GyH

---

Generated by Claude Code

[claude]

Review Summary

Strong fix with excellent root-cause analysis and defensive coding.

Key Strengths:

• Correctly identified upstream commit 8c0b9cd04 as root cause
• Discovered both metal memory AND --tensor-split parsing triggers
• Two-part fix: normalize() + select() with good error messages
• Comprehensive tests (7 cases covering all scenarios)
• Exceptional documentation in CLAUDE.md

Code Quality: No issues found. Format string is literal (safe), float equality check is correct, bounds checking is solid.

Pre-merge: Verify CI passes for macOS-15 Metal jobs before merge.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud