Experimental AMD-focused FP4 quantization and backend work for llama.cpp,
developed on Framework Desktop with AMD Strix Halo 395+ and 128 GB unified RAM.
ROCmFP4 adds new GGUF tensor formats, quantization presets, ROCm/HIP kernels, Vulkan shader support, and reproducible regression guards for long-context MTP inference. The goal is a practical 4-bit format for AMD systems that keeps model coherence protected while improving memory use and decode speed.
Status: experimental research build. Results are hardware-, driver-, model-, and prompt-sensitive. Do not treat these numbers as upstream llama.cpp claims until they are independently reproduced.
ROCmFP4 is a custom 4-bit weight format for GGUF models:
Q4_0_ROCMFP4: dual-scale 4.50 BPW layout using two finite UE4M3 scale bytes per 32-weight block.Q4_0_ROCMFP4_FAST: single-scale 4.25 BPW layout for speed-sensitive tensor roles.- Tensor-aware presets that mix ROCmFP4 layouts with protected higher-precision tensors where quality matters.
- ROCm/HIP vector-dot, copy, dequant, and FlashAttention handling for the new layouts.
- Vulkan decode and MMQ shader support for the same GGUF tensor types.
- MTP regression guards for long-context target/draft decode.
ROCmFP4 is not MXFP4, NVFP4, or a renamed Q4 format. It uses a Codebook10 4-bit value table and finite unsigned E4M3 half-scale semantics tuned for the current AMD backend paths in this tree.
Current strongest reproduced local result:
| Hardware | Model | Backend | Context | Profile | Decode |
|---|---|---|---|---|---|
| Framework AMD Strix Halo 395+, 128 GB unified RAM | Qwen3.6 35B A3B MTP ROCmFP4 STRIX_LEAN | ROCm0 | 262144 | reasoning on, draft-MTP, q8 main KV, q4 draft KV | 104.4 tok/s short, 89.3 tok/s sustained |
| Framework AMD Strix Halo 395+, 128 GB unified RAM | Qwen3.6 27B MTP ROCmFP4 STRIX_LEAN | ROCm0 | 262144 | draft-MTP | 33.6 tok/s short, 28.0 tok/s sustained |
The benchmark policy is intentionally conservative: a microbenchmark win is not promoted unless end-to-end decode guards hold or improve.
| Target | Script | Status |
|---|---|---|
Strix Halo / RDNA3.5 (gfx1151) |
scripts/build-strix-rocmfp4-mtp.sh |
Validated locally on Framework Desktop / Ryzen AI MAX+ 395 |
RDNA3 (gfx1100 class) |
scripts/build-rdna3.sh |
Build target provided; community validation wanted |
RDNA2 (gfx1030 class) |
scripts/build-rdna2.sh |
Build target provided; community validation wanted |
RDNA4 (gfx1200 class) |
scripts/build-rdna4.sh |
Experimental; requires ROCm support for gfx1200 device libraries |
| Vulkan fallback | Manual CMake path | Recommended when HIP support is incomplete or a GPU is not mapped cleanly |
- Validated integration snapshot:
4860505ee - Validation date:
2026-06-13
This snapshot passed the promoted Strix Halo gate with:
- ROCmFP4 quantization tests
- MTMD C API smoke test
- ROCm copy and FlashAttention backend tests
- ROCmFP4 copy, FlashAttention, and runtime regression guards
- Qwen3.6 35B A3B MTP ROCmFP4 smoke test
| Guide | Who it's for |
|---|---|
docs/STRIX-HALO-QUICKSTART.md |
Strix Halo users — full install, quantize, run, validate |
docs/BUILD-AMD-ARCHITECTURES.md |
AMD GPU build flags and scripts, including RDNA2 through RDNA4 plus experimental gfx906 |
docs/ROCmFP4-REPRODUCIBILITY.md |
Regression guards and proof commands |
docs/ROCmFP4-MTP-COMPARISON.md |
Benchmark history and promoted profiles |
ggml/rocmfp4/README.md |
Format details and expert HIP tuning knobs |
ggml/rocmfp4/— ROCmFP4 format definitions, CPU reference quant/dequant, and HIP helper kernelsggml/src/ggml-cuda/— upstream HIP/CUDA backend files with AMD ROCmFP4 integration (HIP builds use this directory even with-DGGML_CUDA=OFF)ggml/src/ggml-vulkan/vulkan-shaders/— Vulkan shader support for ROCmFP4scripts/build-*.sh— build scripts per AMD GPU generationscripts/check-rocmfp4-*.sh— correctness and performance regression guards
git clone https://github.com/charlie12345/rocmfp4-llama.git
cd rocmfp4-llama
git checkout mtp-rocmfp4-strixPick the script that matches your GPU:
| Your GPU | Build command | Output folder |
|---|---|---|
| Strix Halo / RDNA3.5 | env JOBS=16 scripts/build-strix-rocmfp4-mtp.sh |
build-strix-rocmfp4/ |
| RDNA2 (RX 6000) | env JOBS=16 scripts/build-rdna2.sh |
build-rdna2/ |
| RDNA3 (RX 7000, including RX 7600-class cards) | env JOBS=16 scripts/build-rdna3.sh |
build-rdna3/ |
| RDNA4 (RX 9000) | env JOBS=16 scripts/build-rdna4.sh |
build-rdna4/ |
| Vega 20 / gfx906 experimental (MI50 / MI60) | env JOBS=16 scripts/build-gfx906.sh |
build-gfx906/ |
| Windows RDNA2 | build-hip.bat |
build-hip/ |
Not sure which GPU you have? See
docs/BUILD-AMD-ARCHITECTURES.md for the full
gfx target table, runtime environment variables, and Vulkan-only builds.
Strix Halo users: follow
docs/STRIX-HALO-QUICKSTART.md for
prerequisites, MTP flags, and troubleshooting.
Key binaries after a successful build:
bin/llama-cli
bin/llama-server
bin/llama-quantize
bin/llama-bench
bin/test-backend-ops
bin/test-quantize-fns
bin/test-quantize-perf
Start from an F16 or BF16 GGUF source model. Quantizing an already heavily quantized GGUF into ROCmFP4 is useful for smoke tests only; for real quality, use an F16/BF16 source.
Compact Strix profile:
./build-strix-rocmfp4/bin/llama-quantize \
/path/to/source-bf16.gguf \
/path/to/model-ROCmFP4-STRIX_LEAN.gguf \
Q4_0_ROCMFP4_STRIX_LEANQuality-biased Strix profile:
./build-strix-rocmfp4/bin/llama-quantize \
/path/to/source-bf16.gguf \
/path/to/model-ROCmFP4-STRIX.gguf \
Q4_0_ROCMFP4_STRIXPure experimental formats:
./build-strix-rocmfp4/bin/llama-quantize source.gguf out-dual.gguf Q4_0_ROCMFP4
./build-strix-rocmfp4/bin/llama-quantize source.gguf out-fast.gguf Q4_0_ROCMFP4_FASTUse the Strix presets for serious testing. The pure FAST format is smaller and can be faster, but may trade away too much coherence on sensitive tensors.
Example interactive run:
cd rocmfp4-llama
HSA_OVERRIDE_GFX_VERSION=11.5.1 \
GGML_HIP_ENABLE_UNIFIED_MEMORY=1 \
./build-strix-rocmfp4/bin/llama-cli \
-m /path/to/model-ROCmFP4-STRIX_LEAN.gguf \
-dev ROCm0 \
-ngl 999 \
-c 262144 \
-b 512 \
-ub 512 \
-fa on \
-ctk q8_0 \
-ctv q8_0 \
--spec-type draft-mtp \
--spec-draft-n-max 3 \
--spec-draft-n-min 0 \
--spec-draft-p-min 0.0 \
--spec-draft-p-split 0.10 \
--spec-draft-type-k q4_0 \
--spec-draft-type-v q4_0 \
--reasoning on \
--jinjaFor models that do not support MTP or reasoning, remove the --spec-* and
--reasoning flags.
Run the full promoted gate:
env HSA_OVERRIDE_GFX_VERSION=11.5.1 scripts/check-rocmfp4-all-regression.shFocused guards:
scripts/check-rocmfp4-quant-regression.sh
scripts/check-rocmfp4-rocm-runtime-regression.sh
scripts/check-rocmfp4-rocm-fattn-regression.sh
scripts/check-rocmfp4-vulkan-runtime-regression.sh
scripts/check-rocmfp4-qwen-mtp-regression.sh
scripts/check-rocmfp4-qwen35-a3b-mtp-regression.shThe scripts accept environment overrides for model paths, binary paths, context, cache type, MTP settings, and speed floors. See each script for the exact variables.
- Coherence first: tensor-aware presets are preferred over pure speed profiles.
- AMD-specific work must be isolated and measurable.
- ROCm and Vulkan paths must avoid silent fallback where ROCmFP4-specific kernels exist.
- Rejected experiments are recorded so they are not repeatedly rediscovered.
- Claims must cite model, backend, context window, flags, hardware, and date.
- This is not native FP4 tensor-core execution. rocWMMA FP4 input support is not available in the local headers used by this build.
- ROCmFP4 is currently optimized for AMD Strix Halo behavior and may need retuning on other GPUs.
- TurboQuant and TriAttention are not runtime flags in this isolated tree.
- Some scripts reference local model defaults; override
MODEL,ROCMFP4_MODEL, orBASELINE_MODELfor your checkout.
This repository is based on llama.cpp and keeps the upstream MIT license. See
LICENSE for details. Bundled third-party notices are listed in
THIRD_PARTY_NOTICES.md.