fix(fp8): never apply FP8 storage to already-quantized weights

[Pfannkuchensack]

Summary

Stacked on # (which is stacked on the Z-Image PR). No logical dependency — the guard stands alone — but all three touch _should_use_fp8, _apply_fp8_to_nn_module, MainModelDefaultSettings.tsx and the same test file, so merging in order avoids conflicts.

Enabling the fp8_storage toggle on an already-quantized model breaks it. Every quantized-format loader reaches _apply_fp8_layerwise_casting — FLUX/FLUX2/Krea-2/Qwen-Image/WAN/Z-Image GGUF, FLUX bnb-NF4, and the T5/Qwen3/Gemma2 encoders — and the cast there is not a no-op. Verified on real layers:

• GGUF raises ValueError: Operation changed the dtype of GGMLTensor unexpectedly at load time.
• bnb NF4 corrupts silently. bnb.nn.LinearNF4 subclasses nn.Linear, so the isinstance check passes and the packed uint8 payload is cast to float8. Inference still returns finite numbers — the model just produces garbage.

Reproduce the silent one directly (no model download needed):

import torch, bitsandbytes as bnb
from invokeai.backend.model_manager.load.load_default import ModelLoader

l4 = bnb.nn.LinearNF4(256, 256, bias=False)
l4.weight = bnb.nn.Params4bit(torch.randn(256, 256), requires_grad=False, quant_type="nf4")
l4 = l4.cuda()
x = torch.randn(1, 256, device="cuda", dtype=torch.bfloat16)
ref = l4(x).clone()

ModelLoader._apply_fp8_to_nn_module(l4, torch.float8_e4m3fn, torch.bfloat16)
print("max abs diff:", (l4(x).float() - ref.float()).abs().max().item())
# before this PR: ~50   (no error, no warning)
# after:           0.0

Guarded on two levels, because a format check alone is not enough — externally quantized weights can ship under a plain diffusers format (e.g. an SDNQ-quantized Z-Image):

• _should_use_fp8 rejects gguf_quantized, bnb_quantized_nf4b and bnb_quantized_int8b.
• _apply_fp8_to_nn_module skips any module whose params are non-floating-point (bnb's packed uint8) or a torch.Tensor subclass (GGUF's GGMLTensor), regardless of declared format. This also covers the case where a model already has a persisted fp8_storage=true from before this PR.

Frontend hides the toggle for quantized formats, so the UI stops offering a control the backend refuses.

Related Issues / Discussions

Follow-up to #8945 (FP8 storage). Same series as the Z-Image and Anima FP8 PRs.

QA Instructions

Needs a CUDA GPU.

Reproduce the bug on main (both cases need fp8_storage enabled in the model's Default Settings):

1. Any GGUF main model → generation fails at load with Operation changed the dtype of GGMLTensor unexpectedly.
2. A FLUX bnb-NF4 model → no error at all, output is garbage. Or run the snippet above.

Verify the fix:

1. UI — open a GGUF or bnb model in Model Manager → Default Settings. The FP8 Storage switch is gone. Open a non-quantized model (SDXL, FLUX checkpoint, Z-Image diffusers) → the switch is still there and still works.

2. Legacy DB value — the backend must not rely on the UI. Force the old value back in:

   curl -X PATCH http://127.0.0.1:9090/api/v2/models/i/<gguf-model-key> \
     -H "Content-Type: application/json" \
     -d '{"default_settings":{"fp8_storage":true}}'
   

   Then generate with that model. Expect: model loads at its normal size, no FP8 layerwise casting enabled line in the log, no GGMLTensor error.

3. No regression on non-quantized models — a model with FP8 enabled must still log FP8 layerwise casting enabled ... and load at roughly half its usual VRAM.

⚠️ If you test step 2 with a GGUF Z-Image model, denoise still fails afterwards with Multiple dispatch failed for 'torch._ops.aten.where.self' (torch.where(mask, pad_token, feats_cat) with a GGMLTensor pad token). That is a pre-existing, unrelated GGUF/Z-Image incompatibility — it reproduces identically with fp8_storage off, and on main. Use a GGUF FLUX model if you want a run that completes.

Unit tests:

uv run --extra cuda --extra test pytest tests/backend/model_manager/load -q --no-cov

433 passed, 129 skipped locally. The new tests cover the format check (parametrized over all three quantized formats) and the param-level guard (both signals: non-floating-point payload and Tensor subclass), plus a control assertion that an ordinary layer in the same model is still cast.

Merge Plan

Merge after the Anima PR to avoid conflicts in load_default.py, MainModelDefaultSettings.tsx and test_load_default_fp8.py. If the earlier PRs in the series are dropped, this one rebases onto main cleanly on its own — it has no logical dependency on them.

No DB schema, no redux slice, no API schema change. Existing fp8_storage=true values persisted against quantized models stay in the DB and are simply ignored from now on; no migration needed.

Checklist

• The PR has a short but descriptive title, suitable for a changelog
• Tests added / updated (if applicable)
• ❗Changes to a redux slice have a corresponding migration — n/a, no redux changes
• Documentation added / updated (if applicable) — n/a
• Updated What's New copy (if doing a release after this PR) — n/a