fix(model-loaders): stop materializing scaled-fp8 checkpoints in float32 - #9429
Open
Pfannkuchensack wants to merge 4 commits into
Open
fix(model-loaders): stop materializing scaled-fp8 checkpoints in float32#9429Pfannkuchensack wants to merge 4 commits into
Pfannkuchensack wants to merge 4 commits into
Conversation
The Krea-2 and Z-Image loaders dequantized ComfyUI 'scaled fp8' weights with `weight.float() * scale` and left the result in float32 until a much later cast to the compute dtype. That holds the entire model at 4 bytes per parameter: a ~12 GB Krea-2 fp8 checkpoint peaks at ~50 GB of RAM before dropping to ~25 GB, which puts a 32 GB machine into swap during a cold load — before anything moves toward VRAM. Both now multiply in float32 for precision but store the compute dtype immediately, halving the cold-load peak. This is the same fix the FLUX.2 loader already carries; its comment documents the identical symptom (~36 GB vs ~17 GB for a 9B model). The Qwen-Image loader was already correct. _dequantize_scaled_fp8 takes the target dtype as a parameter (defaulting to bfloat16), and the Krea-2 single-file loader resolves the compute dtype before calling it so the weights land in their final type directly instead of being cast twice.
Pfannkuchensack
requested review from
JPPhoto,
blessedcoolant,
dunkeroni and
lstein
as code owners
August 1, 2026 04:11
Collaborator
|
This would fix my issues with krea thus far |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The Krea-2 and Z-Image loaders dequantize ComfyUI "scaled fp8" checkpoints with
weight.float() * scaleand leave the result in float32 until a much later cast to the compute dtype. That holds the entire model at 4 bytes per parameter during the cold load.For a ~12.2 GB Krea-2 fp8 checkpoint (≈12.2B params at 1 byte each) that means:
The ~25 GB figure is corroborated by the official Krea-2-Turbo diffusers transformer, which is 25 GB on disk in bf16.
Both loaders now multiply in float32 for precision but store the compute dtype immediately, so the whole model is never materialized in float32. This is the same fix the FLUX.2 loader already carries — its comment documents the identical symptom ("~36GB vs ~17GB for a 9B model … was the dominant cold-load spike"). The Qwen-Image loader was already correct (it multiplies in
compute_dtype)._dequantize_scaled_fp8gains a target-dtype parameter (defaulting to bfloat16), and the Krea-2 single-file loader resolves the compute dtype before calling it, so weights land in their final type directly instead of being cast twice.Related Issues / Discussions
Reported by a user with 32 GB RAM (~22 GB free) loading a Krea-2 scaled-fp8 checkpoint: RAM filled up during load, before anything moved toward VRAM.
QA Instructions
Requires a ComfyUI "scaled fp8" checkpoint — one with
<layer>.weightinfloat8_e4m3fnplus a<layer>.weight_scalesibling. Krea-2 and Z-Image single-file checkpoints from the usual sources qualify.htop) while loading such a model onmain. Peak RAM is ~4× the file size.main— output is unchanged. The multiply still happens in float32; only the storage dtype of the result changed, and it was going to be cast to that dtype a few steps later anyway.Automated:
pytest tests/backend/model_manager/load/— 426 passed. Includes a new case asserting the result lands in the compute dtype rather than float32, which is the property that bounds the peak.Not covered: the RAM peak itself is not asserted in a test; the guard is the dtype assertion. Measuring peak RSS in CI would be flaky.
Merge Plan
Nothing special. No schema change, no node versions affected — this only touches loader internals.
Checklist
What's Newcopy (if doing a release after this PR)