feat: VRAM swap, VisionOffload strategies, Q2K/Q3K quantization, and low-VRAM GPU support

[soymh]

Summary

This PR brings DeepSeek‑OCR to low-end NVIDIA GPUs (RTX 3050 4GB, GTX 1060, etc.) via a VRAM swap system that manages SAM/CLIP vision model memory on CUDA, plus expanded DSQ quantization support and a configurable VisionOffload strategy enum.

10 commits | 23 files changed | +2140 / −603

---

Key Changes

1. VisionOffload Strategy Enum (crates/core/src/runtime.rs)

Adds Auto (default), Sequential, FullGpu, Cpu modes. --vision-offload CLI/config flag replaces the old boolean vision_swap.

2. Sequential VRAM Swap (crates/infer-deepseek/src/model/swap.rs)

New SequentialVramSwap struct loads SAM (~1.26GB) then CLIP (~0.86GB) one at a time on CUDA, never holding both simultaneously. Auto-detects VRAM <6GB via nvidia-smi / DEEPSEEK_OCR_VRAM_MB env var.

3. Hybrid Parallel Mode (compute_image_embeddings_with_swap)

Global view runs on CPU in parallel with patch crops on CUDA via scoped threads — best speed/VRAM trade-off on tight GPUs (~2.3GB peak for Q4K).

4. Full GPU Mode (compute_image_embeddings_full_gpu)

All vision on CUDA: loads SAM→processes→drops→loads CLIP→processes. ~3.2GB peak. Faster vision prefill when VRAM allows.

5. LM Device Transfer (move_language_to_device, to_device)

UnsafeCell-based LM ownership transfer lets the language model move between CPU/CUDA. Weight structs gain to_device(&self, device) with QMatMul::QTensor raw byte reconstruction.

6. Q2K / Q3K DSQ Quantization (deepseek-ocr-q2k, deepseek-ocr-q3k)

New quantized model entries use Q2_K and Q3_K DSQ snapshots, further reducing memory.

7. Bug Fixes

• Q4_K round-trip corruption: Keep LM on CUDA during FullGpu vision, evict only for sequential mode
• Q2_K precision loss: Protect gate_proj/up_proj by clearing weight_f32 in quantized LinearWeights
• Generate first tokens debug logging: Added trace! logging for first 20 generated tokens

8. Config & CLI

• --vision-offload <auto|sequential|full-gpu|cpu> CLI flag
• --patches-per-batch <N> tune VRAM vs throughput (default 2)
• cpu_patches config option for hybrid mode tuning
• Generated config.toml includes vision_swap = true, patches_per_batch = 2, vision_offload = "auto"

---

Pros

1. Enables 4GB GPUs: DeepSeek‑OCR Q4K now runs on RTX 3050, GTX 1060, etc (~2.3GB peak VRAM)
2. Flexible strategy enum: Auto-detect, sequential hybrid, full-GPU, or CPU-only — one flag fits all scenarios
3. Hybrid parallel speed: Global CPU + patch CUDA in parallel gives the best perf for sub-6GB cards without throttling
4. Full GPU mode: When VRAM allows, keeps everything on CUDA for fastest vision prefill (~3.2GB)
5. Configurable batch size: --patches-per-batch lets users dial in VRAM vs throughput per card
6. Q2K/Q3K support: Further quantization options for aggressive memory reduction
7. Correct Q4_K round-trip: LM stays on CUDA during FullGpu, preventing all-zero decoder corruption
8. Auto-detection: nvidia-smi + env var fallback means zero-config for most users

Cons

1. UnsafeCell in model: language_model_mut() is unsafe — callers must guarantee no concurrent access; could be a footgun for future multi-threaded decode
2. Sequential swap is slower: Loading SAM/CLIP one at a time adds ~1-2s per image vs keeping both on GPU
3. Hybrid mode CPU cost: Global view on CPU can become a bottleneck on systems with weak CPUs
4. QMatMul device transfer hack: qmatmul_to_device reconstructs QTensor from raw bytes — brittle if upstream candle changes internal representation
5. nvidia-smi dependency: VRAM auto-detection spawns a subprocess; DEEPSEEK_OCR_VRAM_MB env var fallback is manual
6. Vision swap only for DeepSeek‑OCR: PaddleOCR‑VL and DotsOCR don't benefit; the added complexity is unused for other backends
7. Limited testing: Tested primarily on Q4K CUDA; FP16 vision, FullGpu, and non‑CUDA paths have less coverage