Skip to content

fusion-model-choice.md

Latest commit

 

History

History
39 lines (25 loc) · 2.71 KB

fusion-model-choice.md

File metadata and controls

39 lines (25 loc) · 2.71 KB
Raw

Fusion model & prompt — production decision

Decision (2026-05-05): KIMI_FUSION_MODEL=kimi-k2.6, FUSION_PROMPT_VARIANT=legacy.

This is what the live demo runs on, and what .env.example ships with.

Why we did not run a formal bench to make this call

Phase 4.1.F shipped a reproducible bench harness (scripts/phase4_bench.py) covering the full (model × prompt_variant) × clusters matrix. We chose not to spend hackathon-grade gateway budget on a live run because the decision is already settled by upstream evidence:

  • Latency is no longer a model-selection criterion. The warm-caching strategy (see TODO.md § "Demo strategy — warm caching") collapses live demo wall-time to disk-read seconds regardless of cold-call latency. Selecting a faster model would buy nothing the cache does not already buy.
  • Quality wins, k2.6 stays the default. The empirical profile in project_kimi_proxy_latency.md (auto-memory; informal Phase 4.1 measurements across 10+ live tests) shows kimi-k2 collapses Fusion latency to ~9 s but misses category-conflict detection, drops 1–2 attribute keys from the merged record, and parse-fails on ~33 % of identical prompts. None of those is acceptable against the Fusion contract documented in hackathon-brief.md.
  • No measurement has shown a strict-dominant alternative on quality to justify spending a bench cycle to confirm a foregone conclusion.

The bench harness is kept as-is for future re-decision; we just don't pay for it now.

How to re-challenge this decision later

Run the harness on five contrasted zones (Saint-Aignan, Orléans, Bordeaux, a coastal commune, a large-EPCI candidate):

uv run python scripts/phase4_bench.py orleans bordeaux saint-aignan sete lyon \
    --models kimi-k2.6 kimi-k2 \
    --variants legacy mechanical \
    --clusters-per-zone 6

CSV output lands in data/bench/fusion-<YYYYMMDD-HHMM>/. Read parse_ok, conflict-category coverage, INSEE precedence inversions, and merged-record key-count delta against the k2.6 baseline.

Conditions that would flip the default

Switch only if a (model, variant) pair is strictly dominant on all four quality metrics defined for Phase 4.1.G:

  1. parse_ok ≥ 95 % post-retry (so end-to-end ≥ 99 %).
  2. 100 % conflict-category detection on cross-source clusters that carry one.
  3. 0 inversions of INSEE > Wikidata > Atout-France precedence on synthetic capacity fixtures.
  4. Merged-record key-count parity with the k2.6 baseline within ±1 key.

A win on three of four is not enough — Fusion's contract is non-negotiable on conflict detection and on the precedence rule.