Skip to content

calibrations: add the zh→en translation-head calibration for the Hunyuan 8bit model - #448

Open
clkao wants to merge 2 commits into
QuentinFuxa:mainfrom
clkao:spacedock-ensign/calibration-zh-en
Open

clkao wants to merge 2 commits into
QuentinFuxa:mainfrom
clkao:spacedock-ensign/calibration-zh-en

Conversation

@clkao

@clkao clkao commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

calibrations: add the zh→en translation-head calibration for the Hunyuan 8bit model

Summary

Upstream ships one translation-head calibration for
mlx-community/Hy-MT2-1.8B-8bit: en→zh. The simultaneous-MT engine loads a
calibration at construction and raises when the file for the requested
direction is missing, so --simultaneous currently works for en→zh only.

This PR adds the zh→en calibration file in the same format, produced with the
same procedure as the en→zh file: alignment-head detection over translation
datasets, translation-score (TS) ranking, stability checks, and a promotion
gate. The heads were detected with the PyTorch bf16 reference
implementation and transferred to the 8bit MLX checkpoint; the transfer was
verified (the 8 calibrated heads are the top-8 by TS score on the MLX
checkpoint, and the attention matrices agree with the PyTorch eager
baseline). The file records the full provenance: pinned model revision,
quantization, source hashes, per-check stability flags, and the promotion
gate result.

User impact

  • --simultaneous with zh→en on Hy-MT2-1.8B-8bit works on a fresh
    install. Without this file the engine raises at startup.
  • No existing behavior changes. The en→zh file is untouched; the loader,
    the engine, and the commit policy are unchanged.

A question for the maintainer (separate decision)

On main, a missing calibration raises at engine construction. An earlier
design used silent deactivation instead: log a warning naming the missing
(model, direction) tuple, disable the simultaneous path, and fall back to
the base translate-on-close behavior. The simultaneous AlignAtt paper
hard-fails in the same situation. Both are defensible; the question is which
the project wants. This PR ships files only and does not change the behavior
either way.

Calibration quality notes

  • Top head: layer 9, head 5 (TS score 0.77 on the 8bit MLX checkpoint,
    0.79 on the PyTorch bf16 reference), the same primary head as the en→zh
    and ja→zh calibrations, which supports these being general alignment
    heads rather than direction-specific ones.
  • 8 heads total, matching the en→zh file's head count.
  • The test suite includes a golden-attention test: the heads are loaded
    from the bundled file and must reproduce the recorded commit decisions of
    the calibrated engine on real captured attention slices. A corrupted or
    reordered head list fails the test (verified with top-1 and top-3
    corruption).

Validation

  • New tests (tests/test_simul_calibration_files.py): the file loads
    through load_calibration with an exact prompt match; the engine
    constructs for zh→en where it raised before (negative control: a missing
    file still raises); the golden-attention reproduction test; the
    corruption-detection test.
  • uv run pytest -q tests/ --ignore=tests/test_pipeline.py: same result as
    main apart from the new tests.
  • ruff check .: passes.
  • Live smoke on Apple Silicon (Qwen3 ASR + Hy-MT2-1.8B-8bit, zh→en,
    simultaneous): drafts release during speech; draft coverage 0.83 over four
    finals with the paper commit policy.

Checklist

  • I searched for an existing issue or discussion and linked it when
    relevant. (Will link the display/event-stream issue if the maintainer
    prefers calibrations discussed there.)
  • I added or updated tests for behavior changes.
  • I updated documentation for user-facing changes. (The file itself
    documents provenance; no user-facing docs change.)
  • I ran ruff check ..
  • I ran the relevant pytest suite.
  • I did not commit credentials, model weights, generated caches, or
    private data.

…8bit model

Upstream main carried only the en→zh calibration, so --simultaneous with
zh→en raised at engine construction (load_calibration is unconditional).
This adds the two missing directions from the completed calibration runs:
- zh→en: 1198 aligned pairs (ccmt), 8 heads, top (9,5) TS=0.794, 3/3
  stability splits stable (max delta 0.0075), promotion gate passed.
- ja→zh: 219 aligned pairs (ccmt + wikimatrix-ja-zh), 8 heads, top (9,5)
  TS=0.891, 3/3 splits stable (max delta 0.0244), promotion gate passed.

Heads were detected on the PyTorch/transformers bf16 base
(tencent/Hy-MT2-1.8B, eager attention) and verified to transfer to the
8bit MLX path: identical ranking, top-head decode-step span-share 0.80.

The loader additionally requires the runtime provenance (pinned revision,
quantization, alignment-source sha256) and per-check stable_vs_full flags;
both are recorded. Prompt blocks match the resolved Hunyuan profile exactly
for each direction (asserted by test).

Note for review: with no calibration file for a direction, engine
construction still raises. The previously-shipped behavior silently
deactivated the simultaneous variant (translate-on-close) and logged a
warning naming the missing tuple — worth deciding whether to restore that
fallback; the AlignAtt4LLM reference implementation hard-fails
(RuntimeError), which is what we originally moved away from.
… attention

The ja→zh export came from a thin calibration corpus (219 used pairs) with
no live validation — it returns when a real calibration backs it.

The file tests were tautological: they proved the loader parses the JSON,
not that the heads are correct. Now the bundled heads must reproduce the
commit decisions the calibrated zh→en engine actually made, via a captured
golden-attention fixture (3 informative apply_commit_policy calls from the
deterministic fixture replay — raw per-head decode-step attention rows plus
the recorded frontier and result). Sensitivity is proven at two tiers:
swapping the top head must move the stabilized argmax trajectory, and
swapping the top three must flip a recorded commit decision — the paper
policy's head averaging is intentionally robust to single-head noise, so
the decision-level control needs a multi-head corruption.
@clkao
clkao requested a review from QuentinFuxa as a code owner September 10, 2026 20:59

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant