Repository navigation
Conversation
…8bit model Upstream main carried only the en→zh calibration, so --simultaneous with zh→en raised at engine construction (load_calibration is unconditional). This adds the two missing directions from the completed calibration runs: - zh→en: 1198 aligned pairs (ccmt), 8 heads, top (9,5) TS=0.794, 3/3 stability splits stable (max delta 0.0075), promotion gate passed. - ja→zh: 219 aligned pairs (ccmt + wikimatrix-ja-zh), 8 heads, top (9,5) TS=0.891, 3/3 splits stable (max delta 0.0244), promotion gate passed. Heads were detected on the PyTorch/transformers bf16 base (tencent/Hy-MT2-1.8B, eager attention) and verified to transfer to the 8bit MLX path: identical ranking, top-head decode-step span-share 0.80. The loader additionally requires the runtime provenance (pinned revision, quantization, alignment-source sha256) and per-check stable_vs_full flags; both are recorded. Prompt blocks match the resolved Hunyuan profile exactly for each direction (asserted by test). Note for review: with no calibration file for a direction, engine construction still raises. The previously-shipped behavior silently deactivated the simultaneous variant (translate-on-close) and logged a warning naming the missing tuple — worth deciding whether to restore that fallback; the AlignAtt4LLM reference implementation hard-fails (RuntimeError), which is what we originally moved away from.
… attention The ja→zh export came from a thin calibration corpus (219 used pairs) with no live validation — it returns when a real calibration backs it. The file tests were tautological: they proved the loader parses the JSON, not that the heads are correct. Now the bundled heads must reproduce the commit decisions the calibrated zh→en engine actually made, via a captured golden-attention fixture (3 informative apply_commit_policy calls from the deterministic fixture replay — raw per-head decode-step attention rows plus the recorded frontier and result). Sensitivity is proven at two tiers: swapping the top head must move the stabilized argmax trajectory, and swapping the top three must flip a recorded commit decision — the paper policy's head averaging is intentionally robust to single-head noise, so the decision-level control needs a multi-head corruption.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
calibrations: add the zh→en translation-head calibration for the Hunyuan 8bit model
Summary
Upstream ships one translation-head calibration for
mlx-community/Hy-MT2-1.8B-8bit: en→zh. The simultaneous-MT engine loads acalibration at construction and raises when the file for the requested
direction is missing, so
--simultaneouscurrently works for en→zh only.This PR adds the zh→en calibration file in the same format, produced with the
same procedure as the en→zh file: alignment-head detection over translation
datasets, translation-score (TS) ranking, stability checks, and a promotion
gate. The heads were detected with the PyTorch bf16 reference
implementation and transferred to the 8bit MLX checkpoint; the transfer was
verified (the 8 calibrated heads are the top-8 by TS score on the MLX
checkpoint, and the attention matrices agree with the PyTorch eager
baseline). The file records the full provenance: pinned model revision,
quantization, source hashes, per-check stability flags, and the promotion
gate result.
User impact
--simultaneouswith zh→en onHy-MT2-1.8B-8bitworks on a freshinstall. Without this file the engine raises at startup.
the engine, and the commit policy are unchanged.
A question for the maintainer (separate decision)
On
main, a missing calibration raises at engine construction. An earlierdesign used silent deactivation instead: log a warning naming the missing
(model, direction) tuple, disable the simultaneous path, and fall back to
the base translate-on-close behavior. The simultaneous AlignAtt paper
hard-fails in the same situation. Both are defensible; the question is which
the project wants. This PR ships files only and does not change the behavior
either way.
Calibration quality notes
0.79 on the PyTorch bf16 reference), the same primary head as the en→zh
and ja→zh calibrations, which supports these being general alignment
heads rather than direction-specific ones.
from the bundled file and must reproduce the recorded commit decisions of
the calibrated engine on real captured attention slices. A corrupted or
reordered head list fails the test (verified with top-1 and top-3
corruption).
Validation
tests/test_simul_calibration_files.py): the file loadsthrough
load_calibrationwith an exact prompt match; the engineconstructs for zh→en where it raised before (negative control: a missing
file still raises); the golden-attention reproduction test; the
corruption-detection test.
uv run pytest -q tests/ --ignore=tests/test_pipeline.py: same result asmainapart from the new tests.ruff check .: passes.Hy-MT2-1.8B-8bit, zh→en,simultaneous): drafts release during speech; draft coverage 0.83 over four
finals with the paper commit policy.
Checklist
relevant. (Will link the display/event-stream issue if the maintainer
prefers calibrations discussed there.)
documents provenance; no user-facing docs change.)
ruff check ..private data.