Repository navigation
Conversation
Caption rendering logic previously had to be re-derived by each consumer from pipeline state fields. This adds one display model that consumes a standardized caption event stream, plus the first view over it. - caption_events.py: four event types (transcription/translation x provisional/final), each carrying the text, the committed and full source snapshots it was produced against, and whether a translation model call produced it. An EventTap fans out to sinks; EventLog records the stream as JSONL for replay and testing. - overlay_model.py: the display rules. Captions are partitioned at sentence terminators (Latin .!? and CJK marks); a completed sentence is never extended by the next sentence's words. Completed sentences queue and each holds for a minimum time; a new draft supersedes only other drafts, never queued committed sentences. The sentence the viewer is watching flips to final style in place; shown text never visibly retracts; reworded drafts are held until a commit resolves them. - overlay.py: the overlay view over the model (composable with other views via MultiRenderer). - display_adapter.py / src_buffer.py / event_diff.py / overlay_events.py: the event-to-display plumbing. - tests + five recorded event streams (tests/golden/): every display rule has a failing counterpart, and recorded streams replay through the real model to assert the reader-visible sequence.
The unified display model needs production emission: the QuentinFuxa#436 split moved the translation loop to translation_processor.py without event call sites, and the ASR-side emission points needed verification. - audio_processor: attach EventTap + DisplayAdapter (optional EventLog via the event_log arg) in __init__; emit transcription_final/provisional at the ASR commit block, the streaming-buffer timeout refresh, and the terminal finish() flush; dedupe consecutive identical ASR provisionals. - translation_processor: run_translation takes the tap and emits translation_final (single or list) and translation_provisional with the display-correctness rules preserved: only when the simul layer is active (the base path's buffer is untranslated source and must never be flashed as an MT draft), deduped per draft text, carrying the committed/source snapshots and the fresh flag (MT call counter advanced). - test_harness: TestState.display carries the event-derived DisplayState snapshot; _collect_results attaches it so views render from the stream. - tests/test_translation_emission.py: fixture-level contract tests for all four emission rules (no live model). - test fixture completeness: the object.__new__-built AudioProcessor fixtures in test_asr_coalescing_boundaries/test_pause_segmentation/ test_backends now set event_tap/_last_asr_prov/_last_mt_prov/ processing_error/sep, mirroring what production __init__ sets.
…s contract The emission block reached into the simul engine's private attributes (_simul_active, _mt_call_count, _committed_text, _source_text) and swallowed read errors with silent empty strings — a coupling the QuentinFuxa#436 translation/display split exists to prevent, and a failure-visibility regression against the QuentinFuxa#440/QuentinFuxa#441 direction. - TranslationProgress dataclass (timed_objects.py): mt_call_count, committed_text, source_text, source_end — the live source-progress state the caption event stream needs. - Capability is a class attribute, matching upstream's class-based backend routing: MlxLlmTranslation.provides_drafts = False (base buffer holds untranslated source and must never flash as a draft); MlxLlmTranslationSimul.provides_drafts = True. - progress() on the base returns the honest empty default; the simul engine overrides it as a pure re-expose of its existing internal state (no renames). - run_translation reads translation state only through a _ProgressReader: public contract attributes with documented defaults for third-party backends (NLLB, the AlignAtt sidecar), and progress() failures become one visible warning per session plus a session-scoped draft-display disable — never a silent empty string. - the simul engine's internals are unchanged: process() semantics, commit modes, and the paper commit policy are untouched; the contract is a pure re-expose.
…iene The JSONL caption event log was only reachable through a dev driver. The flag moves to the server CLI: parse_args defines --event-log, the config carries it, and the AudioProcessor tap (which already read args.event_log) picks it up from the server config path. Also: reword the remaining references to views outside this PR's scope, and finish import ordering so the branch passes ruff clean (origin/main passes ruff clean; the added files must not regress that).
The PR carries the caption semantics, not the clients that render them: the AppKit overlay view, the terminal renderer, and the source reading buffer move to the client branch stacked on this one. The replay instrument now drives the display model directly (no view import) and traces the committed caption line as the hold-drain releases sentences. The stream-comparison assertions are inlined in their test, dropping the event_diff helper and the unused overlay_events module.
…e accumulator OverlayDisplayModel/overlay_model.py named a view-agnostic component after a view, and 'model' collides with the repo's ML-model meaning (model_mapping.py, model_paths.py). The module is now caption_display.py with class CaptionDisplay; the stream-level line reducer (display_adapter.DisplayAdapter) moves into the same module as CaptionLineAccumulator with state CaptionLineState, removing the duplicate DisplayState name. Threading contract documented: events are forwarded synchronously on the event loop thread; both classes are single-threaded from that thread, and views marshal state across to their own threads. Docstrings de-scoped from the removed overlay view. No behavior change: suites return the same pass/fail sets as before the rename.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
feat: caption display semantics and a standardized caption event stream
Summary
The web UI renders the translation buffer as it comes from the MT layer. For
backends that do not produce provisional translations, that buffer holds
untranslated source text, so raw source shows up in the translation line.
Translations arrive as strings managed inside the MT layer: a correction
replaces text the reader already saw, a multi-sentence translation is
replaced wholesale by the next update, and sentence ends are nowhere in the
data. Display behavior also has no tests, because nothing is recorded.
This PR moves sentence derivation and display rules into a layer separate
from the MT layer:
caption_events.pyemits four typed events (transcription andtranslation, provisional and final) from a tap in
AudioProcessor, eachcarrying the text and the source it was produced against.
--event-logrecords the stream as JSONL for replay and testing.
caption_display.pyturns the stream into a display state: captions arederived per sentence, a finished sentence is held and always shown, a
sentence flips to final style in place, and final text never visibly
shrinks or rewords.
provides_draftsand aprogress()method.Backends without drafts (NLLB, the AlignAtt sidecar) need no changes; the
draft path is unreachable for them.
It does not wire the current web UI yet; that is the follow-up.
User impact
results_formatterare unchanged.--event-logis new, off by default, and writes one JSONL line per event.process()semanticsare unchanged.
Validation
test_caption_display.py: each display rule has a test that fails whenthe rule is broken.
test_caption_events.pyandtest_translation_emission.py: stream andemission contracts, no live model needed.
(
tests/golden/zh_long_time_frontier.jsonl) replayed throughCaptionDisplayshows every finished sentence in order, each shown once.uv run pytest -q tests/ --ignore=tests/test_pipeline.py: same result asmainapart from the new tests.Hy-MT2-1.8B-8bit, zh→en): therecorded log replays through
CaptionDisplay.ruff check .passes; it also passes onmain.Checklist
--event-logwill be added.)ruff check ..private data.