Skip to content

Dictation: speak into the terminal, transcribed on your own machine (#13) - #22

Merged
Maxaubert merged 12 commits into
mainfrom
feat/13-dictation
Sep 19, 2026
Merged

Maxaubert merged 12 commits into
mainfrom
feat/13-dictation

Conversation

@Maxaubert

@Maxaubert Maxaubert commented Sep 19, 2026 •

Copy link
Copy Markdown
Owner

Closes #13 for Prism Terminal. Built once in core/, so Prism gets it next (its own PR, after asking, per the standing rule).

Spec and plan: docs/superpowers/specs/2026-09-19-dictation-design.md, docs/superpowers/plans/2026-09-19-dictation.md (approved 2026-09-19).

What it is

Turn it on in Settings > Dictation, hold Right Alt, speak, let go: the words are pasted at the cursor of the shell in front. It never presses Enter. A pill over the terminal shows a live level meter and the text as you speak; the tab wears a mic mark; a short cue sounds at start and stop. Escape cancels. Off by default.

  • Local. whisper.cpp's official Windows build, bundled (8 MB zip, 11 MB unpacked). Audio goes to a server on 127.0.0.1 and is never written to disk.
  • Models (Base recommended on CPU, Small, Large v3 Turbo recommended with a GPU, Large v3) and the optional NVIDIA pack are downloaded on a click into %LOCALAPPDATA%\PrismDictation, shared with Prism, each against a SHA-256 pinned in the catalog. Model urls are pinned to a Hugging Face commit. An unverified file is deleted, never loaded.
  • Right Alt is AltGr on Norwegian and most European keyboards, so the key is read as a 200 ms SOLO hold; any other key in that window means it was typing. Rebindable; hold or toggle.
  • Pause media asks Windows which sessions are Playing, pauses those and resumes exactly those. Not the play/pause key, which would start music that was off.
  • Off means off: no listener armed, no process, no download. Switching it off kills the server.

Found on the way

  • The engine needs Microsoft's C++ runtime (MSVCP140, VCRUNTIME140, VCRUNTIME140_1, VCOMP140; read off the import tables) and neither official zip ships it, so on a fresh Windows it would die at launch without a word. core/tools/fetch-whisper.mjs now copies the four DLLs app-local from the build machine, only if validly signed by Microsoft, plus the MIT notice the zip also lacks. The GPU pack finds them through PATH.
  • The installer had grown 73 MB: the e2e's cached test model was being swept into the asar. Excluded, with the core's source and vendor/. Installer: 113 MB (was 110).
  • The page may now ask Chromium for the microphone, so what it may ask for is a written closed list: audio capture for this window, the clipboard, fullscreen. Never the camera.

Measured

CPU, Base, resident server: about 0.9 s a pass whatever the clip length. RTX 5090 with the official CUDA 12.4 pack: it runs (first run 9 s compiling kernels), then Large v3 answers in 0.36 to 0.5 s.

Tests

  • 577 unit tests (369 before): the hotkey reducer incl. the exact Norwegian AltGr sequence, the cleaner (never emits a newline, as a property), WAV, the catalog, the store against a local HTTP fixture (checksum mismatch, cancel, disk, unpack), the engine with a fake server (GPU fallback, serialised passes, idle stand-down), the media helper protocol, option parity.
  • e2e dictation, real end to end: Chromium's fake microphone plays a known clip, the real bundled engine hears it with the Tiny model, and "ask not what your country can do for you" must arrive on the prompt line with no new prompt after it. Also: off does nothing and runs nothing, AltGr typing never opens the mic, the meter moves, live text appears, the pill does not move the terminal's rows, Escape pastes nothing, the server dies with the switch and with the app. 20 checks.
  • The same scenario passes against the packaged app (PT_E2E_PACKAGED=1), which finds the engine where the installer puts it.
  • All 15 scenarios, typecheck, lint: green. Installed locally: 0.2.0, exe timestamp 17:41 -> 19:41.

After the owner's first hands-on (same evening)

"It works perfectly", then notes on the model manager, all applied: models carry their full names ("Whisper Base"); every row leads with its vendor's mark (OpenAI, NVIDIA; simple-icons' CC0 outlines, drawn in the row's own ink so they wear the theme); "Delete" is ✕ Uninstall and is a button like Download; every Download button is the same (the badge is what recommends); GPU acceleration is Enable / Disable and nothing else (enable downloads it, disable uninstalls it; a separate on/off beside an Uninstall was built first and removed); the row in use wears a 3.5% tint instead of the hover grey. New e2e dictationPage stages an installed model and GPU engine (sparse file, marker) and asserts all of it. Also fixed a race in the links e2e. 16 scenarios green; installed 20:15.

Prism's half is Maxaubert/Prism#162 (its own PR), pinned to core-v0.2.0-rc.5 until this merges.

Only the owner can check

How the cues and the pill feel; Right Alt and AltGr on the physical Norwegian keyboard; pause-media against a real player; downloading a model and the GPU pack from the real network; accuracy with Large v3 on the 5090.

Not in v1

AMD/Intel GPU acceleration (no official build), Parakeet, voice commands, a global hotkey, custom vocabulary, English-only models, auto-stop on silence.

🤖 Generated with Claude Code

Maxaubert and others added 5 commits September 19, 2026 19:15
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Hold Right Alt, speak, release: the words are pasted at the cursor, never sent.
whisper.cpp's official CPU engine is bundled (with Microsoft's C++ runtime and
the MIT notice beside it), models and the NVIDIA pack are downloaded on request
into a folder both apps share, each against a pinned SHA-256. Off by default.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…n drive the packaged app (#13)

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Maxaubert and others added 5 commits September 19, 2026 20:02
…marks, full names, Uninstall, Enable / Disable (#13)

Owner's notes of 2026-09-19. Disabling GPU acceleration is a per-app setting
that stops THIS app using the engine and keeps the 675 MB on disk; Uninstall is
what frees it. Also fixes a race in the links e2e (it could read the span in the
frame before the link decoration was repainted).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…is what recommends (#13)

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…: enable downloads it, disable uninstalls it (#13)

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…e ground (#13)

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Maxaubert and others added 2 commits September 19, 2026 20:51
…, as every CI runner starts it (#13)

Windows PowerShell 5.1 inherits pwsh's PSModulePath and cannot load
Get-AuthenticodeSignature. Found by running Prism's terminal gate on a GitHub
runner for the first time; it would have failed both apps' release builds.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@Maxaubert
Maxaubert merged commit 9614121 into main Sep 19, 2026
1 check passed
@Maxaubert
Maxaubert deleted the feat/13-dictation branch September 19, 2026 20:00
github-actions Bot pushed a commit that referenced this pull request Sep 19, 2026
) (#22)

* docs(dictation): design spec and implementation plan (#13)

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* feat(dictation): the shared contract (#13)

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* feat(dictation): local speech to text, built once in the core (#13)

Hold Right Alt, speak, release: the words are pasted at the cursor, never sent.
whisper.cpp's official CPU engine is bundled (with Microsoft's C++ runtime and
the MIT notice beside it), models and the NVIDIA pack are downloaded on request
into a folder both apps share, each against a pinned SHA-256. Off by default.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* feat(dictation): docs, the real end-to-end scenario, 0.2.0 (#13)

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* build(dictation): keep the e2e cache out of the installer; the e2e can drive the packaged app (#13)

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* feat(dictation): the model manager reads as a list of things: vendor marks, full names, Uninstall, Enable / Disable (#13)

Owner's notes of 2026-09-19. Disabling GPU acceleration is a per-app setting
that stops THIS app using the engine and keeps the 675 MB on disk; Uninstall is
what frees it. Also fixes a race in the links e2e (it could read the span in the
frame before the link decoration was repainted).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* fix(dictation): every model's Download button is the same; the badge is what recommends (#13)

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* fix(dictation): GPU acceleration is Enable / Disable and nothing else: enable downloads it, disable uninstalls it (#13)

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* fix(dictation): Uninstall is a button, the same one Download is (#13)

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* fix(dictation): the row in use wears a whisper of a fill, close to the ground (#13)

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* fix(dictation): the engine fetch works when started from PowerShell 7, as every CI runner starts it (#13)

Windows PowerShell 5.1 inherits pwsh's PSModulePath and cannot load
Get-AuthenticodeSignature. Found by running Prism's terminal gate on a GitHub
runner for the first time; it would have failed both apps' release builds.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* docs(core): the core is released by a workflow now, not by hand (#23)

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Maxaubert added a commit that referenced this pull request Sep 25, 2026
…2e harness (#71) (#72)

Settings: the colour editor saves the whole setup into Custom and is a theme
pick (#8, #29); ThemeSwitchAsk registers once (#22); a hex field commits only
a changed draft (#26); the dictation key is never a key that types or a chord
the terminal owns, and a press elsewhere ends the capture (#27); the editor
takes the focus, traps Tab and hears Escape, the pencil is its own button
(#28); a finished download reads fresh state before picking the model (#30).

Release and CI: release.yml tags the built commit (#11); core-release runs
from main only (#34), waits for terminal-gate by name and its success (#38),
and opens an issue when a core change cannot be released; core-version also
requires the core version above the base's (#39).

Harness: a time limit per scenario and closeApp (#35), profiles removed after
each scenario (#36), the poll's first verdict awaited instead of 6 s (#37).

Core 0.15.3, app 0.18.3. New reviewSettings e2e and unit tests.


Claude-Session: https://claude.ai/code/session_01LJbePcRzre7AusNzPNS2Bk

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
github-actions Bot pushed a commit that referenced this pull request Sep 25, 2026
…2e harness (#71) (#72)

Settings: the colour editor saves the whole setup into Custom and is a theme
pick (#8, #29); ThemeSwitchAsk registers once (#22); a hex field commits only
a changed draft (#26); the dictation key is never a key that types or a chord
the terminal owns, and a press elsewhere ends the capture (#27); the editor
takes the focus, traps Tab and hears Escape, the pencil is its own button
(#28); a finished download reads fresh state before picking the model (#30).

Release and CI: release.yml tags the built commit (#11); core-release runs
from main only (#34), waits for terminal-gate by name and its success (#38),
and opens an issue when a core change cannot be released; core-version also
requires the core version above the base's (#39).

Harness: a time limit per scenario and closeApp (#35), profiles removed after
each scenario (#36), the poll's first verdict awaited instead of 6 s (#37).

Core 0.15.3, app 0.18.3. New reviewSettings e2e and unit tests.


Claude-Session: https://claude.ai/code/session_01LJbePcRzre7AusNzPNS2Bk

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Dictation when using the agents

1 participant