Froggy

A personal, voice-first control plane for your computer: local dictation, commands, a keyboard launcher, and optional agent integrations. Fork or copy it and adapt it to the way you work. The macOS app is the working daily driver; native iOS keyboard and Android IME prototypes exercise each mobile platform's safe insertion path.

This is not a Yap fork and it has no compatibility layer. The current design optimizes for one Apple-silicon Mac and conservative Accessibility-based insertion.

Product direction

This README documents what Froggy does today. MANIFEST.md defines the longer-term product direction: Froggy as the top-level command router for applications, agents, and computers. The implementation boundaries and staged migration are documented in docs/ARCHITECTURE.md and docs/ROADMAP.md.

What works

The first vertical slice is implemented:

1. AVAudioEngine captures the raw microphone while listening is enabled. The experimental Echo Cancellation toggle keeps a private, unmuted Core Audio output tap running, including while the microphone is off, with 200 ms of speaker reference and 600 ms of level history in memory. Quiet starts have no added observation or calibration wait; missing reference alone never delays activation. Only detected recent speaker output triggers Preparing, while the isolated adaptive canceller runs before transcription. The indicator becomes ready when suppression validates or after at most one second of warmup, whichever comes first. The canceller adapts during playback and for a 600 ms echo tail after playback stops, then bypasses DSP and passes microphone samples through unchanged. The output monitor and learned filter remain available so cancellation resumes automatically when playback returns. A weak suppression measurement never shuts off the microphone or repeatedly reopens the warmup gate. Every mic activation reuses learned filter coefficients. Mic off stops microphone I/O and adaptive processing; turning Echo Cancellation off also removes the speaker monitor and retained filter and restores direct raw capture.
2. FluidAudio resamples to 16 kHz and Silero VAD segments utterances locally. Endpointing is transcript-aware: an utterance whose newest streaming partial ends a sentence closes after 0.25 s of silence, while mid-thought speech is held open for 0.15 s more before closing at the original VAD boundary; resumed speech cancels the held endpoint.
3. FluidAudio's Parakeet Unified EN 0.6B transcribes eagerly while speech is still arriving, using the int8 640 ms streaming tier. The offline-batch encoder is retained as a failure fallback.
4. A small deterministic pass handles punctuation and common developer spellings.
5. Eligible non-empty dictation is deterministically normalized and pasted directly into the same application that was focused when speech began.
6. Before pasting, Froggy reads the text on either side of the caret (AXSelectedTextRange plus AXStringForRange, or a slice of AXValue) and attaches the phrase to it: a space when a word or closing punctuation precedes the caret and the phrase starts with one, nothing after whitespace or an opening bracket, a trailing space when the paste lands in front of a word, and the first letter capitalized after a sentence end. A period is added only with positive evidence of a sentence boundary: the text before the caret is Froggy's own previous insertion (so the pause is known), that insertion ended mid-sentence, the speaker paused at least 1.5 s between utterances, and the new phrase does not begin with a continuation word such as and, but, or which. Every ambiguous case gets a plain space. When an app hides its selection (some Electron editors), Froggy's last insertion stands in for the seam only if no key or click has happened since; otherwise the phrase is pasted verbatim. Escaped (literal) utterances are never joined. Each decision is logged under insertion as Seam: …, and Fn-Z undo removes the separators with the text.
7. Three dedicated built-in keys separate intent: F5 starts ambient listening when the mic is off and stops every kind of listening when it is on, Caps Lock opens typed commands, and Fn controls speech.
8. A double tongue click presses Return in the focused text input (see below).

Audio and ordinary dictation stay local by default. The optional OpenAI live transcription setting sends eligible speech to OpenAI as described below. The independently enabled Speech coach sends ambient audio windows for analysis. A fix sends its text context to OpenAI when a Platform API key is available; model downloads require a network connection on first use.

Optional speech coach

Open the GUI (Caps Lock) → Speech coach command, then turn on Enable speech coach for experimental ambient filler feedback. The pane contains the enable toggle, optional coaching guidance, status, and recent feedback. Escape or Back returns to GUI search; leaving the pane does not stop coaching, and opening it does not enable listening. The coach keeps listening across apps, independently of command attention and composer. F5 or “Froggy off” stops both pipelines until explicit activation; disabling only the coach leaves other capture owners intact. Ordinary transcription, commands, and insertion keep their existing path.

Speech coach has two independent feedback paths. A coach-only OpenAI gpt-live-transcribe session streams verbatim transcript deltas to the realtime word monitor (default: “um” and “uh”). New detections slide in from the right; partial words wait for a word boundary, and transcript corrections reconcile visible occurrences. Word detections expire after five seconds by default. This transcript never replaces Froggy's clean command/dictation transcript. Filler recall and actual detection latency still require representative live QA.

Every five seconds, the separate gpt-audio-1.5 analyzer receives that target interval alone as audio, with transcripts from the preceding 25 seconds as text context. Earlier audio is never replayed into a new count. Counts and new advice apply only to the newest interval; filler-like counts must be supported by that interval's verbatim transcript. Audio lets pausing and vocal-variation suggestions use delivery evidence. One request runs at a time; slow requests skip coverage rather than accumulating a backlog, and results more than ten seconds behind the target audio are discarded. Near-silent targets skip analysis. Transient context contains only successfully analyzed target transcripts.

The HUD is overlay text with subtle scrims, separate from the menu bar item and transcript overlay. Each row fits its own contents and shares a right edge with the other rows. The window follows the widest visible row, pinned 20 points from the screen's right edge. The idle mode header stays compact; long suggestions wrap within a width limit, and tall stacks scroll. Its realtime word row sits above factual notes such as “Like” ×3 / 30s and a stack of suggestions. Counts are estimates from distinct five-second targets; partial coverage is identified. The count row disappears 15 seconds after the latest positive audio window by default; silence cannot renew it. An independent clock expires feedback even when capture stops delivering buffers. Every distinct returned suggestion gets a clickable toast, newest first, expiring after 15 seconds by default. Hovering pauses that toast's timer. Repeated evidence is suppressed for 30 seconds; excess rows scroll rather than being dropped. Clicking opens the Speech coach GUI to the original evidence, proposed wording, contextual reasoning, and word definitions. That detail view remains fixed while live feedback continues.

The Passive / Active selector in GUI → Speech coach chooses a saved profile. Passive is the default: word monitoring, filler-like counts, and diction suggestions. Active additionally enables pausing and vocal variation. Each mode has its own word list and category toggles: add “exactly” to Passive's monitored words to flag every occurrence there, independently of Active. Click the HUD's Passive / Active label to toggle modes directly. The header shows the mode and realtime word detections; detailed listening status is available in the coach pane and the waveform's tooltip. Diction prompts require a more precise or contextually better replacement, never variation simply for novelty. Diction evidence must quote the target transcript; suggestions that insert filler words are discarded, and labels are derived from the replacement and original fields. Configure monitored words, categories, optional guidance, and display lifetimes in GUI → Speech coach. Recent suggestions are retained in memory (up to 100) for inspection, with no transcript/audio archive.

Show heads-up display defaults on when coaching is enabled. Drag the waveform handle to adjust its height or move it to another display; it snaps to that display’s right edge. The display and height are remembered independently of the menu bar item and the transcript overlay. The overlay does not take keyboard focus; clicking a suggestion explicitly opens the GUI. Hide it with × or the coach pane's toggle without stopping coaching. F5 / “Froggy off” stops both audio paths; disabling coaching hides the HUD.

This is a separate opt-in upload policy: unaddressed ambient audio can leave the Mac while coaching is enabled. Inert Mode and detected secure input suppress uploads, clear unsent audio/context, and cancel requests. Neither path saves recordings or adds them to the transcript corpus. The realtime socket has a bounded upload queue and reconnects after errors; ordinary dictation remains independent. Fixed boundaries can split words, and the streaming provider lacks word timestamps and speaker identification. Use one close microphone and speaker for this experimental version; microphone audio may include other call participants.

The analyzer and configuration interface are independent of any general agent; Projector integration is deferred. See evaluation and live QA.

Optional OpenAI live transcription

Enable Prefer OpenAI transcription in the menu bar menu, or Settings → Speech model → OpenAI live transcription, to try gpt-live-transcribe. It is off by default and shares the Platform API key in Settings → OpenAI with Fix.

• Only utterances that begin while command attention is armed or the foreground app is in compose mode can upload. Local VAD gates audio; ordinary silence and unaddressed background speech stay local. Eligible speech includes the short VAD pre-roll needed to preserve its onset.
• The first “Froggy” utterance while disarmed stays local. Fn tap arms cloud eligibility; Fn hold alone does not. Eligibility is checked before each upload; attention expiry, leaving the captured app, or ending composer stops further upload. Turning the toggle off cancels pending cloud results immediately.
• Parakeet keeps providing live preview and endpointing. The OpenAI final transcript goes through the usual normalization, command routing, target checks, and single insertion. No automatic Luna cleanup is added.
• Missing keys, API failures, empty cloud results, and slow requests fall back to local recognition. Connection setup is bounded at three seconds and the final result at two seconds after endpoint. Each utterance uses its own socket; there is no continuously uploaded session and no Froggy backend to deploy.
• Diagnostics identify the final recognizer and any fallback. Saved corpus records identify cloud transcripts correctly. This initial experiment sends audio with English/low-delay hints, without editor text or screenshots.

The optional adapter is isolated from the local recognizer and command runtime.

Mobile prototypes

apps/ios contains a buildable containing app and keyboard extension. apps/android contains a buildable settings app and proper InputMethodService, without overlay, Accessibility Service, or INTERNET permission. Both open a real microphone path, discard audio, use an explicitly labeled canned transcript, revalidate the captured editor, and attempt final insertion at most once. Local ASR is intentionally the next slice; current setup and physical-device test instructions live in each app's README.

The two native implementations consume the same schema, transaction table, canonical examples, and normalization goldens in packages/dictation-contracts. They do not share an application runtime.

Deliberate constraints

• Apple silicon and macOS 15 (Sequoia) or newer.
• A source-independent command bus accepts speech, typed input, and resolved gestures, then schedules target-bound execution stacks. Routing is deterministic today; open-ended Fix commands may use the configured model-backed engine stack.
• The application and focused control are captured when speech begins. Froggy revalidates that exact Accessibility element before each effect and can finish a target-bound stack after the app moves to the background. Froggy itself is always blocked.
• Chromium and Electron Accessibility trees are activated once per process when detected. Chrome never receives automatic focus changes. Read-only value and insertion-seam probes never move focus. T3 Code is the sole app-specific adapter: the actual insertion effect may focus its composer, places the caret at the end when it does, and retains its legacy insertion fallback when Electron still reports the focused prompt as an AXGroup.
• Secure text fields are always blocked, and so is any target while macOS Secure Event Input is on (a focused password field, or Terminal with Secure Keyboard Entry). Froggy checks the flag before typing and reports it instead of losing a paste.
• If the captured app/control disappears, changes, or becomes unsafe before an effect, its stack is halted.
• Hosted transcription is optional and off by default. There is no analytics, public update service, onboarding, or compatibility promise. Final text transcripts and target-app metadata are saved locally automatically; saving utterance audio is optional and off by default.

Build and run

Source: endograph/froggy.

The optional maintainer tooling uses buildmac and targetmac as example SSH aliases. Configure them for your machines before using that tooling.

Prerequisites are Xcode 26.6+ and its command-line tools. Endofrog, SSH access, and a paid signing account are not required for local builds.

make mac-test
make mac-run

make mac-run creates and opens apps/mac/.build/Froggy.app. Froggy immediately begins prewarming Silero and Parakeet so the hold shortcut is ready without a first-utterance model delay. The microphone itself stays off until listening is activated. The first activation will:

1. request microphone access;
2. request System Audio Recording access when experimental Echo Cancellation is enabled;
3. request Accessibility access, used only to inspect the focused element and synthesize paste;
4. finish downloading or compiling the local model assets if launch-time prewarming is still in progress.

After granting Accessibility access, stop and relaunch the app if macOS does not recognize it immediately. The build script uses a single available Apple Development signing identity, or FROGGY_CODESIGN_IDENTITY when specified. Without one, it uses ad-hoc signing; code changes may then require removing and re-adding Froggy under System Settings → Privacy & Security → Accessibility.

Froggy starts in the menu bar with the microphone off. The default controls are:

Control	Action
Caps Lock	Open or dismiss command search
Hold Fn	Dictate into the focused input; release to finish
Tap Fn	Arm a spoken command
F5	Start ambient listening, or stop capture when the microphone is active
Escape	Go back from a launcher page, or close it at the root

Use the menu bar item's Settings button to change keys and preferences. Agents and an OpenAI API key are optional for local dictation and native commands. Settings explain the empty agent state and how to assign a Primary agent key. The menu's Accessibility… action opens the Accessibility permission flow. Display controls are always expanded in the menu and remain available in Settings. The menu has no ambient-listening, undo, or Inert Mode controls. Use Settings → Hotkeys → Undo last insertion → + to assign an optional key or chord, globally or for one keyboard. It starts unbound, supports multiple bindings, and uses the same target-checked undo as Fn-Z. Fn-Z remains available. Settings also explain tap versus hold for the Speech key.

Source builds have no update server and make no automatic update requests. The menu explains that updating requires rebuilding. To use your own SSH update service, set both FROGGY_UPDATE_HOST and FROGGY_UPDATE_EXECUTABLE when building. The executable is an absolute path on that host and accepts check or install, using the JSON release protocol implemented by the optional Endofrog tooling. These values are embedded in the bundle: do not set a private endpoint on a binary intended for public distribution. Existing maintainer deployment must supply this configuration explicitly.

Bundled Git metadata is limited to a short build stamp; the app also records build time. It no longer packages hostname, author names, branch, dirty paths, or commit history. This does not scrub compiler paths or symbols from binaries. For a fresh source snapshot, use python3 distribution/export-source.py /tmp/froggy-source.tar.gz --ref HEAD from the reviewed commit. The exporter substitutes the public AGENTS.md and honors .gitattributes exclusions for generated internal reviews. See source distribution for the instruction drift check and export details. This is not a privacy audit.

Optional maintainer deployment with Endofrog

The included Endofrog configuration is the original maintainer's automation setup. Forks can use the local commands above without installing it. To adopt Endofrog, configure your own Endograph runtime, SSH destination, and signing setup; the example host alias is not a public service.

In the maintainer's setup, debug updates are deployed by Endofrog to targetmac. Each deploy waits for the old Froggy process to exit, installs the new bundle, launches a fresh instance, and verifies it is running. No manual launch should be needed; a failed restart is reported with its reason.

Endofrog's runtime uses Codex app-server with gpt-5.6-luna and the service user's ChatGPT login, sharing that account's Codex usage allowance. Its host actions and offline worker sandbox still govern builds and installations. Inception uses a separately configured coding agent.

Endofrog builds and signs without running tests, and separately owns the test suite: coding agents run only the target they touched and ask Endofrog for full runs, test advice, and pruning. Its persistent SwiftPM cache and bundle staging live in apps/mac/.build-endofrog/ in each worktree, separate from peer builds and tests in apps/mac/.build/, so their build locks cannot block deployment. The first build in this separate cache takes longer.

Using the Mac app

Say “Froggy settings” or “Froggy open settings”, or choose Froggy Settings in the GUI, to open the existing settings panel. Repeating the command brings the same window forward.

Froggy owns three independently remappable hotkeys, defaulting to F5, Caps Lock, and Fn. Keyboard settings also offer an optional Primary agent binding with no default key: it opens the configured default agent (or the first registered agent when no valid choice is saved) directly in the agent GUI, including when the GUI is hidden. Pressing it while that agent side already has keyboard focus dismisses the GUI; otherwise it switches to or focuses that agent. Use its + button to assign a key or chord globally or for the selected keyboard. Settings can record a single key or a simultaneous modifier chord for ambient mode, typed commands, or speech. For example, hold Control and Option, then press Space to record Control + Option + Space. A modifier pressed alone is saved on release, so the recorder does not prematurely save the first part of a chord. Modifiers must match exactly; Control + Space and Control + Shift + Space are distinct bindings. Speech starts on the final key-down and a hold ends on that key’s release, even if modifiers are released first. Sequential key combinations are not supported. The Keyboard selector starts on Default, which applies to keyboards without a saved override, including unidentified event sources. Select a connected keyboard and change a key to save its own three hotkeys; Use Default removes that override. Saved keyboards remain listed while disconnected and their mappings return on reconnect. Each action shows its bindings as matching buttons on one row, followed by a small + to add another without replacing the existing key. Click a binding to change it; right-click an additional binding to remove it. Additional bindings saved under Default apply globally, including to keyboards with primary overrides and software-generated shortcuts such as Logi Options+; additions under a named keyboard apply only to that keyboard. Remove an additional binding individually, or use Use Default to clear that keyboard’s primary overrides and additions. For example, keep Fn as the Default Speech key and use + to add Command + Option + 5 under Default for an Options+ button. If a scoped recording receives an unidentified source, it explains that Default is required rather than silently ignoring the input. Froggy also identifies a keyboard’s separate consumer-control service so native media/dictation input can retain the same keyboard scope. For a Logitech K620, select it, click Speech, and press left Control: other keyboards retain their own speech key. Recording for a specific keyboard ignores presses from other keyboards. Identical keyboards without serial numbers share a profile when their model and connection type match. It remaps Caps Lock and the physical F5 Dictation HID usage to private function keys on each keyboard where they are assigned, preventing their native actions. Froggy queries a fresh HID service list once a second so reconnected keyboards receive their new source identities, checks these session-scoped mappings, restores them after wake or reconnect, and restores displaced host mappings on quit. Modifier hotkeys retain their native flags for unregistered shortcuts. When the mic is off, F5 starts ambient listening. When the mic is on for any reason—including Fn, composer, or an unfinished transcription—F5 stops it and discards unfinished transcription; commands already queued or running continue. Composer stays enabled but cannot automatically restart the mic until F5 or Fn explicitly enables it again. Pressing F5 twice within 300 ms toggles composer for the app foregrounded on the second press; the first press remains immediate, while the second resolves the pair and aligns microphone activation with the new Composer state. Entering Composer forces the microphone on, and leaving Composer forces it off, regardless of the microphone state before the pair. Froggy runs as a menu bar accessory without Dock or ⌘Tab entries. Launch and restart do not open the GUI. Caps Lock opens or focuses command mode, switching from the agent side when necessary. When command mode already has keyboard focus, Caps Lock dismisses the GUI. It takes keyboard focus directly, even when no other Froggy window is open, without activating the whole app. After Escape closes it, Caps Lock brings it back in front of the current app. Caps Lock explicitly opens native search for apps and registered actions; closing the GUI leaves Froggy running. Type clip to select Clipboard History first, then press Return to enter clipboard mode in the same GUI. Escape or Back returns from a child pane to plain action search and restores the root search text from before entering that pane; Escape at root closes it; Caps Lock dismisses the focused command GUI from any page. Arrow keys select results and Return opens the selection. ⌘1–⌘9 trigger the first nine command results in their current search order; each row shows its shortcut only while Command is held. These shortcuts apply to root action search while the GUI is focused. Ch finds Chrome. Match closeness, frequency, and recency determine action order. Complete typed commands such as insert hello retain their arguments and use the shared command bus. A command whose whole effect is showing one of the launcher's own surfaces — an agent's conversation, clipboard history, file search, Read Aloud history, speech coach — declares that destination, so running it from the launcher navigates in place, keeping the panel, the field, and keyboard focus, and still runs through the bus; each of those surfaces is an ordinary registered command, so froggy commands lists it too. With the command GUI open, an Fn tap keeps ordinary voice-command behavior and leaves the panel open; recognized speech appears as a visual reference without changing the query. The preview belongs to the current utterance and input: it clears when transcription completes or text is inserted, and late transcripts cannot reappear after closing or switching the GUI. When the agent input owns speech, an Fn tap toggles its listening window after the microphone is ready; a background agent window does not intercept Fn in another app. Holding Fn dictates into the focused GUI input on either side and preserves any existing listening window. The header shows a microphone while command speech is being previewed or dictation targets the GUI; it clears when listening stops. The aura carries the remaining attention time, and expiry leaves the panel open. The header has no voice button: voice is armed by an Fn tap or by the spoken hot word that opened the panel. An Fn hold selects insertion into the focused input, including the GUI. Fn owns speech: a quick tap turns on the mic and immediately arms the same one-shot command override as saying “Froggy,” including inside composer, while the waveform overlay shows the draining cooldown. The listener remains available for 6 seconds after the latest command utterance normally, or 3 seconds while the frontmost app is in composer so unprefixed composition resumes naturally. An active Fn session recalibrates between those windows as focus moves between composer and non-composer apps. Tapping Fn again within 300 ms of the first tap’s release turns on hands-free dictation: the microphone those taps opened stays on and every utterance is inserted into the focused app as it completes, with no key held, until the next Fn press. That ending tap finalizes the capture so the last words are still inserted, then stops listening unless F5 or composer independently owns capture; holding the ending press instead keeps the same insertion going as push-to-talk until release. Neither tap waits on the other: the first tap arms commands at its own release, and only the second tap’s release is read differently, so single tap and hold keep their timing. A second Fn press after the 300 ms window disarms the command arm; holding that second press transfers the tap-owned microphone to push-to-talk insertion, and release finalizes the insertion and stops listening unless F5 or composer independently owns capture. Holding Fn without a prior tap has the same insertion behavior. The second press never independently toggles ambient capture. Audio opens on Fn key-down before tap-versus-hold resolves, and a hold retroactively marks the open utterance as insertion so its first syllable is retained. Fn-Z still undoes the last insertion. Literal typed text uses insert {text}.

The GUI uses the same native Liquid Glass as floating overlays on macOS 26, with system edge treatment and no custom outline or dark tint. Glass is a separate background beneath the search field and results; scrolling content stays clipped inside its rounded edge. The native window shadow is disabled so results do not introduce a rectangular rim around the glass. The blue keyboard aura follows GUI focus and clears when focus moves elsewhere, while the open GUI keeps its draft. The main command and agent inputs have no idle placeholder text; live dictation still appears as it arrives. Root search has no idle branding footer and shrinks to its search field and results; status messages and clipboard actions appear when needed.

The launcher has command and agent sides. Tab flips between them only from their root draft editors; interaction controls and command child panes keep their own navigation. The glyph at the left of the root field is the pointer control for the same switch: the Froggy glyph opens the default agent, and an agent's icon returns to commands. Caps Lock summons command mode or dismisses it when already focused. The optional Primary agent binding does the same for the default agent selected in Settings (the first registered agent when none is selected). Pressing either binding on the other side switches to its destination. The Primary agent binding also switches from any command child pane; returning to command mode restores that pane and its navigation state. Clicking outside or switching apps leaves the GUI open with its draft, page, selection, and scroll position intact; the matching binding focuses that retained window. It uses normal window ordering and can be covered by other apps. On a voice-opened agent panel, the Primary agent binding acquires the editor; Caps Lock switches to commands. The agent side uses the self-advertised SVG icon from its initialize handshake, falling back to the agent-name monogram. Froggy caches the last validated icon in Application Support so it is present before the resident process reconnects; the next successful handshake replaces or removes it. Both sides render one and the same input header — glyph, field, chip — so flipping sides keeps the field, its keyboard focus, and its caret; each side keeps its own text, and only the content below the header changes. The input surface stays on the command panel's frame, keeping the field fixed while the history extends below it like command results. The turn-based agent UX has a separate turn detail above the input, using the available height up to the bottom of the menu bar while the input stays fixed. Click a history row below the input to inspect that turn. Sending a message focuses it immediately with an optimistic message pill, before delivery is acknowledged; the detail then follows the acknowledged turn, including steers. Messages, activity steps, and replies have individual pills instead of one enclosing surface. User and assistant message bubbles hug their content up to 80% of the detail width: user messages sit on the right with a subtle blue glass tint, and assistant messages sit on the left with dark glass. Agent activity and tool-call updates, thinking indicators, and incomplete-reply notices use the same dark glass as assistant replies. The GUI stays fixed and does not accept dragged text or files; typing and paste remain available. Markdown renders throughout the detail, including headings, paragraphs, nested lists, quotes, tables, links, emphasis, and fenced code. An active turn shows a thinking spinner, and new items animate in at the bottom, pushing earlier items upward (respecting Reduce Motion). Selecting an older turn keeps it visible while other turns update. Closing and reopening starts with no detail until a history row is selected or a message is sent. The agent side has no controls row: the history list sits directly under the field's divider. Stop the active turn with “ stop” or the agent's stop capability from the command side, switch agents by typing a name or saying a hot word, and read voice state from the aura and the header. Unread replies are tracked while an agent's reply is not visible, including while another launcher side is open. Partial voice feedback remains visible beneath a nonempty draft. Keyboard focus survives agent submissions and agent switches; Escape, closing the GUI, or flipping to commands releases the conversation arm. A panel opened by voice appears in front without taking keyboard focus until it is clicked or the Primary agent binding is pressed. If it is closed while an agent keeps working, observed reply completion posts a notification.

Agent input uses the same command bus as every other entry point. Return and a double tongue click act only on the focused editor: the message draft sends through agent.<id>.send, while a focused form field submits that rendered form; approvals run only from their explicit action or an exact spoken approval phrase for the interaction selected when speech began. The canonical hot-word grammar decides whether an onset froggy marker runs a command instead. Failed or skipped sends and commands keep the current draft; a command-prefixed draft clears only after a successful result, comparisons ignore surrounding whitespace, newer edits win over late outcomes, and identical unacknowledged messages are not sent twice. Typing an agent's name on the command side exposes its agent.<id> action and carries any remaining text across as the first message. Pending interactions appear inline with one selected item and the same form vocabulary as hosted extension UI; required values and number, integer, option, length, item-count, and range constraints are validated before submission. Array items remain discrete values with stable identities, including items containing commas. Form state is scoped to agent, conversation, interaction ID, and revision, survives pending or failed response admission, and is discarded only when that interaction resolves or is replaced. Responses carry the rendered card's own agent, conversation, interaction ID, and revision through executor admission, so stale cards are rejected instead of rebound. Closing the panel defers interactions. While an agent's window is armed, exact ordinary-speech approval phrases such as “yes” or “go ahead,” and decline phrases such as “no” or “cancel that,” answer the interaction selected at speech onset; literal and push-to-talk insertion never do. A voice-opened panel receives the following utterances without taking native keyboard focus. Composition records the editor focused at onset — the draft, or an exact form revision, field, and array item; switching fields, items, or cards before final transcription rejects those words rather than redirecting them.

An addressed utterance that matches no deterministic capability now opens a handler chooser instead of falling through to Fix. The default agent is first, Fix is second, and other agents follow; Return sends the unchanged text to the selected handler. froggy fix … remains the explicit way to invoke Fix directly. The agent's rose hue appears only in the listening aura; the panel itself stays neutral, and keyboard blue appears only while no agent window is armed.

The activation event tap automatically re-enables itself if macOS disables it after a timeout or user-input interruption, so gestures do not silently stop until relaunch.

Fn-down starts hardware microphone I/O and attaches the speech consumer; when Froggy stops listening, it stops the audio engine and removes its input tap. Once an Fn hold selects push-to-talk insertion, the HUD turns amber through the release grace period. Explicit diagnostic clip observers may share an active speech session, but their last teardown also releases the microphone. Microphone activation and deactivation are silent; the HUD shows listening state. Each start is logged under capture with its audio path, engine start and first-buffer times, and press→audio latency. Global activation uses the same Accessibility permission already required for target detection and insertion.

Key events reach Froggy through a CGEvent tap serviced on its own thread, so a busy main run loop (model loading, a stalled UI) cannot trip the system's tap timeout, and App Nap is disabled for the process. macOS still disables a tap whose callback stalls and lets the offending press through to the system; Froggy re-enables the tap, and an owned key that reaches the parallel event monitors instead is recovered and handled as if the tap had seen it. Both cases are logged as activation diagnostics. The dictation-key and Caps Lock remapping that turns those keys into F19 and F18 is pushed again whenever any HID device (re)appears, in a burst after wake, and on a 30-second check; a mapping found missing is logged the same way, because a dictation key with no mapping reaches macOS Dictation (its chirp) rather than Froggy. Gesture timing uses each key event's own timestamp, and the hold threshold defers to the tap thread's view of the key: a tap whose release the main thread was slow to process (a cold audio engine start, a busy UI) stays a tap. The echo-cancellation status the HUD polls once a second reads capture state and system-tap counters without waiting on either device-startup lock, so cold device setup cannot stall the main thread and queue key events behind it. The capture diagnostics record when usable microphone audio starts flowing (after any speaker warmup) and explain starts that were skipped or stopped (already busy, suppressed, failed, or stopped again because listening was no longer requested). Audio configuration changes and whether the engine resumed after them are logged the same way.

The tap snapshots physical key fields without consulting the keyboard layout; character labels are resolved on the main thread only when recording a custom binding. Echo-cancellation suppression remains finite even for completely silenced output, and the status display omits invalid numeric readings.

Ending dictation without detected speech inserts nothing. Releasing a push-to-talk hold keeps capturing for another 250 ms to retain the end of the last word. Pressing again during this grace period continues the capture; cancellation and microphone-off still stop immediately. After the grace period, Froggy checks any remaining audio (including audio captured while models were preparing) through speech detection before finalizing; silence and background audio without a speech onset never reach transcription. The final partial audio frame is also checked so short spoken words are retained.

The floating or menu-bar-docked controls are called instruments; their expanded details appear in a hover pane. Floating instruments and hover panes use translucent clear Liquid Glass over a translucent dark backing for contrast over bright backgrounds and matching light text and controls (ultra-thin material on older macOS); compact menu-bar rows remain transparent. Panes take clicks like ordinary panels but never take keyboard focus, so the app you were working in keeps it. Panes scroll with small overlay scrollers that float over the content and take no gutter, regardless of the system “Show scroll bars” setting. Hovering the waveform instrument opens its 20 most recent saved transcripts, newest first, with target app and date/time. Clicking anywhere in a transcript row copies its full text and shows a checkmark. The list uses normalized text when available and raw transcription otherwise, survives relaunch, and updates as new transcripts finish. Moving into the pane keeps it open; leaving dismisses it after a short grace period. A low-frequency cursor check repairs missed exit events so a pane cannot remain stuck open after the pointer has left. Docked instruments share one hover pane that switches as the pointer moves between them. A single click acts on whatever is under the pointer, and clicks in a pane's empty areas stay in the pane rather than reaching the app behind it. Panes dismiss shortly after the pointer leaves. During switches, content keeps its final layout while the glass pane moves and resizes; the pane resize and content crossfade run together on the same spring curve, with fixed content dimensions and no stagger or slide. Text does not rewrap during the animation. Clicking the waveform still toggles command listening, and dragging closes the pane without changing capture.

The waveform represents live capture only. Releasing Fn after push-to-talk keeps it visible during the 250 ms capture grace period, then hides it; final transcription and insertion may continue in the background without presenting stale microphone activity.

Press Fn-Z to send one undo command to the application that received Froggy's most recent insertion. Froggy refuses the undo if focus has moved to another application or if it has no unconsumed insertion to undo.

With empty input, the GUI shows up to two previously used launcher actions, most recent first, followed by a divider and screenshots and clipboard entries from the last five minutes, newest first. Reusing an action moves it to the top; unused actions do not fill spare rows. Typing switches to normal search and hides the recent-items section. Search results are reused while the query and catalog are unchanged, including when decay links are saved; saving, renewing, or archiving a decay link invalidates those results.

Tongue-click commands

Non-speech sounds can act as keystrokes while Froggy is listening. The first one is a double tongue click, which presses Return in the verified focused text input; the same editable-element, secure-field, and captured-target revalidation rules used for text insertion apply.

There is no trained model behind this. ClickDetector scans the 16 kHz stream in 4 ms frames alongside Silero and accepts a transient only if it:

• rises at least 6 dB above the loudest of the preceding 48 ms (it must start from quiet);
• has a high first-difference-to-energy ratio, a cheap stand-in for spectral centroid that separates clicks from voiced speech and thumps;
• settles within 80 ms and stays quiet for another 40 ms, which rejects plosives (voicing follows them) and sustained sounds. A quieter rebound that begins within 32 ms of the primary transient may decay for up to 152 ms from onset as room reverberation; it must remain at least 8 dB below the primary peak. Louder rebounds retain the strict 80 ms path.

DoubleClickPairer fires when two clicks arrive 60–500 ms apart and then ignores clicks for 400 ms so a triple click fires once. Keyboard and trackpad presses sound the same as tongue clicks through the built-in microphone, so InputActivityMonitor records physical key and button presses and the pipeline discards clicks heard within 750 ms of one; keystrokes Froggy synthesizes itself are excluded so a double click right after an insertion still works. A double click is a sound-gesture shortcut for the same Enter command recognized by "Froggy enter." If heard while an utterance is still open, it cuts and finalizes that utterance at the first click instead of waiting out Silero's trailing-silence timeout. The Enter command joins the ordered utterance/command queue behind that utterance, so it cannot run until all earlier transcription and insertion work completes. The click audio itself is not transcribed.

Every accepted click candidate and rate-limited near-miss is logged in the diagnostics dashboard (click category, debug level) with its duration, peak, attack, high-frequency ratio, and failed gates. The built-in thresholds are 6 dB attack, -48 dBFS peak, 0.20 high-frequency ratio, and 80 ms maximum duration. Thresholds can be overridden without a rebuild; they are re-read each time listening starts and the effective values are logged under click:

Settings → Sound gestures exposes Echo-tolerant click detection. It is on by default, updates the active detector immediately, and controls whether quieter room reflections receive the extended settling window. The numeric reverberation keys below remain developer tuning controls rather than the only way to discover or disable the behavior.

defaults write com.rt2zz.froggy clickAttackDB 6           # dB above the preceding 48 ms (default 6)
defaults write com.rt2zz.froggy clickMinimumPeakDBFS -48  # default -48
defaults write com.rt2zz.froggy clickMinimumHighFrequencyRatio 0.2   # default 0.20
defaults write com.rt2zz.froggy clickMaxDurationMs 80     # default 80
defaults write com.rt2zz.froggy clickReverberationTailMs 152  # advanced echo-tail tuning
defaults write com.rt2zz.froggy clickReverberationAttenuationDB 8  # echo must be this much quieter
defaults write com.rt2zz.froggy clickPairMaxIntervalMs 700  # default 500
defaults delete com.rt2zz.froggy clickAttackDB            # back to the default

The full numeric tuning key list is FroggyDefaults.ClickTuning.all (also clickPostQuietMs, clickReleaseDB, clickReleaseFromPeakDB, clickReverberationTailMs, clickReverberationAttenuationDB, clickPairMinIntervalMs, clickPairRefractoryMs, clickInputSuppressionMs); the related user preference key is clickReverberationEnabled. Known gaps: when no utterance is open, Silero may still open one on the clicks themselves; the Return is already queued ahead of it, so the cost is one wasted transcription of click audio. Other people's keyboards nearby are not gated, only presses this Mac receives.

Register another instance explicitly with register remote froggy buildmac (or another SSH host), or simply register buildmac. list remote froggy (also list remote froggys) opens the Remote Froggys child route in the GUI, showing registered computers, SSH hosts, and supported capabilities. Registration stays in that route with visible progress and a success or failure message; its search query is retained after failure for retrying. Esc or Back returns to all actions. unregister remote froggy buildmac or a row's Unregister button removes a peer. Registration is shared across Froggy capabilities, initially supports SSH, and verifies the remote instance's identity and advertised capabilities. No computers are registered by default. In the Remote Froggys screen, search for an SSH host to reveal Register when no registered computer exactly matches its host or name. Select it and press Enter (or click it) to register; there is no separate registration field.

Remote connections discover the running Froggy app in the SSH user’s account, including ~/Applications or a custom location. Start exactly one current copy on that computer before registering or searching; the install path is not stored.

Clipboard history searches registered remote Froggys live over SSH. The machine filter defaults to Auto. Select This Mac and/or one or more remotes to search only those machines. Typing a registered host prefix such as buildmac: in clipboard search selects that remote; buildmac: purple otter selects BuildMac and searches for purple otter. The prefix becomes the machine selection and leaves the search field, using the same 2-second policy as the menu. Unknown prefixes remain literal text. Explicit selection waits for every selected remote before publishing one integrated list, with a 2-second deadline. At that deadline, available results appear with unavailable/timed-out hosts and a Retry option; late replies are discarded. Changing the query, content type, or machine selection starts a new search. Auto retains the 100 ms policy below. Each query waits up to 100 ms: remote matches received in time join the initial results; later matches appear behind Load remote matches. The displayed list stays fixed, and loading remote matches preserves the selected item. Rows identify their source computer. Only registered peers advertising clipboard search are queried.

BuildMac must run Froggy to record copies and have the current app installed at /Applications/Froggy.app. Search uses existing SSH authentication and verified host keys. Previews stay in memory only; full content transfers on Paste or Copy, without being saved to local Froggy history. Text and image representations can be pasted; remote file entries are searchable but file transfer is not supported. Remote entries cannot be edited or deleted from the client. Offline BuildMac never blocks local history.

Recent screenshots and clipboard suggestions

With an empty search field, the GUI shows its usual suggested actions, then a horizontal divider and one combined list of local screenshots and clipboard entries from the last five minutes, newest first. Rows show a thumbnail or text preview, relative age, and the Enter action: Paste for clipboard entries, Copy Image for screenshots. New arrivals keep the selected item stable; expired items leave suggestions without deleting their history. The divider is hidden when there are no recent items. A screenshot also copied as an image or single file appears once, using the clipboard row and copy timestamp.

Search Screenshots (also “Froggy search screenshots”) opens a GUI subpage of saved screenshots in reverse creation order. Typing filters filenames, with every word required in any order. Enter / ⌘C copies the image; ⌘Enter opens Preview. The context menu also offers Copy File and Reveal in Finder. Escape returns to the main search. Older clipboard entries remain in Clipboard History.

Screenshot discovery uses the macOS Screenshot app's configured save folder, falling back to Desktop. Only files marked with macOS screenshot metadata are included, so renaming a screenshot still works. Subfolders, captures moved out of that folder, and clipboard-only captures are not part of screenshot search; clipboard-only images remain available in Clipboard History. The folder setting and its contents refresh every two seconds; screenshots appear once saved.

A persistent helper process performs directory reads, metadata checks, image comparison and thumbnail rendering. It starts in the background with Froggy. Opening the GUI reads cached suggestions immediately and never waits for the helper or synchronously polls the clipboard. Cold results arrive asynchronously; helper failures are visible in screenshot search and retry in the background. Screenshot files stay in their original folder; previews are held only in memory.

File search

Open File Search from the GUI to search Spotlight-indexed filenames in your home folder on this Mac and registered remotes. The blank page lists recently used files: what Spotlight recorded as used in the last 30 days anywhere under home, plus files changed in the last 3 days in Desktop, Documents, Downloads, Pictures, Movies, and Music, merged newest first, at most 20 rows per machine. The changed list is limited to those folders because a date query across a large home measured 20 to 40 seconds on its first run in each process. Typing searches filenames in-process through Spotlight, every word required in any order, debounced 120 ms so slower typing cannot pile up queries; a blank query runs at once. Spotlight stops at 1,000 candidates before filtering, the path is the only attribute read, and at most 200 rows are shown. Rows stay on screen until the newer query publishes, and a superseded query never publishes. Search scopes are the visible folders directly under home, so ~/Library never spends the candidate cap; items directly under home are matched locally. Results respect macOS folder permissions for Froggy locally and the SSH session remotely. Protected folders such as Downloads may be absent until access is granted in macOS Privacy & Security settings. The machine filter supports Auto, This Mac, one or multiple remotes, and buildmac: prefixes. Auto gives remotes 100 ms before requiring Load remote matches; selecting machines waits for their integrated results, up to 2 seconds, with Retry for unavailable hosts. Selection stays fixed when late matches arrive.

Select a file and press ⌘C to copy it to the clipboard, with no file-size cap. Local copies use the original file reference. Remote copies stream over SSH into Froggy's private local cache before placing a file reference on the clipboard for Finder paste; memory use stays bounded and available disk space still applies. Folders cannot be copied through this route.

Enter first asks how to open files: Copy URI, Open in Finder, or an installed Cursor, VS Code, Zed, or TextEdit. The choice persists as the Enter default. ⌘Enter always opens the same picker; check Use this as default to replace the saved choice, or leave it unchecked for a one-time action. The picker keeps the selected file fixed and Escape cancels without changing the default. If the saved editor is uninstalled, Enter asks again.

Copy URI produces a file: or sftp://host/… URI. Open in Zed uses Zed's native SSH workspace (zed://ssh/buildmac/…) for remote files and folders, preserving the registered SSH alias and its configured hostname, username, key, and jump host. It does not require SMB. Zed handles its normal remote-server setup and connection UI.

Open in Finder reuses a mounted SMB share or connects to the appropriate share on demand, then reveals the original file locally. It uses the remote address from the working SSH connection, since SMB does not understand SSH aliases. macOS may ask you to sign in to the share. Cursor, VS Code, and TextEdit use the same SMB path. If sharing is unavailable, Froggy explains how to enable File Sharing and share the containing folder; SSH access alone does not enable SMB. Froggy never enables sharing or changes permissions implicitly, and does not substitute a downloaded editing copy. Copy File over SSH does not require SMB.

Search returns up to 200 metadata results per machine; refine the filename to narrow them. Hidden paths, Library, node_modules, and common credential files are excluded. Both Macs need a current Froggy build. Downloads are on demand and remain in ~/Library/Caches/Froggy/FileTransfers; search does not replicate files.

Addressing, composer, and hot-word overrides

The user is always talking to Froggy, and interpretation is never a global setting. Push-to-talk and typed input are always addressed to Froggy. Ambient (continuous) speech is addressed only when it starts with a registered hot word, while attention is armed, or while the focused app is in composer; everything else is dropped, shown in the transcript overlay as IGNORED rather than silently discarded. A hot word spoken after other words ("I mentioned Froggy enter") delimits only an utterance that was already admitted; on its own it addresses nothing. Everything routing reads is captured when the utterance begins: the app in front, where dictation goes, the attention window, any one-shot arm, and composer membership. A gesture, panel switch, or composer change during transcription never changes the meaning of speech already under way. Addressed speech is a command by default — enter presses Return without the hot word while attention is armed. Saying "Froggy compose" or "Froggy compose mode", or double-tapping F5, puts the focused app in composer: unprefixed addressed speech is inserted into it as normalized dictation, spoken commands require the hot word, and typed GUI input is always a command (insert … is the explicit typed insertion), and any existing command arm/countdown expires immediately. Saying "Froggy command mode" returns the active app to command interpretation; "Froggy stop compose", "Froggy stop composer", and "Froggy stop composing" are equivalent. A frontmost composer app automatically owns ambient microphone activation; switching to a non-composer app releases that ownership, while F5-owned ambient listening remains independent. Pressing F5 while composer capture is active stops the mic without exiting composer and suppresses automatic reacquisition until F5 or Fn explicitly enables capture again. Composer is sticky per app, including across quitting/reopening that app and restarting Froggy. Commands, tab/window switches, focus changes, and attention expiry never disable it. It ends only when explicitly turned off using a command-mode/stop-composer command, repeated double F5 in that app, or the menu's compose-off action. An explicit command-mode or stop-composer command leaves a live microphone stream running as ordinary ambient listening; microphone-off remains a separate command. While composing, the HUD and microphone stay active in amber unless command attention is armed, when the waveform turns green, the transcript row reads COMPOSE, and the composer app's front window gets an amber border.

Ambient attention. In ambient listening the hot word arms Froggy for the same 6-second default window as an Fn tap, shortened to 3 seconds while the frontmost app is in composer: every utterance is then addressed, like a conversation, until the window passes with no actionable command, when Froggy disarms — the mic and hot-word detection stay live, but unprefixed speech is ignored until the next "Froggy". Every addressed command that resolves re-arms attention, including one spoken under push-to-talk; dictation does not, so an Fn-hold insertion opens no window. Unprefixed speech the window admits must name a registered command: anything else is shown as UNMATCHED, runs nothing, never reaches a hosted fix, and does not extend the window. Open-ended requests for Fix need the hot word or an Fn tap. A focus change recalibrates a live window for the new app's length, measured from when it was armed, and lets a lapsed window expire; it never extends one. The waveform overlay shows the armed window as a draining strip; the strip disappears when attention is disarmed.

Agent attention and the launcher's editor. Saying a registered agent's name on its own (for example, a fictional agent named "Atlas") opens that agent and arms a thirty-second window addressed to it; an Fn tap on an open agent side arms the same window, and a second tap disarms it. The launcher's field is an ordinary insertion target: Froggy captures it at speech onset exactly as it captures an application's text field, and the agent side is a composer for it, so unprefixed addressed speech composes there through the same insertion path as any other field — normalization, spacing at the seam, undo, and journaling included. An Fn hold inserts one utterance; a double tongue click presses Return, which submits the focused editor. Close, side flip, a second tap, and hosted-view or clarification dismissal end the window, and the microphone follows it: the window is the only lease. Nothing sends by itself. Only utterances for that field renew the window; a "Froggy …" command inside it is a one-utterance override that neither renews nor cancels it, a bare "Froggy" arms nothing, and a delayed utterance can only renew the session it began under, never a newer one or one that was disarmed. While agent attention is armed a frontmost composer app is suspended: its membership survives, its speech goes to the agent, and it resumes when the window expires. If a secure text field or Secure Event Input is observed when an utterance begins, during its capture, or right before a message leaves, an agent-bound utterance is suppressed entirely: no transcript, audio, or corpus entry is published, cloud transcription is cancelled, the HUD shows "Speech blocked · secure input", and agent attention disarms until something re-arms it. Local commands such as "Froggy off" are unaffected.

Saying "Froggy" on its own additionally applies a one-shot command override to the next utterance. The hot word can also prefix the command in the same breath: "Froggy enter" executes Enter immediately after its command utterance finishes transcribing, while "Froggy" on its own arms the following utterance. Repeating the hot word creates another ordered step in the same target-bound execution stack: "1, 2, 3, 5. Froggy fill in the missing numbers. Froggy enter" inserts the first utterance, resolves and applies the edit, then presses Enter. Luna can also plan that same dependency from one command utterance: "1, 2, 3, 5. Froggy fill in the missing number then enter" returns the edit plus a validated input.enter follow-up. Generated follow-ups may use any executable Froggy command and may include another Fix; every stack is bounded to 20 total steps and three Fix resolutions. A failed, cancelled, invalid, over-budget, or no-effect prerequisite halts the remaining stack. Stacks for different applications can resolve concurrently; stacks for the same application serialize. Their queued, running, completed, and failed states appear in the HUD and diagnostics. While armed, the menu-bar icon switches to a command glyph, the live transcript is labeled COMMAND, and the menu offers Disarm. Arming persists until the next utterance, until the attention window lapses, or until listening stops; a lone "Froggy" followed by a long silence arms nothing.

The executable vocabulary lives in one capability registry shared by voice, the launcher, diagnostics, execution, and model-generated follow-up validation. Natural voice and typed phrases resolve to a capability ID plus schema-validated values; users do not have to speak a rigid argument syntax. The current commands are enter, insert, Fix, use Codex Computer, use Codex CLI, show Codex Computer session, new Codex Computer session, open Codex app, compose and stop composing (compose.start / compose.stop), off, pause media (media.pause), play media (media.resume), read aloud (reading.read), stop reading, skip ahead, and skip back (reading.stop / reading.next / reading.previous), tab N (system.select_tab), and application launch/focus/hide/quit. Pause/play target macOS's current Now Playing session without focusing its application, so a YouTube video in a background Chrome tab can be controlled; Froggy checks the current state, sends the distinct pause or play command rather than toggling, and reports failure if macOS does not confirm the requested state. "Froggy tab 3" presses ⌘3 in the captured application, even while it is in the background; "go to tab", "switch to tab", digits, spoken numbers (including ASR homophones like "to" and "for"), and "last tab" (⌘9) are accepted, and the shortcut inherits each app's own ⌘-digit semantics, so numbers above nine are not deterministic commands. "Froggy open Safari", "Froggy switch to Notes", "Froggy hide Music", and "Froggy quit Preview" use macOS application services. Application verbs also tolerate a missing space when capitalization or a digit marks the boundary: "OpenT3 code" launches T3 Code, and "openSafari" launches Safari. The app name and any command tail keep their original text; ordinary words such as "opening" are not split. Application names are matched against installed apps by exact, partial, fuzzy, acronym, and conservative phonetic forms: "open Chrome" resolves Google Chrome, while the longer match in "open Chrome Canary and start compose" wins and the remaining words are planned as registered commands by Luna after Canary is foregrounded. Close matches open a Raycast-style launcher clarification showing the transcript and app choices; the local choice count improves the same phrase next time and Reset All Settings clears it. Any phrase command can be chained the same way: "Froggy compose mode and insert test 123" starts composer, then Luna plans insert test 123 as registered commands that run once composer is on. The tail must follow "and", "then", or punctuation closing the command phrase, so "compose an email to Bob" stays one open-ended Fix request; commands that take text (insert, Fix, use Codex CLI) own everything after their verb. A simple command works offline; a chained command tail requires the configured Luna API key. "Froggy insert hello comma world" inserts hello, world; direct keyboard input uses the same insert hello comma world command. Saying "never mind" anywhere in a command cancels that whole interaction before any effect is queued, including earlier hot-word-delimited steps; explicit insert, composer or push-to-talk insertion, and escaped literal input keep the words as text. "Froggy off" stops microphone capture without cancelling commands already running. Enter presses Return in the captured eligible target; "return," "press enter," and "hit enter" are accepted aliases. Escape remains an input override rather than an executable capability: it makes the next utterance insert verbatim (no hot-word scan and no deterministic normalization, so "new line" stays as words); "Froggy escape, tell Froggy hello" inserts tell Froggy hello immediately. This is how the word "Froggy" itself, or any phrase that would otherwise be a command, gets typed.

The desktop command family names its target explicitly:

• Use Codex Computer {task} sends a task to the current Computer conversation.
• Show Codex Computer Session opens that conversation for inspection without submitting a task. “Codex Computer status” and “check Codex Computer” are equivalent.
• New Codex Computer Session starts a fresh conversation for future tasks. “Reset Codex Computer session” is equivalent.

These are separate launcher actions. The session commands execute directly; Use Codex Computer opens a task prefix for you to complete. For example: “Froggy, use Codex Computer, research three grills and put the best one in my cart, but don't buy it.” The desktop adapter prefers the Codex app, dismisses any Computer Use overlay, submits through its composer, and restores the previous app's focus. It retains its existing ChatGPT app fallback. Launcher suggestions hide desktop handoffs when neither app is installed, and hide CLI handoffs when no supported Codex executable is found. Explicit typed or spoken commands still explain missing dependencies. Suggestions refresh when the launcher opens and use that snapshot while typing. Each handoff checks again at execution time; installation does not imply authentication or Accessibility permission. Accessibility permission and the desktop app's Computer Use tools are required. The adapter reuses the current conversation; if you switch conversations manually, use New Codex Computer Session before the next task.

Use Codex CLI {task} runs a separate non-interactive task through codex exec --json, supplies the task over stdin, and waits for completion or failure. It uses the workspace-write sandbox and existing Codex authentication. Executable discovery checks installed Codex/ChatGPT bundles, Froggy's process PATH, Homebrew, and ~/.local/bin, ~/.bun/bin, ~/.volta/bin, ~/.npm-global/bin, and ~/Library/pnpm. A custom version-manager installation must be on Froggy's process PATH; Finder launches may have a different PATH from your terminal. A missing executable reports where Froggy searched. For example: “Froggy, use Codex CLI, inspect this repository and run its tests.” The run must return a validated completed, blocked, or failed outcome. Open Codex App opens the desktop app for inspection; it does not claim to show the CLI run's session. Froggy surfaces missing installations, unsuccessful runs, blocked/failed outcomes, and invalid results in the UI and diagnostics.

The GUI keeps its presentation and focus ownership in a dedicated main-actor launcher session. Switching or closing it synchronously invalidates old editor targets; native focus changes update the session directly. This separation does not change keyboard or voice behavior and adds no background queue.

Agents

A registered agent is a resident conversational delegate addressed by name. An agent is installed by writing ~/.froggy/agents/<id>/froggy-agent.json; that directory is watched, as the extensions root is, so installing, editing, or removing a manifest takes effect while Froggy runs. A change settles for half a second, then the rescan replaces changed agents outright (the old child stops, the new one starts), unregisters removed ones, and leaves an agent whose manifest did not change running untouched. A broken manifest keeps the agent listed as unavailable with its reason. agents.reload rescans on request from voice, the launcher, or froggyctl invoke agents.reload and reports how many agents are available; agents.status shows each agent and any reason it is unavailable. Reloads are serialized across watcher and explicit requests, and the GUI follows replacement agent instances. A fresh adapter restores pending interactions from an authoritative snapshot; only reconnects retaining their live state resume directly from a cursor.

Registering an agent publishes five capabilities that voice, the launcher, and froggyctl reach like any other: agent.<id> opens its conversation, agent.<id>.send {text} delivers one message (steering the active turn or starting one; it completes when the agent's adapter acknowledges delivery, never when the turn ends), agent.<id>.stop cancels the turn Froggy last authoritatively observed as active, agent.<id>.status reports its state, and agent.<id>.respond answers the host-selected pending question or approval with structured values. Turn state, pending-interaction order, and selection have one shared projection for the panel and executor; delivery acknowledgements record assignment but never overwrite newer versioned state. Froggy captures the full destination with the stop target or selected interaction and revision when each original or dynamically generated batch is admitted, then validates that same destination at transmission rather than accepting host identity as command arguments or reconstructing it from a turn ID. Spoken, "Atlas summarize this" sends, "Atlas" alone opens, "Atlas stop" and "Atlas status" select those controls, and "Froggy atlas summarize this" resolves through the send alias like any command. In one utterance, "Atlas summarize this, froggy enter" sends first and presses Enter in the captured app only after the send is acknowledged; a rejected or unknown delivery halts what follows. Messages to one destination leave in execution-dependency order without holding the focused-app lane while they wait, including dynamically planned sends. Spoken content is not published until every initial or generated send in its stack passes the transmission-time secure-input and Inert Mode check. Inert Mode suppresses send, stop, and respond but still opens and reports status.

Delayed GUI insertions, replacements, key presses, and undo require the same open editor session and destination captured when the work began. Closing, reopening, or switching that destination rejects stale work and stops its remaining commands. Saved agent commands use the same target admission and secure-input checks as direct commands. Agent response values are recorded only if agent-content recording remains enabled continuously from admission through acknowledgement.

Future direction: replace one-process-per-command codex exec runs with a persistent Codex app-server session and Froggy-owned MCP browser/computer tools. That would preserve conversational context, stream richer progress into the HUD, and match the tool-oriented architecture used by agent interfaces such as T3 Code without depending on Codex's private desktop Computer Use protocol.

Detection rides on the streaming transcript rather than a separate keyword-spotting model, so it lands ~0.7–0.9 s after the word is spoken at the current 640 ms tier. FluidAudio's auxiliary Parakeet CTC 110M keyword spotter runs over the first 1.5 s of an utterance whose first word resembles Froggy and can correct a mis-heard first word before routing; it does not gate transcription, and if that optional model cannot load, ordinary transcription remains available. Because audio is always being transcribed, detection latency only delays the indicator; it never loses speech. While ambient capture is disarmed and the app is not in composer, the transcript overlay withholds streaming partials until the final lands (as IGNORED or as whatever it resolved to), unless a partial starts to sound like the hot word. Hot words are a registry, not a literal: each entry has a phrase, the transcript spellings Parakeet has produced for it, acoustic aliases for the CTC spotter, a match rule, and a binding. "Froggy" is the first entry, bound to command interpretation: its canonical spelling delimits commands anywhere in an admitted utterance and may repeat, while lower-confidence variants (frauggy, froggie, frog e, …) count only at utterance onset or after terminal punctuation. Agent names are onset-only entries bound to that agent, and any other capability can be bound to a phrase with the remainder as its text argument. Spellings are compared as normalized words, so no two entries can share one. Words inside balanced double quotes are text: they never match a hot word, select "stop" or "status", or cancel with "never mind". One parse over the whole finalized utterance produces an ordered plan before anything runs; "never mind" in any command or message segment cancels the entire plan, except inside an explicit insert. Every registered phrase feeds vocabulary boosting and the onset-miss correction, which repairs a mis-heard first word to the phrase the spotter heard. Grow the spellings from what Parakeet actually emits, visible as near-misses in the asr category.

Always-on keyword detector (shadow mode). FroggyCore bundles a small Core ML keyword model (FroggyHotWord.mlmodelc, trained from the transcript corpus by tools/hotword) that scores the last second of microphone audio every 100 ms on the CPU. Each registry entry may name its own bundled model; the detector runs one per entry and each firing carries the entry id. Installed apps load it from Contents/Resources/Froggy_FroggyCore.bundle; missing model resources are reported as a detector preparation failure. Today it only reports: each firing is a hotword diagnostics event with its score, stream position, compute time, and the utterance open at that moment, so its decisions can be compared with the transcript-based ones. It does not arm, route, or gate transcription. The plan is to make it the sole listener while ambient capture is disarmed and Parakeet is idle, once its recall and false-accept rate are measured in real use; see docs/benchmarks/ for the current numbers.

Fix

Say “Froggy fix” to lightly improve grammar, punctuation, clarity, and transcription mistakes while preserving meaning and voice. Clean, clean up, rewrite, and revise invoke the same command. Ping is no longer a command.

A selection defines the edit boundary. With no selection, Fix edits the whole focused input. Add an instruction: “Fix; make this more formal.” Froggy gives the model the complete readable text as context, but applies its replacement only to the chosen range. A replacement can be longer or shorter than the original selection. Dictation utterance boundaries and recent transcripts do not determine scope or enter the model context.

Scopes are resolved locally before the model runs: “fix the last paragraph,” “fix that last sentence,” “fix this paragraph,” “fix the first two sentences,” or “fix the last five words.” First/last scopes operate within a selection when one exists. This/current refers to the unit containing the caret. “Fix the whole input” explicitly overrides a selection. Scope and direction combine: “fix the last paragraph; make it shorter.” Unsupported or missing scopes stop with a request to select the intended text; they never expand to the whole input.

Editable text is replaced in place with the normal Undo path. Froggy captures text and selection before calling the model, then revalidates the app, control, full text, and selection before applying the range replacement. Changes while the model works stop the edit. Native controls must expose readable text and a writable selection range; unsupported controls fail safely rather than using Select All or undoing an earlier dictation. Native application edits are read back to check that the expected complete value landed. Froggy's GUI editor uses its own range replacement and undo.

Read-only selections produce Fixed text in Clipboard History (clip in the GUI), where the existing copy and paste actions retrieve it. The result is saved before completion, without changing the system clipboard. This explicitly retained result follows clipboard history retention independently of execution-history content capture. Some browser selections expose only selected text; in those cases that is all the available context.

Settings → Fix engines orders OpenAI GPT-5.6 Luna, Apple Foundation Models (macOS 26 with Apple Intelligence), and a final deterministic cleanup. Hosted calls send the full readable context, chosen range, and instruction using the configured OpenAI Platform API key. Audio is not sent by this command. The deterministic engine can perform basic cleanup, but refuses custom instructions that need a language model. An unchanged model result stops without asking another engine to make a stronger edit. Fix models return replacement text only; they cannot generate submission, agent handoff, or other follow-up commands. Explicit subsequent commands still run through the normal command stack.

Single-click editing remains disabled until click detection is reliable enough.

Voice proximity and direction

On a Mac without a usable microphone, recording reports that no input is available; the GUI and clipboard history remain usable.

TargetMac exposes its built-in MacBook Air microphone to applications as one 48 kHz input channel. That mono signal cannot reveal direction of arrival: any physical mic-array processing and beamforming happens before Froggy receives audio.

Each completed utterance now records RMS level, peak level, an adaptive noise-floor estimate, and estimated signal-to-noise ratio in both Settings → Diagnostics and the diagnostics dashboard. This gives us real samples from TargetMac before choosing thresholds; none of these measurements gate transcription yet.

A future near-field filter can combine those measurements with spectral balance and reverberation estimates into a calibrated likely near / uncertain / likely far score. It would need per-device calibration and should initially be a conservative classifier feature. It cannot measure physical distance, and a loud nearby speaker or a quiet user can defeat an amplitude-based gate. True directional filtering would require a multichannel external microphone whose individual channels are exposed to Core Audio, or hardware that performs its own directional/near-field filtering.

Other commands:

make mac-build    # compile the macOS executable
make mac-bundle   # debug .app
make mac-release  # optimized .app
make mac-check    # build and run the fast default Mac suite
make mac-test FILTER=CommandExecutorTests # run one suite (or test-name regex)
make mac-integration-test # extension build/reload/process end-to-end test
make mac-clean
make ios-check    # contract tests + unsigned simulator app/keyboard build
make android-check # contract tests + debug APK; requires JDK 17 and Android SDK 36
make check        # check all three native applications

make mac-test runs the same default suite as make mac-check; swift test builds the package and tests in one invocation. The default excludes only the extension end-to-end test, which repeatedly compiles real Swift packages. Extension manifest, scaffold-name, and watcher tests remain in the default. Run make mac-integration-test when explicitly requested or when a specific extension building, loading, invocation, watching, or SDK/templates change needs coverage that focused tests cannot provide. Bare swift test --package-path apps/mac still runs the entire Mac suite.

Tests exist for behavior that is hard to prove or finicky, not for asserting general correctness. In the optional maintainer setup, Endofrog owns the suite: it runs full suites on request (endo --agent endofrog send "test: ..."), advises on whether a behavior deserves a test (consult: ...), and prunes the suite at most once per day after a fully green run. That setup restricts coding agents to the test target they changed. Forks without Endofrog run tests locally using the commands above and choose focused or broader coverage according to the change. Documentation, copy, and cosmetic/layout edits need no automated tests; inspect the diff and use direct UI QA when useful. See AGENTS.md for the agent validation policy.

Development signing

The build script automatically uses the sole valid Apple Development identity in the login keychain. This gives the app a stable team-based designated requirement so macOS privacy grants can survive ordinary rebuilds. If more than one development identity exists, select one explicitly:

FROGGY_CODESIGN_IDENTITY="Apple Development: Your Name (TEAMID)" make mac-bundle

Without a valid identity, local builds continue with an explicit warning and ad-hoc signing. Those builds may need Accessibility permission granted again whenever their code changes.

Useful spike utterances

With a text field focused in T3 Code, Chrome, or another accessible application:

• is this working question mark
• open package json in t three code
• write a short note explaining why the deployment is delayed
• write a TypeScript function that groups users by organization as a code block

Show microphone (menu-bar pane and Settings → Display) defaults on and shows a draggable Microphone instrument at native menu-item width. Asymmetric top-left and bottom-right corner brackets, with moderately longer horizontal strokes than vertical arms, frame a small dot: solid while Froggy's microphone capture is on and dim while it is off. When muted, the dot, brackets, and slash all use 40% opacity. A short slash fits inside the brackets when Froggy's current input reports muted, even if another input or virtual audio device cannot be muted. The pane separately reports whether all connected inputs are muted. Device names and hardware mute state appear in the hover pane. Hover or click for two selectors: System changes macOS's default input, while Froggy follows System or remembers an independent input by stable Core Audio UID. If Froggy's preferred mic disconnects or cannot open, it temporarily uses the system default without forgetting the preference; reconnecting returns to the preferred mic. Voluntary Froggy changes wait until the current utterance ends when the active mic is still available. The Froggy selector always displays the actual current input, including the system fallback when the preferred mic is unavailable. The pane shows pending switches, fallback, and errors. With capture off it says Ready to use. The pane appears immediately, but its per-device level meters wait until it has remained open for 500 milliseconds before sampling; leaving sooner cancels the pending microphone activation. Selecting a device does not activate an idle microphone.

Each distinct selected device has Test microphone controls. Mute all microphones / Unmute all microphones affects every connected input for every app, independently of which inputs System and Froggy selected. Every input is attempted; unsupported hardware and failed writes are reported explicitly, and partial mute never displays as all-muted in the pane. The button and hotkey toggle back to unmute when all software-controllable inputs are muted, even if an unsupported virtual input remains. The pane provides an unmute action after partial success. These are one-time operations on currently connected devices; newly connected microphones are not automatically muted. Gain is not exposed or changed. Changes made by hardware or other apps update the pane and icon. Tests explicitly record five seconds from the named microphone, with a live meter and manual playback. Test audio stays in memory, is replaced by the next test, is never transcribed or saved to history, and never falls back to another microphone. Closing the pane cancels a recording or playback. The ordinary live meter reuses Froggy's existing stream and does not activate a second microphone. Sound Settings and Audio MIDI Setup links appear directly in the pane. Settings → Hotkeys → Mute all microphones uses the same key recorder as the other hotkeys: click +, then press any key or chord (release a modifier to bind it alone). Add multiple bindings, scope them to a keyboard or Default, click to re-record, or right-click to remove. It starts unbound and shares conflict checks with the other hotkeys. Every binding targets all connected microphones. Device mute is independent of F5's capture toggle and Fn's speech gestures.

The instrument shares floating glass, docking magnets, placement persistence, and the common hover pane. Microphone Settings opens it through the launcher or voice. The registered microphone.control capability carries selection, mute, and test actions through normal execution history and Inert Mode; microphone.list exposes connected names, UIDs, System selection, and hardware control support for CLI use. Reset All Settings clears Froggy's input override, instrument placement, and mute shortcut; it does not change the Mac's current input or hardware gain/mute settings.

Both menu bar variants use a monochrome frog matching the favicon, without its neon glow or background tile. The icon adapts to the menu bar appearance and stays the same across listening and interpretation states. The menu shows the last transcript, target, normalized output, and pipeline latency so false positives and performance regressions are visible; final transcripts and their metadata are also saved locally. Display controls include: Show commit in menu bar to add the build hash beside the menu bar icon (off by default), Show transcript instrument for the independent activity feed, Show last command for the latest command pill, Show media controls for playback, Show status icons for Keep Awake, Show performance diagnostics for Froggy's own CPU and memory, and Show dev info for build metadata (six-character stamps: abc12+ when dirty, abc123 when clean). Show waveform instrument independently shows a compact, draggable waveform. It starts in the menu bar left of the camera cutout (centered on displays without a cutout), remembers its placement and display, and shares the other overlays’ menu-bar/top-center magnets, neighbor snaps, and Shift bypass. It uses glass while floating and a transparent menu-bar footprint while docked. Its attention countdown travels with it, inside the bottom of the row. Clicking the waveform acts like an Fn tap: it starts the microphone and arms the next utterance as a command, including in composer; clicking again disarms that tap session. It shares Fn’s command timeout and microphone ownership. Dragging moves the waveform without activating the microphone. Reset All Settings clears its visibility and placement. The option also enables the mode glow spanning each display’s top and bottom edges, which always shows the same color as the side auras: amber, blue, green, and violet while command attention is armed on the auras’ hold-then-fade timeline. It is hidden entirely while Froggy’s input device is muted, unless the GUI is armed. The glow originates from a thin, one-point light at the physical top and bottom edges of each screen, with a faint halo fading inward to transparent over sixteen points. The top edge leads; the bottom is 55% as bright. There is no aura around the cutout or waveform and no glow at the menu bar’s bottom edge. The screen-edge glow is hidden while the microphone is off or preparing. The disabled waveform stays fully opaque white, including during microphone preparation. The bars are always white. While the microphone is on they follow its loudness: the newest level sets the center bar and older levels trail outward, rising quickly and falling more slowly; silence settles to five dots. Behind them sits a soft pool of the current mode color, dimmest at rest and brightening and swelling as you speak, more slowly than the bars. A system-muted input device turns the pool cool grey and suppresses the violet attention crossfade, matching the screen-edge glows. Ambient green uses a brighter, more saturated pool with 60% opacity at rest and 86% at peak loudness, so listening stays visible even in silence. Insertion amber in the waveform uses 48% opacity at rest and 74% at peak loudness; screen-edge amber is unchanged. While command attention is armed the pool crossfades to the same violet the screen-edge glows use and brightens well past any mode color, then dissolves back as the window winds down, on the glows’ hold-then-fade curve; the countdown strip is violet too. It fades to nothing at the row’s edges, so a docked waveform never washes over the overlay beside it, and it goes out entirely when the microphone is off. The bars are four points wide in every state, so a bar at rest is the same circle whether the microphone is on or off and turning it on never resizes or shifts them. Bar motion, the countdown strip, the top and bottom edge glows, the side-edge listening glows, and their fades are Core Animation animations interpolated by the render server, so Froggy does no per-frame work while they move. The composer border stays grey until microphone audio is ready for dictation, then turns amber. Microphone-off keeps the screen-edge glow hidden; explicitly armed keyboard input keeps its independent blue feedback in the countdown strip and both screen-edge glows. The top light has 80% opacity with a halo peaking at 16%. Activation fades in over 180 milliseconds (instant with Reduce Motion), then the light stays steady while its color follows the input mode. Turning off hides it immediately. The countdown drains across a thin two-point strip at the bottom of the waveform and softens while paused, returning to full brightness when it resumes. Command attention and the temporary Fn microphone timeout pause during a detected utterance, then resume with their remaining time at the speech endpoint; the countdown strip freezes with them. A resolved command still refreshes attention normally. Command arming takes precedence over composer amber in the countdown strip and the screen-edge glows, which crossfade to violet while command attention is armed and dissolve back to the current app’s normal interpretation color as the window winds down (typed keyboard input retains blue); the bars themselves stay white. The waveform can be moved between displays; the mode glow still spans every display’s full width. This option defaults off and can coexist with the transcript overlay. Command launch uses the command window and waveform feedback; there is no separate circular I-beam/status indicator.

Icon-only overlays use a shared square menu-bar footprint: collapsed media controls, collapsed last command, and Keep Awake are each one menu-bar row wide on their display. Text and expanded controls add space beyond that base footprint.

Froggy always has a native menu bar item. Show commit in menu bar adds the six-character build stamp beside the frog icon and is off by default. Both appearances open the same Froggy menu. This setting is independent of Show dev info. The menu shows a selectable build stamp. Hovering the menu bar item shows the build when enabled; echo-cancellation status remains in the menu and Settings. Reset All Settings restores the simple icon. Native menu bar placement and automatic sizing supply the side margins around its content. It is not a separate floating overlay.

Show performance diagnostics, in the menubar panel and Settings → Display, defaults off. When enabled, a draggable glass pane samples Froggy's own process once a second: CPU as a percent of one core with a two-minute trend line, memory footprint (the figure Activity Monitor shows) with its trend and the peak since launch, thread count, wakeups per second, CPU time and uptime since launch, and the thermal state plus Low Power Mode. Sampling reads kernel counters only and never touches the audio or command paths; nothing is sampled while the pane is hidden. The compact row shows only the CPU percentage and memory value, without labels. Its fixed 104-point width scales with the display font; fixed value slots and monospaced digits keep the row and numbers steady as samples change. Hovering opens the full details below it, with the Froggy logo beside Performance to identify these as Froggy’s own process metrics; moving into the pane keeps it open, and moving away dismisses it after the same short delay and fade as the other overlays. The docked row shares the media/last-command hover pane, including sliding transitions and its persistent hover highlight. Floating performance uses the same hover behavior with its own glass pane. Dragging dismisses details and shares the other overlays’ menu-bar/top-center magnets, neighbor snaps, center attachments, and native status-icon attachment group; hold Shift to bypass magnets. The docked row is transparent and keeps its compact footprint while details are open. Placement and display are remembered; Reset All Settings clears them along with visibility. There is no memory or CPU budget yet; the pane exists so regressions are visible while Froggy is used.

Show last command (menu-bar pane and Settings → Display) defaults off and independently shows a draggable pill. It retains the latest issued command or composer insertion until the next issuance, including after disarming or stopping the microphone; it shows just the ⌘ command-key icon before any command this session. Each new issuance shows its name beside ⌘ for five seconds, then collapses back to the icon. Completion updates and hover do not extend that deadline. The name uses 8-point semibold text in a closely fitted capsule, with the ⌘ icon outside the capsule. Its variable width is capped at 24 characters including the ellipsis. The completion status remains available in the hover details. Hover opens a separate rounded Liquid Glass pane beneath the pill with the full command name, whitespace-normalized argument, and completion status, including after the five-second timeout; long arguments scroll. Moving between the pill and pane keeps it open. Dragging or disabling the option closes the pane. The pill uses the same saved placement, menu-bar/top-center magnets, neighbor snaps, status-icon attachment group, and Shift bypass as media controls. A neighbor snap remembers the touching edge: a neighbor on the left keeps the pill’s left edge fixed so names grow rightward; a neighbor on the right keeps its right edge fixed so names grow leftward. The five-second collapse follows the same edge. The ⌘ icon occupies a fixed slot at the anchored edge, so showing, replacing, and hiding the text never moves it. Text grows rightward from a left anchor and leftward from a right anchor. Native status-group attachments grow leftward; the top-center magnet centers the icon itself. The pinned edge survives relaunch; Shift or dragging away clears it. At a display edge, the name truncates to the available width rather than moving over its neighbor. The text capsule has glass while floating; the rest of the row stays transparent, including around the icon; hover never expands its menu-bar hit area. Visibility and placement reset with Reset All Settings. Command contents are session-only.

Listening edge glows: three overlapping glows spill inward from the midpoint of each display’s left and right edges whenever microphone audio is ready or the GUI is armed. A faint narrow spine tapers into long tails along the edge, linking three softly overlapping lobes. The initial shape is established at full size before animation begins, so first activation uses the same 0.35-second fade as later activations. Microphone loudness makes both auras breathe gently inward, at most a fifth wider and slightly taller, rising over 80 milliseconds and relaxing over 280 milliseconds; silence restores their resting size without disturbing the attention fade. Reduce Motion makes these level changes immediate. The glows continuously drift between randomly chosen sizes and positions over 3–5 seconds, assigning small, medium, and large lobes across the three positions; any position can be largest. Mode changes quickly reshape them over 0.45 seconds from the currently displayed shape, then resume slow movement; both edges move together. While the GUI stays focused, microphone mode changes only recolor the bubbles, preserving their sizes and ongoing gentle drift. The lobes randomly trade warmer, near-base, and cooler hue variations, up to 12.6 degrees from the mode hue. These offsets drift smoothly with the shape, keeping the three colors distinct while retaining the mode’s overall color. Green indicates unarmed ambient listening and reaches only half as far inward from the edge. When Froggy's input device is muted at the system level, the aura turns a cool grey at that same shallow depth: Froggy is capturing silence, so no violet attention, amber, or voice orb appears until the microphone is unmuted, while blue GUI arming still shows. Violet colors the aura at full depth when command attention is armed, including an Fn command tap or the hotword; a focused GUI keeps its top bubble blue. While listening is active, meaning command attention, a composer, or the armed GUI, a small, sharply edged white orb sits in the middle of each aura and brightens and grows with microphone loudness on the same swell timing, ignoring anything quieter than −36 dBFS so room noise and keyboard clatter that would not be transcribed leave it dark, growing to more than twice its resting size at peak loudness and holding at least a third of its brightness for the whole of a detected utterance so talking stays lit between words. The orb tracks the speech envelope closely, attacking over 60 milliseconds and releasing over 180 milliseconds so each syllable shows as its own pulse. For as long as an utterance is detected it also breathes faintly on its own, a slow uneven loop of a few percent in size and brightness beneath the envelope, and eases back to rest at the endpoint. Reduce Motion disables the breathing. The orb has a solid center with a short falloff and sits outside the aura's loudness swell, so the mode color around it stays clearly visible. It goes dark in silence, never appears during ambient green, and disappears with the microphone. Amber indicates insertion. While the GUI has keyboard focus, the top aura bubble stays blue and is 25% wider and taller than its normal drifting size; the lower two retain the microphone mode color. With the microphone off or preparing, all three are blue. Losing GUI focus removes the blue override. In blue GUI mode, both side auras travel upward over 0.45 seconds so the GUI input header sits roughly 25% down from the top of the aura, staying at that height as results expand below it. Returning to voice mode moves them back to the screen midpoint. Reduce Motion makes this repositioning immediate. Mode colors blend smoothly over 0.3 seconds, continuing from the displayed color if the mode changes again; Reduce Motion disables the continuous movement and makes mode changes immediate. Voice colors require microphone readiness. The untimed one-shot command arm holds the full violet aura; a timed arm follows the attention window. That attention color and extra intensity hold for the first 60% of the attention window, then dissolve over the final 40% with a smootherstep curve: the whole aura gently starts fading, moves quickly through the color change, and softly returns to its base color and depth. The normal mode glow remains intact. The fade holds during speech and softly refills on rearming. There is no moving countdown boundary, dot, outline, or pulsing; the waveform retains its precise attention countdown. The whole glow gently fades in over 0.35 seconds and fades out over 0.4 seconds when the microphone is off or preparing and the GUI is not armed. Rapid toggles reverse from the current opacity without flashing; Reduce Motion makes visibility changes immediate. The glows work independently of the waveform overlay, pass clicks and drags through, and never take focus. They reposition when displays change. There is no pointer-following or text-caret overlay. Green indicates a ready ambient microphone awaiting the hotword; the normal hotword/attention rules still apply.

Docked waveform, last-command, performance, media, and status-icon rows show a subtle rounded highlight on hover, matching the pressed appearance of a native macOS menu-bar item. For media, performance, and last-command rows, the highlight stays active while the attached pane is showing, including pointer transfer, dismissal delay, and the media pane’s fade-out. It clears once both the pointer and pane are gone, or when the row is undocked. Moving between docked media, performance, and last command switches one shared pane: it slides and resizes beneath the new item while the content crossfades. The active item keeps its highlight. Moving into the pane keeps it open, and native media menus hold its contents steady until the selection ends. Reduce Motion makes pane transitions immediate.

Center menu-bar magnet: drag a menu-bar overlay near the center to attach it. On a display with a camera cutout, items attach 12 points away from its left and right edges, leaving a small inset on both sides. The inset survives resizing and display changes; adjacent items within each group remain flush. On a display without a cutout, the innermost left item is centered in the menu bar and the right group sits immediately to its right. Additional left items extend leftwards. Side and order are remembered when changing displays or reconnecting a display with a cutout, so the same arrangement maps back to the original sides. Hidden items leave room for visible neighbors without losing their place; if no left item is visible, the first right item temporarily occupies the center. This applies to the media controls, waveform overlay, performance pane, last command, and status icons. Hold Shift or drag away to detach. Reset All Settings clears the center attachments.

Froggy owns the center of the menu bar. Overlays never attach to, follow, or measure native status items, and nothing reads the window list; the right side of the bar belongs to macOS and other apps, and a docked Froggy row placed there simply stays where it was dropped.

Show transcript instrument is off by default. When enabled, it displays the former activity rows below the live-pane header in a separate floating glass window with a small Live activity header. It defaults to top center below the menu bar and remembers its own position and display. It stays visible whenever enabled, including when its feed is empty. When empty, only the header remains. New rows grow downward from its own top-left anchor; arrival, expiry, and app changes never move that anchor or resize the menu bar item. Pending speech appears newest-first as Transcript: …, then updates in place to Insert: … or Ignored: … once classified. Commands use chips: the command-mode green verb (Launch) and grey argument (Chrome), dimmed while queued; a failure reason or returned Fix result adds text. Rows recognized by OpenAI carry an OpenAI chip. Clicking a row copies its complete label and value and briefly highlights it. Each entry expires ten seconds after its latest update; later lifecycle updates can re-add it. Row changes do not animate panel layout.

Transcripts and Read Aloud use native Liquid Glass with adaptive foreground colors when floating. Their typography and single-row height follow the display’s menu-bar metrics and system font, refreshing on display changes. Playback removes the glass effect entirely when docked at single-row height, leaving transparent rows without boundary lines or floating shadows. Hidden or floating windows are not neighbor snap targets. Docked media controls keep their compact menu-bar footprint; hovering opens a separate Liquid Glass pane directly below the menu bar, with rounded corners on all four sides and extra bottom padding. The pane stays above status and transcripts. The independent transcript window keeps rounded glass. The overlays share native drag tracking. Playback retains its menu-bar and point magnets; holding Shift bypasses magnets. Playback collapses while dragging so snaps use its compact footprint. Its drag stays held through docking and playback updates until mouse-up; buttons retain their normal click behavior. Menu-bar magnets show no rounded drag preview, and release or cancellation clears previews. Reset All Settings clears their visibility and placement preferences without deleting saved transcripts or the Keychain API key.

Transcript corpus

Froggy automatically saves every non-empty final speech transcript locally, including raw text, normalized text, target application metadata, acoustics, recognizer provenance, insertion outcome, and timing. Empty results and abandoned partials are discarded. Text saving does not require audio saving.

Settings includes Save utterance audio, off by default. When enabled at the start of an utterance, its segmented audio is saved alongside the final transcript; changing the setting affects subsequent utterances. When enabled, short contextual clips for detected double-clicks and click near-misses are also stored separately under Click Gestures/<capture-uuid>. Continuous microphone audio is never retained.

Records live under ~/Library/Application Support/Froggy/Transcript Corpus/Utterances/<utterance-uuid>. Each directory contains a versioned record.json and, only when audio was saved, audio.wav. The record's optional audioFile identifies an attached clip. Undo feedback is appended separately to annotations.jsonl. Settings shows saved item count and size, with Reveal and confirmed Delete All controls. Delete All removes saved text, audio, metadata, and annotations; future final transcripts continue to save automatically. Nothing in this history is uploaded.

Diagnostics dashboard

Settings → Diagnostics shows the latest target, route, speech timing, acoustic levels, click detection result, dropped-utterance count, and OpenAI session count when available. These troubleshooting details stay out of the menubar panel.

Choose Open Diagnostics from the menu-bar panel to open the live web console. It shows the newest 500 structured events, including lifecycle state, VAD boundaries and audio duration, target Accessibility metadata, transcripts, normalization and insertion output, timings, insertion results, errors, and dropped-utterance counts.

The dashboard's Test Mode section presents the same event stream as a chronological log, including transcriptions, command lifecycle events, diagnostics, and insertion results. A fixed text input at the bottom appends an inserted record to that stream when submitted. These synthetic records are local diagnostic data; they do not inject text into another application. If a test session is active, they are also written to its events.jsonl.

Each utterance has a UUID-correlated latency trace. The dashboard summarizes recent p50/p90 latency and renders stacked bars for endpoint detection, queueing, post-endpoint ASR wait, insertion, and otherwise-unattributed overhead. Streaming traces additionally report time to first partial, feed compute, finalization, maximum worker backlog, feed and partial counts, and whether offline fallback ran. The timing event also includes capture-to-pipeline backlog, cumulative VAD compute, post-VAD work, normalization, endpoint-to-completion time, estimated last-voice-to-completion time, ASR throughput relative to audio duration, cold/warm run index, and terminal outcome. Startup timing separately reports VAD and ASR initialization. ASR and insertion emit Points of Interest signposts for inspection alongside CPU and Core ML activity in Instruments.

For microphone startup measurements, quit Froggy and launch the signed app with open -W -n -a /Applications/Froggy.app --args --audio-startup-benchmark /tmp/froggy-audio-startup.jsonl (use a new output path each time). This diagnostic replaces the normal app session, runs 20 raw-microphone trials alternating full stop/start and pause/resume after 1- and 5-second idle periods, writes setup, prepare, engine-start, and first-buffer timings, then exits. It requires existing microphone permission and saves no audio. Relaunch Froggy normally afterward. The benchmark does not change the normal microphone lifecycle.

For deliberate QA, start a Test Session from the dashboard. While it is active, Froggy writes the structured event stream to events.jsonl, and the dashboard can label any event as correct, false positive, missed intent, bad transcript, bad transform, near, or far. Corrections and notes go to feedback.jsonl. Sessions live under ~/Library/Application Support/Froggy/Test Sessions.

Utterance audio is off by default and cannot be enabled midway through a session. If Save utterance audio is explicitly selected before starting, each segmented utterance is written as a 16 kHz mono PCM WAV alongside the JSONL. Stopping the session writes final counts to manifest.json. Nothing is uploaded.

The server binds only to the stable loopback endpoint 127.0.0.1:19737. Event and test-session APIs require a random per-install bearer token persisted in local preferences; the menu puts that token in the URL fragment so it is not included in normal page requests. The stable port and token let an existing browser tab refresh normally after Froggy relaunches. If another process owns port 19737, Froggy reports the collision instead of silently choosing a different URL. Outside an explicitly started test session, diagnostic events are held in memory and disappear when the app quits. Raw audio is retained only for explicit test-session audio capture or when Save transcribed utterances is enabled and the utterance produces a non-empty final transcript.

To view a dashboard running on another Mac, forward its loopback port over SSH rather than exposing the server to the LAN:

ssh -L 19737:127.0.0.1:19737 <host>

Then open the menu-generated dashboard URL locally, preserving its #token=... fragment. If local port 19737 is already occupied, use another local port on the left side of the tunnel and replace only the browser URL's port.

Architecture

microphone
  → 16 kHz resampling
  ├→ click detector ── double click → queued Enter command
  → Silero VAD
  ├→ Parakeet Unified EN 0.6B (640 ms eager stream + live partials)
  → ordered completion queue (one active + two waiting)
  → stream finalization (offline batch on stream failure)
  → deterministic normalization
  → revalidate target → paste literal dictation

The ASR choice is not coupled to Yap. StreamingUnifiedAsrManager is configured as [70, 7, 1], FluidAudio's 640 ms efficiency tier. A dedicated FIFO worker owns the manager, so inference never blocks microphone ingestion or Silero endpoint detection. Partial revisions are surfaced while speaking; after Silero closes the utterance, the completion queue waits only for remaining stream work and final decoding. If streaming fails, UnifiedAsrManager retries the complete captured utterance with the int8 offline encoder.

There is no semantic routing layer on the dictation path. Foundation Models is used only by the optional fix engine when that engine is enabled and available.

Audio ingestion continues while an earlier utterance is being transcribed or routed. Utterances are processed in speech order; if inference falls more than two complete utterances behind, the newest utterance is explicitly reported as dropped instead of allowing memory use and latency to grow without bound.

Inert Mode keeps audio analysis, sound-gesture detection, local transcription, routing, overlays, diagnostics, and optional corpus capture running, while suppressing all text insertion, key presses, undo/replace actions, hosted fix resolution, selective OpenAI transcription, speech-coach uploads, and app handoffs. It can run without Accessibility permission and is intended for safe QA or transcript-only use.

What this spike should answer

Use it for a few real sessions and track:

• missed speech and bad VAD boundaries;
• Parakeet errors on project names and programming vocabulary;
• ambient-speech false positives;
• unintended ambient speech inserted while literal mode is active;
• time-to-insertion for literal normalized dictation;
• focus detection failures across native, Chromium, and Electron applications.

Those results should drive the next step. Activation-key+Z provides a one-keystroke undo, and opt-in test sessions provide the labeled dataset. Semantic transformation can return later as an explicit post-trigger without delaying ordinary dictation.

Repository layout

Froggy is a native-app monorepo. Implemented applications live under apps/; reusable code and language-neutral behavior contracts live under packages/. The daily-driver macOS app is in apps/mac; the constraint prototypes are in apps/ios and apps/android. Cross-platform behavior fixtures live in packages/dictation-contracts; the extension SDK and wire protocol live in packages/froggy-extension-sdk. The planned Apple speech extraction is not yet materialized because the prototypes have not proved which macOS implementation pieces should move. See docs/MOBILE.md and the coding-agent runbook in docs/MOBILE_QA.md.

Dependency

FluidAudio is pinned exactly to 0.15.6 in apps/mac/Package.swift and is licensed under Apache-2.0.

Read aloud

Select text in your app, then say "Froggy read aloud" ("read this", "read that to me", and "read it aloud" also work), or run Read Aloud from Caps Lock search. It reads the selection from the app you were using when the command started, even if the launcher has since taken focus. If no nonblank selection is available, it falls back to the clipboard. Selection lookup uses Accessibility, including browser text markers; apps that do not expose selected text use the clipboard. It never issues Copy or modifies the clipboard, and secure fields are excluded. Add an optional prompt after the command: "read aloud skip code blocks" or "read aloud translate to German". Everything after the trigger is one reading instruction, including phrases joined by "and". A prompt sends the selected or clipboard text and instruction to OpenAI's gpt-5.6-luna to prepare a spoken script before synthesis; it does not change the clipboard. Translation and other preparation failures are shown instead of falling back to the original.

Without a prompt, markdown is prepared offline: headings and list items become sentences, link and emphasis syntax disappears, bare URLs become "link to host", and code blocks and tables are announced with counts instead of being read. The script streams paragraph by paragraph through OpenAI's gpt-4o-mini-tts (voice Marin). Audio starts as the first PCM arrives, with later paragraphs fetched ahead of playback. This uses the same OpenAI key discovery as Fix. Missing keys fail visibly. If no selection is available, empty/non-text clipboards and clipboard content marked concealed also fail visibly. Froggy's restored transient clipboard remains readable.

When Show media controls is enabled, the Liquid Glass media control widget shows readings or the current macOS Now Playing session. It remains available even when the waveform overlay is disabled and when no media is selected. At rest it shows one clickable, 18-point timeline circle. Floating, the circle is a pie timeline with a tiny play/pause icon in its center. Docked in the menu bar, the circle has no play/pause glyph: it shows the source (album artwork, the menu-bar frog for readings, the app icon, or three still level bars when nothing is playing) clipped inside a solid two-point white ring matching the other menu-bar icons, and progress is drawn as a white arc around the ring over a dimmer track. The docked source icon is greyscale until the hover pane opens, and the whole control dims to about half while paused. Clicking the circle pauses or resumes the selected media; dragging it moves the overlay without toggling playback, including when media is unavailable. Progress uses adaptive neutral whites and greys throughout, including the pie slice and buffering pulse. Initial preparation uses a spinner. Once audio starts arriving, the floating circle gently pulses without a progress slice, and the docked ring pulses, while the total duration is unknown; Reduce Motion keeps them still. When synthesis finishes, the floating pie slice and the docked arc show playback progress against the complete duration.

Hover expands the glass while the clickable pie stays at the same screen position. The expanded pane is 280 scaled points wide, with the timestamp at the top right. Back, full-size play/pause, and forward are centered together, with speed immediately to their left. Floating controls put this transport row beneath the pie; the docked hover pane uses a single transport row. The app icon appears only on the expanded row below the controls, directly left of the title or reading's first sentence. The track selector appears only when more than one track is available. Clicking the artwork or title brings the selected source forward. For a scriptable browser (Chrome and its variants, Brave, Edge, Vivaldi, Opera, and Safari) Froggy asks the browser itself for its tab titles, then selects the tab whose title matches the playing media and raises its window, so a background tab comes to the front rather than whatever tab was already showing. An exact title wins over a " - " one, and a leading unread count is ignored. This needs Froggy to be allowed to control that browser under System Settings → Privacy & Security → Automation; if that is refused the widget says so. Otherwise, and for non-browser apps, it restores/raises a uniquely matching player window when Accessibility exposes one, and with no window match it activates the owner app. For Read Aloud, the artwork/title opens reading history. Collapse waits for the animation to finish. Reduce Motion disables size animation. For readings, hover reveals back/forward 10 seconds, elapsed and buffered time at the top right, and the 0.5×–2× speed menu left of the centered skips. There is no close button; Show media controls in Settings → Display or the menu-bar pane controls visibility. The speed label has no caret. The expanded pane fits the controls and title, adding height only for an error message. The controls stay expanded while the speed menu is open. Glass uses the native macOS 26 material (thin material on macOS 15), including its edge treatment. The glass effect wraps floating controls so the system can adapt foreground contrast with the surface. Docked controls retain their normal foreground and stay mounted above a separate glass background shown only while expanded. The compact menu row has no glass effect; expansion preserves the same anchored animation as the floating pane. In the expanded view, preparation keeps the spinner; until synthesis finishes, the circle stays unsliced and the timestamps show elapsed and buffered time.

The entire glass surface captures clicks. Enabled playback buttons and the speed menu show a pointing hand; disabled buttons and non-control areas use an arrow. Drag any non-control area, including the progress indicator, to move it. The title opens reading history. The overlay defaults to top center below the menu bar; it magnetizes to top center or along the menu-bar rail, with its top flush to the screen edge. Hold Shift to bypass magnets. Placement and the chosen magnet are remembered across readings and launches and re-resolved on display changes. Hover expansion preserves the saved top-center anchor even at screen edges, so play/pause never shifts under the pointer. Errors expand the panel and remain visible until dismissed.

Playback speed is saved when changed, including through voice or system media controls, and applied before the first audio of every new reading after a relaunch. Reset All Settings clears placement and restores 1× for subsequent readings. Speed changes preserve pitch; seeks clamp to downloaded audio and preserve pause and speed. Stopping or starting a new reading cancels previous preparation and synthesis.

Read Aloud History is a page in the main GUI, available from launcher search or by clicking the title below the playback controls. It lists previous readings, shows unfinished resume positions, and lets you play, pause, search, or delete a reading and its cached audio. Playback positions at zero or the end are not bookmarks; those readings play from the beginning.

Identical source text with the same reading prompt reuses one history record and resumes its saved timestamp. Different prompts (for example, different target languages) create separate readings. Source text, prompts, prepared paragraphs, completed PCM paragraphs, and resume positions are stored locally in ~/Library/Application Support/Froggy/ReadAloud. Completed audio is reused without an API call; an interrupted paragraph is regenerated when needed. Positions are saved periodically, immediately on transport changes or stop, and on normal app exit. Fully cached readings work without an API key.

Show media controls in the menu-bar pane (also in Settings → Display) is off by default. When enabled, the overlay remains visible while playing, paused, or idle—even with no media selected. Empty controls show “No media playing” and cannot send playback actions. There is no inactivity timeout. Turning the toggle off hides the overlay without stopping playback or clearing session/history state; playback commands and new media cannot override it. Stop ends the current reading and preserves its history, while the enabled controls remain available.

Froggy also becomes macOS's Now Playing session, so media keys, AirPods, and Control Center control the reading. "Froggy pause media" and "Froggy play media" follow the same selected source as the widget; "Froggy stop reading" (also "stop talking", "be quiet") ends it. "Froggy skip ahead" ("next paragraph") and "Froggy skip back" ("previous paragraph") keep their paragraph jumps, as do next/previous track. "seek reading 10", "seek reading -10", and "reading speed 1.5" expose the overlay's transport through the same command system. Seeking uses seconds of source audio at every speed. System skip controls also use 10 seconds and publish the current playback speed.

Playback uses a private output engine and ends after the last paragraph. Froggy does not duck or barge in on its own speech: ambient listening can hear playback like other speaker audio, so use a hot-word or Fn command to interrupt.

Keep Awake and status icons

Three commands control idle sleep on this Mac, from speech, GUI search, or the status icon's menu:

• keep awake keeps the Mac and display awake indefinitely.
• keep awake 1hr keeps them awake for one hour. Spoken keep awake one hour and keep awake for one hour run the same command. Repeating it starts a fresh hour; keep awake removes the timer.
• allow sleep releases Froggy's request so normal idle sleep can resume. It does not immediately put the Mac to sleep or override other apps' requests.

Show status icons, in the menubar panel and Settings → Display, defaults off. When enabled, it shows a solid white sun while Keep Awake is active. The row disappears when Keep Awake is off or its timer expires. Start Keep Awake through voice or GUI search to show it again. Hover shows indefinite status or the one-hour session's end time. Clicking or right-clicking opens exactly the three actions above, with the current state checked. Failures appear in the ordinary command result, and on the icon and its tooltip while the row is visible.

The row remembers its own display and position, uses native glass while floating, and becomes transparent when docked in the menu bar. It shares the media controls' menu-bar and top-center magnets and snaps beside docked media controls or waveform overlay. Hold Shift to bypass magnets. Hiding the row does not change Keep Awake; Reset All Settings clears its visibility and placement. Quitting Froggy releases the request, and relaunch starts with Keep Awake off. Explicit sleep and closing a laptop lid still follow macOS behavior.

General media controls

The playback widget also follows macOS's selected Now Playing session, including media in background browser tabs. Its compact row is one clickable circle: play/pause plus pie progress when floating, the source icon inside a progress ring when docked; the expanded controls show one track list, with compact album artwork (or the app icon), title, and artist for each controllable app session plus Read Aloud. The selected control target is highlighted in that same list. All items retain full opacity and artwork color, including paused items. Longer lists scroll. Each row has an Open externally icon on the right that opens its source without changing the selected control target; Read Aloud opens Reading History. Choosing a track changes what the widget and media commands control without starting or stopping playback. The choice stays selected until it disappears or a reading is explicitly started. The list selects media sessions, not songs in an app’s upcoming queue. When docked, hover opens a separate pane below the compact source-and-progress row. Timing, skips, reading speed/stop, artwork/title, and the track list stay in that lower pane, leaving neighboring menu-bar items accessible. Its rounded menu-style surface meets the bottom of the menu bar, with extra padding below the list. The menu bar item opens a native macOS menu with the system’s rounded border, glass, and shadow. Moving into the pane or using its menus keeps it open. Floating controls retain their existing expansion transition. macOS's advertised capabilities determine skip behavior: ±10 seconds when timed skipping or seeking is supported, otherwise next/previous track when available. Symbols and tooltips distinguish the two; unsupported directions are disabled. Skips are verified against updated position or track metadata, and a timed skip preserves paused playback. Unknown duration stays indeterminate; external media does not show reading's buffering, speed, or stop controls. Play/pause still uses distinct, verified commands.

A preparing or playing reading takes priority. Pausing keeps it selected, so Play resumes that reading. Starting another app's media while the reading is paused hands control to that app; pausing the external media keeps that source selected. Resuming a reading through history takes priority again. Finishing or stopping a reading reveals the system session, without starting any other audio. When there is no system session, a completed reading remains available while Show media controls is enabled. These automatic rules apply until a track is explicitly chosen in the picker.

Media metadata and small artwork thumbnails are read locally through a bounded native helper hosted by system Perl, using the private MediaRemote framework, roughly once a second. The picker exposes one controllable session per app; macOS does not reliably route separate controls to every tab or player within an app. No browser scripting permissions are needed and no media metadata is uploaded, with one exception: for YouTube in a browser (detected from a window titled "

Extensions

Froggy runs locally authored Swift extensions from ~/.froggy/extensions/<id>/. Each is a Swift package with a froggy-extension.json manifest that declares its capabilities: ID, title, summary, spoken aliases, whether it takes the text after its trigger, and effects. Capabilities join the same registry as built-in commands, so they are found by Caps Lock search, run by voice with the hot word, and can chain a tail like any other command. Every command takes at most one argument, the text after its trigger; the extension parses it itself.

Froggy ships a small library of ready-made extensions inside the app. Say "Froggy install messages extension" to install one by name, or say "Froggy install extension" (or pick Install Extension from Caps Lock search) to choose from a list in the panel. Froggy copies the package into the extensions directory, builds it, and publishes its commands. The install is a directory of symlinks into the bundled library rather than a copy, so a Froggy update is live on the next load with nothing to migrate, and on a development Mac whose SDK link points at the repository, editing a library package under packages/froggy-extension-sdk/Library rebuilds the installed extension in place. To customize a library extension, remove the link and copy the package in. Installing over an existing package is refused so your edits are never overwritten. froggy-ext library and froggy-ext install <id> do the same from the terminal (built from apps/mac as the froggy-ext product, also bundled at /Applications/Froggy.app/Contents/MacOS/froggy-ext).

To have a coding agent make an extension, give it the bundled authoring skill at ~/.froggy/sdk/Skills/froggy-extensions/SKILL.md. Install that skill into an agent's skill directory with froggy-ext install-skill <directory>; for example, froggy-ext install-skill ~/.codex/skills. This adds a symlink and refuses to overwrite an existing skill. Then ask the agent to make and install the extension you want.

froggy-ext new <id> creates a starter in the ignored draft directory ~/.froggy/extensions/.draft-<id>/. The agent edits the manifest and sources, then runs froggy-ext publish <id> to validate the manifest and atomically rename the finished package into ~/.froggy/extensions/<id>/. Publishing refuses to overwrite an existing extension. Placing a complete package there manually works too: this is a trusted native-code directory, and placement authorizes compilation and handshake execution, not invocation of its commands.

Froggy discovers the package automatically, builds it, and registers its commands. froggy-ext wait <id> waits up to three minutes for that exact source revision to be active in the running app and emits JSON; failures exit nonzero with diagnostics. If the sources change while waiting, rerun it for the new revision. froggy-ext status reads app-owned live JSON without building; stale or missing app status is an error. Say “Froggy extension status” to see builds, errors, and ready commands in Froggy's panel. These status views explicitly distinguish a previous build still in use from a successful update.

Edit installed packages directly for updates. Froggy watches the extensions directory and rebuilds edited packages with the Swift toolchain, so the next invocation uses the new code without restarting Froggy. The SDK ships inside Froggy.app; Froggy links ~/.froggy/sdk to it at launch and packages depend on ../../sdk. A Froggy update rebuilds extensions against its SDK; one that no longer compiles shows the compiler error and goes unavailable in search, or keeps its previous build in use when it has one.

Each invocation starts the extension's executable, speaks JSON-RPC over its stdin and stdout, and ends the process afterwards. A crash or hang fails only that command, bounded by a two-minute limit, and ordinary commands keep working. Extensions can report progress and log; extension diagnostics appear under the extensions category in the dashboard. froggy-ext check <id> and froggy-ext list build and report state from the terminal; those independent CLI builds do not prove the app loaded the package. Use wait for that.

The library ships memory, messages, processes (Kill Process: "Froggy kill Chrome" quits the application whole, a name matching one process signals it, and anything else shows the table by CPU and memory with Quit and Force Quit), chrome (Search Chrome Tabs: "Froggy chrome tab gmail" switches to the one matching tab or lists the matches; needs Automation access to Google Chrome, which macOS asks for once), and colors (Pick Color samples a pixel with the system loupe and copies its hex; Picked Colors lists the last fifty and copies hex, RGB, or HSL).

The t3code extension creates inquiries and searches T3 Code servers on this Mac and its registered Froggy remotes. Install with “Froggy install t3code extension”. No separate configuration is needed: new inquiries read T3’s project/global model and runtime defaults on each command. New inquiries open one composer with a project picker, message, and optional model override. The picker includes projects from this Mac and registered Froggy remotes, labeled by machine and path. Spoken project/model/message values prefill the pane; duplicate project names across servers require an explicit selection. The current desktop project is not exposed through this adapter’s API. “Froggy t3code new thread with astra saying teach me about moon phases” prefills that composer; Create and send creates the thread on the selected project’s server and submits the inquiry. Typed commands can use a colon instead of “saying”, and in PROJECT before the delimiter selects another project. “Froggy t3code find recent thread about moon phases saying elaborate on the moon and tides” searches active thread titles and message content, then sends the follow-up through the machine that owns the selected thread. Parallel search results show machine, project, date, and excerpt. Several matches, a reached result limit, or an unavailable machine open a chooser; incomplete coverage is visible. A unique match is used directly only when every machine responded. Omit the follow-up to find a thread without sending anything. “Froggy t3code open thread TITLE-OR-ID” resolves a thread without sending. t3 code is also accepted in every command.

Remote search uses Froggy’s existing registered SSH remotes, not T3’s desktop connection catalog. Each remote Mac needs its native T3 installation and local server, plus the helper installed by the extension package’s install-mesh-helper.sh (rerun after extension updates). No separate server list or model configuration is needed. The helper keeps T3 credentials on that Mac; queries, results, and follow-ups travel over SSH. Results deduplicate by T3 environment and thread identity. A machine without a helper or running server is reported unavailable. The composer retains its message when a read-only project/model check fails. Creation and submission are never retried automatically.

T3 receives messages by thread ID through its own protocol, without clipboard insertion or simulated Return. Froggy brings the native T3 Code app forward and names the destination, but does not navigate its window to that thread; select it in T3 if needed. “Submitted” means T3 acknowledged the command, not that its model finished. An unacknowledged submission is never retried automatically. New inquiries use the project checkout and T3’s effective model and runtime permissions. The adapter does not create new worktrees or read the removed defaultThreadEnvMode setting. The extension discovers ~/.t3/userdata/server-runtime.json, verifies the local environment identity, and uses T3's bundled auth session issue CLI. Its bearer session expires after 30 days and is stored with owner-only permissions in the extension data directory; expired or revoked sessions reconnect automatically before any thread mutation. Command history records thread and command identities and acknowledgements, without copying prompts, excerpts, or credentials.

The host offers five operations. Clipboard read returns the current clipboard classified into text, web links, file URLs, and a PNG of an image up to 8 MB, refusing concealed password-manager content. Present shows a view (below). Contacts search looks names up through Froggy's Contacts permission. Model complete runs one strict structured-output call through Froggy's OpenAI key: the extension sends instructions, input, and a JSON schema and gets the parsed value back, so extensions never carry credentials or choose a provider. Color sample shows the system color loupe from Froggy's own process and returns sRGB components, so an extension needs no application context of its own. Each extension also gets a data directory at ~/Library/Application Support/Froggy/Extensions/<id>/, created before every invocation and handed over in the handshake, for anything it keeps between runs; it survives rebuilds and uninstalls.

An extension can ask Froggy to show a form, list, or detail while its command runs. Froggy renders it in the Caps Lock panel with its own keyboard conventions: Return runs the default action, Escape cancels, arrows move a list selection, and the panel closes when the extension answers. Dismissing the view cancels the command and ends its process. A view may wait up to ten minutes for you; the two-minute execution limit does not run while it is open.

The library's Messages extension is the example: after installing it, say "Froggy send message to Mom saying I'm running late". A name is looked up in Contacts through Froggy, which asks for Contacts access once; several matching people or several numbers open a list to choose from, and a phone number or email is used as spoken. The message is handed to Messages by AppleScript with the handle and body passed as arguments. Say "Froggy message" or leave out a part and the prefilled form opens instead. The result reads "Handed to Messages", not "sent": AppleScript cannot confirm delivery. It needs Messages signed in to iMessage and macOS Automation permission for the extension, which macOS asks for on first use and may ask again after a rebuild. SMS is not attempted; it would need Text Message Forwarding from an iPhone. Set FROGGY_MESSAGES_MOCK_OUTBOX to a file path to test the command without sending anything.

The library's Memory extension is a save-for-later bank. Copy a link, file, image, or text and say "Froggy save this", optionally with context and tags: "Froggy save this about rate limiting, tags swift and networking". A link or path spoken in the command is saved instead of the clipboard. Speech tags follow tag, tags, or hashtag; typed #swift works too. URL fragments are preserved when parsing tags. Links and files are bookmarks: the original app owns them, and moving or deleting an external file can break its bookmark. Clipboard text is saved in full, including whitespace; image copies live in the bank's blobs/ directory under relative filenames, so moving or restoring the whole bank preserves its images. Preview and model limits never truncate saved originals. A link's page title is fetched with a four-second request timeout. With nothing to save, a form asks for a link, path, or text.

"Froggy recall" shows one memory at random, "Froggy memories" lists them all with a filter, and "Froggy find memories about X" first searches every record's title, tags, context, reference, and full saved text locally, matching words without case or accent distinctions. With five or more local matches, Froggy's configured model ranks the top 40 of them using bounded excerpts. With fewer, the model searches the whole bank by meaning, newest 400 first, so a question that shares no words with a memory can still find it. Local matches are never removed by the model's answer; missing credentials or a provider failure fall back to local ordering. Every result view says what was searched and whether the model took part. Search does not fetch linked pages or read referenced files.

Views offer Open, Copy, and Forget; Open hands links, files, and saved images to the system, Copy restores saved content to the clipboard, and Forget permanently removes the record and its owned image, never an external file. These interactive capabilities declare clipboard mutation, foreground disruption, and destructive effects; Find also declares provider use.

The bank is one JSON-lines snapshot at ~/Library/Application Support/Froggy/Extensions/memory/memories.jsonl, readable by any tool or agent. Writers must hold an exclusive flock on the adjacent .lock file for the entire read/modify/atomic-replace operation; never append to the index or replace the lock file. The extension uses the same lock for reads. Every update replaces the complete index atomically. A line that does not decode, fails validation, or repeats an id is moved to memories.quarantine.jsonl beside the index the next time the bank is read, and every command that follows says how many lines it moved, so a bad hand edit never hides the rest of the bank and the line stays available for repair. An index that cannot be read at all still fails visibly and blocks writes. Image records use a blob filename; external bookmarks use reference.

Clipboard history in command search

Open with Caps Lock → clip → Return. Clipboard mode shares the same command panel and keyboard-input session. Escape or the Back button returns to plain command search, even when the clipboard search has text in it. Clicking outside the command GUI dismisses it and leaves focus on the app you clicked.

History records text, rich text, images, links, emails, colors, and grouped files from every application on this Mac while Froggy runs, even when its command GUI is closed. It also captures the current clipboard on startup. macOS does not expose older copies made before recording began. Search by content, source, or renamed title; image text recognition runs on-device. The preview pane shows content and metadata. Return pastes the selection through the shared command executor into the captured safe text target; ⌘Return copies it and ⇧⌘Return pastes plain text. Inert Mode suppresses paste effects. ⌘P cycles type filters, ⌘. pins/unpins, ⌘X deletes, and ⌘K opens additional actions including rename, edit, image-text copy, and recording controls. ⌘, opens clipboard settings.

History lives in ~/Library/Application Support/Froggy/Clipboard, defaults to 90 days, and is capped at 1,000 unpinned entries / 200 MB. Pinned entries do not expire; copies over 20 MB are skipped. Confidential/transient clipboard types, excluded password-manager apps, and Froggy's temporary insertion clipboard are ignored. Settings controls retention, excluded apps, recording, and plain-text preference. There is no clipboard network sync or remote preview fetching.

Optional development updates

When an SSH update endpoint is explicitly configured at build time, Froggy checks it at launch, after system wake, display wake, or session unlock (with retries while the connection recovers), and every 15 minutes while running. A newer published build adds a 9.8-point white circle centered over the frog in either menubar appearance, with a transparent ! cut through both the circle and the frog stroke. The menubar panel includes Check for Updates and, when needed, Update now, plus the installed build's date and time and the last successful check time. Updating asks Endofrog to install its latest signed bundle and restart Froggy; checking alone never installs. SSH to the configured host must work without an interactive password prompt. Unconfigured source builds never check for updates.

Endofrog publishes each successful signed build before attempting installation, so a sleeping laptop can catch up after waking. “Latest” means the latest build Endofrog successfully published, including dirty changes, rather than unbuilt source edits. A unique build ID distinguishes builds that share a visible hash. Failed checks preserve a known update, show a retry message, and leave Froggy usable. The update service retains the latest and previous bundles on the build host.

Command-line control

Every registered command is automatically available through froggy, including commands added or reloaded by extensions. The CLI reads the running app's capability registry and submits structured invocations to its shared command bus and executor. There is no separate CLI allowlist or implementation.

On launch, Froggy installs ~/.local/bin/froggy as a symlink to its bundled Contents/MacOS/froggyctl executable and refreshes that link if the app moves. Keep ~/.local/bin on your shell's PATH. Existing files or links to unrelated commands are left alone, and installation conflicts are shown in Froggy. The distinct internal filename avoids colliding with the GUI's Froggy executable on case-insensitive filesystems. The updated app must be running in your logged-in session; the CLI does not launch another app or microphone.

froggy commands
froggy search reading
froggy describe reading.speed
froggy invoke reading.speed --args '{"text":"1.5"}' --wait
froggy targets
froggy invoke text.insert --args '{"text":"hello world"}' --target app:com.apple.TextEdit --wait
froggy applications Chrome
froggy files invoice
froggy clipboard invoice
froggy readings invoice

commands and search include unavailable capabilities and their reasons. describe returns parameter names, required flags, enumerated values where applicable, effects, required context, and target requirements. Arguments are a JSON object of strings and are validated by the same schema and registration used for execution. Text remains data; it is not parsed as a spoken command or executed as shell code. Complex command-specific text formats are described by the capability; file commands take a file-search entry serialized as the text string. Clipboard paste accepts its entry ID as text and optional "plain":"true". Reading resume accepts a saved reading ID as text. Clipboard and reading searches return up to 100 local matches.

Commands requiring an application or focused context require --target: frontmost, pid:1234, or app:com.apple.TextEdit. targets lists running applications with PID selectors. The app captures that target when it receives the request and applies the existing target, secure-field, and effect checks when executing. It does not silently substitute whichever app becomes focused later. A command may still need the GUI for a form, chooser, or clarification; the normal Froggy presenter handles those interactions.

invoke normally returns a queued or running execution with its UUID. The CLI also writes the UUID to stderr before sending, so a lost connection leaves an identity to investigate. Supply --id UUID to choose it yourself. Reusing an ID still retained by the host is rejected, not re-executed.

froggy status UUID
froggy wait UUID --timeout 120
froggy cancel UUID
froggy events --limit 100

Responses are JSON. --wait on invoke waits for the complete execution stack; wait does the same for an existing ID. Exit status is 0 for successful requests, 1 for errors, failed or cancelled executions, and 2 for a wait timeout. A timeout or interrupted CLI does not cancel the command. cancel requests cancellation of that stack only; inspect the eventual terminal state to see what happened. Completion describes Froggy's observed execution, including any handoff, and does not prove delegated work or inserted text was independently verified.

Live results include command summaries and content blocks and are held only in memory: at most 128 active CLI executions and 256 retained execution records. Result bodies are limited to 1 MiB per execution; oversized bodies are omitted with contentOmitted: "size_limit" and a shortened summary. Old completed records are evicted as new work arrives; restarting Froggy clears all live results. Unknown/evicted IDs do not imply success or failure. Use the durable event journal for retained lifecycle metadata; do not automatically retry an invocation after a lost response. Reading CLI results does not persist their content in the journal.

The local Unix socket lives at ~/.froggy/control/command.sock, in a private user-owned directory. Both ends verify the peer's user ID. One running Froggy instance owns the endpoint; failure to start it is visible in Froggy and leaves ordinary controls available. This is local control, not a new remote mesh protocol.

Durable execution history

Froggy records runtime-managed commands and insertions in a local append-only SQLite journal at ~/Library/Application Support/Froggy/EventJournal.sqlite. This includes ordinary dictation, literal insertion, explicit insert commands, clipboard paste, Fix edits, and undo. Records connect command/insertion identities to their execution stack, origin, target, ordering, and observed outcome. Extension commands receive the same lifecycle records. The undo shortcut now uses the registered text.undo command, checks the captured insertion identity, and records which utterance it undid. “Undo last insertion” is also available from voice and the launcher.

Read a page without launching the GUI or microphone:

/Applications/Froggy.app/Contents/MacOS/Froggy --events --limit 100
/Applications/Froggy.app/Contents/MacOS/Froggy --events --after 'CURSOR' --follow
/Applications/Froggy.app/Contents/MacOS/Froggy --event-artifact UUID > output.bin

Each history stdout line is a JSON page with events, cursor, and hasMore. The one-shot command returns at most --limit records (default 500, maximum 1000); pass its cursor to fetch the next page. Follow drains existing history and then polls committed records every 200 ms until interrupted. Save a page's cursor after consuming it to resume later. --event-artifact writes only the selected artifact's bytes. Both commands accept --journal PATH to inspect a copy. Invalid cursors, absent artifacts, and storage errors produce JSON on stderr and exit status 1. A journal is created on the first runtime execution; reading before it exists reports an error.

History reads never execute commands. Incomplete work after a crash has an unknown outcome. A completed delegation means the handoff returned; successful insertion means the adapter returned, not that Froggy proved the final text. Cancelled or failed actions may have partial effects. Dismissed extension commands are recorded as cancelled.

Settings → Execution history → Include inserted text in execution history defaults off. Enable it to record the actual text of successful dictation, explicit insertion, and Fix edits. Consent must be on at both start and completion. Text up to 4 KiB is inline; larger text up to 8 MiB becomes an artifact. Larger text is omitted with a reason. Turning the setting off stops future capture; existing history remains. This is separate from Froggy's saved transcript corpus. Clipboard data, command arguments, arbitrary errors, and result bodies are not automatically copied into the journal.

Extensions can explicitly record typed content through the SDK:

let receipt = try await host.recordEvent("search.completed", schemaVersion: 1,
    payload: ["count": .number(3)],
    artifacts: [.init(name: "results.json", mediaType: "application/json", data: resultsData)])

Froggy supplies the producer namespace, invocation/stack links, identity, sequence, and causal link. receipt.id and receipt.cursor acknowledge a committed event. Extension payloads must deliberately omit credentials and unfiltered arguments/results; the inserted-text setting does not govern explicit extension events. Use the event's own schema version for its payload. Large content belongs in artifacts: up to eight, totaling 8 MiB per event, with a 64 KiB event-envelope limit. Artifact bytes and their event commit together; Froggy never fetches a producer-supplied path or URL. Losing a reply does not prove the event was absent, so callers should not blindly retry recording.

Storage failures are visible and do not disable ordinary controls; events during an outage may be missing. An explicit extension recording request throws if it cannot return a durable receipt. Invalid extension content is rejected without reporting a storage outage or advancing its causal history. History currently has no automatic pruning. Journal read commands never submit work.

Decay links

A decay link is a named command with its arguments saved on this Mac. It stays available in voice and launcher search until 30 days after its last successful use, then moves to the searchable archive. Successful use restarts the clock; searching, failed commands, and Inert Mode do not. Nothing is deleted on expiry. Saved decay links use a bent alias-arrow icon in command search to distinguish them from ordinary commands.

• Select or copy a complete web URL, then say “Froggy make this a decay link called Foo.” Say “Froggy Foo” to run it later. Selection takes precedence over the clipboard.
• “Froggy open URL https://example.com” opens a web URL in your default browser.
• “Froggy create decay link” opens a form for the name, any single registered command with arguments, and the number of days without use (default 30).
• “Froggy decay link archive” opens a searchable list of expired links. Restore restarts the clock without executing the saved command.

Names must be unique, including archived names, and cannot already resolve to a command. Links cannot wrap other decay commands. Saved commands retain their original effect and target requirements. URLs currently support HTTP and HTTPS. The local ~/Library/Application Support/Froggy/DecayLinks.json file stores names, command IDs, arguments, and decay dates; these contents are not automatically copied into execution history.

Text expansion snippets

Run Snippets (snippets, manage snippets, or search snippets) to search, edit, import, export, and configure your library. Create Snippet (create snippet, add snippet, or new snippet) opens a child pane in the command GUI with Name, Trigger, multiline Text, and a Submit button. Opening these commands and moving between their child panes keeps the existing GUI window and keyboard focus in place. The Snippets library uses the same native searchable list and child forms for edits and expansion options. Back/Escape returns to the parent pane or root search; dismissed forms never save. Validation errors keep the draft in the form. The panes share the clipboard browser’s glass surface and header back arrow; The snippet editor is a conversational form that reads as one sentence: “Create snippet Foo that inserts Bar when I type Baz”. Words and blanks flow and wrap together like prose in heavier, larger type; each blank is an underlined inline input whose placeholder is a single word. Other forms keep their existing styling. Say or type that sentence to prefill the editor; review the captured text and trigger, then Submit to save. A blank trigger still creates a launcher-only snippet. There is no separate snippets window. Both are registered commands available from voice, launcher search, and the CLI (snippets.manage and snippets.create). Snippets are not part of Settings.

The library stores named plain-text snippets with optional, case-sensitive triggers. Any trigger is allowed: ;ae, @@, myemail, or another sequence of up to 128 characters; no special prefix is required. Names and nonempty triggers must be unique. Text can contain multiple lines and up to 65,000 characters. Empty triggers make launcher-only snippets.

Automatic expansion works independently of voice capture and composer. Choose Immediately (the default), After delimiter (keep delimiter), or After delimiter (discard delimiter). Delimiters are spaces and punctuation; Return and Tab retain their submit/navigation behavior. Expand within words is off by default. Immediate mode expands a shorter trigger before a longer trigger beginning with it; the Snippets pane flags this overlap. Use delimiter mode for overlapping triggers. Disabling automatic expansion preserves manual insertion through launcher search, snippet <name>, and the registered snippet.saved.<id> CLI capabilities. Snippet text is inserted verbatim.

Import/export uses Raycast's JSON array of name, text, and optional keyword objects (format and behavior reference). Imports append atomically and reject duplicate names/triggers. This first version supports plain text; Raycast dynamic placeholders remain literal text. Snippets are stored locally in ~/Library/Application Support/Froggy/Snippets.json.

Expansion uses the existing global keyboard tap and Accessibility permission. It replaces only a verified trigger immediately before an unselected caret; apps must expose that text and caret through Accessibility. Unsupported fields, password fields, Secure Keyboard Entry, focus/navigation changes, and stale matches are left alone. Continuing to type before a queued expansion starts cancels that expansion and preserves the typed characters. Froggy never buffers or suppresses ordinary typing. Use launcher insertion when a custom editor cannot expose its text. Turn off matching Raycast expansions when using Froggy for the same triggers, to avoid two expanders acting on one shortcut.

Expansions and manual snippet insertions run through the shared effect gate and durable execution journal. Ordinary typing is never persisted. Snippet content uses the existing Include inserted text in execution history opt-in.

License

Froggy's original source is available under the MIT License. Dependencies and third-party model assets retain their respective licenses. The MIT license does not establish the provenance or redistribution rights of training data or bundled model weights.

Opening an agent GUI sends an optional preparation hint through the journaled agent.<id>.prepare command. Repeated visibility updates and resident reconnects do not send it. Agents advertising guiOpened can prepare context before the next message; the receiving agent controls whether and how it prepares context. Preparation respects Inert Mode and secure input and does not submit a message.

Resident-agent status diagnostics are adapter-defined: Froggy accepts arbitrary reasonCode strings and ignores unused provider connection fields. Execution state and destination identities remain strictly validated. Agent names and hot words come from each installed manifest.