Repository navigation
fix(talk): Gemini 3.8 Live over gateway-relay: model contracts, cancellation recovery, consult transcript ordering - #1
Closed
saariuslystoned wants to merge 3 commits into
Closed
saariuslystoned wants to merge 3 commits into
saariuslystoned wants to merge 3 commits into
Conversation
`gemini-3.8-live-extended-thinking` closes the session (1007) on function response scheduling and abandons a consult after a `willContinue` interim, so it now gets NON_BLOCKING tools with one final result per call and no interim, plus `thinkingLevel` low|medium|high. `gemini-3.8-live` keeps the async contract but gets no thinking config, which it rejects at setup. Extended Thinking also gains a real barge-in: an empty client-content turn interrupts generation, which the server confirms with `interrupted` and `turnComplete`. The model predicates and thinking-config builder move to a sibling module so the provider stays within its line cap. Verified against the live API on 2026-09-19 with a headless probe; see the PR. Related: openclaw#150204 openclaw#150205 openclaw#137006 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… confirm a cancellation Gemini Live has no response.cancel. Its bridge only interrupts through input audio or, on Extended Thinking, a client turn, and reports the response done at its own turnComplete. The relay waited one second for that confirmation and then closed the whole session, so on Google every stop or barge-in ended the call (openclaw#150205). When the drain deadline passes, the relay now completes the cancellation locally: the drain releases microphone audio and tool results, the stall is logged with the client reason, and the interrupted generation's remaining audio, transcripts, and tool calls are discarded until the provider reports it done or a replacement response starts (bounded at 30 s). The next microphone frame re-arms a turn for the following reply. The deadline lives in a sibling module and its tests in a sibling file to respect the relay line caps. Closes openclaw#150205 Related: openclaw#137006 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… is admitted A gateway-relay consult pre-persists its prompt as a keyed user turn and the embedded run adopts it when it starts. An assistant voice final persisted in that window moves the session's current-turn anchor and the run fails with "Session transcript keyed user is outside the current turn" (openclaw#150204). On Gemini 3.8 the interim "checking" response made this nearly systematic. The relay now holds assistant finals' session appends from consult admission until the consult settles and appends them in order afterwards; holds nest across overlapping consults and release on close. Clients, echo tracking, confirmations, and spoken run control still see every transcript immediately, and user finals are not held. Closes openclaw#150204 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
saariuslystoned
marked this pull request as ready for review
September 19, 2026 02:33
Owner
Author
|
Superseded by the upstream PRs openclaw#152413, openclaw#152427 and openclaw#152428, which carry the same three commits split per fix with updated evidence. |
saariuslystoned
pushed a commit
that referenced
this pull request
Sep 26, 2026
…agent in chat (openclaw#158120) ## What Problem This Solves Fixes: owners who ask their agent in chat to change a key, a config value, or one of their own skills get refused, sent to a dashboard, or told to file a Workshop proposal. Examples: "you can't post API keys here", switching the embeddings provider routed to the web-search wizard, only proposals for handwritten skills. ## User Impact An owner can hand the agent an API key or token in chat, ask it to change config such as the embeddings provider, and have it edit skills they own. Session permission modes are unchanged: Full Access applies, restricted sessions ask. - **Keys from chat.** Masked setup flows still keep keys out of model context and remain the default. If the user already pasted a key or token, the agent stores it in the shared secret store and points the config key at it with a `store` SecretRef instead of refusing. It never echoes the value back. The pasted message already reached the model provider and transcript; redaction covers later logs and output only, and the docs say so. - **Existing store entries are never touched.** Each save inserts a new entry named after the config key plus a random suffix (`GATEWAY_REMOTE_TOKEN_9B139B5E231299BC`). Nothing is overwritten, revived, or deleted, and a new name can never match anything already pointing into the store, including a stale reference to a removed and purged entry. Replacing a key leaves its previous entry for `openclaw secrets store rm`. Rotating a key keeps its configured store provider alias, and the audit records the alias actually used. - **Embeddings.** The agent now treats the memory embeddings provider, model, and key as `memory.search.*` config, not the web-search setup wizard. - **Skills.** When the user asks, the agent edits skills they own directly: repository skill source, workspace `skills/`, project `.agents/skills/`, and configured extra skill directories. Bundled, ClawHub-installed, and plugin-provided skills are replaced by their owners' updates. For those, the agent says so and offers to capture the change as a Workshop skill. - **Tone.** The "never request / paste credentials in chat" lines are gone from the `openclaw` tools and system-agent prompts. "Never echo secret values" stays, and so do factual pointers for flows that genuinely need a UI: channel sign-in, provider OAuth/accounts, and model onboarding. No config option, schema, or protocol change. The Full Access permission-policy floor from the first revision moved to openclaw#158142 for its own security review. ## Why This Change Was Made After openclaw#149870, approved config writes may target any path. What still blocked owners was model-facing text telling the agent to refuse credentials, plus the missing ability to store a chat-provided value anywhere but plaintext config. `config_set_ref` gains an optional `secret` argument (read without trimming; only emptiness is checked). With it, the system agent: 1. registers the value for redaction when the proposal is built; 2. keeps the key's existing store provider alias when it has one; 3. sends one `secrets.writeForConfigRef` command to the SQLite state worker with the requester's live-authority guard. The host re-checks that guard at the worker's transaction and commit admission (`createSqliteWorkerWriteAdmission`), so a run stopped while the command is queued writes nothing. The transaction inserts a new row under a freshly minted `NAME_<16 random hex>`; 4. writes the ref through the existing config writer, which re-checks authority. If that write fails (before or after the writer commits), OpenClaw rereads the config and the error says the key was saved as `<NAME>` and whether the config key points at it. There is no automatic delete: another consumer may have linked the fresh entry, or the writer may have committed before failing; 5. the normal config reload picks up the new ref, since its id always changes. Nothing new runs SQLite on the Gateway main thread. <details> <summary>Out of scope / follow-ups</summary> - Found while proving this: in Full Access, after a delegated change applies, the next agent turn in the same chat fails with `SQLite database already belongs to another worker backend`. It reproduces on unmodified `origin/main` (`71bb516`) with a `logging.level` change followed by one more message. This PR does not fix it. - Built-in provider sign-in and model onboarding stay handoffs; they own live verification of the active inference route. - Other secret-store set/delete paths remain synchronous migration debt, as `worker-access.md` already records. </details> ## Evidence Real Telegram Test Server (Convex-leased userbot, fresh Gateway, QA mock provider, Full Access, tester is owner), first revision: | | Screenshot | |---|---| | Token given in chat, applied with no approval prompt and no refusal (synthetic QA token) |  | ### Final effects at this head qa-channel scenario `system-agent-owner-trust` passes through a real Gateway and state worker. The Gateway is seeded with an unrelated `GATEWAY_REMOTE_TOKEN` entry, then: 1. A command-allowed **non-owner** (`bob`) sends the key. The `openclaw` tool is owner-only. 2. The **owner** (`alice`, Full Access) sends it. Captured step details (redacted by the Gateway; the store ref id prints as `__OPENCLAW_REDACTED__`): ```json { "nonOwnerEntryNames": ["GATEWAY_REMOTE_TOKEN"], "storedRef": { "source": "store", "provider": "default", "id": "__OPENCLAW_REDACTED__" }, "storeEntryNames": ["GATEWAY_REMOTE_TOKEN", "GATEWAY_REMOTE_TOKEN_9B139B5E231299BC"] } ``` - After the non-owner turn: only the seeded entry exists and `gateway.remote.token` is unset. - After the owner turn: `gateway.remote.token` is a `store` SecretRef, the token is in its own minted entry (`GATEWAY_REMOTE_TOKEN_9B139B5E231299BC`), and the seeded entry's `updatedAt`/`updatedBy` are unchanged. No approval prompt was posted, and the token is absent from chat, config, and the store listing. **Revoked request**, through the production worker (Node main thread, real broker, `writeSecretStoreEntryForConfigRef`). The requester's guard passes the caller's check, then reports the run stopped: ```text seeded: [ 'GATEWAY_REMOTE_TOKEN (cli)' ] revoked request rejected: requesting run is no longer active after revoked request: [ 'GATEWAY_REMOTE_TOKEN (cli)' ] owner request saved as: GATEWAY_REMOTE_TOKEN_F1657B971F691824 after owner request: [ 'GATEWAY_REMOTE_TOKEN (cli)', 'GATEWAY_REMOTE_TOKEN_F1657B971F691824 (openclaw)' ] seeded value intact: true ``` Tests (each fails without the behavior it covers): - production worker path (`secret-store-config-ref.worker.test.ts`, forked database-worker lane with the real broker): a chat secret gets its own minted entry beside a live `GATEWAY_REMOTE_TOKEN` without touching it; a requester revoked after the caller's check writes nothing; - store kernel: a refusal at commit admission rolls the transaction back; each save mints a new `NAME_<hex>` and leaves the key's previous entry unchanged; a stale name whose entry was removed and purged still resolves to nothing after a chat save; - operations: stored and referenced with no value in output or audit; authority gone before the store write writes nothing; a failed config write names the saved entry and leaves it in place; rotating a key keeps its configured store provider alias, and the audit records it; - tool: proposes a store write without repeating the key, preserving leading and trailing whitespace. Measured single-worker wall time per new or materially changed test file at this head (`node scripts/run-vitest.mjs run <file>`, local M-series; vitest Duration includes import and setup): | File | Tests | Wall | Vitest duration | |---|---:|---:|---:| | `src/secrets/store/secret-store-config-ref.worker.test.ts` (new, database-worker lane) | 2 | 14 s | 2.05 s | | `src/secrets/store/secret-store.test.ts` | 32 | 16 s | 13.37 s | | `src/system-agent/operations.test.ts` | 43 | 18 s | 15.30 s | | `src/agents/tools/system-agent-tool.test.ts` | 36 | 15 s | 12.89 s | QA scenarios: `system-agent-owner-trust` (mock-openai) runs in about 27 s after build; `skill-owner-direct-edit-live` is live-frontier only and took about 3 min with `claude-cli/claude-sonnet-4-6`. Wording pins for the removed lecture text were deleted. The focused store, worker, exclusivity, operations, tool, approval, and delegate suites pass. `node scripts/check-changed.mjs` passes every gate except core lint, which fails only on three files this PR does not touch (`server-chat-metadata-lifecycle.integration.test.ts`, `session-companion-ask.ts`, `app-sidebar-session-list-render.ts` over `max-lines` on the base); oxlint on the changed files is clean. Security decision: a Full Access owner's pasted key goes to the Gateway-wide team store without a separate approval. Maintainer (@obviyus) accepted this in the PR conversation. **Rotation with a second consumer**, through the production worker (Node main thread, real broker). A second consumer references the key's first entry before the next save lands; value fingerprints only: ```text owner saves key #1 -> MODELS_PROVIDERS_OPENAI_API_KEY_6D96F6E92B59E935 (sha256:4a5c5a4aa8de) second consumer now references MODELS_PROVIDERS_OPENAI_API_KEY_6D96F6E92B59E935 (e.g. linked while the next save is queued) owner saves key openclaw#2 -> MODELS_PROVIDERS_OPENAI_API_KEY_4066F18ABAAE6903 (sha256:28bc4e3fe10d) second consumer's entry MODELS_PROVIDERS_OPENAI_API_KEY_6D96F6E92B59E935 after rotation: sha256:4a5c5a4aa8de unchanged: true ``` **Config write fails after the save, with a second consumer on the fresh entry**, through the production worker and the system-agent apply path (fingerprints only): ```text owner result: Saved the secret as GATEWAY_REMOTE_TOKEN_6B68C031E571F81C, but could not point gateway.remote.token at it: config write failed after commit (rollbackStatus: not-restored). Retry, or remove the entry with `openclaw secrets store rm GATEWAY_REMOTE_TOKEN_6B68C031E571F81C`. config gateway.remote.token: null second consumer's entry GATEWAY_REMOTE_TOKEN_6B68C031E571F81C: sha256:09ae5b4fd36b second consumer keeps the credential: true ``` **Stale reference to a removed and purged entry**, production worker for purge and save: ```text purged rows: 1 stale ref GATEWAY_REMOTE_TOKEN after purge: SECRET_STORE_NOT_FOUND chat save -> GATEWAY_REMOTE_TOKEN_8F698B5396990ADD resolves (value hidden) stale ref GATEWAY_REMOTE_TOKEN after chat save: SECRET_STORE_NOT_FOUND ``` **Owned-skill edit with a live model.** New scenario `skill-owner-direct-edit-live` (live-frontier; run with `claude-cli/claude-sonnet-4-6`, subscription auth) passes at this head. It seeds workspace skill `qa-owner-greeting` replying `OWNER-GREETING-V1`, and the owner asks in plain words: "Please change my qa-owner-greeting skill so it replies OWNER-GREETING-V2 instead of OWNER-GREETING-V1." Captured result: Skill file after the turn: ```markdown --- name: qa-owner-greeting description: Greets the owner with a fixed marker --- When the user asks for the owner greeting, reply with exactly: OWNER-GREETING-V2 ``` Agent reply: "Let me find the skill file. Done. The `qa-owner-greeting` skill now replies `OWNER-GREETING-V2` instead of `OWNER-GREETING-V1`." The model edited the skill file in place and confirmed it; it did not refuse or file a Workshop proposal. Co-authored-by: Ayaan Zaidi <hi@obviy.us>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What Problem This Solves
Talk over gateway-relay cannot use Gemini 3.8 Live today:
gemini-3.8-live-extended-thinking: the first agent consult closes the Live session with1007 Function response scheduling is not supported for this model; a configuredthinkingLevelis ignored.gemini-3.8-live: any thinking setting fails at setup with1007 Thinking level is not supported for this model.Session transcript keyed user is outside the current turn([Bug]: Talk on gemini-3.8-live: agent consult fails with "Session transcript keyed user is outside the current turn" (voice transcript lands before run adoption) openclaw/openclaw#150204).User Impact
Operators can point
talk.realtime.modelatgemini-3.8-live-extended-thinking(orgemini-3.8-live) and get working spoken agent consults, the configured reasoning depth, stops and interruptions that keep the call alive, and long sessions. Providers that confirm cancellations (OpenAI) are unchanged.Why This Change Was Made
Three focused commits, each meant to stand alone as an upstream PR:
realtime-voice-model-contract.ts: Extended Thinking gets NON_BLOCKING tools, noscheduling, no interim "working" response,thinkingLevellow|medium|high, and a realhandleBargeIn(an empty client-content turn, which the server confirms withinterrupted+turnComplete);gemini-3.8-livekeeps the async contract and gets no thinking config.Live protocol facts behind commit 1 were measured against the real API on 2026-09-19 with a headless probe (raw event logs kept): scheduling closes the session; a
willContinueinterim is accepted but the model abandons the call and never speaks a later final; one final response works even after a 25 s silent wait;activityStartwith automatic VAD (the approach in openclaw#137006) is silently ignored; the empty client turn interrupts cleanly.Evidence
Unit:
vitest run extensions/google/realtime-voice-provider.test.ts extensions/google/realtime-voice-provider.gemini38.test.ts src/gateway/talk→ 34 files, 705 tests passed.tsgo:coreandtsgo:extensionsclean.check:changedlanes (conflict markers, line-cap ratchet, max-lines suppression ratchet, assertion-safety ratchet) all OK.Phone (Pixel 10 Pro Fold, official debug app built from v2026.9.3, gateway built from this branch,
gemini-3.8-live-extended-thinking, thinkingLevel high, speech played from a Mac speaker), all on one relay session unless noted:tool.call→tool.result→ spoken "The current time on the gateway machine is 10:13 PM EDT on Friday, September 18th, 2026." Nokeyed user is outside the current turn, noisError; the pre-call filler was appended after the consult.turn.cancelledat "8", next turn "Understood. Stopping the count now.", session live, no relay warnings.session.closed, no warnings). The unconfirmed-cancel recovery is covered bycancellation-recovery.test.tsand the provider interrupt by the probe; a Control UI stop-button run is still open.Probe results and the other captures: proof branch.
Full proof packet (captures with manifests, gateway log excerpts, probe event logs, PROOF.md) lives in the swarm-intercom run dir
runs/openclaw-relay-scout-20260919/.🤖 Generated with Claude Code