Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
457 changes: 457 additions & 0 deletions CHANGELOG.md

Large diffs are not rendered by default.

Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file modified dist/webbrain-zim-xapian-v0.95-corresponding-source.zip
Binary file not shown.
4 changes: 2 additions & 2 deletions docs/accessibility-tree-and-refs.md
Original file line number Diff line number Diff line change
Expand Up @@ -233,8 +233,8 @@ When an embedded app or form remains difficult to inspect or target reliably,
`promote_iframe` navigates the **current run tab** to that child frame's own
standalone URL. Subsequent tools operate on the standalone page, and normal
browser Back history is preserved so `go_back` or the browser Back button can
return to the embedding page. This is a same-tab handoff, not a background
`new_tab`.
return to the embedding page. This is a same-tab handoff, not background-tab
creation.

Use this workflow **before editing** the iframe:

Expand Down
1 change: 0 additions & 1 deletion docs/agent-tools.md
Original file line number Diff line number Diff line change
Expand Up @@ -80,7 +80,6 @@ Legend: **Yes** = available · **-** = not available · **C** = Chrome only ·
| `press_keys` | No | Yes | Yes | Yes | - |
| `navigate` | No | Yes | Yes | Yes | - |
| `wait_for_element` | No | Yes | Yes | Yes | - |
| `new_tab` | No | Yes | Yes | Yes | - |
| `promote_iframe` | No | No | Yes | Yes | - |
| `scratchpad_write` | No | Yes | Yes | Yes | - |
| `progress_update` | No | Yes | Yes | Yes | - |
Expand Down
13 changes: 10 additions & 3 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -331,7 +331,7 @@ trace and diagnostic exports as privacy-sensitive data.
|---|---|---|
| `get_accessibility_tree`, `click_ax`, `type_ax`, `set_field`, `hover` | content script message | Injected page context |
| `click`, `type_text`, `press_keys`, `scroll`, `read_page`, etc. | content script message | Injected page context |
| `navigate`, `new_tab`, `go_back`, `go_forward` | `chrome.tabs` / `browser.tabs` API | Background script |
| `navigate`, `go_back`, `go_forward` | `chrome.tabs` / `browser.tabs` API | Background script |
| `fetch_url`, `research_url`, `list_downloads`, etc. | `network-tools.js` | Service worker |
| Enabled skill tools | `skills.js` registry + `executeHttpSkillTool()` | Service worker |
| `list_webmcp_tools`, `execute_webmcp_tool` | experimental CDP `WebMCP` domain | Chrome service worker + page-registered callback |
Expand All @@ -347,16 +347,19 @@ trace and diagnostic exports as privacy-sensitive data.
| `execute_js` | bounded CDP `Runtime.evaluate` (Chrome) / content script (Firefox) | Dev-only page JavaScript |
| `read_console`, `inspect_network_requests` | mode-scoped bounded CDP Runtime/Log/Network buffers | Chrome Dev-only diagnostics |
| `inspect_event_listeners` | permission-gated content target marker + CDP `DOMDebugger.getEventListeners` | Chrome Dev-only listener diagnosis |
| `read_email_verification_message` | service worker + bounded accessibility reads | Mid/Full only after the OTP skill is active; directly scopes verified already-open message routes, requires Act/Dev plus mailbox-host click permission to open an inbox item, and completes or rejects bounded message continuations |
| `get_shadow_dom`, `shadow_dom_query`, `get_frames` | content/CDP helpers | Full Act advanced fallbacks; also added to Mid in Dev mode |

Browser-tab creation, enumeration, activation, and run retargeting are not general model-callable capabilities. To inspect another URL, the agent uses an available URL reader; to interact with it, it navigates the current run tab. Explicit separate-tab requests are surfaced as a limitation rather than silently converted into current-tab navigation. The only private-tab exception is the single OTP-skill-gated reader above: the runtime chooses an already-open supported mailbox without exposing the tab catalog, and any message-opening helper is inactive and disposable. Internal research/helper tabs and normal page-authored `target=_blank` behavior remain separate infrastructure.

Chrome CSS patch records include the top-level `documentId` and a patch-specific CSS marker. Full navigation clears persisted records, and `remove_injected_css` checks the live document before calling `removeCSS`, preventing an old patch ID from removing equivalent CSS on a replacement page. If navigation races either identity check during injection, WebBrain removes that patch's exact uniquely marked CSS from the replacement document before discarding its record. Chrome `execute_js` passes a 15-second timeout to CDP. Dev diagnostic event handlers are registered before either agent-loop variant starts and own their debugger session across turns, so ordinary run cleanup preserves their bounded buffers. Leaving the panel-wide Dev mode drains every tab in the CDP client's active-diagnostics registry, removes the handlers and buffers, and sends `Runtime.disable`, `Log.disable`, and `Network.disable` so Chrome also stops domain-level diagnostic work; conversation and tab cleanup additionally detach the debugger.

### Step 6a: Skills and Dynamic Tool Exposure

Settings -> Skills stores enabled skills in `customSkills` (`chrome.storage.local`
or `browser.storage.local`). On startup, `background.js` loads packaged default
skills from `skills/*`, adds any missing default (currently FreeSkillz.xyz, the
prompt-only email verification-code helper, and Humanizer), and refreshes an
email verification-code helper, and Humanizer), and refreshes an
existing
built-in skill record when the packaged copy changes. If the user removes a
default skill, its removal tombstone prevents it from being silently re-added;
Expand Down Expand Up @@ -394,6 +397,10 @@ new default IDs can still be migrated into existing installations.
compatible schemas to `getToolsForMode(...)` at LLM-call time, respecting
mode, tier, and site adapter. Download-job tools remain
hidden in Ask and require their normal permission gate in action modes.
The bundled OTP helper is a narrower built-in exception rather than a declared
network tool: once that exact skill is active on Mid/Full, the runtime appends
one fixed browser-neutral schema. Compact never receives it, and imported
skills cannot claim its reserved name.

Loading is idempotent and multiple relevant skills can be active in one run.
The loader's trusted instruction permits activation only for the user's request
Expand Down Expand Up @@ -452,7 +459,7 @@ tracks as a successful video or hand ffmpeg work to the user.

| User intent | Expected skill | Catalog modes | Notes |
| --- | --- | --- | --- |
| Find, read, copy, or enter a code visible in browser email/message content | OTP / verification-code helper | Ask, Act, Dev | Prompt-only; after loading it guides existing page tools. |
| Find, read, copy, or enter a code visible in browser email/message content | OTP / verification-code helper | Ask, Act, Dev | Guides narrow current-page reads; on Mid/Full, loading it also exposes one fixed reader for an already-open signed-in supported webmail tab. |
| Create and use a temporary mailbox for an unimportant signup | Disposable email (Mail.tm) | Act, Dev | Not shown to Ask. It may overlap with OTP during a verification flow, so both can be loaded. |
| Read a YouTube transcript, fetch a blocked NYTimes article, or resolve/download supported public media | FreeSkillz.xyz | Ask, Act, Dev | Ask can load the skill but still cannot see its Act-only `download_public_media` tool. |
| Draft or rewrite an email reply, message, or post the user will send | Humanizer | Ask, Act, Dev | Prompt-only; preactivated on webmail adapters and on the explicit Humanize selected-text shortcut, otherwise routed by catalog. Returns final text only. |
Expand Down
4 changes: 2 additions & 2 deletions docs/claude-chrome-comparison.md
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,7 @@ get_interactive_elements, click, type_text, press_keys, scroll, navigate,
go_back, go_forward, extract_data, inspect_element_styles, wait_for_element,
inject_css, remove_injected_css, patch_element, revert_patch, execute_js,
read_console, inspect_network_requests, inspect_event_listeners,
highlight_element, wait_for_stable, schedule_resume, schedule_task, get_selection, find_text, new_tab,
highlight_element, wait_for_stable, schedule_resume, schedule_task, get_selection, find_text,
promote_iframe,
done, clarify, get_shadow_dom, shadow_dom_query, get_frames, iframe_read,
iframe_click, iframe_type, fetch_url, research_url, list_downloads,
Expand All @@ -95,7 +95,7 @@ core surface, including Dev-only `execute_js`, is shared.
|---|---|
| AX-first DOM control | `get_accessibility_tree`, `click_ax`, `type_ax`, `set_field`, `set_checked`, `hover`, `drag_drop` |
| Legacy DOM fallback | `get_interactive_elements`, `click`, `type_text`, `press_keys`, `scroll`, `wait_for_element`, `wait_for_stable` |
| Navigation and tabs | `navigate`, `go_back`, `go_forward`, `new_tab`, `promote_iframe` |
| Navigation | `navigate`, `go_back`, `go_forward`, `promote_iframe` |
| Reading/extraction | `read_page`, `read_pdf`, `read_page_source`, `extract_data`, `inspect_element_styles`, `get_selection`, `find_text` |
| Dev editing and diagnostics | `inject_css`, `remove_injected_css`, `patch_element`, `revert_patch`, `execute_js`, `read_console`, `inspect_network_requests`, `inspect_event_listeners`, `highlight_element` |
| Shadow DOM and frames | `get_shadow_dom`, `shadow_dom_query` on Chrome, `get_frames`, `iframe_read`, `iframe_click`, `iframe_type`, `promote_iframe` |
Expand Down
4 changes: 2 additions & 2 deletions docs/fr/accessibility-tree-and-refs.md
Original file line number Diff line number Diff line change
Expand Up @@ -186,8 +186,8 @@ Lorsqu'une application ou un formulaire intégré reste difficile à inspecter o
courant** vers l'URL autonome de cette frame enfant. Les outils suivants agissent
sur la page autonome, et l'historique Retour normal du navigateur est conservé :
`go_back` ou le bouton Retour permet de revenir à la page qui contenait
l'iframe. Il s'agit d'un transfert dans le même onglet, et non d'un `new_tab` en
arrière-plan.
l'iframe. Il s'agit d'un transfert dans le même onglet, et non de la création
d'un onglet en arrière-plan.

Utilisez ce flux de travail **avant toute modification** dans l'iframe :

Expand Down
1 change: 0 additions & 1 deletion docs/fr/agent-tools.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,6 @@ uniquement · **Dev** = module Dev (fournisseurs Mid/Full ; pas Compact).
| `press_keys` | Non | Oui | Oui | Oui | - |
| `navigate` | Non | Oui | Oui | Oui | - |
| `wait_for_element` | Non | Oui | Oui | Oui | - |
| `new_tab` | Non | Oui | Oui | Oui | - |
| `promote_iframe` | Non | Non | Oui | Oui | - |
| `scratchpad_write` | Non | Oui | Oui | Oui | - |
| `progress_update` | Non | Oui | Oui | Oui | - |
Expand Down
2 changes: 1 addition & 1 deletion docs/fr/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -185,7 +185,7 @@ while (steps < maxSteps) {
|---|---|---|
| `get_accessibility_tree`, `click_ax`, `type_ax`, `set_field`, `hover` | message de script de contenu | Contexte de page injecté |
| `click`, `type_text`, `press_keys`, `scroll`, `read_page`, etc. | message de script de contenu | Contexte de page injecté |
| `navigate`, `new_tab`, `go_back`, `go_forward` | API `chrome.tabs` / `browser.tabs` | Script d'arrière-plan |
| `navigate`, `go_back`, `go_forward` | API `chrome.tabs` / `browser.tabs` | Script d'arrière-plan |
| `fetch_url`, `research_url`, `list_downloads`, etc. | `network-tools.js` | Service worker |
| Outils de compétence activés | Registre `skills.js` + `executeHttpSkillTool()` | Service worker |
| `done` | agent.js — capture une capture d'écran de vérification + sonde d'état de page | Service worker + CDP |
Expand Down
4 changes: 2 additions & 2 deletions docs/fr/claude-chrome-comparison.md
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,7 @@ get_interactive_elements, click, type_text, press_keys, scroll, navigate,
go_back, go_forward, extract_data, inspect_element_styles, wait_for_element,
inject_css, remove_injected_css, patch_element, revert_patch, execute_js,
read_console, inspect_network_requests, inspect_event_listeners,
highlight_element, wait_for_stable, schedule_resume, schedule_task, get_selection, find_text, new_tab,
highlight_element, wait_for_stable, schedule_resume, schedule_task, get_selection, find_text,
promote_iframe,
done, clarify, get_shadow_dom, shadow_dom_query, get_frames, iframe_read,
iframe_click, iframe_type, fetch_url, research_url, list_downloads,
Expand All @@ -97,7 +97,7 @@ de la surface de base, y compris `execute_js` réservé au mode Dev, est partag
|---|---|
| Contrôle DOM AX prioritaire | `get_accessibility_tree`, `click_ax`, `type_ax`, `set_field`, `set_checked`, `hover`, `drag_drop` |
| Repli DOM hérité | `get_interactive_elements`, `click`, `type_text`, `press_keys`, `scroll`, `wait_for_element`, `wait_for_stable` |
| Navigation et onglets | `navigate`, `go_back`, `go_forward`, `new_tab`, `promote_iframe` |
| Navigation | `navigate`, `go_back`, `go_forward`, `promote_iframe` |
| Lecture/extraction | `read_page`, `read_pdf`, `read_page_source`, `extract_data`, `inspect_element_styles`, `get_selection`, `find_text` |
| Édition et diagnostic Dev | `inject_css`, `remove_injected_css`, `patch_element`, `revert_patch`, `execute_js`, `read_console`, `inspect_network_requests`, `inspect_event_listeners`, `highlight_element` |
| Shadow DOM et cadres | `get_shadow_dom`, `shadow_dom_query` sur Chrome, `get_frames`, `iframe_read`, `iframe_click`, `iframe_type`, `promote_iframe` |
Expand Down
18 changes: 13 additions & 5 deletions docs/fr/skills.md
Original file line number Diff line number Diff line change
Expand Up @@ -93,11 +93,19 @@ d'article sans identifiants.

#### Assistant OTP / code de vérification

Ne se charge que pour les demandes pertinentes et ne déclare aucun outil réseau.
Sur l'onglet de l'exécution active, il privilégie le texte sélectionné ou un
sous-arbre borné de l'arbre d'accessibilité, retient le code de service pertinent
le plus récent, exclut l'accès aux SMS et aux applications natives, et respecte
la gestion stricte des secrets.
Ne se charge que pour les demandes pertinentes et ne déclare aucun outil réseau
externe. En Mid et Full, il ajoute un lecteur interne limité pour un onglet de
webmail compatible, déjà ouvert et connecté. `inspect` ne modifie pas la boîte
et reste disponible dans Ask. Comme l'ouverture peut marquer le message comme
lu, `open_message` exige Act/Dev ainsi que l'autorisation de clic pour l'hôte de
la boîte ; une copie temporaire inactive est alors créée, toutes les
continuations bornées du message sont lues (sinon l'opération échoue de manière
fermée), puis la copie est fermée. Le modèle reçoit des références de message opaques, jamais le
catalogue des onglets, l'URL de la boîte ni les références d'accessibilité.
Compact ne reçoit aucun outil de compétence ou inter-onglets. Sur l'onglet de
l'exécution active, il privilégie toujours le texte sélectionné ou un sous-arbre
borné, exclut l'accès aux SMS et aux applications natives, et respecte la
gestion stricte des secrets.

Lorsqu'il est utilisé, le contenu de page délimité et le code sont inclus dans la
requête normale envoyée au fournisseur LLM que vous avez configuré. Si
Expand Down
28 changes: 20 additions & 8 deletions docs/privacy-and-data-flow.md
Original file line number Diff line number Diff line change
Expand Up @@ -429,14 +429,26 @@ stop this data flow entirely.

The "OTP / verification-code helper (email)" skill
(`skills/otp-verification-code-helper.md`) is explicitly Ask/Act compatible,
is prompt-only, and declares no external tool or endpoint. It guides WebBrain's
existing page-reading tools to
prefer selected text or a bounded, message-scoped accessibility-tree subtree on
the active run tab when finding a recent, service-matching code. It cannot list
or switch to background tabs, read SMS, phone notifications, native apps, or
another device, and it forbids private mailbox APIs or sign-in bypasses. The
skill itself creates no additional network request. When the user asks WebBrain
to read a code, however, the scoped page content and extracted code are included
and declares no external endpoint. It guides WebBrain's existing page-reading
tools to prefer selected text or a bounded, message-scoped accessibility-tree
subtree on the active run tab. On Mid/Full, and only after that exact skill is
active, one fixed internal tool may inspect an already-open signed-in supported
webmail tab. The runtime enumerates tabs locally but returns only a provider and
bounded service-matching previews with opaque message references; disclosure
requires the full normalized service identity or all sufficiently discriminative
service tokens. It does not send the tab catalog, mailbox URL, or accessibility
references to the model. A provider-verified already-open message is scoped and
read directly across the supported providers. If an inbox candidate must be
opened, the operation is unavailable in Ask because opening can mark the email
read. In Act/Dev it receives the normal click permission for the mailbox host,
clones the mailbox URL into a temporary inactive tab, consumes every exact bounded
message-scoped continuation or fails closed without paginating the wider mailbox,
and closes the clone.
That clone can make ordinary authenticated requests to the same webmail origin;
the helper does not call a mailbox API or external skill endpoint. Compact has
no such tool. The skill cannot read SMS, phone notifications, native apps, or
another device, and it forbids sign-in bypasses. When the user asks WebBrain to
read a code, the scoped page content and extracted code are included
in the normal request to the user's configured LLM provider as part of the
current conversation. When Record traces is enabled, the raw page-reading tool
result and model response are also retained locally in the `webbrain_traces`
Expand Down
2 changes: 1 addition & 1 deletion docs/security-model.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ Differences below.)

| Permission | Risk | Mitigation |
|---|---|---|
| `<all_urls>` | Content script injection anywhere — the agent can read and interact with any page the user visits | The user must explicitly switch to an action mode (Act or Dev) before clicks/types/navigation. Ask mode is read-only. The agent never auto-activates on new tabs. |
| `<all_urls>` | Content script injection anywhere — the agent can read and interact with any page the user visits | The user must explicitly switch to an action mode (Act or Dev) before clicks/types/navigation. Ask mode is read-only. General browser-tab creation, enumeration, activation, and run retargeting are not exposed as model-callable tools. The one OTP-skill-gated Mid/Full reader internally selects an already-open supported mailbox and returns only bounded service-matching content; it exposes no tab catalog. Candidate inspection is read-only, while opening a selected message requires Act/Dev and mailbox-host click permission because it may mark mail read. Its disposable inactive clone consumes exact message continuations to completion or fails closed. |
| `debugger` | CDP access provides trusted events and full DOM/network control on any tab | The debugger is attached only to the target tab while a CDP-backed run or mode-scoped Dev diagnostics own it. Run cleanup closes run-scoped WebMCP state and detaches unless Dev diagnostics remain active; conversation cleanup and tab removal drain all owners and detach. |
| `webRequest` | Can observe XHR/fetch metadata for requests made by the active page | API mutation observer is off by default; when enabled, it keeps only a bounded in-memory per-tab buffer for repeated-click shortcut hints and opaque same-origin replay. |
| `downloads` | Can save files to the user's Downloads folder without prompting | Only the agent's explicit download-capable tool calls (`download_files`, `download_file`, `download_resource_from_page`, `download_social_media`, download-job skill tools) use this, and each is gated by the capability × origin permission prompt. |
Expand Down
18 changes: 14 additions & 4 deletions docs/skills.md
Original file line number Diff line number Diff line change
Expand Up @@ -81,10 +81,20 @@ fallback.

#### OTP / verification-code helper

Loads only for relevant requests and declares no network tool. On the active run
tab it prefers selected text or a bounded accessibility-tree subtree, matches
the newest relevant service code, excludes SMS/native-app access, and honors
Strict secret handling.
Loads only for relevant requests and declares no external network tool. On Mid
and Full it adds one narrow internal reader for an already open, signed-in
supported webmail tab. A provider-verified already-open message is read directly.
Candidate discovery otherwise leaves the mailbox untouched; candidate disclosure
requires the full normalized service identity or all sufficiently discriminative
service tokens. `inspect` remains read-only in Ask. Because opening can mark an
email read, `open_message` requires Act/Dev and normal click permission for the
mailbox host; the selected inbox item opens only in a temporary inactive duplicate
that is closed after every bounded message-scoped continuation is consumed.
Incomplete, changed, or unscoped continuations fail closed. The model receives opaque message
references, not a tab catalog, mailbox URLs, or accessibility references. Compact
keeps no skill or cross-tab tool surface. On the active run tab the skill still
prefers selected text or a bounded accessibility-tree subtree, excludes
SMS/native-app access, and honors Strict secret handling.

When used, the scoped page content and the code are included in the normal
request to your configured LLM provider. If **Record traces** is enabled, raw
Expand Down
Loading
Loading