Remove browser tab tools and enhance OTP mailbox handling - #2964
Merged
Conversation
The step-1 benchmark still required new_tab, so a model running against the current schemas could not produce the designated answer and the prompt-mandated limitation response scored as wrong — skewing every model comparison downstream of it. - 009 and 089 now expect the unsupported-capability response: explain that tabs cannot be opened and offer current-tab navigation, matching BROWSER_TAB_LIMITATION in the prompt tiers. - 087 no longer asks which link to open in a new tab, and the 002/003/007 rubrics stop offering new_tab as an alternative path. - Scenario 006 seeded an assistant new_tab call into its replayed history; that third retry is now fetch_url on the same dead URL. Two pre-existing instances of the same drift surfaced while fixing this: golden 065 and scenarios 011/060 still named the retired screenshot tool, and scenario 030's generator entry had fallen behind its hand-edited JSON. All are corrected, so both generators are idempotent. test/run.js now asserts that every idealFirstToolCall, idealNextToolCall, and seeded tool call names a tool some browser / mode / tier actually offers, so retiring a tool cannot silently leave the benchmark unanswerable again. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The README claimed the enriched payload carries 35 function schemas. The real count depends on browser, mode, and tier — 14 for Ask, 49 for Chrome Act/full, 60 for Chrome Dev/full — so any fixed number is wrong for most invocations and goes stale on the next tool change, which is how it drifted to 35. Describe what the array holds instead. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two review findings on the tab-tool removal. Removing the built-in definitions also dropped new_tab, list_tabs, and activate_tab from AGENT_TOOL_NAMES, and RETIRED_AGENT_TOOL_NAMES never picked them up — so RESERVED_AGENT_TOOL_NAMES stopped covering them and an enabled skill manifest could re-declare all three. Confirmed in both builds before the fix. They are retired names now, which keeps the merge in getToolsForMode from contradicting BROWSER_TAB_LIMITATION. Golden 065 asked for /screenshot, not inspect_viewport. inspect_viewport supplies a read-only view to the model and captures, saves, or attaches nothing for the user, while the prompts route an explicit capture request to the /screenshot slash command. Expecting the tool call rewarded an internal inspection and scored the product-correct answer wrong. The scenario seeds that moved to inspect_viewport in the previous commit are the other case — the agent looking at the page itself — and stay as they are. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The union built one set across every browser/mode/tier, so a fixture passed whenever its tool existed on some surface. That masked Dev-only tools inside Act fixtures, which is how eleven scenarios replaying a CSP-rejected execute_js went unnoticed: execute_js is Dev-only in both builds, yet csp-blocked-eval (021-030) and scenario 020 declared mode act, seeding a call the Act surface never offers. Each fixture is now checked against its own declared mode at full tier on both browsers. Mid and Compact stay out of it on purpose: they expose fewer tools by design, so a golden naming something they drop is the prompt-size tradeoff the corpus exists to measure. Requiring the tool at full tier is what separates "this smaller tier omits it" from "we retired it and nothing can call it" — the 054/download_files case is the former and still passes at act/full. Verified the tightened check fails on both shapes the union let through: a Dev-only seed in an Act scenario, and a Dev-only idealFirstToolCall in an Act golden. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Declaring csp-blocked-eval as Dev meant --tier compact threw inside buildScenarioPayload for those eleven scenarios. The worker pool caught it per scenario so the run still finished, but they landed as verdict "error" — indistinguishable from a timeout or an HTTP 500, and counted against the model in the scoreboard. A scenario whose mode has no payload at the tier under test never reached a model, so it is neither a pass nor a failure. The runner now asks isRunnableModeTier first, sends nothing, and records verdict "skipped" with the reason. Freeze mode replays a captured prompt and ignores the tier, so it never skips. The decision lives in lib/score.mjs with the rest of the taxonomy, and the reason is persisted in the saved result, so regrade.mjs reproduces the verdict rather than reading the absent output as "empty". Skips come out of the denominators in safety-report.mjs and regrade.mjs and out of the per-category totals the runner prints. Verified against a compact run: Dev scenarios skip with no request, an Act scenario in the same run still executes, and --mode act runs the skipped ones anyway. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The shared BROWSER_TAB_LIMITATION landed in the Ask prompt directly under the line that says Ask cannot click, type, or navigate, and told the model to offer current-tab navigation anyway. Ask exposes no navigate tool in either build, so a user who accepted that offer got a promise the mode could not keep and a second handoff to make. Act and Dev keep the navigation fallback, which they can actually perform. Ask now points at the URL-reading tools it does have (fetch_url, research_url) and offers the Act handoff when the user really needs a tab opened. The prompt test now enforces the split rather than checking one shared string: Act prompts must offer current-tab navigation, the Ask prompt must not, and Ask must still expose a URL reader and no navigate — so the wording cannot drift from the tools behind it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Remove browser tab management tools
When a research escalation is active, clicking Stop only aborted the detached tab, leaving the originating run alive. Now looks up sourceTabId via researchEscalationSourceTab and passes the correct tab to agent.abort(). Test assertion updated to match.
open_message now dispatches only against a one-use grant minted by the permission gate, so a call that never passed the capability x host prompt cannot create a helper tab or click in the mailbox. The gate and the handler also derive the session key the same way, from the raw service argument, so astral characters can no longer split them across a UTF-16 slice. Gmail routes are parsed instead of pattern matched: an open message needs a thread-id-shaped tail, which keeps #inbox/p2 and #search/github/p2 out of the message branch that returned a whole listing as one message. Threads opened from a later page still resolve. Mailbox reads peek the abort flag rather than consuming it, so a stop during a read still ends the run. They also refuse to read a tab that is navigating away from the mailbox, which stops a foreign URL from becoming the host the click prompt is charged to. Failures after the helper tab opens now say the session is gone and that inspect has to run again, and provider error strings lose their accessibility refs before they reach the model. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GKsX7CYmaEp2rjeVs9EBAL
_otpEmailOpenFailure copied its input with a spread, so any field a helper result happened to carry would have gone to the model along with the error. Nothing reaches it that way today, but the results it wraps are the same shape that carries the accessibility tree, the tab record, and the live mailbox URL, and the model is not allowed to see any of those. It now builds the response from a fixed set of fields instead. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GKsX7CYmaEp2rjeVs9EBAL
Add skill-gated cross-tab OTP email reader
|
@esokullu is attempting to deploy a commit to the esokullu's projects Team on Vercel. A member of the Team first needs to authorize it. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.