Skip to content

RELEASE.md

Latest commit

 

History

History
427 lines (285 loc) · 71.7 KB

RELEASE.md

File metadata and controls

427 lines (285 loc) · 71.7 KB
Raw

Release and package verification

Related docs:

  • ../README.md
  • REQUIREMENTS.md
  • ARCHITECTURE.md
  • TOOL_CONTRACT.md
  • ELECTRON.md
  • platform-smoke.md
  • SUPPORT_MATRIX.md
  • Bounded agent_browser outcome metadata on details (resultCategory, successCategory, failureCategory, optional nextActions, optional pageChangeSummary with per-step summaries on batch): contract in TOOL_CONTRACT.md; maintainer checklists under “Tool result categories” and “Page-change summaries” in ../AGENTS.md
  • Post-success get text selector visibility (RQ-0074): optional details.selectorTextVisibility / selectorTextVisibilityAll, visible warnings, and inspect-visible-text-candidates* next actions after read-only visibility probes—SUPPORT_MATRIX.md, TOOL_CONTRACT.md, and ../AGENTS.md maintainer checklist
  • Managed-session outcomes (RQ-0077): after extension-managed implicit or fresh --session injection reaches process execution, details.managedSessionOutcome records the transition (created / replaced / unchanged / closed on success; preserved / abandoned when a plan fails before a new session becomes current). Failing sessionMode: "fresh" calls and successful replacements whose old-session close fails also append model-visible Managed session outcome: …—TOOL_CONTRACT.md, COMMAND_REFERENCE.md, SUPPORT_MATRIX.md, and ../AGENTS.md maintainer checklist
  • Stateful context commands (cookies, storage, auth, dialog, frame, state) and aggregate batch results: model-facing details.data is summarized or redacted per TOOL_CONTRACT.md; aggregate batch replaces top-level details.data with a compact per-step matrix (success, argv-redacted command, redacted result or scrubbed error) while full per-step payloads, artifacts, and categories remain on batchSteps[]—operational notes in COMMAND_REFERENCE.md, assembly in extensions/agent-browser/lib/results/presentation/batch.ts

Purpose

Provide one concrete maintainer workflow for validating repo state, package contents, and install guidance before publishing pi-agent-browser-native.

Native Windows: best-effort

The owner explicitly decided on 2026-10-22: “Make native Windows explicitly a best effort.” This supersedes #191 / PR291's restored blocking policy, not its failure evidence. Native Windows is not a first-class required PR, scheduled-default, or release qualification target. Known process-lock and restored Electron cleanup failures remain unresolved; no runtime repair is claimed.

Routine Pi compatibility retains one latest-official-host Windows build/import/process-transport smoke with job-level continue-on-error: true. Its failed steps, TAP and artifacts remain failed evidence, not a pass inferred from overall workflow success. It does not run check:compat, full unit/serial or consumer suites, build the corrected checker, or invoke models. Broader official/fork native Windows qualification is available only from manual Pi compatibility dispatch with windows-qualification=true; those job-level best-effort lanes retain their existing environment self-check, full contracts, Git/npm consumers and failed artifacts. Release callers cannot opt into these deep lanes. Missing or skipped evidence is not passed.

Required macOS/Linux, latest official/fork host/API/consumer qualification, Native lifecycle Linux, quality, Node floors, assertion/deadline/generation/ownership/security contracts remain unchanged. Local release/prepublish and default platform doctor/run select configured required macOS/Ubuntu targets only; optional Windows/Parallels setup cannot block that path. Explicit windows-native commands remain strict and nonzero on failure. Branch-protection/settings changes are separate and are not made by this PR.

Windows contract-test children enable only NODE_DEBUG=pi-agent-browser-policy inside the repository-owned scripts/project.mjs launcher used by npm test and the default/quality verification gates. This assignment happens after the pinned qualifier creates its isolated base environment; setting NODE_DEBUG or NODE_OPTIONS only on the outer workflow would be stripped and is not evidence that diagnostics ran. Build/preparation stages and ordinary runtime consumers do not enable the channel. The npm test gate retains complete parallel TAP and single-worker serial TAP stages, ordered and fail-blocking. Failed Windows jobs print at most 400 exact PI-AGENT-BROWSER-POLICY <pid>: stderr matches (including TAP-wrapped child stderr), report total/shown/omitted counts, and retain the complete qualification.log alongside the unchanged bounded failed-TAP diagnostics. Neither collection nor debug enablement changes coordination behavior, retries, test selection, or budgets.

Current 0.6.16 rollout waiver

This heading retains historical links. The owner's 2026-09-21 “waive windows” instruction made Windows qualification nonblocking for that rollout, even with available runners. It did not establish a Windows pass. The later #191 restoration replaced that CI exception at the time. The current best-effort support decision above now supersedes that restoration policy; neither decision erases failures. Do not reuse the September waiver as current authority. Known policy-lock/restored Electron failures and bounded repeat results remain recorded in the support matrix.

Linux/macOS, declared Node floors, official/fork host and consumer-artifact checks, Native lifecycle Linux, and the composed release/prepublishOnly scripts remain required. Retain all failed attempts. A hosted Windows contract pass is not a native browser dogfood or complete release-matrix pass.

Docs/workflow-only changes may reuse prior live/platform evidence when the complete compiled runtime, runtime dependencies and relevant tests are unchanged. Changed published files require a new retained tarball checked by verify-package.mjs --smoke-pi, including runtime-only SDK and actual CLI loading. Publish those exact checked bytes without repacking.

Pi 1.0 upgrade and qualification

This contract requires Pi 1.0.0. Do not install or activate it in running pre-1.0 sessions; coordinate the runtime cutover and restart participating agents first. Keep runtime activation separate from isolated checkout/SDK qualification. Preserve auth/profile/state files, explicit artifacts and Pi journals when updating or rolling back; browser mutations are not reversible through a package rollback.

Qualify the exact official and fork host graphs independently, including SDK and actual CLI loading, explicit/defaultTools/loader/initial-resume/tree/reload selections, structured success/failure/redaction and unchanged JSON. Inspect regular and fullscreen native TUI results, including failures and images. Keep the existing 250 ms import/factory and 1,000 ms lock budgets; synthetic research timings are not final-source release evidence or universal performance guarantees. Apply normal required macOS/Linux release gates; Windows is separately labeled best-effort. Host identity and actual supported APIs, not a version label alone, determine qualification.

0.7 upgrade and rollback

The 0.7 tool contract is breaking: direct argv stays compact, script moves to agent_browser_code.code with a persistent browser, the job DSL is replaced by native batch/code, and five advanced tools use flat schemas after native activation. Publish with the migration map, not a parallel legacy public route. Preserve saved auth, explicit files, and Pi journals. Existing strict pre-0.7 isolated-script leases are only read for cleanup.

Before activating the update, finish or inspect in-flight mutations/recordings and restart every cooperating Pi process onto the same package. /reload alone is not reliable for picking up rebuilt compiled code. Mixed versions do not share the new execution-ordering guarantee.

Rollback is package-level: restore the previously installed 0.6 package from your saved local copy, then fully restart participating Pi processes with their existing session files. Do not delete auth/profile/state files, artifacts, or transcripts. The older package cannot interpret all new inner-transition records; inspect the live URL and obtain fresh refs before further actions. Rollback does not reverse submitted browser actions, stop external clients, or restore JavaScript variables.

Changed-path qualification

The minimum Pi runtime is 1.0.0, matching the native APIs used by the new tool contract. Wildcard peers remain host-owned dependency declarations, not support for older runtimes.

Qualify official Pi 1.0 and the fork independently with the existing host/package/lifecycle infrastructure. Required evidence for the redesign includes compact schema and additive activation/history, native prompt sections, shared direct/code auth continuity, fresh JavaScript globals, selected real images and conservative geometry, visible handled failures/exact recovery, real cross-process helper/action and whole-cell ordering, cancellation/reaping, and ordered transition replay after interruption/branch changes. Retain native batch and advanced action/QA/Electron/source outcome coverage. Small synthetic timing comparisons are not universal performance claims or sandbox certification.

Routine GitHub PR and release-call checks include a nonblocking Windows build/import/process smoke; deep official/fork Windows qualification and its environment self-check are manual-only. Retain every failed attempt, including historical waived diagnostics, without claiming a Windows pass from an aggregate that ignored failures. These focused checks complement, rather than replace, the local release and publication gates.

Pre-release checks

From the repository root:

npm install
npm run doctor
npm run check:platform-smoke
npm run smoke:platform:ubuntu-image
npm run smoke:platform:doctor
npm run verify -- release

npm run doctor is a read-only first-run diagnostic for PATH, targeted upstream version, the minimum Pi runtime floor, and duplicate package/checkout source conflicts. The package keeps Pi core imports as wildcard peerDependencies because installed Pi package docs require the host Pi install to provide those packages, while the doctor fails setup when pi --version is below the enforced floor. It does not replace upstream agent-browser doctor for browser runtime health and does not edit Pi settings.

For PR-ready local confidence before release-only lifecycle and platform cost, run:

npm run verify -- pre-pr

pre-pr composes the default gate with npm run verify -- package: clean dist/ build, corrected-checker preparation, maintained-code scope and suppression policy, strict lint, Oxfmt, TypeScript, the full unit/fake suite with declaration-isolation probes, generated docs, live command-reference sampling, and package-content verification. CI additionally runs npm run quality:mutations to prove real acceptance-failure propagation; see CODE_QUALITY.md. It intentionally does not run lifecycle, packaged Pi smoke, Crabbox platform smoke, startup-profile, real-upstream or dogfood modes.

npm run verify -- release runs:

  1. npm run verify for generated playbook drift, TypeScript, unit/fake coverage, command-reference generated-block drift, and live command-reference verification against the targeted upstream on PATH
  2. npm run verify -- lifecycle, which launches the configured-source lifecycle harness for /reload, exact --session-id relaunch, managed-session continuity, persisted spill reachability, and native returned-failure behavior
  3. npm run verify -- package-pi, which first validates package contents via npm pack --json --dry-run and then smoke-loads the packed package in Pi isolation
  4. npm run smoke:platform:doctor and the required Crabbox matrix from platform-smoke.md: macOS SSH and Ubuntu local-container targets running fast target-local platform-build plus browser-dogfood-smoke

When using the documented, maintainer-approved local macOS alternative, run and inspect that target-local suite separately alongside the remaining release checks. The default composition still requires macOS/Ubuntu; alternate evidence must not be labeled a passing Crabbox matrix.

npm publish runs npm’s prepublishOnly script from package.json, which executes the same npm run verify -- release gate and then npm pack --dry-run. That concatenated gate is everything in the default npm run verify step (generated playbook drift, clean dist/ build, TypeScript, the unit/fake suite, generated command-reference blocks, and live upstream command-reference sampling against the targeted agent-browser on PATH), the configured-source lifecycle harness, the packaged Pi smoke in package-pi, and the release-blocking Crabbox platform matrix. Using npm publish --ignore-scripts skips that contract intentionally.

prepublishOnly intentionally does not run the standalone host-only npm run verify -- startup-profile, npm run verify -- real-upstream, npm run verify -- dogfood modes; those remain separate npm run verify modes in scripts/project.mjs. The platform matrix includes its own fast target-local build/package gate and browser dogfood suite, and is automated through the release slice.

Run the opt-in startup profiler whenever package layout, the compiled entrypoint, top-level imports, schema registration, or prompt/config startup logic changes:

npm run build
npm run verify -- startup-profile --samples 3

The profiler first clean-builds dist/, then records direct package entrypoint import/factory timing in fresh Node processes and writes .artifacts/startup-profile/latest.json before reporting any budget failure. Failed reports retain every sample and a nonzero exit. Per-sample CPU, resource, hardware, load and artifact-identity receipts follow the startup evidence contract; their diagnostic collection does not move the existing wall-clock boundaries, prewarm the extension, or change the strict 250 ms maximum over three fresh regression samples. The report includes a safety block confirming it did not launch Pi, tmux, mise, npm, browsers, or agent-browser. Full Pi TUI ready-prompt profiling is intentionally excluded because repeated real Pi/tmux launches proved too invasive for routine verification on the operator machine.

For a deterministic host-only real-browser wrapper smoke without model choice in the loop, run:

npm run verify -- dogfood

For direct Crabbox diagnostics outside the full release compose, run the required platform gate (check:platform-smoke, smoke:platform:ubuntu-image, smoke:platform:doctor, smoke:platform:all) from platform-smoke.md, then inspect provider leases:

crabbox list --provider local-container
crabbox list --provider parallels

The Crabbox gate is only green when suite assertions and artifact manifests under .artifacts/platform-smoke/ are green and no unexpected lease/clone remains.

The deterministic dogfood mode clean-builds the compiled package, uses the extension harness and real agent-browser on PATH against a deterministic loopback HTTP fixture, then verifies persistent browser code aggregation, flat advanced QA/actions, native batch, screenshot artifact verification, and explicit session close. Use npm run verify -- dogfood --keep-artifacts or --artifact-dir <path> only while debugging, then delete retained screenshots. This smoke complements, but does not replace, human-readable interactive transcript evidence.

Every release also requires interactive tmux-driven Pi dogfood with the native agent_browser tool against real sites. For extension-focused release smokes, use pi --approve --no-extensions --no-skills -e . from the trusted checkout before publish so auto-loaded dogfood/QA skills cannot replace the bounded smoke workflow; omit --approve only when the smoke is explicitly testing Pi's Project Trust prompt. Run separate skill-enabled dogfood only when validating skill routing or report-generation behavior. Drive prompts with tmux send-keys, exercise at least one simple static site and one real documentation/product site, include advanced QA, code, or native batch when those surfaces changed, close every opened browser session, remove screenshots/temp artifacts, and record the outcome in the release notes or support-matrix evidence. Do not paste raw multi-line prompts into a tmux Pi pane: plain newlines submit separate queued user messages. For scripted smoke driving, collapse prompt files to one line before sending (PROMPT=$(tr '\n' ' ' < /tmp/smoke-prompt.md); tmux send-keys -t "$SESSION":0.0 -l "$PROMPT"; tmux send-keys -t "$SESSION":0.0 Enter). For manual multi-line editing, use Pi's external editor shortcut (Ctrl+G) or configure tmux extended keys so Pi can receive Shift+Enter for newlines; see the installed Pi docs/tmux.md guidance. Automated localhost, fake-upstream, and deterministic dogfood gates do not replace this human-readable live-site transcript evidence. When code changes, add a persisted-session agent_browser_code pass that aggregates at least two real pages, exercises a conditional branch, compares selected output with an ordinary-call baseline, verifies selected screenshot bytes reach the model, and checks browser/auth continuity on a later direct/code call with fresh JS globals. Review ordered custom transition entries and interrupted/replayed state, not only the outer result. Exercise handled failures, timeout/cancellation, whole-cell ordering, and child reaping; close only test-owned browsers afterward and retain Pi transcripts. Never use --no-session for code dogfood. When agent_browser_web_search or package config changed, add one key-free smoke proving the optional tool is absent without config, one fake/unit-backed smoke in the default suite, and one opt-in live Exa or Brave Search check with a real key while confirming the key does not appear in transcripts, stdout/stderr, config status, PR text, or artifacts. When electron.* surfaces, attached-session diagnostics, or qa.attached changed, add a local Electron pass against a disposable fixture app, never the operator's installed desktop apps such as Slack or Obsidian: wrapper launches open real app windows on the operator's desktop even with isolated profiles. Install the stock electron npm package into a temp directory (run its install.js if npm skipped the binary download) next to a minimal app whose hidden window (new BrowserWindow({ show: false }), plus app.setActivationPolicy('prohibited') and app.dock?.hide() on macOS) loads a known page so nothing appears on the operator's desktop, then run electron.list (discovery only) → electron.launch with executablePath set to that package's Electron binary and appArgs set to the fixture app directory (expect isolated profile behavior) → snapshot -i or electron.probe / qa.attached → electron.cleanup with the returned launchId, verifying status/mismatch guidance if you simulate a dead renderer or stale refs. Delete the fixture afterwards. For dense-dashboard stress coverage, use the public Grafana stress checklist below; it is a maintainer workflow, not bundled product skill or recipe runtime.

When reviewing saved session JSONL after a failed smoke or QA reclassification, expect browser tool rows to carry isError: true whenever details.resultCategory is failure. The model-visible observation must also report failure. Canonical JSON stays parseable and unchanged. Final public results return isError directly, with a prose failure notice outside explicit JSON, so Pi's transcript matches the wrapper contract (TOOL_CONTRACT.md). Preserve a normal Pi session directory for those checks; avoiding --no-session keeps this evidence intact (AGENTS.md preferred validation workflow).

The configured-source lifecycle regression harness is required before release because it launches an interactive Pi 1.0.0+ process under tmux with --approve and validates /reload, full relaunch with the same exact --session-id, managed-session continuity, persisted artifacts, compiled-entrypoint pickup after process restart, and native returned-failure behavior. Branch-backed session_tree rehydration and cleanup ownership are validated by focused extension harness tests:

npm run verify -- lifecycle

Use npm run verify -- lifecycle --keep-artifacts when debugging failures, then remove retained artifacts after inspection.

Public Grafana stress checklist

Use this optional-but-recommended checklist when a release touches dashboard behavior, snapshots, refs, scroll, comboboxes, artifacts, network diagnostics, recording, or prompt guidance. It keeps the useful public Grafana dogfood target repeatable without bundling private dogfood/VFR skills or adding a reusable browser recipe layer.

Target:

https://play.grafana.org/d/nodes/linux-node-overview?var-datasource=grafanacloud-prom

Public panels may show No data. Verify the rendered dashboard and controls; this UI checklist does not certify metric ingestion.

Minimum pass:

  1. Open the URL with the native agent_browser tool in a fresh session.
  2. Run snapshot -i; confirm the output is useful on a dense dashboard, including high-value controls and bounded spill behavior when needed.
  3. Exercise one dashboard scroll path. If page-level scroll does not move visible content, confirm details.scrollNoop / next actions or equivalent guidance points to snapshot/screenshot verification and nested-scroll recovery.
  4. Exercise one explicit combobox-targeted action such as a role/name agent_browser_action on a dashboard variable. If it only focuses the field, confirm details.comboboxFocus / next actions point to snapshot -i, press ArrowDown, and press Enter when the closed-state evidence qualifies.
  5. Capture at least one screenshot artifact and verify details.artifactVerification before using the file.
  6. If ffmpeg is on PATH, run record start / visible interaction / record stop and verify the video artifact. Native 0.37 checks ffmpeg before start; older natives may instead return pending output plus details.recordingDependencyWarning. Neither is usable recording evidence. Short/cold 0.37 captures have failed on Ubuntu while explicit 12-second fixture captures encoded successfully; record capture duration and every failed attempt, and never treat an empty file as a captured frame.
  7. Inspect network requests, console, and errors summaries. Treat Grafana Play-side noise such as analytics/Sentry requests, public-demo 403s, and console errors as site noise unless the wrapper leaks secrets, hides actionable failed rows, misclassifies artifacts, or suggests unsafe follow-ups.
  8. Close the browser session and delete temporary screenshots, HARs, recordings, and scratch reports after extracting any release evidence.

Record release evidence as a short note with: date, package/checkout source, target URL, browser command families exercised, artifacts collected and cleaned up, known Grafana-side noise observed, and any product findings converted into CueLoop tasks. Do not commit private dogfood scripts, VFR harness files, raw browser profiles, HARs, videos, or .dogfood/ run output as product docs.

Public Sauce Demo checkout smoke prompt

Use this validation prompt after changing click enrichment, tab pinning, ref preflight, form-fill batching, artifact handling, recording, or prompt guidance. It is intentionally more stateful than example.com and uses a natural user-style request so the transcript shows what the agent chooses on its own. Do not mention agent_browser, snapshots, refs, batch, eval, or upstream command names in the prompt; those are evaluator expectations, not user instructions.

Run it in an isolated checkout session with skills disabled so the run validates the extension browser workflow instead of external dogfood/QA skill routing. It is fine to restrict active tools at launch so the checkout extension is the only browser surface, but keep those launch details out of the user prompt:

pi --approve --no-extensions --no-skills -e . --model openai-codex/gpt-6-astra:low --tools agent_browser,agent_browser_code,agent_browser_tools --session-dir "$SESSION_DIR"

Repeat with the task's chosen Astra reasoning level when validating instruction-following robustness. Use unique temp paths for each run and delete them afterward. Run separate skill-enabled dogfood sessions only when the thing under test is skill integration, not this bounded release smoke.

Submit the prompt as one Pi message. In tmux automation, write it to a temp file with placeholders replaced, collapse newlines to spaces, and send that one line; for manual multiline entry, use Pi's Ctrl+G external editor or a tmux setup that preserves Shift+Enter newlines. Do not paste the raw block into a tmux pane line-by-line.

Copy/paste prompt, replacing the two artifact placeholders with exact absolute paths:

Please run a bounded release smoke check on the public Sauce Demo store. This is not an exploratory bug hunt or dogfood report.

Site: https://www.saucedemo.com/
Demo credentials: standard_user / secret_sauce

Use a clean browser context, not my personal Chrome profile.

Scenario:
- Log in.
- Sort products by price low to high.
- Add at least two products to the cart.
- Open the cart.
- Start checkout with a fake name and postal code.
- Stop on the checkout overview page; do not place the order.

Please gather enough evidence to support the smoke result:
- Save a screenshot here: <ABSOLUTE_SCREENSHOT_PATH>.png
- Save a short screen recording here if recording is available: <ABSOLUTE_RECORDING_PATH>.webm
- Include the final page title/URL, the selected sort order, cart contents, item total/tax/total, and any browser-side network, console, or page-error issues you see.
- Clean up by closing the browser when finished.

Return a concise PASS/FAIL report with evidence and any tool or workflow issues you noticed. Do not create a dogfood-output report directory.

Evaluator expectations after the queued Sauce Demo fixes: the agent should independently choose efficient, safe browser operations; native add-to-cart clicks should mutate cart state without the agent authoring eval/DOM-click fallbacks (the wrapper may fail with details.clickDispatch when upstream reports click success but no trusted DOM event reached the target; that probe covers direct click calls on uniquely named button-like refs and XPath selectors, so batch rows, CSS selectors, and Sauce's repeated "Add to cart" refs are reported as unverified instead); same-snapshot form fills may be batched safely when the agent chooses that route; the selected sort order should be verified; checkout must stop before Finish and must not place the order; the agent must not attempt Finish or another likely final submit action because prompt stop-boundaries are agent responsibility rather than wrapper-enforced business-intent policy; screenshot and recording must use the requested paths or be explicitly reported unavailable, and close should be blocked with details.promptGuard.reason: "requested-artifacts-missing-before-close" until required screenshot paths are verified; network requests may show public-demo telemetry 401s; console may report offline-cache logs; errors should show no page errors; and the browser session plus temp artifacts should be cleaned up after evidence is recorded. A run that reaches checkout-complete.html or silently substitutes artifact paths is a workflow failure even if other store flow steps work. Known upstream defect: agent-browser 0.38.1 and 0.39.0 can report Sauce add-to-cart click success while delivering no trusted DOM events when the click follows an idle gap of about a second. Before calling an empty cart a wrapper regression, run the bare CLI control: log in, select lohi, sleep 1 s, click '#add-to-cart-sauce-labs-onesie', then read .shopping_cart_badge; if the bare control also leaves the cart empty, record the cart/checkout steps as blocked upstream.

What package verification checks

npm run verify -- package confirms that:

  • no repo-local .pi/extensions/agent-browser.ts autoload shim is present
  • LICENSE exists in the repo and the packed tarball
  • canonical published docs are present
  • npm pack --json --dry-run runs the package prepare build once and packs the compiled dist/extensions/agent-browser/index.js entrypoint
  • GitHub/source installs use the same prepare build; when Pi installs with npm install --omit=dev, scripts/prepare.mjs installs source-build dev dependencies with lifecycle scripts disabled before building so Pi can load the ignored compiled entrypoint from a fresh clone
  • the package-level doctor command and capability baseline are present
  • the bundled extension entrypoint and emitted runtime files are present, including the code worker and modules retained for command-line consumers and the publish contract
  • source-only, agent-only, and superseded docs are absent from the tarball

npm run verify -- package-pi runs the same package-content checks and additionally confirms that:

  • the packed package can be loaded through Pi SDK resource loading with the same isolation principle as pi --no-extensions -e <package-source>
  • direct/code/loader tools and inactive advanced tools register without requiring optional search config; explicit host selection remains honored
  • any optional companion tools remain governed by their own configuration gates
  • the registered agent_browser source resolves inside the extracted packed package path, not the working checkout
  • the packaged agent_browser tool can be executed through Pi's loaded native tool definition with a deterministic fake upstream agent-browser --version binary

The packaged execution smoke intentionally uses a temporary fake agent-browser binary and the --version inspection path. It proves first invocation of the packaged Pi tool without launching a real browser. Real browser coverage remains part of local checkout validation and post-publish install validation.

Current forbidden packed files include:

  • AGENTS.md
  • .pi/extensions/agent-browser.ts
  • TypeScript extension source and other test/repo-only maintenance files

For a full packed file listing:

npm run verify -- package --list-files

Local development validation

Before publishing, validate both local-checkout modes without mixing their assumptions.

Quick isolated checkout smoke test

  1. Install agent-browser separately.
  2. Launch pi --approve --no-extensions -e . from this trusted repository root. Omit --approve only when testing Pi's Project Trust prompt.
  3. --no-extensions disables automatic extension loading only; settings, configured package resolution, and other resource types remain active. For isolated test settings, use temporary HOME and PI_CODING_AGENT_DIR directories with PI_OFFLINE=1, providing only the model credentials the smoke needs.
  4. Confirm the checkout package loads the compiled dist/extensions/agent-browser/index.js entrypoint (run npm run build first after source edits).
  5. Run a smoke prompt that exercises agent_browser.
  6. Restart Pi after extension edits; configured-source /reload is a separate validation mode.

For expanded-surface validation, the smoke prompt should cover native tool invocation rather than shelling out to agent-browser: --version, --help, skills list, skills get core --full, open with sessionMode: "fresh", snapshot -i, click, flat agent_browser_action after enabling action (including unique accessible select and explicit session routing), eval --stdin, batch via stdin, agent_browser_code, advanced QA, and experimental source/network tools after loader activation, screenshot <path>, explicit --session … open plus --session … close, network requests, console / errors, diff snapshot, stream status plus stream disable, dashboard start plus dashboard stop, and chat <message> (credential failure is acceptable evidence of wrapper pass-through when AI_GATEWAY_API_KEY is intentionally unset). Clean up any opened browser session with close, remove temporary files, and kill the tmux session before ending validation.

This checklist assumes a real agent-browser on PATH. It complements, but does not overlap, npm run verify -- lifecycle: that harness swaps in a fake upstream binary and focuses on /reload, exact --session-id relaunch, managed-session continuity, spill-path persistence, and native returned isError and JSON-preservation semantics (scripts/verify-lifecycle.mjs), not the full command matrix above.

When a smoke or dogfood run fails after sessionMode: "fresh" (missing binary, timeout, upstream error, or qa preset reclassification), read details.managedSessionOutcome before assuming which managed session the next default sessionMode: "auto" call will follow; the same struct can appear without the extra Managed session outcome: … prose line on "auto" failures. Field-level semantics and append ordering relative to other diagnostic tails are documented in TOOL_CONTRACT.md and the session-mode notes in COMMAND_REFERENCE.md.

Configured-source lifecycle validation

Run the automated harness for deterministic configured-source lifecycle regression coverage (required before publish together with the other Pre-release checks):

npm run verify -- lifecycle

The harness creates an isolated PI_CODING_AGENT_DIR, writes settings with exactly one temporary configured package source, runs pi in tmux with --approve, default model zai/glm-5.2, and a deterministic --session-id, puts a deterministic fake agent-browser first on PATH, starts an explicit sessionMode: "fresh" managed browser, drives /reload, closes Pi, and relaunches with the same exact session id instead of typing /resume. It also asserts the JSONL session header id, same-page managed-session continuity, compiled JS code pickup after full process relaunch, persisted spill reachability, and native returned isError and JSON-preservation semantics for a QA reclassification. Per-step tmux waits default to 180000 ms (three minutes) in scripts/verify-lifecycle.mjs (DEFAULT_TIMEOUT_MS); override with --timeout-ms <ms> when slower models or cold starts need more headroom. Override the model when needed:

npm run verify -- lifecycle --model openai-codex/gpt-5.5:minimal

Combine flags in one invocation when both apply (order after lifecycle is flexible as long as each value-taking flag is immediately followed by its value):

npm run verify -- lifecycle --model openai-codex/gpt-5.5:minimal --timeout-ms 600000

It retains Pi transcripts and evidence, and removes disposable runtime inputs after success. It does not replace occasional real-browser manual smoke testing.

Lifecycle triage: page checks require the first completed tool result to report a successful expected command and observed data.url (open) or data.origin (snapshot); recovery text and remembered targets are not page evidence. A cold post-quit tab-drift remains a continuity failure: upstream restores storage, and the wrapper must reopen the recorded URL after confirming shutdown before the first current-page read. Verify both empty storage and origin storage at non-root URLs and hash-routed URLs, including after tab list or explicit HTTP reads start the daemon and in batches with non-page prefixes, plus fresh refs and main-frame scope. A blank page, an origin root, or a URL mentioned only in an error is not the remembered page. Explicit URL QA recovery does not satisfy the same-page snapshot requirement; URL reopening does not recover unsaved forms, JavaScript memory, or history. A timeout on sentinel v2 after exact-session relaunch means the new compiled entrypoint did not load after process restart. A reload-step timeout or missing post-reload snapshot often means Pi rejected reload while the TUI still showed Working… (Wait for the current response to finish before reloading), even when the session JSONL already has a final assistant message. Re-run with --keep-artifacts --verbose, inspect the retained pane capture, and confirm the configured model follows tool prompts reliably. Slower models may need a higher --timeout-ms than the 180000 ms default.

Environment and automation pitfalls

These show up often in cloud dev boxes and scripted smokes; they are maintainer notes, not product defects.

Topic What to watch for Mitigation
Pi CLI vs repo devDependencies Global pi older than the minimum Pi runtime floor for the release can change TUI behavior, /reload, package installs, and tool routing during lifecycle or checkout smokes. Run npm run doctor and align pi with the current audited baseline before release gates (pi update or install the matching version). The published peer range stays wildcard per Pi package docs, and the doctor enforces the minimum Pi runtime floor before package validation.
npm lockfile (packageManager) package.json pins npm@12.2.0 for the Node 24.21.0 runtime baseline. Older npm versions can rewrite optional-platform metadata in package-lock.json; private registry proxies can also leak organization URLs into resolved entries. Use the npm version declared in packageManager when refreshing the lockfile; do not commit unrelated lockfile churn or WorkOS/private-registry URLs.
pi -p / print mode Non-interactive pi -p may hang or emit no stdout for long real-browser smokes without a TTY. Use tmux-driven interactive pi for release evidence and checkout smokes; reserve -p for short, non-browser checks.
Real-browser cleanup real-upstream, Sauce Demo, and live-site runs can leave defunct Chrome/agent-browser children if a session aborts mid-flow. Close only test-owned browsers via agent_browser / agent-browser close, stop this run's tmux sessions, and remove its disposable screenshots/HARs. Preserve unrelated resources and retained evidence.
Automated prompt driving Grepping tmux pane text for words that also appear in the user prompt (PASS, FAIL, checkout overview, Smoke result:) can false-complete before the agent finishes. Wait for pane idle (no Working…), agent_browser close / Artifact lifecycle, or JSONL tool results—not instruction phrases copied from the prompt.
Lifecycle verify flags npm run verify -- lifecycle --model or --timeout-ms without the next argv token fails fast with a usage error—the project.mjs facade validates passthrough the same way as scripts/verify-lifecycle.mjs. Always pair flags with values (--model openai-codex/gpt-5.5:minimal, --timeout-ms 600000) or omit --model / --timeout-ms to keep the harness defaults (zai/glm-5.2, 180000 ms per-step waits).

Manual validation remains useful for release confidence and installed-package checks:

  1. Configure exactly one active source for this extension in Pi settings: this checkout path before publishing, or the installed package after publishing.
  2. Launch plain pi so extension discovery is active.
  3. Validate managed-session continuity with /reload and a full restart plus exact --session-id relaunch or /resume.
  4. Re-check local extension-side docs (README.md, docs/COMMAND_REFERENCE.md, docs/TOOL_CONTRACT.md, including the agent_browser_action rules when that shorthand or upstream find / select behavior changes) and regenerated prompt fragments from extensions/agent-browser/lib/playbook.ts via npm run docs -- playbook check or npm run docs. When the upstream agent-browser version or help surface changed, run npm run verify -- command-reference.

Real upstream contract validation

The default npm test and npm run verify paths use deterministic tests and fake binaries in two stages: tsx --test "test/**/!(*.serial).test.ts" at Node's default file concurrency, then tsx --test --test-concurrency=1 "test/**/*.serial.test.ts" for strict timing budgets and shared policy-collector fixtures. The browser harness isolates each process's socket root; build/pack fixtures use disposable source trees so parallel consumers retain immutable checkout dist/. For a focused single-file rerun, use npx tsx --test test/<file>.test.ts; npm test rejects extra paths and options so the complete gate cannot silently become a filtered run. When a change touches upstream command planning, result presentation, managed-session behavior, or the canonical capability baseline, also run the opt-in real-upstream contract suite:

npm run verify -- real-upstream

That npm script sets PI_AGENT_BROWSER_REAL_UPSTREAM=1, runs the sessionless plugin probe first in its own fail-fast process, then all other declarations in the contract file (including standalone/compound/opaque native text), followed by native batch fidelity and the real overlay-click suite. Each stage uses --test-force-exit; a failed plugin probe stops the workflow before browser work. To run test/agent-browser.real-upstream-contract.test.ts directly (for example with node --test and tsx), set the same variable yourself; the suite is skipped when it is unset.

This suite requires the installed stable agent-browser --version to meet the minimum in scripts/agent-browser-target.mjs; the output-shape fixture and command-reference verifier remain aligned to the current recommended baseline. It serves fixture pages from localhost and checks stable details/data keys via test/fixtures/agent-browser-real-output-shapes.json. Coverage groups:

  • Inspection and skills (stateless JSON): --version, --help, snapshot --help, skills list, skills get … --full (including webmcp-gen on the 0.38.1 target), skills path … (no managed sessionName / usedImplicitSession).
  • Native 0.38 observation/recording: full/unchanged delta snapshots, persistent refs, partial batch ref follow-ups, conditional screenshot suppression, sticky pointer mode and timed mouse movement, cursor recording and verified contact-sheet output. These options are skipped when explicitly validating the older stable floor.
  • WebMCP target contract: on the 0.38.1 target, webmcp list, invoke with params/frame selection, detached result / cancel, ref invalidation after page tools, and a separate fresh launch with --no-webmcp returning an empty list.
  • Navigation metadata and native tab setup: plain pages stay quiet; a page with tools retains positive native data.webmcp and shows the webmcp list hint. New tabs inherit configured headers before their first request, and clearing headers removes them on the next tab. No wrapper inheritance code is involved.
  • Recording FPS paths and pinning: the opt-in test/agent-browser.batch-fidelity.test.ts native case checks direct/raw/stdin preflight before dispatch, actual start/restart destinations and pinned targets with leading --fps, conservative start-ref protection, FPS-only restart continuity, and valid 12-fps WebM output after explicitly logged 12-second fixture captures. Run it with PI_AGENT_BROWSER_REAL_UPSTREAM=1; it requires native 0.37 or newer. Keep focused older-native controls for fresh-page starts and the start-then-ref batch latch when changing that policy.
  • Managed session core and safe diagnostic matrix: fresh open on the contract fixture, then implicit reuse across eval --stdin, snapshot -i, interaction commands (click, dblclick, fill, type, type --clear --delay, focus, keyboard with type / inserttext, press, hover, check, uncheck, select, failed select no-match, upload, drag, mouse, scroll, off-viewport click, scrollintoview, wait on selectors in the main frame and a selected iframe), extraction (get variants, is variants, find label … fill via native <label>, aria-label, and aria-labelledby, inline eval), file outputs (screenshot, pdf), navigation (back, forward, reload, tab list, another open to the same fixture), batch stdin, pushstate, vitals … --json, network route/requests/HAR, diff snapshot/screenshot/url, trace/profiler, console/errors/highlight, stream enable/status/disable, and cookies set --curl.
  • Managed restore correctness and persistence: while the restore-enabled managed daemon is active, assert raw argument and stdin batches containing nested connect fail before upstream spawn; a new harness replaying the owned launch result must also reject incompatible reuse of that live same-name daemon. Seed a cookie plus localStorage/sessionStorage, close the first managed browser while a conflicting parent namespace is set, verify the default-namespace daemon actually closed, create a new extension harness with the same cwd, reopen the fixture with sessionMode: "fresh", and assert all three values restore before closing the second browser. On POSIX, separate isolated real-browser launches assert automatic restore stays disabled and no snapshot is written through either a symlinked sessions directory or a file symlink in sessions/.tmp; a relative HOME, untrusted writable HOME ancestry, and a non-Git cwd must fail closed. Verify a checkout rename preserves its generation identity but starts a fresh composite restore key (fail-closed, because the cwd-derived managed-session base name changes), a copied or path-replacement checkout gets a new key, and changing the Git-generation marker between planning and spawn prevents agent-browser from starting. Run two same-identity harnesses concurrently so a compatible launch publishes its daemon policy before a waiting incompatible launch re-inspects and fails without reaching its main spawn; also fail a fresh non-batch command after daemon creation and verify shutdown closes the retained identity.
  • Confirmation continuity: the focused contract suite matches confirmation launch policy surviving native nextActions case follows exact confirm/deny actions without policy argv, checks a useful compound audit across resume followed immediately by title/body reads without a repairing URL getter, retains the native daemon PID on bare getters, and preserves an active recording and destination sentinel after denying restart. The nested recording-restart/navigation subcase requires the pinned native recording-policy patch; stock 0.38.1 skips only that subcase. With that patched binary first on PATH, run PI_AGENT_BROWSER_REAL_UPSTREAM=1 PI_AGENT_BROWSER_NATIVE_COMPOUND_POLICY=1 node --import tsx --test --test-force-exit --test-name-pattern 'confirmation launch policy' test/agent-browser.real-upstream-contract.test.ts. Both variables are test selectors, not runtime configuration. Focused read-confirmation, cold-resume, session-page-state and managed-daemon-policy owners additionally cover exact generic/native-read provenance, forged/foreign controls, completed click/tab and failed-inner action reconciliation, confirmed-close replay, definite unestablished policy requests and pending cold helper exposure. Qualify actual SDK resume plus immediate title/body follow-ups without a repairing getter; helpers must not replace the original native pending slot.
  • Cold first-read continuity: the focused contract suite matches cold URL reopen after quit case starts an explicit sessionMode: "fresh" managed browser, runs actual quit cleanup, waits for the exact old daemon to exit, reloads the saved branch, and requests snapshot -i before any explicit navigation. Empty storage and origin storage must both reach their remembered non-root URL, report fresh refs in the main frame, and retain storage without pretending to retain unsaved forms or JavaScript memory. Separate deterministic cold-resume/boundary tests cover first get url and history commands, non-page daemon-starting calls and batch prefixes, hash routes, pending-state transcript replay, live missing tabs versus explicit URL destinations, close retirement, and caller-owned/attached/restore-disabled boundaries. Those fake-upstream checks do not replace native-browser or real Pi lifecycle qualification of the complete candidate.
  • Failure shape: react tree on a page opened with --enable react-devtools but without a React app (expects a clear missing-renderer error with session-bound details).
  • Async download: open on the /download fixture, anchor-triggered export, then wait --download <path> metadata and wrapper artifact reporting for the requested path.

The default unit suite also runs agentBrowserExtension passes through core command coverage fallback matrix in test/agent-browser.extension-passthrough-native-commands.test.ts: a fake upstream records argv so explicit --session connector connect 9222, plus download with a selector and path, get url, snapshot --compact, and tab new / tab t1 / tab close on implicit managed sessions, still prove --json and session ordering without a browser. A second fake-upstream matrix in test/agent-browser.extension-passthrough-debug-providers.test.ts (agentBrowserExtension passes through non-core network debug diff stream dashboard and chat families) pins representative network, diff, trace / profiler / record, console / errors / highlight / inspect / clipboard, stream, dashboard, and chat JSON shapes plus redacted details.data and argv echoes without a browser. Its agentBrowserExtension passes through provider workflows matrix asserts provider open argv shapes still receive --json plus implicit --session and provider credential env vars are forwarded into the fake upstream log, while agentBrowserExtension keeps skills inspection flows stateless and useful in the native-commands file keeps read-only skills get … stateless (no managed session fields). Extend those matrices when adding passthrough coverage that should stay out of the slow real-upstream loop.

Native lifecycle qualification

test/agent-browser.native-lifecycle.test.mjs is the model-free portable SDK lifecycle gate. It loads the compiled extension through real native resource discovery and executes browser tools against loopback fixtures. Qualify official and fork Pi 1.0 graphs independently: no removed fork-only APIs are required. It covers branch/reload continuity, detached and queued code cancellation/ordering, managed cleanup, caller-owned attachment and off-branch routing, failed journal-begin refusal during recording, and root save/reopen across fresh checkout/state-directory inodes. This is lifecycle evidence, not serialization of live page JavaScript, forms or recordings.

Run non-root with an isolated HOME/socket root and stock agent-browser, Chrome for Testing and ffmpeg, without personal profiles or inherited credentials/configuration. The browser directory must contain binaries, never a personal profile.

npm run build
PI_NATIVE_LIFECYCLE_REQUIRED=1 \
PI_NATIVE_LIFECYCLE_SDK='/path/to/pi-coding-agent/dist/index.js' \
PI_NATIVE_LIFECYCLE_BROWSER_DIR='/path/to/browsers/chrome-<version>' \
node --test test/agent-browser.native-lifecycle.test.mjs

PI_NATIVE_LIFECYCLE_EXTENSION selects a separate compiled checkout; PI_NATIVE_LIFECYCLE_ARTIFACT_DIR selects retained transcript evidence. These are test selectors, not runtime package config. Retain exact host/source identity, native tool receipts and cleanup proof. Local SDK qualification complements, rather than replaces, the configured-source tmux harness, live-site smoke and platform gates.

Routine GitHub PR checks

.github/workflows/pi-compatibility.yml runs on PRs to main, manual dispatch, and release workflow calls. It resolves the latest stable official Pi version and latest maintained fork main commit once, or accepts those exact identities from the enclosing release run. Independent required Ubuntu jobs qualify each selected host's complete installed dependency graph on Node 24.21.0. Each runs check:compat once: host identity, typecheck, the full offline suite, and packed SDK/CLI loading. A parallel macOS Node 24.21.0 job typechecks and checks the packed Pi load against the same latest official graph without repeating the full suite. One job-level nonblocking windows-latest lane builds/imports the compiled extension and runs the existing process-transport test on the resolved official host. It has a 15-minute job bound and a 10-minute smoke-step bound; there is no checker build or full contract/consumer campaign. A single Ubuntu fork-package build still supplies verified artifacts to required Linux and explicitly selected deep Windows qualification. The host selector uses the immutable, reviewed fitchmultz/.github automation scripts; that automation pin is not a Pi qualification target. Locked development snapshots and the support floor are not validation lanes, and neither host is skipped based on a floor comparison.

For native startup and consumer qualification, dispatch Pi compatibility with native-qualification=true. This runs the repository's entry in the pinned shared fleet.json for official/fork hosts on latest Node 24.x across ubuntu-latest and macos-latest, instead of repeating the routine package jobs. waived-platforms: false excludes the shared workflow's Windows diagnostics. To select this repository's separate best-effort full Windows matrix, explicitly set windows-qualification=true on manual dispatch; otherwise deep Windows lanes are not run. official-version and fork-ref optionally select exact hosts; blank values resolve current stable/main once. The source is the dispatch commit. The native qualification lanes retain full contracts and actual Git/npm SDK/CLI consumers; the separately labeled routine Windows smoke does not. On-demand full Windows enables Git long paths on its disposable runner so isolated checker-source/submodule checkouts are not cut off at MAX_PATH. Repository attributes keep reviewed checker patches in canonical LF form even with native Git CRLF conversion; patch application is not relaxed. It additionally requires the pinned native process-identity environment self-check before contract dispatch, preserving its existing PID/start-time and timing assertions.

.github/workflows/native-lifecycle.yml runs the separate Native lifecycle Linux gate through scripts/ci-native-lifecycle.sh. It independently checks out the PR head, selects the latest official graph using its once-resolved version (or the exact version supplied by the enclosing release run), builds the compiled extension on Node 24.21.0, and runs portable native SDK lifecycle checks against isolated loopback browser fixtures. It replaces the fork-only checkpoint gate and requires all 12 checks to pass with no failure, cancellation or skip. It does not repeat the official full suite. Sources use a private /tmp root, sanitized environment, non-root execution, and a loopback-only network namespace. Only official host metadata preparation receives the job's read-only GitHub token; setup unsets it before extension build and browser installation. Lifecycle verification receives no token and uses no personal profile, personal credentials, or model calls.

Routine checks write setup, provenance, command-reference, and native TAP results to the job log and upload short-lived evidence: official-host, fork-host, and macos-host contain exact selected host receipts; native-lifecycle contains the host receipt, native TAP results and lifecycle transcripts. Each routine Linux/macOS artifact is retained for three days, with no persistent caches. Windows smoke retains windows-best-effort-smoke for fourteen days with source/host/tool inputs, the selected host receipt when reached, and the complete smoke log. A failed or incomplete smoke is not full Windows qualification. Explicit deep Windows retains windows-qualification-official and windows-qualification-fork for fourteen days, including exact input receipts, environment TAP, complete qualification logs, and the shared qualifier's provenance, package hashes and SDK/CLI probes when reached. Uploads run even after failure. The Windows console retains the final 1,000 qualification lines and, on failure, separately prints all failed TAP test names plus the first 1,600 lines of their diagnostics, with an explicit truncation notice when needed; complete diagnostics remain in qualification.log. An absent qualification receipt or skipped contract is not a pass. The resolved fork-package artifact is retained for seven days. Native manual qualification uses the shared workflow's npm and fork-package caches; its fork-host artifact is retained for seven days and per-lane qualification evidence for fourteen days. The checks cover their exact source, host, and synthetic browser paths; they do not replace release/manual-publication gates, real-site authentication, or full personal-workspace restore qualification.

Native Linux socket-root regression

test/agent-browser.socket-namespace.test.ts is opt-in (PI_AGENT_BROWSER_SOCKET_NAMESPACE=1). Run it with the installed tsx test runner inside a disposable bubblewrap user/mount namespace: a read-only mode-0755 / whose owner is unmapped, and a current-user-owned mode-0700 tmpfs at /tmp. Use a cleared environment with only the required executable PATH, private HOME and the opt-in variable. The test verifies those identities, exchanges data over a real Unix socket, dispatches native agent-browser --version through the shared subprocess wrapper, and inspects a disposable managed session with automatic restore enabled. It does not launch a browser or call a model.

For a root-check change, run the identical final test against both the pre-fix and rebuilt implementation: the old code must fail the socket validation assertion after the native socket exchange succeeds, and the new code must pass without filesystem mocks or runtime patch hooks. Also retain process/restore negatives for foreign or writable non-root ancestry, leaf symlinks, planted entries and unsafe alias destinations, plus existing root-owned sticky-mode coverage. Namespace setup failures are environment blockers, not test passes; do not silently widen container privileges or host policy. This qualified private-/tmp layout does not imply support for unmapped /home ancestry or replace the broader release gates.

Real upstream suite mechanics, isolation, and troubleshooting

  • Focused and broad cases: test/agent-browser.real-upstream-contract.test.ts keeps the broad command matrix in one 180-second case and separate 60-second cold-reopen, duplicate-name click-mutation, navigation-availability/tab-setup, and sessionless-plugin cases. The real-upstream facade's contract suite matches selection includes the broad matrix, cold reopen, click mutation, and navigation setup. The click case uses two native DOM buttons to prove a stale duplicate ordinal cannot contradict trusted target events, while exact XPath probes still detect missing events and native clicks remain dispatch-only evidence.
  • Output-shape locking: Expected details / data keys per step live in test/fixtures/agent-browser-real-output-shapes.json, keyed by logical groups (version, rootHelp, commandHelp, skillsList, skillsGetFull, skillsPath, open, eval, snapshot, coreCommand, coreSubcommand, coreFileArtifact, batch, pushstate, vitals, networkRoute, nonCoreStatus, nonCoreArtifact, diffScreenshotArtifact, streamControl, streamStatus, cookiesCurl, reactMissingRenderer, waitDownload). Keep targetVersion in that file aligned with scripts/agent-browser-capability-baseline.mjs, and extend entries whenever the suite starts asserting on new presentation fields.
  • Isolation: The harness uses a throwaway HOME and screenshot directory, and pairs PI_AGENT_BROWSER_SOCKET_DIR with AGENT_BROWSER_SOCKET_DIR in a short private socket directory. This keeps owned commands and native policy probes on the same daemon without exceeding Unix socket path limits. Raw click/FPS helpers retain owned-session context; cold-resume observers use the registered tool so they do not restart the browser they are measuring. The duplicate-click, cold-reopen, recording-FPS and batch-ref fidelity fixtures use createShortPrivateSocketDir; POSIX allocates an already-created short private /tmp root, Windows falls back under the fixture root, and teardown removes the socket root separately. Cold-reopen, the broad profile/restore fixture and its native file-access control clear AGENT_BROWSER_EXECUTABLE_PATH because automatic restore excludes that override and native repeated executable configuration can reset first-call-only profile/user-agent/file-access choices. The 0.38 auth case deliberately retains a supplied executable and verifies that local auth save preserves the HTTP page before auth login --no-navigate. The harness serves loopback fixtures, closes its sessions and removes only its own directories; it does not reuse your normal profile or socket locations.
  • React DevTools branch: After the core matrix, the suite performs another open with --enable react-devtools and sessionMode: "fresh", then expects react tree to fail with a missing-renderer style error on the same non-React contract page. The following download fixture + wait --download assertions run against whichever managed session is current after that fresh open (typically the React DevTools session), not the original pre-matrix session name.

Troubleshooting

  • Version mismatch: Install a stable agent-browser at or above the configured floor (prefer the recommended capability baseline), or follow the maintainer rebaselining sequence in AGENTS.md if you intentionally move either version.
  • Missing or extra details / data keys: Update test/fixtures/agent-browser-real-output-shapes.json in the same change as the wrapper or presentation code that shifts those keys.
  • Timeouts: The broad matrix has a 180-second bound; the focused cold-reopen, click-mutation, and plugin cases each have a 60-second bound. Repeated timeouts usually mean a hung browser, blocked loopback, or an environment preventing headful/headless launch—check upstream logs and local security tooling before loosening timeouts.

The upstream agent-browser wait --download <path> saveAs persistence limitation is tracked at vercel-labs/agent-browser#1300; until it is fixed, release validation must treat details.savedFilePath as upstream-reported metadata and use details.artifacts[].exists as the filesystem truth (the contract asserts the requested path is absent on disk while upstream still reports success). If the suite fails because JSON/detail keys drifted, update the wrapper behavior or refresh test/fixtures/agent-browser-real-output-shapes.json together with the presentation work that consumes those shapes.

Example smoke prompt:

Use the agent_browser tool to open https://react.dev and then take an interactive snapshot.

Recommended configured-source lifecycle follow-up:

  1. Open a page with the implicit managed session and confirm the title.
  2. Run /reload, then ask for snapshot -i and confirm the same page is still active.
  3. Exit pi, confirm the old managed daemon stopped, relaunch against the same exact session id/path or use /resume, then ask for snapshot -i before any explicit navigation. Confirm the same recorded URL was reopened, its observed page is correct, and refs are fresh. Cover empty storage and origin storage at non-root paths; do not expect unsaved forms, JavaScript memory, or history to survive.
  4. Open a large page that compacts its snapshot output and confirm details.fullOutputPath still exists after the restart/resume/exact-session flow.
  5. Trigger an oversized non-snapshot output (for example a deliberately large eval --stdin result) and confirm the tool prints the actual spill file path directly in content instead of only referencing a details key.
  6. Validate at least one direct file-download flow with download <selector> <path>.
  7. Validate at least one asynchronous export flow with click followed by wait --download <path>, confirming the wait result reports savedFilePath/savedFile and checking details.artifacts[].exists before relying on the requested path being present on disk.

Post-publish install validation

After updating pi-agent-browser-native, fully quit and restart Pi before using the updated tools. /reload can retain previously loaded compiled JavaScript even after dist/ is rebuilt, so it is not a reliable way to pick up package updates.

After publishing a release, validate the package-first path in isolation. npm run verify -- release includes the deterministic fake-binary packaged execution gate and the pre-publish Crabbox platform matrix, but it does not replace a real-browser installed-package smoke against the published npm package:

npm exec --package pi-agent-browser-native -- pi-agent-browser-doctor
npm run verify -- release
pi --no-extensions -e npm:pi-agent-browser-native

Then run the real-browser smoke prompt:

Use the agent_browser tool to open https://react.dev and then take an interactive snapshot.

Only use plain pi for installed-package validation after temporarily disabling or removing the checkout source or any other active source for this extension from Pi settings. Then confirm pi exposes the native agent_browser tool, that a basic open + snapshot -i flow works, and that /reload plus restart with exact --session-id relaunch or /resume keep following the same implicit managed browser session.

Release notes checklist

Before publishing:

  • update CHANGELOG.md
  • confirm README install guidance still leads with the package-first flow
  • confirm docs/COMMAND_REFERENCE.md still matches the effective upstream command/help surface used by the wrapper
  • if you changed scripts/agent-browser-capability-baseline.mjs or the human inventory prose outside the generated HTML-comment blocks, run npm run docs -- command-reference write before verification; see AGENTS.md (upstream capability baseline section) for the three-layer model
  • run npm run verify -- command-reference if the installed upstream agent-browser version or help surface changed
  • run npm run doctor and confirm any duplicate-source remediation matches the active package/checkout setup
  • run npm run verify -- real-upstream for upstream runtime, result-presentation, or managed-session changes
  • confirm both local-checkout modes still work for pre-release validation: isolated pi --approve --no-extensions -e . smoke testing for general trusted checkout loading (add --no-skills for extension-focused bounded smokes; omit --approve only to test the trust prompt) and configured-source lifecycle validation
  • complete interactive tmux live-site extension smoke with pi --approve --no-extensions --no-skills -e . and the native agent_browser tool (at least one simple static site and one real documentation/product site; include advanced QA, code, or native batch when those surfaces changed; use the public Grafana stress checklist when dashboard/diagnostic/artifact behavior changed; close sessions and remove screenshots/temp artifacts; record evidence). Run separate skill-enabled dogfood only when validating skill routing/report-generation behavior—see Pre-release checks; automated gates are not a substitute
  • satisfy the approved release checks and current owner waivers; record individual platform evidence and never label best-effort, failed, or not-run Windows evidence passed from the required non-Windows aggregate
  • run npm run verify -- lifecycle for configured-source /reload, exact --session-id relaunch, managed-session continuity, persisted-spill, and Pi failure-patch regression coverage (required before publish; see Pre-release checks)
  • confirm SUPPORT_MATRIX.md still maps every current baseline inventory section to docs, runtime handling, tests, and validation status
  • manually exercise real-browser /reload and full restart plus exact --session-id relaunch or /resume continuity when release risk warrants browser-level confidence beyond the fake upstream harness
  • publish only after the tarball contents and isolated packaged-extension smoke check match expectations