Webstone owns scoring, execution adapters, saved evidence and reports. Public task definitions, source attribution and reference bindings are versioned on Hugging Face.
| Responsibility | Owner |
|---|---|
| Web Fetch task model, deterministic acceptance and offline scoring | src/lexbench_webfetch/ in this repository |
| Search acceptance, relevance metrics and offline scoring | src/lexbench_websearch/ in this repository |
| Public task definitions and source/reference bindings | The LexMount benchmark datasets on Hugging Face |
| Download verification and dataset selection | src/harness/benchmarks/ and its data lock |
| Provider calls, browser execution, evidence and reports | Existing track modules under src/harness/ |
| Qualified WCXB inputs and curated/public replay definitions | Their HF editions; upstream projects retain source attribution |
| WebMainBench and WebArena evaluator code | The selected scorer/evaluator dependency and its recorded source revision |
The two scorer APIs and console commands ship in the Webstone distribution. They require no browser installation or provider credentials. Former standalone scorer repositories retain source history and attribution; current installation and scoring use Webstone.
The README introduces the design principles and workflow diagrams. Benchmark integrations also need the following scoring and review evidence.
Retain the scorer identity and its input evidence separately from computed verdicts. For Search relevance, use named human annotations by default. If a model judge is explicitly selected, preserve its model, prompt, rubric, candidate evidence, raw responses and grades; label model-judged results accordingly. See the Search scoring guide and Web Fetch scoring corrections for their executable contracts.
Independent verification is opt-in. Declare its scope and retain the reviewer, protocol and hashes of the evidence reviewed. The default scope is machine failures; reviewing passes or the full matrix requires an explicit request. Reports state reviewed/total coverage and accepted unreviewed passes.
When verification finds a defect, correct the shared cause in the reference, scorer or execution stage. Retain original evidence and official scores; record corrections separately and rescore or rerun the affected scope under the revised contract. A scorer-only correction can reuse valid saved outputs. Use the existing Web Fetch review workflow and replay protocol for track-specific handling.
Current datasets are published separately:
| Dataset | Purpose |
|---|---|
| Product extraction benchmark (Zyte) | 140 frozen product pages, original labels and separately identified source-qualified annotations |
| LexBench-Web-Fetch | URL-to-content tasks, acceptance contracts and reference bindings |
| LexBench-Web-Search | Scenario queries and source qualifications; factual QA inputs belong to SimpleQA Verified |
| LexBench-SimpleQA-Verified | A separate retrieval adaptation of the publisher's factual questions |
| LexBench-WebMainBench | Corrected frozen-HTML extraction inputs and references |
| LexBench-WCXB | Source-qualified extraction inputs, preserving selected upstream HTML and reference bytes |
| LexBench-Headless-Browser | Executable CDP, driver and web-platform tasks; scenario specifications retained as source inputs |
| LexBench-Trajectory-Replay | 30 public-web tasks written in Webstone, with fixed actions and exact checks |
| LexBench-WebArena-Replay | Full official task catalog; smaller replay editions retained as versioned source inputs |
Dataset cards explain scope, schema, source terms and loading. Configuration counts and selected revisions come from the dataset registry and data lock. WebArena's full task configuration retains both portable and direct-only tasks; inclusion in that configuration does not establish portable replay qualification. The official evaluator remains a separately pinned dependency.
Install Webstone, then prepare its pinned public definitions:
uv sync --frozen
uv run --frozen python -m harness.benchmarks.data
uv run --frozen lexbench-webfetch list
uv run --frozen lexbench-webfetch validate
uv run --frozen lexbench-websearch listWhen upgrading an existing checkout, uv sync --frozen removes the retired standalone scorer distributions before installing the shared distribution.
Downloads use full Hugging Face commits and verify the archive and every file. Public downloads need no access key. Cached definitions work with HF_HUB_OFFLINE=1; WEBSTONE_DATASET_CACHE overrides the default ~/.webstone/datasets/. Changing the evidence directory does not change the selected dataset cache.
If validation reports extra, missing or corrupted cached files, run uv run --frozen python -m harness.benchmarks.data --repair <track> while online. Repair preserves the invalid directory beside the snapshot, then downloads and verifies the locked edition again. Offline repair leaves the existing files intact.
The public Fetch definitions contain compact task contracts and reference hashes. Full third-party captures still require an approved matching input copy:
uv run --frozen lexbench-webfetch export \
--suite webfetch --source /absolute/path/approved-inputs \
--output /absolute/path/new-input-directory
uv run --frozen webstone validate \
--dataset /absolute/path/new-input-directory/dataset.jsonThe export verifies the full dataset and declared reference files before materializing them. A formal live-site comparison uses the certified runner and the selected track's release gates. Ordinary CLI runs are diagnostic. For offline scoring of an existing answer, use lexbench-webfetch score or lexbench-websearch score; their help lists the task, response and evidence inputs.
The data lock selects dataset commits, records archive/file hashes and retains earlier snapshots. Fetch and Search import bindings preserve logical historical paths and exact definition hashes. Existing experiment profiles and certificates retain their original identities.
Select a new published edition through one command:
make pin-dataset DATASET=webfetch REV=<full-Hugging-Face-commit>The same command selects the other dataset keys registered in the lock, including websearch, simpleqa-verified, webmainbench, wcxb, web-extract, trajectory-replay and webarena-replay. The command rejects a commit that is not on the dataset's main history, such as an unmerged Hugging Face pull-request ref or a force-pushed commit, and verifies the release before updating the lock and applicable import bindings. It reads the history through HF_ENDPOINT with optional HF_TOKEN. Keep current-version selections in these files, not in README prose.
Author dataset changes in a checkout of the corresponding Hugging Face dataset. Commit the definitions and materialize any Git LFS files, then use uv run --frozen python -m harness.benchmarks.publication --help to build the inspected commit. --create-pr HF_REPOSITORY opens a Hugging Face pull request for review; it does not update dataset main or create a release tag. Install the optional dataset-publish extra and authenticate with HF_TOKEN for uploads; readers need no token. Preserve upstream attribution and redistribution terms. Scores, provider responses, model grades and private run directories do not belong in a dataset release.
Web Fetch revisions preserve their original task files. score_only_annotation_revision
cannot change the request. An explicitly selected task_scope_revision may also
replace a nonempty task_prompt; it still cannot change task identity, URL,
retrieval route or output type. Both forms require exact parent hashes, per-task
field-change hashes, independent source bindings and passing regression controls.
Select a reviewed revision through the existing Web Fetch export --revision
command and retain its new dataset identity. A scope revision changes the task
contract. For local analysis of saved answers, use the existing
revise_local_snapshot owner with explicit rule_revision, authority and paired
regression IDs, then bind its receipt as the saved matrix's offline_revision.
This retains the original calls and rules and applies each changed case to every
method. The narrower certified scoring-correction command does not authorize
arbitrary prompt changes. A local re-evaluation does not establish that old calls
executed the revised task or make changed input contracts directly comparable:
a formal release requires a breaking version (major, or minor before 1.0) and
remains a separate comparison.
Changed-prompt offline revisions cannot enter a saved matrix's final primary
ranking. Keep original-contract results and revised-rule sensitivity analysis
separate, including per-method effects and any counterfactual retry differences.
All-method failure is a review trigger, not evidence that a requirement is wrong;
scope corrections from method-independent task evidence and apply the same rule
to equivalent tasks, including tasks that already passed.
Neither a revision PR nor an export creates a release tag. Merge revision support
first, then the reviewed dataset PR; select its verified merged commit in a
separate make pin-dataset DATASET=webfetch REV=<full-HF-commit> change.
After review, merge the HF pull request and validate its merged commit. For a
formal dataset release, assign a vMAJOR.MINOR.PATCH tag in that dataset repository
and retain its exact target commit. Preserve existing historical tags and never
move a published tag. Dataset version numbers are independent of Webstone's
software version; an ordinary commit does not require a new release number.
Keep the dataset's version history and increment/comparison rules in its own
VERSIONING.md and link it from the dataset card. The publisher preserves this
committed policy when rebuilding a source or Hub snapshot; uncommitted edits do
not enter the release. Web Fetch's 430-task and 550-task populations require
separate major series; equal counts alone do not identify a historical edition.
Update Webstone's data lock and import bindings in a GitHub PR using the
reviewed full HF commit. Neither a dataset merge nor a moved branch updates
existing runs or the selected input version automatically.
For tracks that select a default dataset and an external scorer, the release
manifest owns default_dataset and the scorer's repository and full commit.
The publisher reads them from the committed source registry, or preserves them
from a Hub source manifest. Pinning a new edition selects those settings together
with its inputs and rejects missing required settings instead of inheriting an
older edition's scorer.
For resource-only or raw-file tasks, declare content_score_applicable: false when no text answer is required; file/resource checks remain active. Fix scoring rules here and keep positive/negative controls with their scorer. Preserve prior measurements; a scoring or reference correction applies to the declared affected scope through the existing revision receipts. src/harness/lexbench/ remains the historical headless-browser execution/report adapter, separate from the two scorer packages.
Webstone wrote the 30 public-web replay tasks, including their actions and exact assertions. Headless scenario designs informed some interactions; its original executable cases use different fixtures, operations and acceptance checks. The replay task JSON freezes actions and expected outputs, while live target pages can still change.
WebArena replay has separate official tasks, self-hosted application state and evaluator rules. Its full task catalog is the current viewer population; qualified recordings and corrected-task editions remain versioned source inputs for the public replay runtime. The original Headless driver runtime remains upstream-owned. Follow each dataset card for its input format and runtime contract.
WebArena Replay 1.0.1 corrects the host values in Webstone's recorded replay inputs, which named private recording hosts. Exported task questions, actions and evaluator inputs are unchanged, so this is a patch edition. Its definitions change only the hashes that bind those corrected sources. The release index records the edition, and the data lock keeps the previous edition in history.
Corrected inputs use the recorded container host, so they follow the same port-based rebinding as other recorded inputs and reach the sites in your site map. Older evidence still uses the earlier hosts. To replay or redact it, list those hosts in WEBARENA_RECORDING_GATEWAYS.
The correction changes bytes that earlier evidence binds:
| Earlier material | Effect | What to do |
|---|---|---|
| Stored attempts for tasks 280 and 282 | Their correction record digest changed | Re-evaluate, or check them against a private copy of the original bytes |
| A run or ordered sweep started before the correction | It stops on changed inputs | Start a new run or sweep |
| A frozen cohort manifest | It stops when a recorded source revision changed | Freeze and qualify a new manifest |
| Qualification receipts for corrected candidates | They refer to the earlier candidate bytes | Requalify those tasks |
| An instance profile that binds the bridge or an earlier dataset hash | It stops with a hash error | Update the bound hash, or keep selecting the earlier edition by revision |
Published reports and measured evidence are not changed. A profile may keep selecting an earlier edition through its recorded revision.
Every track identifies its benchmark code by the full source commit. A release tag may provide a readable label, but the selected tag must resolve to a recorded commit. Freeze task, reference and scoring-file hashes as well; a package version or a moving branch name alone does not identify the evaluation.
Use the existing owner for each selection:
| Benchmark | Version authority |
|---|---|
| LexBench datasets | The Webstone version and commit: src/harness/benchmarks/data-lock.json, its import bindings and the scorer they ship |
| WebMainBench | The data lock's corrected corpus, its upstream/correction provenance and the selected run profile's scorer commit |
| WCXB | Qualified HF edition in the data lock, source manifest and unchanged HTML/reference hashes |
| WebArena replay data | HF edition in the data lock and selected task/trajectory bindings |
| WebArena evaluator for Browser Use and replay | Pinned evaluator dependency and runtime source verification |
| Curated replay | HF edition in the data lock, Webstone source commit and selected task/trajectory hashes |
Read and validate the selected binding before execution; retain it with the run and report. For data distributed as files, preserve content hashes even when no upstream commit is available. Do not invent a source commit for an old snapshot.
Current-use documentation links to these owners instead of repeating selected
versions or commits. Upstream main links in guides are navigation, not execution
inputs. Historical experiments and source attributions keep their immutable
citations; those records must not follow a newer dependency selection.
Updating the installed benchmark does not select a new dataset for an existing report.
The Webstone version names the Web Fetch benchmark edition: its data lock
selects the dataset commit and its code holds the scoring rules (see
version rules). Run
uv run --frozen lexbench-webfetch version to print that version and the
scoring-source hash; unreleased scoring code prints X.Y.Z+unreleased. Runs
record the scoring-source hash and full commit; report builds also record the
Webstone version. Reviewing or fixing code does not publish a release.
Every run pins the full HF commit and input hashes, plus the software/scorer
commit and scoring-source hashes. A dataset release tag is a readable label for
that immutable commit; it never replaces the recorded commit. Intermediate
snapshots remain reproducible without a tag. Matching version labels alone do
not establish comparability: retain the selected task IDs, acceptance rules,
execution controls and evidence alongside those identities.
Webstone publishes software versions as vMAJOR.MINOR.PATCH and records the
full Git commit for execution and report-building code.
Reports reference their benchmark edition. A published build also records the hosting platform's immutable version ID.
Report home pages link to the Webstone build version and the selected bench edition. Their JSON bindings retain code commits and source/input hashes. Rebuilding with a newer Webstone release changes the build identity, not the saved call identity or benchmark edition. Custom datasets without a frozen edition retain their hashes rather than inheriting the installed Webstone version.
| Change | Webstone version | Result comparison |
|---|---|---|
| Documentation or presentation only | Patch | Direct when execution controls also match |
| Reviewed scoring/reference correction with the same task inputs | Minor | Re-score both saved cohorts under one edition |
| Added/removed cases, changed requests, output scope or routes | Major (minor before 1.0, described as breaking) | Separate full-population results |
The lexbench_webfetch module owns lexbench-webfetch release and compare-releases. Historical benchmark version contracts remain available with their original source revisions.
Freeze benchmark-release.json beside the compiled inputs before calls. For --benchmark lexmount-webfetch, give release --version the installed, released Webstone version; the command rejects any other. Custom benchmarks keep their own version numbers.
The certified Web Fetch runner accepts --benchmark-release; it also discovers
the sidecar beside the selected dataset. Release creation records the full scorer
commit alongside the independently computed source hash. The runner checks tasks and the active scorer
before external work and retains the binding in execution metadata. The original
call edition remains in benchmark-input-release.json; a report may select a newer
scoring edition after rescoring, retaining both files.
An instance profile can select an edition explicitly:
{
"benchmark_release": {
"path": "benchmark-release.json",
"sha256": "<SHA-256 of the selected sidecar>"
}
}Paths are relative to the external profile. The existing build copies this edition into the report manifest and package and displays a link beside the home-page scope. Checks reject mismatched populations, acceptance rules, scoring sources, missing sidecars and incorrect version labels. Historical hash-bound reports remain usable without inventing a version label. An explicit historical edition preserves the saved scorer identity during a presentation rebuild; it does not run the current scorer over old results.
Reuse unchanged saved outputs for a minor correction, including prior successes, then recompute scores under one edition. Use the existing revision receipts; do not overwrite the older report. Breaking editions may retain diagnostic task-overlap analyses, but their complete throughputs/pass rates are separate measurements. Latency still requires matching runtime controls and the existing same-window comparison contract.
Local reports are optional outputs from saved evidence. The public CLI supports run --no-report and rescore --no-report; report <run-directory> can build HTML later without new calls. Local use requires no report-hosting account. Formal release checks and packaging apply when preparing a release; upload requires an explicit publication request and authorization. See local reports and optional sharing.
Each report track uses its maintained build, check and package entry points. Browser Use and Browser Trajectory Replay accept an instance profile at each boundary. Keep the profile and measured evidence outside Git. Select a new output directory; changing participants, sample size or report content does not require another renderer or command script.
The native Web Fetch report also accepts an external profile. Bind the frozen
dataset, complete participant matrix, execution settings and evidence hashes in
that profile. Put report prose in the evidence tree and bind it through
comparison.report_interpretation; a report refresh does not require changing
Python strings. The normal build and check retain certification and review gates.
The diagnostic build is a separate evidence state and does not certify results.
Saved-matrix specifications default to presentation: "audit". An explicitly
selected presentation: "final" removes audit-workflow banners only after every
declared result has complete, judged call evidence. It preserves scores, source
bindings, limitations and evidence links; it does not confer independent review
or benchmark certification. The source-aware check also verifies this selection
against the external specification. Incomplete matrices cannot use final mode.
uv run --frozen python scripts/build_report.py \
--profile /absolute/instance/profile.json \
--evidence /absolute/instance/evidence --output /absolute/instance/report
uv run --frozen python scripts/check_report.py \
/absolute/instance/report --profile /absolute/instance/profile.jsonA profile stored outside Git has the same validation rules as a repository profile. Repository defaults select the maintained published report; they are optional convenience paths, not a requirement to commit new report instances.
Browser Use profiles register models, browser labels and measurement controls.
task_selection.path and task_selection.sha256 bind the selection file; its hash
must match selection_sha256. The selection declares its own metadata.tier_counts.
official_dataset.path and official_dataset.sha256 can bind the exact historical
official task file, matching official_data_checksum and the recorded calls.
This allows rebuilding an older measurement after the installed benchmark has
changed. failure_cause_review.source_path may name an external, hash-bound review.
Keep original attempts and task contracts unchanged.
An optional report_history object binds a JSON file through path and sha256.
Relative paths use the authority root; absolute paths can select external content.
A new instance without this binding has no release history. It never inherits the
maintained report's releases or review rounds. Build, check and package use the
same binding.
uv run --frozen python -m harness.browser_use report \
--profile /absolute/instance/profile.json \
--evidence /absolute/model-evidence --output /absolute/instance/report
uv run --frozen python -m harness.browser_use check \
/absolute/instance/report --profile /absolute/instance/profile.json \
--evidence /absolute/model-evidence --require-complete
uv run --frozen python -m harness.browser_use package \
--report /absolute/instance/report --profile /absolute/instance/profile.json \
--evidence /absolute/model-evidence --output /absolute/instance/report.zip \
--attestation /absolute/instance/attestation.json \
--visual-receipt /absolute/instance/visual.jsonRepeat --evidence for additional model cohorts. The renderer derives comparison
counts and failure examples from the selected results. Missing task-type or
execution metadata stays unavailable; it is never inferred from historical IDs.
Trajectory Replay profiles bind the dataset, participant registry, execution
protocol and reference validity. The failure review must bind the exact saved
results. Without an authored content binding, the maintained renderer generates
an interpretation from the current results, including complete failure
distributions and specimen links. An optional authored interpretation uses
--content, --content-bindings and --interpretation-audit; these inputs must
belong to the selected evidence.
uv run --frozen python -m harness.trajectory_replay report \
--profile /absolute/instance/profile.json --results /absolute/results.json \
--review /absolute/failure-review.json --output /absolute/instance/report
uv run --frozen python -m harness.trajectory_replay check \
/absolute/instance/report --profile /absolute/instance/profile.json
uv run --frozen python -m harness.trajectory_replay package \
--report /absolute/instance/report --profile /absolute/instance/profile.json \
--evidence /absolute/results.json --review /absolute/failure-review.json \
--output /absolute/instance/report.zip \
--attestation /absolute/instance/attestation.json \
--visual-receipt /absolute/instance/visual.jsonBoth packers run integrity, link and browser checks before installing an archive. The replay packer automatically uses the existing compact representation above its file-count limit. It verifies every task/result page and evidence payload against the uncompressed report, then exercises exported routes in every saved language. Compression preserves all logical results and physical-call evidence.
The Make targets accept PROFILE, OUTPUT and VISUAL_RECEIPT for these tracks;
replay build/package targets also accept REVIEW, and packaging accepts ARCHIVE
and ATTESTATION. Packaging creates a local archive and does not publish it.
For WCXB, select a profile-bound evidence directory and a fresh output with
scripts/build_report.py --profile /absolute/instance/profile.json. Keep its
report-content-binding.json beside the profile. providers declares the matrix;
participants can supply labels, bilingual routes, interface examples and account
requirements. An optional paired_comparison names baseline and candidate
participants. A declared timing correction still requires complete paired evidence;
ordinary cohorts do not inherit another report’s timing or optional-feature claims. Run
scripts/check_report.py with that profile and uv run --frozen --extra headless-eval python -m harness.web_fetch wcxb-visual
against the output. The existing publication entry point also supports local
packaging without credentials or a network upload:
uv run --frozen python -m harness.web_fetch wcxb-publish --package-only \
--profile /absolute/instance/profile.json \
--source /absolute/instance/report --work /absolute/instance/packageA published-snapshot-audit profile rebuilds a saved Web Fetch report through the
current renderer. Bind its complete exported file manifest, original and corrected
results, historical verdicts, initial recomputation, scoring receipt, review,
interpretation and diagram under inputs. An optional bound method_registry
provides historical participant IDs, engines and labels; release_history supplies
the report's structured history. Keep these instance files outside Git.
score_published_snapshot verifies the full task-by-method matrix, preserves call
identities and timing, and computes scores without contacting providers. It accepts
an exact hash-verified recovered original, otherwise identifies a privacy-projected
input separately. A projected file alone is not evidence that redaction changed a
verdict. Compare the old scorer on the same bytes before assigning that cause.
Build with scripts/build_report.py --profile <profile> --evidence <snapshot> --output <new-report> --language zh. This historical format supports Chinese and
provides an English not-generated notice. Check with scripts/check_report.py <new-report> --profile <profile> --visual-receipt <private-receipt>, then package
through uv run --frozen python -m harness.web_fetch site-package <new-report> <package.zip> --snapshot-audit.
Run these Python entries through uv run --frozen python. The mode retains the
original routes and metric populations and labels the new report as a saved-input
audit; it does not certify new calls or replace the original release. The ordinary
packaging path rejects this evidence state without the explicit audit flag.
When publication is requested and authorized, publish through uv run --frozen python -m harness.web_fetch site-publish --snapshot-audit with a separate
publication target, receipt and operation journal. Use --create for a new report
and set visibility explicitly. A later update uses that report's own site and
expected version. Keep the original report's target unchanged.
A report can bind participant documentation through
comparison.method_registry. Each participant selects its execution engine
and label. Optional report_access contains account, configuration,
env_vars and links; optional report_interface contains entry, auth,
params, fields, request, source and notes. These objects follow the
shared interface renderer's existing format. Use them for cohort-specific
instructions and examples. Measured version fields and binary hashes remain
part of each saved call; a participant name is not a version receipt.
Shared adapter documentation describes interfaces by engine. Adding a version or renaming a participant requires data changes, without another renderer entry.
The runtime defaults to the qualified HF edition selected by the data lock. Its source manifest records upstream identities and qualification exclusions; selected HTML and reference bytes are retained unchanged. Historical profiles keep their original manifests and input hashes. The upstream WCXB project remains the source attribution. Use Webstone's existing runner and scorer for current execution.
The WebMainBench runner defaults to the corrected corpus selected by the data lock. Its scoring command prepares the exact declared scorer checkout when --scorer-repo is omitted, and rejects a changed commit or dirty tracked files. Install the webmainbench-eval extra for its metric dependencies. Explicit dataset files retain the original-corpus hash and scorer defaults; provide the matching hash and scorer commit when selecting another historical edition. Existing profiles and saved evidence remain unchanged.
The local lexbench_websearch module loads its pinned public definitions through the shared dataset loader. The registry's current mapping selects named task suites; explicit dataset, field and hash references are resolved before collection. Historical standalone profiles preserve their original bytes and hashes.
Search providers, annotation collection, latency/billing evidence, audits and reports remain in src/harness/web_search/. Report profiles and measured results stay beside external evidence. Change task data on Hugging Face and scorer code here; keep the existing source qualifications and paired scoring controls. The Search dataset and SimpleQA retrieval edition retain source attribution and derivation details.