Simple is the best; reject over-design.
PRs are welcome for adapters, execution fixes, tests, documentation and translations. Public task definitions are versioned on Hugging Face; scoring code is maintained here; see benchmark boundaries. Run the offline example, then open the guide for the track you want to change.
Include the task ID, method and versions, OS, command and sanitized error. Remove keys, cookies and session data.
| Change | Where it belongs |
|---|---|
| Browser or provider adapter | Existing execution modules; see the provider contract |
| LexBench-Web-Fetch task requirements | Hugging Face task definitions, applied to every method |
| Scoring fix | Shared scorer, scenario regressions and a recorded correction reason |
| New browser, version or sample size | Configuration or registry; reuse the runner and renderer |
| New report instance | An external instance profile, evidence and report content; reuse its build and release commands |
| Reusable logic | src/harness/; scripts/ stays a thin command entry point |
Keep measured counts, host facts and report-specific interpretation out of shared renderers: read them from the selected profile and bound evidence. Published historical reports retain their own content and provenance.
Use absolute Python imports and keep ordinary imports at the top of the module. Existing scorer modules retain import spelling required by published source-hash receipts; use absolute imports in new modules. Keep an import local when loading an optional dependency, preparing a runtime module, avoiding a circular import or preserving test isolation.
Test both accepted and rejected scenarios. Fix scoring rules rather than editing saved verdicts. Refactoring must preserve historical measurements and certificates.
Keep the Chinese README aligned when setup or usage changes. Additional README translations are welcome; other docs do not need duplicate translations. Reports follow their track's language settings.
Benchmark changes follow benchmark editions.
The test for any string: did the project write it, or did it come from the web or a dataset?
- The project wrote it. Identifiers, comments, docs, logs, errors, catalog keys and report interface text have English as their source. The Chinese report edition comes from language catalogs, selected through the report language contract.
- It came from outside. Dataset queries and tasks, frozen web pages, human references, measured responses and quotations keep their original language byte for byte. Never translate, normalize or re-punctuate them; language checks must exempt them.
Test fixtures that stand in for outside text keep its language. Frozen acceptance and audit artifacts are hash-bound experiment inputs whoever wrote them; see reference inputs.
Select a published dataset with make pin-dataset DATASET=<track> REV=<full-HF-commit>.
It rejects commits outside the dataset's main history and verifies the archive and file hashes before updating the data lock and import bindings. Scorer changes belong in this repository and use the existing scenario regressions.
Historical report certificates retain their original evaluator identities.
Register Web Fetch products in src/harness/web_fetch/data/report-methods.json;
executable versions and run parameters come from saved evidence. See
method metadata and
adding a method or release.
Use <type>(<scope>): <summary> for PR titles and new commit subjects. The scope is optional; use the module or feature affected, such as scoring or webfetch.
Choose the type that describes the change:
| Type | Change |
|---|---|
fix |
Correct existing behavior |
feat |
Add supported behavior |
docs |
Change documentation |
refactor |
Restructure code without changing behavior |
test |
Add or correct tests |
perf |
Improve performance |
build / ci |
Change packaging, dependencies or automation |
chore |
Other maintenance |
Write the summary in English, start with an imperative verb, and describe the concrete change. Aim for 72 characters or fewer, including the prefix; omit a trailing period. Use ! before the colon for a breaking change and explain the migration in the description.
Examples:
fix(scoring): require facts in the returned bodyfix(webfetch): disclose build provenance for partial rerunsdocs(contributing): define PR and commit title conventions
Avoid titles such as Fix review feedback, Update code or Misc improvements; name the behavior or document that changes. A PR title describes the complete diff; each commit subject describes that commit. Update the PR title when its scope changes. If squash merging is used, use a subject that follows the same format. Apply this convention to new work; do not rewrite shared commits solely to rename them.
make setup-test
make ciDevelopment checks use the Python version in .python-version (3.11) and Node.js 22. The base example does not need Node.js. make ci also needs ShellCheck and downloads Chromium through Playwright. make help lists the targets; make clean removes build outputs and caches; make format FILES="path.py ..." formats selected files only, because Ruff formatting is not applied repository-wide. Text file reads and writes name encoding="utf-8"; tests/test_explicit_utf8_io.py enforces this and lists the hash-bound sources that must wait for their next authorized revision.
Use focused checks while editing:
make docs-check
make test-python TEST_PATHS=tests/web_search
make test-python TEST_PATHS=tests/test_scoring_regressions.py PYTEST_ARGS='-k reference'
make test-python PYTEST_ARGS='-n 4 --dist worksteal --durations=20'TEST_PATHS selects files or directories; PYTEST_ARGS adds pytest options;
PYTEST_ARGS='-m "not slow"' skips the slowest tests while iterating, and CI still runs them.
Without TEST_PATHS, Python tests cover the entire suite. make docs-check
checks README navigation, citations, track naming and source-language policy;
it needs neither Node.js nor downloaded benchmark data.
CI splits Python tests by a stable hash of their node IDs across two runners,
with four workers per runner. Browser-test modules run once in browser-check;
static, Node and fixture checks run once. Local
make ci remains serial unless PYTEST_ARGS requests workers; focused checks
are usually enough while editing, followed by the complete checks before release.
CI uses this documentation check for PRs changing only the root README,
contribution/security guidance, AGENTS.md, Markdown under docs/, or
PNG/JPEG/WebP/GIF images under docs/assets/.
Code, test, dependency, dataset-card and unknown paths keep the full
test, installation and browser gates. Main pushes and tags always run all gates.
Checks include every commit in the PR; a final documentation commit does not
reduce the scope of earlier code changes. The required Python 3.11 status
rejects failed, cancelled or unexpectedly skipped checks.
| Check | What it covers |
|---|---|
make lint-check |
Basic Python errors, loop-variable closures and unchecked zip across the repository, and the configured Ruff rules for the packages in [tool.ruff] include |
make package-check |
Base CLI, then source archive → wheel → base/optional installation outside the checkout (src/harness/package_gate.py) |
make shell-check |
ShellCheck on deploy/ scripts |
make browser-check |
Chromium reference capture, report interactions and Web Search visual checks |
make test-reference-inputs |
Strict historical-input checks after importing approved reference copies |
| Track release gate | Evidence, report presentation and browser checks before packaging or publishing |
make ci runs what CI runs on a code change: ci-tests, package-check and browser-check. It runs without historical pages: only tests requiring absent historical inputs are skipped. Corrupt installed inputs fail; strict reference mode also fails on missing or incomplete sets.
tests/test_golden_reports.py builds each report track's synthetic example and compares every route with the SHA-256 in tests/golden/. Any change to rendered output fails it. If the change is intended, regenerate the snapshots and commit them with the change, so reviewers see which routes changed:
UPDATE_GOLDEN=1 uv run --frozen --all-extras python -m pytest tests/test_golden_reports.pyTo see the changed lines, set GOLDEN_KEEP=<directory> on two revisions and compare the kept builds with diff -r.
For report releases, follow the release gates. Real services need the environments listed in their track guides.
What the package gate tests
The gate builds a source archive, then a wheel from that archive. Standard python -m pip installs both formats in a clean virtual environment outside the checkout, checks dependencies with pip check, and runs the installed command before the wheel smoke tests. It compares installed modules and resources byte for byte and exercises TypeScript tool bridges, synthetic TEDS revisions, LangChain extraction and Crawl4AI downloads against local HTTP fixtures, including HTTP failures. Missing, stale or extra installed files fail the gate. Node tests use node:module.stripTypeScriptTypes.
Do not commit .env, tokens, cookies, provider outputs, run directories, generated reports or private reviews. Keep third-party sources, versions and licenses. A task definition does not grant permission to redistribute its source page.
Maintainer reference: source exports and historical inputs
A current-source export excludes old commits that may still contain externalized material. Review development history before distributing it; deleting a current file does not remove its earlier copies.
git archive --format=tar.gz --output=/tmp/webstone-source.tar.gz HEADThe archive excludes .git, untracked files and local installations. Extract it, follow README setup and run make ci with an empty WEBSTONE_DATA_HOME. Historical certifier sources retained by manifests are read-only, hash-bound text inputs, not executable entry points or dependencies on old Git commits. See third-party content and reference inputs.
Contributions use the Apache License 2.0. Contributors retain copyright in their contributions; submitting a contribution grants the applicable license and does not transfer ownership to LexMount. Preserve existing copyright notices and third-party terms described in THIRD_PARTY.
Follow the release guide for software compatibility, version changes and release checks. Report builds have their own identities; the Webstone version names the Web Fetch benchmark edition. AGENTS.md owns when a version may change.