feat(web-extract): support source-qualified URL benchmarks and independent reports

.agents/skills/benchmark-report/references/evaluation-design.md

@@ -36,6 +36,22 @@ A report claiming a before/after build speed difference must use a predeclared p
 
 ## References and volatile sources
 
+Select tasks by independently verified page and reference validity, never by a
+participant's accessibility or score. A checker timeout, rate limit, security
+block, or HTTP error leaves validity unresolved; it is not proof that the page
+has disappeared. Retain that candidate for independent verification rather than
+silently replacing it with an easier page. Exclude or correct a task only with
+source evidence of removal, wrong content, or erroneous/stale annotations, and
+apply that decision to every participant. A provider's failure on a valid task
+remains a measured failure.
+
+For a live cohort, an independently captured reference observation may remain
+authoritative during a predeclared, bounded freshness window. Freeze its original
+URL, observation time and input/reference hashes before calls. If a later source
+check is blocked, retain the task and record the failed check and the age of the
+reference observation. A successful check showing changed answers overrides this
+fallback; expired or unsupported observations cannot qualify the task.
+
 Prefer invariant contracts for changing pages: stable labels, required fields, record shape, types, count bounds, uniqueness, and independently observed explanatory text. Do not freeze ranks, prices, member counts, or current titles unless the benchmark is explicitly a time-bound snapshot.
 
 Ground truth must be independent of the provider under test. Retain source location, bytes or a durable observation, hash, capture method, and scope. Partial annotations support only the requirements they cover. They are not a full-body reference. When authoring or expanding a task, inspect its actual reference text against the reader's main content needs. A dashboard's labels do not establish its values; a feed's navigation does not establish its entries. Select applicable fact, body-region, record, pattern, resource and file checks through the shared scorer. Give the source-complete answer and a page shell or incomplete answer opposite outcomes before freezing the new version. Preserve earlier input versions.

configs/report-templates.json

@@ -81,14 +81,16 @@
         "research-and-compatibility",
         "full-cohort-progress",
         "synthetic",
-        "field-extraction-cohort"
+        "field-extraction-cohort",
+        "field-extraction-suite"
       ],
       "release_check": "uv run --frozen python -m harness.web_extract.cli check /absolute/report --release --visual-receipt /absolute/visual-receipt.json",
       "package": "uv run --frozen python -m harness.web_extract.cli package /absolute/report --output /absolute/report.zip --visual-receipt /absolute/visual-receipt.json",
       "benchmark": "src/harness/benchmarks/cards/zyte-product-extraction.json",
       "cohort_example_build": "uv run --frozen python -m harness.web_extract.example --output /tmp/extract-cohort-example --mode full-cohort-progress",
       "publish": "uv run --frozen python -m harness.web_extract.cli publish --source /absolute/report --target /absolute/publication-target.json --work /absolute/new-publication --visual-receipt /absolute/visual-receipt.json",
-      "record_example_build": "uv run --frozen python -m harness.web_extract.example --output /tmp/extract-record-example --mode field-extraction-cohort"
+      "record_example_build": "uv run --frozen python -m harness.web_extract.example --output /tmp/extract-record-example --mode field-extraction-cohort",
+      "suite_example_build": "uv run --frozen python -m harness.web_extract.example --output /tmp/extract-suite-example --mode field-extraction-suite"
     }
   }
 }

examples/web_extract/README.md

@@ -14,7 +14,7 @@ shared cache, works offline after download, and preserves existing hash-bound in
 
 ## Scope and naming
 
-Web Extract measures structured field extraction from archived HTML: the caller
+Web Extract measures structured field extraction from archived pages or live URLs: the caller
 requests fields such as price, stock or author, and the service returns field values.
 Web scraping is the broader term for collecting website data; extraction is the
 part measured here. [Zyte calls it automatic extraction](https://docs.zyte.com/zyte-api/usage/extract/index.html).
@@ -76,7 +76,8 @@ Browser infrastructure requiring a separately connected model is outside this
 comparison. Browserbase JSON uses only the Browserbase API key. The separately
 registered Stagehand route uses its native Model Gateway and needs a project ID;
 it does not read an OpenAI key. Bright Data's site-specific extractor remains discoverable as a candidate, but
-is rejected by the full archived-page plan because it cannot parse arbitrary HTML.
+is outside arbitrary-schema cohorts because it requires a supported site-specific
+extractor. Browser infrastructure alone does not supply the requested field values.
 
 New cohorts give every native API the same public URL for each frozen page.
 Each dataset has its own hosting service. Browser Eval can find and reuse a
@@ -428,6 +429,61 @@ and [versioned preview URLs](https://developers.cloudflare.com/pages/configurati
 
 ## Multi-value field reports
 
+### Source-qualified external cohorts
+
+Use the `records` adapter for another benchmark with explicit field schemas and
+source-validated references. Its definition stays outside Git and binds the source
+versions, sampling rules, page hashes, schemas and reference hashes. References are
+used for local scoring and never sent to an extraction API.
+
+A definition may set `extraction_instructions` to declare its task scope. This
+text is frozen in the dataset profile, plan and request contract. List-query
+benchmarks should request all matching records; the default single-record
+instructions remain available for detail-page benchmarks. Field descriptions
+carry the original question when an API accepts a schema without a separate query.
+
+```sh
+uv run --frozen --extra web-extract-eval python -m harness.web_extract.cli \
+  --home /absolute/data/cohort prepare --benchmark records \
+  --source /absolute/data/cohort-definition.json
+```
+
+The definition declares `input_mode: snapshot` or `input_mode: live`:
+
+| Mode | URL sent to the native API | Source validation |
+| --- | --- | --- |
+| `snapshot` | The immutable hosted page URL selected by `host` | HF revision and the complete HTML/resource inventory before and after collection |
+| `live` | The original website URL in the task | Original reference values checked on the source before and after collection |
+
+Live cohorts reject snapshot bindings. Saved HTML is private source evidence;
+providers still visit the original site. Unavailable pages or references that no
+longer appear stop certification, while completed provider calls remain saved.
+An optional `live_reference_policy.maximum_observation_age_seconds` (at most
+86,400 seconds) permits a previously captured independent source when a fresh
+access attempt fails. Each case then freezes `qualification.source_url` and a
+timezone-aware `qualification.observed_at`. Preserve a header-only charset in
+`qualification.encoding`, such as `utf-8`; it is bound with the captured bytes
+and used when checking them. A successfully observed page with unsupported
+reference values still stops certification. Access failure alone does not
+establish that the question or answer is invalid.
+Source checks run at most four origins concurrently, with serial requests and a
+one-second pause within each origin. They do not contribute to provider latency.
+Transient network errors and HTTP 408/429/5xx responses allow up to three source
+checks, with bounded backoff recorded in the receipt. Missing reference values do
+not trigger retries. This policy never retries provider extraction for a better score.
+
+Snapshots can include captured styles, images and other resources. Their hashes
+are part of the whole-snapshot version; changing an asset changes that version
+even when the main HTML is unchanged. Scripts and external resources remain
+blocked by response headers.
+
+The shared scorer reports page-hit macro F1, value-level micro F1 and exact-field
+set rate. Text matching preserves content and units while normalizing whitespace.
+Explicitly typed URL fields resolve relative links against the source URL, using
+the same matching rule in aggregate scores and result details. A cohort using
+these rules must disclose them as an adaptation unless they reproduce the
+upstream evaluator.
+
 The same report owner supports registered record schemas through
 `mode: field-extraction-cohort`. The instance content binds every planned
 participant to its certified source or journal checkpoint, with source-plan
@@ -450,8 +506,10 @@ release gate requires all participants complete and a hash-bound visual receipt.
 
 An explicit report-scope omission sets `include_in_comparison: false` and supplies
 `exclusion_reason` on that participant. Its calls, progress, detail panels and
-timings remain available, while quality and speed leadership use only the declared
-comparison participants. Release checks require every compared participant to be
+timings remain available, while quality uses only the declared comparison
+participants. Timing and price tables retain every planned participant; timing
+extrema use the stated all-call population, including account rejections.
+Release checks require every compared participant to be
 complete and reject an omitted participant without a reason. Source execution
 plans and certificates are never rewritten to change this presentation scope.
 
@@ -489,3 +547,29 @@ A repair inherits its parent's instructions and native options unless the plan e
 overrides the native options. The instruction-delivery format is frozen too: historical
 record runs with a different format require a new cohort instead of a mixed repair. Bare run IDs resolve under `<home>/runs/`; explicit relative paths
 resolve from `<home>`, and absolute paths retain their meaning.
+
+## Independent benchmarks in one report
+
+Use the same renderer's `field-extraction-suite` mode to publish several cohorts
+under one entry. Each cohort keeps its own dataset, scorer, result inventory,
+timing population and denominator. The overview does not average quality scores
+across benchmarks. Run the network-free example with:
+
+```sh
+uv run --frozen python -m harness.web_extract.example --output /tmp/extract-suite --mode field-extraction-suite
+uv run --frozen python -m harness.web_extract.cli check /tmp/extract-suite
+```
+
+The suite content lists `cohorts`, each with `id`, `label`, `home`, `content` and
+`profile`. Its child content declares a matching `site_prefix`. The instance
+profile binds the suite content hash as usual. Nested pages, evidence dialogs,
+the English notice and the return link use the same hosted-root navigation.
+The release receipt binds a visual receipt for every child under `cohorts`.
+
+A participant can reuse several disjoint certified scopes through an evidence
+selection's `segments` array. Each segment identifies its original `home`,
+`run`, `plan` and `plan_sha256`. Assembly verifies each original certificate and
+requires identical task definitions, schemas, instructions and HTML for the
+selected IDs. The scopes must cover the declared dataset exactly once. This
+allows an expanded cohort to retain earlier calls without selecting the best
+answer, changing evidence or repeating completed calls.

src/harness/web_extract/analysis.py

@@ -169,7 +169,7 @@ def failure_stage(row):
 
 def record_field_results(reference, row, *, text_matching="literal", unavailable_fields=()):
     """Expose exact-set and official page-hit outcomes without changing saved scores."""
-    from harness.web_extract.swde import canonical_text
+    from harness.web_extract.swde import record_value_key
 
     if text_matching not in ("literal", "whitespace"):
         raise ValueError("Unsupported record text matching policy")
@@ -184,16 +184,14 @@ def record_field_results(reference, row, *, text_matching="literal", unavailable
         actual = prediction.get(field, [])
         valid = (
             {
-                canonical_text(value) if text_matching == "whitespace" else value
+                record_value_key(reference, field, value, text_matching)
                 for value in actual
                 if isinstance(value, str)
             }
             if isinstance(actual, list)
             else set()
         )
-        expected = {
-            canonical_text(value) if text_matching == "whitespace" else value for value in gold
-        }
+        expected = {record_value_key(reference, field, value, text_matching) for value in gold}
         if field in unavailable:
             result[field] = {
                 "expected": [],

src/harness/web_extract/assets/report.js

@@ -307,12 +307,14 @@ for (const table of document.querySelectorAll("table[data-sortable]")) {
   }
 }
 // Compare the complete same-page population, before row-level diagnosis filters.
+const fieldOutcome = (row, field) => row.dataset.fieldOutcomes === undefined
+  ? row.dataset[field] : JSON.parse(row.dataset.fieldOutcomes)[field];
 const qualityOutcome = (row, field) => {
   if (!row || ["pending", "unavailable"].includes(row.dataset.status)) return null;
   if (!field && row.dataset.scoredFields === "0") return null;
   if (field) {
-    if (row.dataset[field] === undefined || row.dataset[field] === "unavailable") return null;
-    return (!row.dataset.stage || row.dataset.stage === "fields") && row.dataset[field] === "correct";
+    if (fieldOutcome(row, field) === undefined || fieldOutcome(row, field) === "unavailable") return null;
+    return (!row.dataset.stage || row.dataset.stage === "fields") && fieldOutcome(row, field) === "correct";
   }
   return row.dataset.status !== "fail" && row.dataset.outcome !== "wrong";
 };
@@ -361,10 +363,10 @@ if (form) {
     let shown = 0;
     for (const row of resultRows) {
       const field = controls.field || "";
-      const applicable = !field || row.dataset[field] !== undefined;
-      const bad = field ? (Boolean(row.dataset.stage) && row.dataset.stage !== "fields") || row.dataset[field] !== "correct" : row.dataset.status === "fail" || row.dataset.outcome === "wrong";
+      const applicable = !field || fieldOutcome(row, field) !== undefined;
+      const bad = field ? (Boolean(row.dataset.stage) && row.dataset.stage !== "fields") || fieldOutcome(row, field) !== "correct" : row.dataset.status === "fail" || row.dataset.outcome === "wrong";
       const status = controls.status || "all";
-      const scored = field ? row.dataset[field] !== "unavailable" : row.dataset.status !== "unavailable";
+      const scored = field ? fieldOutcome(row, field) !== "unavailable" : row.dataset.status !== "unavailable";
       const show =
         applicable &&
         (!comparedCases || (comparedCases.has(row.dataset.case) && [baseline, ...competitors].includes(row.dataset.method))) &&

src/harness/web_extract/certification.py

@@ -4,23 +4,24 @@
 from pathlib import Path
 
 from harness.reporting.templates import metric_contract
-from harness.web_extract.dataset import benchmark_config, task_bindings
+from harness.web_extract.dataset import benchmark_config, input_mode, task_bindings
 from harness.web_extract.pipeline import (
-    RECORD_INSTRUCTIONS,
     available_methods,
     data_home,
     digest,
     instruction_schema,
     load_run,
     probe,
     read_json,
+    record_instructions,
     require_external_output,
     save_json,
     score_run,
     selected_profile,
     validate_dataset,
     validate_fixture_base,
     validate_instruction_schema,
+    validate_live_route,
 )
 
 
@@ -36,6 +37,9 @@ def create_plan(home, methods, output, *, snapshots=None):
     if any(registry[m].get("retired") for m in methods):
         raise ValueError("Retired reference interfaces cannot enter a new cohort")
     validate_dataset(home)
+    live = input_mode(home) == "live"
+    if live and snapshots is not None:
+        raise ValueError("Live cohorts use original URLs, not a snapshot registry")
     truth = read_json(home / "dataset/dataset/ground-truth.json")
     plan = {
         "schema": "archived-product-full-cohort/1",
@@ -74,8 +78,13 @@ def create_plan(home, methods, output, *, snapshots=None):
         plan.update(task_bindings(home))
         for participant in plan["participants"].values():
             participant["maximum_physical_calls"] = len(truth) + 20
-    if snapshots is None and any(
-        registry[m]["input"] == "url" or "input_route" in registry[m]["options"] for m in methods
+    if (
+        not live
+        and snapshots is None
+        and any(
+            registry[m]["input"] == "url" or "input_route" in registry[m]["options"]
+            for m in methods
+        )
     ):
         raise ValueError(
             "URL-capable interfaces require --snapshots; upload-only methods may omit it"
@@ -103,10 +112,19 @@ def create_plan(home, methods, output, *, snapshots=None):
         plan["resource_decision"]["memory"] = (
             "Remote memory unavailable; local client memory does not measure the provider."
         )
+    if live:
+        plan["input_mode"] = "live"
+        for method, participant in plan["participants"].items():
+            participant["input"] = "url"
+            if registry[method]["input"] != "url":
+                if "input_route" not in registry[method].get("options", []):
+                    raise ValueError("A live cohort requires native URL extraction")
+                participant["native_options"] = {"input_route": "url"}
     adapter = benchmark_config(home)["adapter"]
     if adapter is not None:
+        plan["extraction_instructions"] = record_instructions(home)
         for case in truth.values():
-            schema = instruction_schema(adapter.task_schema(home, case), RECORD_INSTRUCTIONS)
+            schema = instruction_schema(adapter.task_schema(home, case), record_instructions(home))
             for method in methods:
                 validate_instruction_schema(method, schema)
     save_json(output, plan, exclusive=True)
@@ -116,6 +134,7 @@ def create_plan(home, methods, output, *, snapshots=None):
 def run(home: Path, profile: Path, method: str, run_dir: Path, fixture_base=None):
     require_external_output(run_dir)
     plan = read_json(profile)
+    live = plan.get("input_mode") == "live"
     if plan.get("schema") == "archived-extraction-repair/1":
         from harness.web_extract.repair import run as repair_run
 
@@ -146,18 +165,28 @@ def run(home: Path, profile: Path, method: str, run_dir: Path, fixture_base=None
     participant = plan["participants"][method]
     if participant.get("input") == "html" and plan.get("public_snapshots"):
         fixture_base = None
-    if available_methods()[method]["input"] == "url" and (
-        not fixture_base or not fixture_base.startswith("https://")
+    if (
+        not live
+        and available_methods()[method]["input"] == "url"
+        and (not fixture_base or not fixture_base.startswith("https://"))
     ):
         raise ValueError("URL participant requires frozen HTTPS fixtures before collection")
     records = benchmark_config(home)["adapter"] is not None
+    if live:
+        if participant.get("input") != "url":
+            raise ValueError("Live participants must declare URL input")
+        validate_live_route(method, participant.get("native_options"))
+    if records and plan.get("extraction_instructions") != record_instructions(home):
+        raise ValueError("Frozen extraction instructions differ from the dataset")
     if records and plan.get("schema") != "archived-extraction-full-cohort/1":
         raise ValueError("Frozen plan schema differs from the selected record benchmark")
     if records or plan.get("task_adapter_sha256"):
         bindings = task_bindings(home)
         if not bindings or any(plan.get(key) != value for key, value in bindings.items()):
             raise ValueError("Frozen task adapter or dataset profile changed")
     validate_dataset(home)
+    if live != (input_mode(home) == "live"):
+        raise ValueError("Frozen input mode differs from the selected dataset")
     truth = read_json(home / "dataset/dataset/ground-truth.json")
     case_hash = digest(("\n".join(sorted(truth)) + "\n").encode())
     if plan.get("dataset_manifest_sha256") != digest((home / "dataset/manifest.json").read_bytes()):