feat(web-extract): support source-qualified URL benchmarks and independent reports

.agents/skills/benchmark-report/references/evaluation-design.md

@@ -36,6 +36,22 @@ A report claiming a before/after build speed difference must use a predeclared p
 
 ## References and volatile sources
 
+Select tasks by independently verified page and reference validity, never by a
+participant's accessibility or score. A checker timeout, rate limit, security
+block, or HTTP error leaves validity unresolved; it is not proof that the page
+has disappeared. Retain that candidate for independent verification rather than
+silently replacing it with an easier page. Exclude or correct a task only with
+source evidence of removal, wrong content, or erroneous/stale annotations, and
+apply that decision to every participant. A provider's failure on a valid task
+remains a measured failure.
+
+For a live cohort, an independently captured reference observation may remain
+authoritative during a predeclared, bounded freshness window. Freeze its original
+URL, observation time and input/reference hashes before calls. If a later source
+check is blocked, retain the task and record the failed check and the age of the
+reference observation. A successful check showing changed answers overrides this
+fallback; expired or unsupported observations cannot qualify the task.
+
 Prefer invariant contracts for changing pages: stable labels, required fields, record shape, types, count bounds, uniqueness, and independently observed explanatory text. Do not freeze ranks, prices, member counts, or current titles unless the benchmark is explicitly a time-bound snapshot.
 
 Ground truth must be independent of the provider under test. Retain source location, bytes or a durable observation, hash, capture method, and scope. Partial annotations support only the requirements they cover. They are not a full-body reference. When authoring or expanding a task, inspect its actual reference text against the reader's main content needs. A dashboard's labels do not establish its values; a feed's navigation does not establish its entries. Select applicable fact, body-region, record, pattern, resource and file checks through the shared scorer. Give the source-complete answer and a page shell or incomplete answer opposite outcomes before freezing the new version. Preserve earlier input versions.

configs/report-templates.json

@@ -81,14 +81,16 @@
         "research-and-compatibility",
         "full-cohort-progress",
         "synthetic",
-        "field-extraction-cohort"
+        "field-extraction-cohort",
+        "field-extraction-suite"
       ],
       "release_check": "uv run --frozen python -m harness.web_extract.cli check /absolute/report --release --visual-receipt /absolute/visual-receipt.json",
       "package": "uv run --frozen python -m harness.web_extract.cli package /absolute/report --output /absolute/report.zip --visual-receipt /absolute/visual-receipt.json",
       "benchmark": "src/harness/benchmarks/cards/zyte-product-extraction.json",
       "cohort_example_build": "uv run --frozen python -m harness.web_extract.example --output /tmp/extract-cohort-example --mode full-cohort-progress",
       "publish": "uv run --frozen python -m harness.web_extract.cli publish --source /absolute/report --target /absolute/publication-target.json --work /absolute/new-publication --visual-receipt /absolute/visual-receipt.json",
-      "record_example_build": "uv run --frozen python -m harness.web_extract.example --output /tmp/extract-record-example --mode field-extraction-cohort"
+      "record_example_build": "uv run --frozen python -m harness.web_extract.example --output /tmp/extract-record-example --mode field-extraction-cohort",
+      "suite_example_build": "uv run --frozen python -m harness.web_extract.example --output /tmp/extract-suite-example --mode field-extraction-suite"
     }
   }
 }

examples/web_extract/README.md

@@ -14,7 +14,7 @@ shared cache, works offline after download, and preserves existing hash-bound in
 
 ## Scope and naming
 
-Web Extract measures structured field extraction from archived HTML: the caller
+Web Extract measures structured field extraction from archived pages or live URLs: the caller
 requests fields such as price, stock or author, and the service returns field values.
 Web scraping is the broader term for collecting website data; extraction is the
 part measured here. [Zyte calls it automatic extraction](https://docs.zyte.com/zyte-api/usage/extract/index.html).
@@ -76,7 +76,8 @@ Browser infrastructure requiring a separately connected model is outside this
 comparison. Browserbase JSON uses only the Browserbase API key. The separately
 registered Stagehand route uses its native Model Gateway and needs a project ID;
 it does not read an OpenAI key. Bright Data's site-specific extractor remains discoverable as a candidate, but
-is rejected by the full archived-page plan because it cannot parse arbitrary HTML.
+is outside arbitrary-schema cohorts because it requires a supported site-specific
+extractor. Browser infrastructure alone does not supply the requested field values.
 
 New cohorts give every native API the same public URL for each frozen page.
 Each dataset has its own hosting service. Browser Eval can find and reuse a
@@ -106,7 +107,11 @@ to select another HF edition; its full resolved commit is frozen in the result.
 Discovery searches historical Cloudflare deployments as well as the current one.
 Each managed snapshot has one public `snapshots.json` declaring the HF dataset
 repository, full source commit and complete HTML/file inventory hashes. Discovery
-checks this one manifest rather than downloading individual pages. The result
+uses this manifest to identify the dataset version. On first use, it also hashes
+every served page and resource. The verified deployment ID and whole-input hash
+are cached under the configured data home; later runs reuse that proof only after
+Cloudflare confirms the same immutable deployment. Concurrent first users share
+one audit, and static downloads reuse network connections. The result
 records both the selected HF commit and the snapshot's source commit; an identical
 HTML inventory allows reuse across metadata-only HF revisions. Snapshots without
 this declaration require an approved new deployment. Matching inputs are
@@ -125,6 +130,11 @@ the same case URL; upload-only interfaces receive the identical frozen HTML and
 can share the plan. A plan containing only upload-only methods can omit the binding. Before and after calls, managed Cloudflare snapshots use
 the frozen whole-dataset manifest and its hash; a changed manifest blocks certification.
 External hosts additionally have every selected page hash-checked before and after calls.
+An unconfirmed Cloudflare binding follows the same byte checks. The management API
+must confirm the immutable deployment identity, and a complete byte-verification
+proof must bind that deployment to the selected input inventory, before a
+whole-manifest check can replace HTML and resource downloads. A new deployment
+or changed inventory requires another byte check.
 Page-by-page verification remains an explicit byte-audit command and is not used
 to identify a managed snapshot version.
 Keep raw HTML, original benchmark names, sources and licenses; exclude reference
@@ -428,6 +438,67 @@ and [versioned preview URLs](https://developers.cloudflare.com/pages/configurati
 
 ## Multi-value field reports
 
+### Source-qualified external cohorts
+
+Use the `records` adapter for another benchmark with explicit field schemas and
+source-validated references. Its definition stays outside Git and binds the source
+versions, sampling rules, page hashes, schemas and reference hashes. References are
+used for local scoring and never sent to an extraction API.
+
+A definition may set `extraction_instructions` to declare its task scope. This
+text is frozen in the dataset profile, plan and request contract. List-query
+benchmarks should request all matching records; the default single-record
+instructions remain available for detail-page benchmarks. Field descriptions
+carry the original question when an API accepts a schema without a separate query.
+
+```sh
+uv run --frozen --extra web-extract-eval python -m harness.web_extract.cli \
+  --home /absolute/data/cohort prepare --benchmark records \
+  --source /absolute/data/cohort-definition.json
+```
+
+The definition declares `input_mode: snapshot` or `input_mode: live`:
+
+| Mode | URL sent to the native API | Source validation |
+| --- | --- | --- |
+| `snapshot` | The immutable hosted page URL selected by `host` | HF revision and the complete HTML/resource inventory before and after collection |
+| `live` | The original website URL in the task | Original reference values checked on the source before and after collection |
+
+Live cohorts reject snapshot bindings. Saved HTML is private source evidence;
+providers still visit the original site. Unavailable pages or references that no
+longer appear stop certification, while completed provider calls remain saved.
+An optional `live_reference_policy.maximum_observation_age_seconds` (at most
+86,400 seconds) permits a previously captured independent source when a fresh
+access attempt fails. Each case then freezes `qualification.source_url` and a
+timezone-aware `qualification.observed_at`. Preserve a header-only charset in
+`qualification.encoding`, such as `utf-8`; it is bound with the captured bytes
+and used when checking them. A successfully observed page with unsupported
+reference values still stops certification. Access failure alone does not
+establish that the question or answer is invalid.
+Live cohorts resume through their original certified run. The archived repair
+entry point rejects live inputs so a retry cannot substitute a hosted page or
+HTML upload for the original URL. Explicit HTTP-200 challenge pages count as
+access failures and follow the same bounded capture policy; a changed source
+page with missing answers still fails reference qualification.
+
+Source checks run at most four origins concurrently, with serial requests and a
+one-second pause within each origin. They do not contribute to provider latency.
+Transient network errors and HTTP 408/429/5xx responses allow up to three source
+checks, with bounded backoff recorded in the receipt. Missing reference values do
+not trigger retries. This policy never retries provider extraction for a better score.
+
+Snapshots can include captured styles, images and other resources. Their hashes
+are part of the whole-snapshot version; changing an asset changes that version
+even when the main HTML is unchanged. Scripts and external resources remain
+blocked by response headers.
+
+The shared scorer reports page-hit macro F1, value-level micro F1 and exact-field
+set rate. Text matching preserves content and units while normalizing whitespace.
+Explicitly typed URL fields resolve relative links against the source URL, using
+the same matching rule in aggregate scores and result details. A cohort using
+these rules must disclose them as an adaptation unless they reproduce the
+upstream evaluator.
+
 The same report owner supports registered record schemas through
 `mode: field-extraction-cohort`. The instance content binds every planned
 participant to its certified source or journal checkpoint, with source-plan
@@ -450,8 +521,10 @@ release gate requires all participants complete and a hash-bound visual receipt.
 
 An explicit report-scope omission sets `include_in_comparison: false` and supplies
 `exclusion_reason` on that participant. Its calls, progress, detail panels and
-timings remain available, while quality and speed leadership use only the declared
-comparison participants. Release checks require every compared participant to be
+timings remain available, while quality uses only the declared comparison
+participants. Timing and price tables retain every planned participant; timing
+extrema use the stated all-call population, including account rejections.
+Release checks require every compared participant to be
 complete and reject an omitted participant without a reason. Source execution
 plans and certificates are never rewritten to change this presentation scope.
 
@@ -489,3 +562,29 @@ A repair inherits its parent's instructions and native options unless the plan e
 overrides the native options. The instruction-delivery format is frozen too: historical
 record runs with a different format require a new cohort instead of a mixed repair. Bare run IDs resolve under `<home>/runs/`; explicit relative paths
 resolve from `<home>`, and absolute paths retain their meaning.
+
+## Independent benchmarks in one report
+
+Use the same renderer's `field-extraction-suite` mode to publish several cohorts
+under one entry. Each cohort keeps its own dataset, scorer, result inventory,
+timing population and denominator. The overview does not average quality scores
+across benchmarks. Run the network-free example with:
+
+```sh
+uv run --frozen python -m harness.web_extract.example --output /tmp/extract-suite --mode field-extraction-suite
+uv run --frozen python -m harness.web_extract.cli check /tmp/extract-suite
+```
+
+The suite content lists `cohorts`, each with `id`, `label`, `home`, `content` and
+`profile`. Its child content declares a matching `site_prefix`. The instance
+profile binds the suite content hash as usual. Nested pages, evidence dialogs,
+the English notice and the return link use the same hosted-root navigation.
+The release receipt binds a visual receipt for every child under `cohorts`.
+
+A participant can reuse several disjoint certified scopes through an evidence
+selection's `segments` array. Each segment identifies its original `home`,
+`run`, `plan` and `plan_sha256`. Assembly verifies each original certificate and
+requires identical task definitions, schemas, instructions and HTML for the
+selected IDs. The scopes must cover the declared dataset exactly once. This
+allows an expanded cohort to retain earlier calls without selecting the best
+answer, changing evidence or repeating completed calls.

src/harness/web_extract/analysis.py

@@ -169,7 +169,7 @@ def failure_stage(row):
 
 def record_field_results(reference, row, *, text_matching="literal", unavailable_fields=()):
     """Expose exact-set and official page-hit outcomes without changing saved scores."""
-    from harness.web_extract.swde import canonical_text
+    from harness.web_extract.swde import record_value_key
 
     if text_matching not in ("literal", "whitespace"):
         raise ValueError("Unsupported record text matching policy")
@@ -184,16 +184,14 @@ def record_field_results(reference, row, *, text_matching="literal", unavailable
         actual = prediction.get(field, [])
         valid = (
             {
-                canonical_text(value) if text_matching == "whitespace" else value
+                record_value_key(reference, field, value, text_matching)
                 for value in actual
                 if isinstance(value, str)
             }
             if isinstance(actual, list)
             else set()
         )
-        expected = {
-            canonical_text(value) if text_matching == "whitespace" else value for value in gold
-        }
+        expected = {record_value_key(reference, field, value, text_matching) for value in gold}
         if field in unavailable:
             result[field] = {
                 "expected": [],

src/harness/web_extract/assets/report.js

@@ -307,12 +307,16 @@ for (const table of document.querySelectorAll("table[data-sortable]")) {
   }
 }
 // Compare the complete same-page population, before row-level diagnosis filters.
+const fieldOutcome = (row, field) => {
+  const outcomes = row.dataset.fieldOutcomes === undefined ? row.dataset : JSON.parse(row.dataset.fieldOutcomes);
+  return Object.prototype.hasOwnProperty.call(outcomes, field) ? outcomes[field] : undefined;
+};
 const qualityOutcome = (row, field) => {
   if (!row || ["pending", "unavailable"].includes(row.dataset.status)) return null;
   if (!field && row.dataset.scoredFields === "0") return null;
   if (field) {
-    if (row.dataset[field] === undefined || row.dataset[field] === "unavailable") return null;
-    return (!row.dataset.stage || row.dataset.stage === "fields") && row.dataset[field] === "correct";
+    if (fieldOutcome(row, field) === undefined || fieldOutcome(row, field) === "unavailable") return null;
+    return (!row.dataset.stage || row.dataset.stage === "fields") && fieldOutcome(row, field) === "correct";
   }
   return row.dataset.status !== "fail" && row.dataset.outcome !== "wrong";
 };
@@ -361,10 +365,10 @@ if (form) {
     let shown = 0;
     for (const row of resultRows) {
       const field = controls.field || "";
-      const applicable = !field || row.dataset[field] !== undefined;
-      const bad = field ? (Boolean(row.dataset.stage) && row.dataset.stage !== "fields") || row.dataset[field] !== "correct" : row.dataset.status === "fail" || row.dataset.outcome === "wrong";
+      const applicable = !field || fieldOutcome(row, field) !== undefined;
+      const bad = field ? (Boolean(row.dataset.stage) && row.dataset.stage !== "fields") || fieldOutcome(row, field) !== "correct" : row.dataset.status === "fail" || row.dataset.outcome === "wrong";
       const status = controls.status || "all";
-      const scored = field ? row.dataset[field] !== "unavailable" : row.dataset.status !== "unavailable";
+      const scored = field ? fieldOutcome(row, field) !== "unavailable" : row.dataset.status !== "unavailable";
       const show =
         applicable &&
         (!comparedCases || (comparedCases.has(row.dataset.case) && [baseline, ...competitors].includes(row.dataset.method))) &&