Repository navigation
feat(web-extract): support source-qualified URL benchmarks and independent reports - #260
lanyue-llk wants to merge 27 commits into
Conversation
![lexmount-ocr[bot]](https://avatars.githubusercontent.com/u/277637559?s=60&v=4){kind=small}
There was a problem hiding this comment.
This review found 9 issue(s), including 9 blocking issue(s).
There was a problem hiding this comment.
This review found 9 issue(s), including 9 blocking issue(s).
There was a problem hiding this comment.
This review found 9 issue(s), including 9 blocking issue(s).
There was a problem hiding this comment.
This review found 5 issue(s), including 5 blocking issue(s).
There was a problem hiding this comment.
This review found 6 issue(s), including 6 blocking issue(s).
There was a problem hiding this comment.
This review found 3 issue(s), including 3 blocking issue(s).
There was a problem hiding this comment.
This review found 3 issue(s), including 3 blocking issue(s).
lanyue-llk
changed the title
There was a problem hiding this comment.
This review found 3 issue(s), including 3 blocking issue(s).
src/harness/web_extract/report_suite.py(token_budget_reached): stopped group "src/harness/web_extract/report_suite.py" mid-review: used 511079 tokens exceeds budget 500000
There was a problem hiding this comment.
This review found 2 issue(s), including 2 blocking issue(s).
src/harness/web_extract/report_suite.py(token_budget_reached): stopped group "src/harness/web_extract/report_suite.py" mid-review: used 558951 tokens exceeds budget 500000
There was a problem hiding this comment.
This review found 3 issue(s), including 3 blocking issue(s).
There was a problem hiding this comment.
This review found 6 issue(s), including 6 blocking issue(s).
There was a problem hiding this comment.
This review found 4 issue(s), including 4 blocking issue(s).
There was a problem hiding this comment.
This review found 3 issue(s), including 3 blocking issue(s).
There was a problem hiding this comment.
This review found 5 issue(s), including 5 blocking issue(s).
There was a problem hiding this comment.
This review found 1 issue(s), including 1 blocking issue(s).
src/harness/web_extract/snapshots.py(token_budget_reached): stopped group "src/harness/web_extract/snapshots.py" mid-review: used 572409 tokens exceeds budget 500000
There was a problem hiding this comment.
✅ Re-review complete · eed0365
No findings in this incremental review.
Web Extract can now evaluate source-qualified URL tasks with custom schemas and list queries, then publish several independent benchmarks through the existing report renderer. Live tasks use the original website; archived tasks use a version-bound public HTML service. Providers receive field requirements, never reference answers.
The shared adapter freezes instructions, source/reference hashes, captured resources and input routes before collection. Source access blocks stay separate from invalid references. Certified, disjoint task scopes can be assembled without repeating successful calls. Live inputs cannot enter archived repair paths, and URL routes no longer decode unused HTML. The portable evaluator and runtime use the same text and URL matching rules.
Cloudflare reuse requires both an API-confirmed immutable deployment and a complete page/resource byte-verification proof for the selected inventory. Concurrent first users share one audit, static downloads reuse connections, and later runs reuse the proof. External or unconfirmed hosts still require direct byte checks. Repair collection and offline verification share the same receipt contract.
Reports keep each benchmark's denominator, quality metrics, latency and normalized prices separate. Arbitrary schema field names retain their identity in result filters; hosted navigation and on-demand evidence work from nested cohort pages. Output paths and reserved language directories are rejected before writes.
Validation covers source text/URL equivalence, malformed answers, reference multiplicity, native live routes, resumable collection, managed/external resource integrity, concurrent snapshot users, certified scope assembly, nested report navigation, pricing and evidence controls. Python and JavaScript behavior checks pass. The delivered report passed 176 desktop/narrow and light/dark page states; hosted success/error drilldown and keyboard navigation were exercised.
Report and ImageGen evaluation-flow diagram (organization access). The report contains Expanded SWDE 210 tasks, NewsXLM 189 tasks and LiveWeb-IE 200 tasks. Oxylabs' 342 quota-rejected archived tasks are retained as account failures and excluded from those quality comparisons. Dataset inputs, provider responses, report prose and illustrations remain outside Git.