Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,19 @@ All notable changes to this project are documented here. The format follows

## [Unreleased]

### Added
- **A page as the target.** A bare address on any host that is not a social platform
— pasted into the TUI bar or given to `check --url` — is read as the document
itself instead of being refused as "a page, not a post". The page is fetched up
the same ladder its sources are, cut into one paragraph per block, and cites the
way a post does: each paragraph's sentences are checked against the pages that
paragraph links to. A page that prints a reference list is paired by citation
number like a paper. A page the ladder could not read ends the run in the ladder's
own words (`UNVERIFIED (blocked)`, `… (unreachable)`, …), with the setting to type
when the browser step was the missing one; a page with no links is reported rather
than passed as clean; the page's own address is never one of its sources
(`docs/superpowers/specs/2026-09-17-page-target-design.md`).

## [0.4.3] - 2026-09-17

The TUI says what to type when it stops you, and every tag gets a GitHub Release.
Expand Down
17 changes: 15 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -243,8 +243,21 @@ proofpath check --url https://news.ycombinator.com/item?id=8863
proofpath check - # paste the text of a post that cannot be read
```

A page is not a post: paste the claim as text with the address inside it, and the page
is fetched as that claim's source.
Any other address is a **page**, and a page is read as the document itself:

```bash
proofpath check --url https://en.wikipedia.org/wiki/AlphaFold
```

The page is fetched up the same ladder its sources are — same permissions, same
consent prompt for the browser step — and cut into paragraphs. A page that prints a
reference list (a journal's article page) is paired by citation number like a paper; any
other page cites by linking, like a post: each paragraph's sentences are checked against
the pages that paragraph links to. A page the ladder could not read ends the run with the
ladder's own words (`UNVERIFIED (blocked)`, `UNVERIFIED (unreachable)`, …), never with a
verdict. A profile, a subreddit or a platform's front page is still "a page, not a post":
there is no one post there to check. To check one claim *about* a page rather than the
page's own words, paste the claim as text with the address inside it.

Every sentence of the post is checked against the pages its links point to, and the post's
own words are never allowed to stand as their own evidence.
Expand Down
15 changes: 15 additions & 0 deletions docs/superpowers/OPEN-ITEMS.md
Original file line number Diff line number Diff line change
Expand Up @@ -510,3 +510,18 @@ Findings from inspecting the installed base environment, and the decisions they
| 15.9 | The Reddit token exchange follows redirects with basic auth attached | httpx re-applies the credential along a redirect chain. The URL is a fixed Reddit constant, so it needs Reddit itself to be the attacker, but `follow_redirects=False` on that one call costs nothing |
| 15.5 | `NO_REASON_RECORDED` is a user-visible state outside §15 | it is a guard that should never print; if it can print, it belongs in the table |
| 15.6 | `NOT_READ_HERE` is provably unreachable through `verify` | kept deliberately for a sixth platform, like `checker_for`'s `arxiv:` row |

## 16. Page targets — v0.4.4, 2026-09-17

A bare address on any host that is not a platform is read as the document itself
(`2026-09-17-page-target-design.md`).

### New open items

| # | Item | Note |
|---|---|---|
| 16.1 | No cap on a link-heavy page | a Wikipedia article carries hundreds of links and every paragraph's links are fetched as sources; coverage is reported and the run can be cancelled, but a cap with a reported remainder would be kinder |
| 16.2 | A page's words and `href`s inside `PAGE_SKIP_TAGS` are dropped with the element | `aside`, `form`, `button`, `svg`, `template` on top of the ladder's boilerplate; a site that puts its citations in an `aside` loses them silently |
| 16.5 | `find_bibliography` knows five headings | a page (or paper) whose list sits under "Sources", "Notes", "Footnotes" or "Citations" is not seen to print one; a page then cites by its links and says so (`MARKERS_SET_ASIDE`), a paper reports its markers unresolved. Widening `_BIBLIOGRAPHY_HEADING` needs the eval sets re-run, so it is not done here |
| 16.4 | A page's self-link rule ignores the query | `_page_key` drops `?utm_source=` and `?page=2` alike, so a page that keys its articles by query (`?id=123`) would drop a link to a sibling article as a link to itself; no such page has been seen |
| 16.3 | Scheme-less links in pasted text are not references | `resolve._HTTP_URL` wants `http(s)://`; `nature.com/articles/…` in a pasted paragraph is verified against nothing |
1 change: 1 addition & 0 deletions docs/superpowers/specs/2026-09-10-proofpath-design.md
Original file line number Diff line number Diff line change
Expand Up @@ -518,6 +518,7 @@ uv tool install proofpath # everything except the browser engine
proofpath # bare command → interactive TUI
proofpath check paper.pdf # academic provider (default)
proofpath check --url https://bsky.app/... # social provider
proofpath check --url https://example.org/story # any other page, read as the document (2026-09-17-page-target-design.md)
proofpath check draft.md --format sarif # inline problems in VS Code
proofpath check paper.pdf --judge llm # opt-in second opinion

Expand Down
56 changes: 56 additions & 0 deletions docs/superpowers/specs/2026-09-17-page-target-design.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
# A page as the target — "paste any link, it just works"

**Status:** approved 2026-09-17; implementation follows this document. Amends spec
§13 (`check --url`) of `2026-09-10-proofpath-design.md`. No honesty state changes;
one refusal (`NOT_A_POST`) narrows to the platforms it was written for.

## 1. Why

Pasting one address into the TUI bar, or `check --url`, routed it as a *social post*.
Only a Bluesky, Hacker News, Reddit or Mastodon post address passed; every other host
— a journal's article page, a news story, a blog, Wikipedia — was refused before any
request with `IngestError: <url> is a page, not a post …`, and the run showed as
*failed*. The bar's own placeholder says "paste a file path or URL", so the refusal
read as a bug. Links *inside* pasted text already worked for any site.

## 2. Decision (2026-09-17, with the user)

A bare address on a host that is not a platform is **the document itself**: the page
is read up the fetch ladder, its paragraphs are the claims, and its links (or the
reference list it prints) are its sources. The alternatives — ask for the claim after
fetching, or only reword the error — were offered and declined.

## 3. The changes

| Where | What |
|---|---|
| `verify._bare_address` / `_address_document` | one address → `_post_document` when `is_social(url)` **or** `social.is_read_here(url)` (a Mastodon status on an unlisted instance is known by shape), else `_page_document` |
| `verify._page_document` | network denied → refused in `NETWORK_DENIED`'s words; else `fetcher.fetch(url, counts_as_source=False, use_cache=False)`; a ladder failure → `IngestError("<url> could not be read: <Outcome>: <deciding step>")` — the Wayback miss every wall ends on is not the deciding step — plus `config set permissions.install_browser …` when the browser was the missing step; else `ingest.from_page` and, when the page prints no reference list, `ingest.with_carried_links`. Unlike `_pasted`, a citation marker alone does not decide it: "[1]" in one paragraph of a long page must not cost the page every link it carries, and a `linked` document reads a bracketed number as prose (`claims.extract`). The override is reported as a parse note (`MARKERS_SET_ASIDE`), so a list under a heading the finder does not know ("Sources") is not set aside silently (rule 6) |
| `fetch.Fetcher.fetch(use_cache=)` | a cache hit is text alone (`body=b""`); the page needs its markup because its links are its bibliography |
| `fetch.CONTENT_ROOTS` | the `article` → `main` → `body` choice, applied by `extract_text` (via `content_root`) and by the page reader on its own tree, so a source's text and a page's paragraphs are cut from one element |
| `ingest.from_page` → `Page` | HTML walked with lxml: one paragraph per block (`PAGE_BLOCK_TAGS`) that holds no other block; a container's words *between* its blocks (a `div` used as a paragraph, text around `<br>`) are paragraphs of their own; `aside`/`form`/`button`/`svg`/`template` contribute neither words nor links on top of the ladder's boilerplate. Each paragraph's `href`s are resolved against the address the page was served from (against the asked-for address for a Wayback copy, whose raw snapshot keeps links unrewritten) and kept per line, together with any address the paragraph prints in full. A link to the page itself — either address, query ignored (`_page_key`) — is dropped and named (`SELF_LINK_DROPPED`). PDF: `_pdf_document` from bytes, the same reader as a file. Plain text: `from_text` plus printed addresses. Anything else: refused by content type. `document.name` is the address asked for |
| `ingest.with_carried_links` | `with_link_references` for a page: the carried links, one reference per distinct address in order of first appearance, placed at the paragraph's first line (what `claims.pair_links` keys on); kind `linked` |
| `document.Kind` | `+ "page"` (own bibliography or nothing; **not** in `LINK_CITED`) |
| `verify.prepare` | "`<kind>` carries no links; nothing to verify against" for a post *or a page* with no references (rule 6) |
| `verify._parser_for` / stage line | a page opens and closes as `fetch ladder`; pasted text with links (kind `linked` too) still closes as `text` |

Unchanged: the ladder, the consent gate, permissions, per-site rules (none exist and
none are added). A profile, subreddit or platform front page keeps `NOT_A_POST`.

## 4. Product rules, checked

- Rule 2: every ladder outcome reaches the user verbatim; a blocked page ends the
run, it never becomes a silent verdict.
- Rule 4: the network permission is read before the page is fetched, as for a post.
- Rule 6: a page with no links is reported, not passed as clean.
- Rule 1: a page's own address — as an anchor, with a tracking query, printed in
full, or under the address a redirect or Wayback copy served it from — is never
one of its sources.

## 5. Open

- No cap on link-heavy pages (a Wikipedia article can carry hundreds of links): every
paragraph's links are fetched. Coverage is reported and the run can be cancelled.
Recorded in OPEN-ITEMS.
- Scheme-less links (`nature.com/…`) in pasted text are still not references
(`resolve._HTTP_URL`); separate.
4 changes: 4 additions & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,9 @@ dependencies = [
"httpx>=0.28",
"curl_cffi>=0.16.1",
"scrapling>=0.4.15",
# Pulled in by scrapling already; declared because ingest.py walks a page's
# tree with it directly.
"lxml>=5.0",
"protego>=0.3", # parses robots.txt for fetch-ladder steps 1-2
"pymupdf>=1.25",
"python-docx>=1.1",
Expand All @@ -64,6 +67,7 @@ dev = [
"pytest>=8.3",
"pytest-cov>=6.0",
"mypy>=1.14",
"lxml-stubs>=0.5", # types for the page walk in ingest.py
"respx>=0.22",
# Validates the SARIF writer's output against the vendored 2.1.0 schema.
"jsonschema>=4.23",
Expand Down
9 changes: 6 additions & 3 deletions src/proofpath/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -504,7 +504,10 @@ def check(
str | None,
typer.Option(
"--url",
help="A post to check instead of a file: Bluesky, Hacker News, Reddit or Mastodon.",
help=(
"An address to check instead of a file: a post (Bluesky, Hacker News, "
"Reddit, Mastodon) or any other page, read as the document itself."
),
),
] = None,
fmt: Annotated[Format, typer.Option("--format", help="Output format.")] = Format.TEXT,
Expand Down Expand Up @@ -681,8 +684,8 @@ def _flush_streams() -> None:
def _check_target(out: ui.Ui, target: str | None, url: str | None) -> tuple[Path | str, str | None]:
"""The document and the name its locations carry.

``--url`` is a post, handed on as the address it is: what can be read at one is
``verify``'s to decide, not the CLI's. ``-`` is the document itself, arriving on
``--url`` is a post or a page, handed on as the address it is: what can be read
at one is ``verify``'s to decide, not the CLI's. ``-`` is the document itself, arriving on
stdin and named for where it came from; anything else is a file that has to
exist, and a file names itself.
"""
Expand Down
2 changes: 1 addition & 1 deletion src/proofpath/document.py
Original file line number Diff line number Diff line change
Expand Up @@ -88,7 +88,7 @@ class PageError:
# and pairing one of those by position would put a verdict on a source no sentence
# cited (product rule 1). Said in the kind rather than guessed from the shape of the
# document, because the guess is what got that wrong.
Kind = Literal["pdf", "docx", "markdown", "text", "post", "linked"]
Kind = Literal["pdf", "docx", "markdown", "text", "post", "linked", "page"]
KIND_VALUES: tuple[str, ...] = get_args(Kind)
#: The kinds whose references are the links in their own body.
LINK_CITED: frozenset[str] = frozenset({"post", "linked"})
Expand Down
58 changes: 42 additions & 16 deletions src/proofpath/fetch.py
Original file line number Diff line number Diff line change
Expand Up @@ -258,23 +258,39 @@ def extract_text(body: bytes, kind: Kind) -> str:
with pymupdf.open(stream=body, filetype="pdf") as doc: # type: ignore[no-untyped-call]
return "\f".join(page.get_text() for page in doc)
if kind == "html":
if not body.strip():
root = content_root(body)
if root is None:
return ""
page = Selector(body)
# The first article or main element is the content; the body is the fallback.
for selector in ("article", "main", "body"):
found = page.css(selector)
if found:
text: str = found[0].get_all_text(
separator="\n", strip=True, ignore_tags=BOILERPLATE_TAGS
)
return text
return ""
text: str = root.get_all_text(separator="\n", strip=True, ignore_tags=BOILERPLATE_TAGS)
return text
if kind == "text":
return body.decode("utf-8", errors="replace")
return ""


#: Where a page's content lives, first match wins: the page reader in ``ingest``
#: cuts its paragraphs from the same element ``extract_text`` grades a source on.
CONTENT_ROOTS = ("article", "main", "body")


def content_root(body: bytes) -> Selector | None:
"""The element a page's content lives in, or ``None`` for an empty page.

The first ``article`` or ``main`` element is the content; the ``body`` is the
fallback (``CONTENT_ROOTS``, which the page reader in ``ingest`` applies to its
own tree, so a source's text and a page's paragraphs are cut from one element).
"""
if not body.strip():
return None
page = Selector(body)
for selector in CONTENT_ROOTS:
found = page.css(selector)
if found:
root: Selector = found[0]
return root
return None


def html_title(body: bytes) -> str:
if not body.strip():
return ""
Expand Down Expand Up @@ -397,24 +413,34 @@ def summary(self) -> FetchStats:
return FetchStats(counts=dict(self._counts), browser_skipped=self._gate.skipped)

def fetch(
self, url: str, *, text_kind: str = "fulltext", counts_as_source: bool = True
self,
url: str,
*,
text_kind: str = "fulltext",
counts_as_source: bool = True,
use_cache: bool = True,
) -> Fetched:
"""One climb. A bare URL is one source, so a browser refusal here is one
skipped source; the open-access chain fetches several locations for one
DOI and counts that source itself (``counts_as_source=False``)."""
result = self._climb(url, text_kind)
DOI and counts that source itself (``counts_as_source=False``).

``use_cache=False`` climbs past a cached copy. A hit carries the text alone
(``body=b""``), which is all a *source* needs; a page read as the document
itself needs its markup back, because its links are its bibliography.
"""
result = self._climb(url, text_kind, use_cache=use_cache)
self._counts[result.outcome] += 1
if counts_as_source and result.outcome is Outcome.BLOCKED_NO_BROWSER:
self._gate.skipped += 1
return result

def _climb(self, url: str, text_kind: str) -> Fetched:
def _climb(self, url: str, text_kind: str, *, use_cache: bool = True) -> Fetched:
notes: list[str] = []
if self.network_note:
notes.append(self.network_note)
if not self.network_allowed:
return _failed(url, 0, Outcome.NETWORK_DENIED, _no_response(url), notes)
if self._cache is not None:
if use_cache and self._cache is not None:
cached = self._cache.get_raw_text(f"url:{url}")
if cached: # only non-empty text is ever stored, so empty means miss
notes.append("cache: hit")
Expand Down
Loading
Loading