Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 30 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,36 @@ All notable changes to this project are documented here. The format follows

## [Unreleased]

### Added
- **`Tavily API:` in `/config`.** The search section of the settings panel has a key
row: press Enter, paste the key, Enter again. It is saved to the `.env` in your
config directory (mode 0600 on macOS/Linux), never to `config.toml` or the project's
`.env`, and it is never shown: the row says only whether a key is saved and where.
Backspace removes it. Saving a key does not turn search on: while
`search.provider` is not `tavily` the row says the key is unused, and with
`tavily` selected and no key it says one is needed.
- **A note on claims too short to pin down.** A claim like "openai won" is supported
by almost any page about any win. A claim with two content words or fewer, and no
linking verb (`is`, `are`, `was`, …) between them, now carries a "short claim"
note on each of its verdict rows, SARIF included. The note tells you to check
that the quoted passage is about the event you meant. It is a rule, not a model,
and it changes no verdict, tier or exit code.

### Fixed
- **Wikipedia pages no longer read as `UNVERIFIED (blocked)`.** The bot-wall check
matched `captcha` inside MediaWiki's inline configuration script, so every Wikipedia
page was reported blocked at all four ladder steps despite answering 200. Challenge
phrases are now matched outside `<script>`/`<style>` contents and only on small
pages. `cf-chl` and Cloudflare's `_cf_chl_opt` still mark a wall wherever they
appear. Other `cf_chl` tokens (such as `__cf_chl_f_tk`) count only on small
pages, because they linger in the URLs of real articles.
- **A verdict judged against an abstract now says so on its own row.** A short page
or an abstract-only source already had its `LOW CONFIDENCE (abstract only)` row, but
the claim's verdict row (`SUPPORTED (found by proofpath)`, `NOT SUPPORTED`, …) did
not, so read alone it looked like a full-text verdict. It now carries "judged
against the abstract only, not the full text", in the terminal, Markdown, JSON and
SARIF.

## [0.4.8] - 2026-09-29

A text that cites nothing can now be checked: proofpath searches the web for it. And
Expand Down
6 changes: 5 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -321,7 +321,11 @@ proofpath config set search.provider tavily
echo 'TAVILY_API_KEY=tvly-…' >> .env
```

or `search.provider searxng` plus `search.base_url` for your own instance. Setting
In the TUI, `/config` does both: set `search.provider` to `tavily`, then paste the key
into the `Tavily API:` row. It is saved to the `.env` in your config directory, never
shown and never written to `config.toml`.

Or use `search.provider searxng` plus `search.base_url` for your own instance. Setting
the provider **is** the consent — search then runs automatically on source-less text;
`--no-search` turns it off for one run, and `permissions.web_search = deny` turns it
off for good.
Expand Down
18 changes: 18 additions & 0 deletions docs/superpowers/OPEN-ITEMS.md
Original file line number Diff line number Diff line change
Expand Up @@ -560,3 +560,21 @@ Two readers added on the Hacker News and Mastodon patterns respectively
| 18.3 | A Lobste.rs comment permalink's story is never read | `/c/<id>` reads the comment alone (its parent is other people's words, as on Hacker News). A user who pastes `/s/<id>/<slug>#c_<id>` meaning "this thread" gets the one comment; the address says so, but the report does not |
| 18.4 | `_is_own_story` sees only absolute anchors | a relative `href="/s/<id>"` in a Lobste.rs description has no host and passes the filter; downstream `find_url` reads no address out of it either, so nothing is fetched, but the link is not named among the dropped ones |
| 18.5 | `ProviderError.code` is read by one caller | the token is parsed for every 4xx/5xx a `PoliteClient` raises; Crossref, OpenAlex and the rest could use it the same way the Lemmy reader does, and today none does |

## 19. A live `/check openai won` — 2026-09-30

A two-word claim run through evidence search surfaced three things. Two are fixed on
`fix/challenge-markers-and-tavily-key`: every Wikipedia page read as `UNVERIFIED
(blocked)` because the challenge scan matched `captcha` inside MediaWiki's inline
config script (`fetch.py`: phrase markers are now matched outside `<script>`/`<style>`
and only on small pages), and a verdict resting on an abstract-grade read did not say
so on its own row (`verify.py`: the verdict finding now carries a line saying so). The
third needed a design decision and is now decided and implemented (19.1).

### New open items

| # | Item | Note |
|---|---|---|
| 19.1 | ~~Underspecified claims pass too easily~~ **Decided and implemented 2026-09-30** | "openai won" was `SUPPORTED (found by proofpath)`, medium, on a single vanity-award page ("OpenAI's recognition with a 2026 Global Recognition Award"). The passage does entail the claim, so rules 1–3 hold; but a claim with no object ("won *what*?") is satisfied by any win. **Decided (the user): rule-based, note only.** `claims.is_underspecified` counts content words in the text the models check (`claim.hypothesis or claim.text`, so a translated claim is counted in English): hashtags, handles, punctuation and function words (the sentence openers plus auxiliaries, negation, prepositions, pronouns) are dropped; a number is one term ("12.5") and so is a run of capitalised words, a name ("Elon Musk"). **≤ 2 content terms is underspecified** (`UNDERSPECIFIED_MAX_WORDS`), **unless a linking verb stands between two content terms** — a complete predication ("The Earth is flat", "Revenue was 12.5%"). "openai won", "Elon Musk won", "OpenAI failed" and "It is true" are underspecified; "OpenAI won the award" is not. Every verdict finding on such a claim (all kinds, cited or found) carries `report.UNDERSPECIFIED` in its detail, after `ABSTRACT_BASIS` and before `FOUND_BY_PROOFPATH`; the SARIF message appends it the same way. No judge, no new dependency, and kind, level, tier and exit code are unchanged. A supported *cited* claim still produces no finding, so it gets no note. Known limits: a passive with a linking verb ("OpenAI was acquired") counts as complete, and a cited Turkish sentence is counted untranslated, so it reads slightly shorter. In title case or all caps (every content word capitalised) no names are joined, so a verb is never swallowed into one. Known small misses, documented and left: a sentence-initial adverb joins the name after it ("Yesterday Elon Musk won" is flagged), a dotted abbreviation splits ("U.S. won" is not flagged), a "BREAKING:" prefix counts as a content word, and a two-word name in title case ("Elon Musk Won") is three terms, so it is not flagged. The note is worded for every verdict kind (`underspecified claim: it names no object, time or scope, so many different events match it; check that the passage is about the one meant`). Source credibility (vanity awards, content farms) stays out of scope (§3). Evidence-search spec §2 records the rule |
| 19.2 | A bot wall inside a site's full template can read as `ok` | challenge *phrases* now count only on pages under `fetch.CHALLENGE_MAX_WORDS` (300) extracted words, so a long real article that says "CAPTCHA" is not blocked. A wall served inside the site's own layout, with menus not in `<nav>`, can exceed 300 words and be read as content (review, 2026-09-30: a 364-word probe). A title/h1 signal was considered and rejected: it would block real pages titled "CAPTCHA". `cf-chl` stays unconditional. Cloudflare's underscore traces (`_cf_chl_opt`, `__cf_chl_f_tk`) are not matched by `cf-chl` — a candidate structural marker |
| 19.3 | The markdown `## Checked` table does not mark abstract-grade support | a cited claim SUPPORTED by an abstract produces no finding (by design), so its row in `report.py`'s Checked table is the only place it appears, and it carries no abstract mark. The source's own `LOW CONFIDENCE (abstract only)` finding and the coverage line still state it; a per-row mark would make the table honest on its own (review, 2026-09-30) |
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ proofpath saying whether it found evidence for what the text claims.
| Which sentences | **A heuristic check-worthiness filter plus a cap**: at most 5 claims, each with up to 3 results. Sentences left unsearched are counted in coverage (rule 6) |
| Short text (added 2026-09-28) | When **no** sentence passes the filter, every sentence is searched, up to the cap. A one-line post ("OpenAI battı") is exactly the claim the user wants checked, so it is never filtered away |
| Language (added 2026-09-28) | The models are English-only (`nli-deberta-v3-base`, `bge-small-en-v1.5`). A claim that does not look English is **translated by the judge** when `--judge` is on: the English rendering is used for the queries and is the text the models check, while the report shows the user's sentence with a `checked as: …` note. With no judge, or a judge that did not answer, the claim is `UNVERIFIED (language not supported)` and is not searched. That is never a silent NEI |
| Underspecified claims (added 2026-09-30, OPEN-ITEMS 19.1) | **A rule-based note, never a verdict change.** A claim is underspecified when the text the models check (the English hypothesis when translated) has ≤ 2 content terms — hashtags, handles, punctuation and function words dropped, a number one term, a run of capitalised words one name — and no linking verb between two content terms ("The Earth is flat" is a complete predication; "openai won" is entailed by any page about any win). Each of its verdict findings, cited or found, carries `underspecified claim: it names no object, time or scope, so many different events match it; check that the passage is about the one meant`, after the abstract line and before the provenance tag, and the SARIF message appends it. Kind, level, tier and exit code are unchanged, and no judge or NLP dependency is used. Known limits: a passive ("OpenAI was acquired") counts as complete, and a cited Turkish sentence is counted untranslated |
| Trigger | **Automatic.** When the text has no source and a search provider is configured, it is searched without asking. Configuring the key or URL is the consent. `--no-search` turns it off for one run, and `permissions.web_search = deny` turns it off for good |

Rejected, and kept rejected: letting `--judge` answer when there is no source. That makes
Expand Down
114 changes: 114 additions & 0 deletions src/proofpath/claims.py
Original file line number Diff line number Diff line change
Expand Up @@ -448,6 +448,120 @@ def is_checkworthy(text: str) -> bool:
)


# A claim that states too little to pin down (OPEN-ITEMS 19.1): "openai won" is entailed
# by any page about any win, so its verdict weighs far less than a specific claim's.
# Counted on content words -- every term left once hashtags, handles, punctuation and
# function words are gone. A figure is one term ("12.5"), and so is a run of
# capitalised words, which is a name ("Elon Musk", "Global Recognition Award"). Two or
# fewer is underspecified: the same floor as ``CHECKWORTHY_MIN_WORDS``, but counted on
# terms that say something, so "OpenAI has won" and "Elon Musk won" (two) are caught
# while "OpenAI won the award" (three) is not. Deliberately blunt and explainable: it
# only annotates a verdict, never changes one.
UNDERSPECIFIED_MAX_WORDS = 2
# A decimal or grouped figure is one term ("86.4", "1,000"); anything else is a word
# as ``_WORDS`` reads it. The lookahead keeps "3rd" one word rather than "3" + "rd".
_TERMS = re.compile(r"\d+(?:[.,]\d+)*(?!\w)|\w+(?:['’]\w+)*") # noqa: RUF001 - a real apostrophe
# What may stand between two capitalised words of one name: "Hewlett-Packard".
_NAME_GAP = re.compile(r"[\s-]+")
# A linking verb between two content words makes a complete predication: "The Earth is
# flat" says all it means to say, where "OpenAI won" leaves out what was won. "It is
# true" is still short: its subject is not a content word.
_LINKING = frozenset(
{"is", "are", "was", "were", "be", "been", "being", "am", "isn't", "aren't", "wasn't",
"weren't"}
) # fmt: skip
# Words that carry no content of their own: the sentence openers above, plus
# auxiliaries, negation, prepositions and pronouns. A verb that is not an auxiliary
# stays a content word: "won" is what the claim says happened.
_FUNCTION_WORDS = (
_OPENERS
| _LINKING
| frozenset(
{
"has", "have", "had", "having", "do", "does", "did", "will", "would", "shall",
"should", "can", "could", "may", "might", "must", "not", "no", "never", "of",
"to", "from", "by", "with", "about", "into", "onto", "over", "under", "than",
"then", "or", "nor", "also", "just", "very", "really", "too", "which", "who",
"whom", "whose", "me", "him", "us", "them", "your", "itself", "themselves",
"hasn't", "haven't", "hadn't", "doesn't", "don't", "didn't", "won't",
"wouldn't", "can't", "cannot", "couldn't", "shouldn't", "it's", "that's",
"there's",
}
)
) # fmt: skip


def _folded_term(term: str) -> str:
return term.casefold().replace("’", "'") # noqa: RUF001 - a real apostrophe


def _is_function(term: str) -> bool:
return _folded_term(term) in _FUNCTION_WORDS


def _terms(text: str) -> list[str]:
"""Words and figures in order, with a run of capitalised content words joined into
one term. A function word never joins a name, so a capitalised sentence opener
("The Earth") stays out of it, and a capitalised first word followed by a
lowercase one is only the start of the sentence.

In title case or all caps ("OpenAI Wins The Global Award", "OPENAI WON THE AWARD")
a capital says nothing about names, and joining would swallow the verb; when every
content word is capitalised, no words are joined and each counts on its own.
"""
words = [match.group() for match in _TERMS.finditer(text)]
content = [word for word in words if not _is_function(word)]
names = not all(word[:1].isupper() for word in content)
terms: list[str] = []
end = 0
for match in _TERMS.finditer(text):
term = match.group()
if (
names
and terms
and term[:1].isupper()
and terms[-1][:1].isupper()
and not _is_function(term)
and not _is_function(terms[-1])
and _NAME_GAP.fullmatch(text[end : match.start()])
):
terms[-1] = f"{terms[-1]} {term}"
else:
terms.append(term)
end = match.end()
return terms


def is_underspecified(text: str) -> bool:
"""Whether ``text`` names at most ``UNDERSPECIFIED_MAX_WORDS`` content terms and is
not a complete predication (a linking verb between two content terms).

The rules read English. A claim that was not translated is counted as written: a
cited Turkish sentence, whose suffixes fold what English says in several words into
one, reads a little shorter than its translation would. Known, and small: a cited
sentence is rarely two words long.

Other known small misses, left as they are: a sentence-initial capitalised adverb
joins the name after it ("Yesterday Elon Musk won" is flagged), a dotted
abbreviation splits into letters ("U.S. won" is not flagged), and a "BREAKING:"
prefix counts as a content word.
"""
terms = _terms(_TAGS.sub(" ", text))
content = [not _is_function(term) for term in terms]
if sum(content) > UNDERSPECIFIED_MAX_WORDS:
return False
return not any(
_folded_term(term) in _LINKING and any(content[:at]) and any(content[at + 1 :])
for at, term in enumerate(terms)
)


def underspecified(claim: Claim) -> bool:
"""``is_underspecified`` on the text the models check: the English rendering of a
translated claim, or the claim itself."""
return is_underspecified(claim.hypothesis or claim.text)


def checkworthy(doc: Document, *, limit: int) -> Checkworthy:
"""Claims for a document that cites nothing, to be searched for.

Expand Down
53 changes: 50 additions & 3 deletions src/proofpath/fetch.py
Original file line number Diff line number Diff line change
Expand Up @@ -42,14 +42,52 @@
RETRIES = 2 # same budget as PoliteClient: three attempts on 429/5xx

# A 200 that is really a bot wall. Matched case-insensitively against HTML bodies.
CHALLENGE_MARKERS = (
"cf-chl",
# Structural: Cloudflare challenge machinery that only a live wall renders -- the
# widget ids (#cf-chl-widget-...) and the challenge options object
# (window._cf_chl_opt). They are trusted anywhere in the raw head, scripts
# included, on a page of any size: on some walls a script is their only trace.
CHALLENGE_STRUCTURAL_MARKERS = ("cf-chl", "_cf_chl_opt")
# Cloudflare's challenge URL tokens (?__cf_chl_f_tk=, __cf_chl_rt_tk,
# __cf_chl_jschl_tk__). A wall's form action carries them, but so do real pages
# after the challenge was passed: the clearance redirect's URL echoed in og:url or
# a canonical link, or an outbound link copied from a challenged address. So they
# are matched against the raw head but count only like a phrase, on a page under
# CHALLENGE_MAX_WORDS -- a wall is small, so no wall is lost to the gate.
CHALLENGE_TOKEN_MARKERS = ("cf_chl",)
# Phrases a wall shows its reader. Matched against the head with the *contents* of
# every <script> and <style> removed, because real pages name these words in code
# -- every Wikipedia page's inline MediaWiki config (RLCONF) says "hcaptcha", which
# once made all of Wikipedia read as blocked. The opening tags stay, attributes and
# all: a DataDome or reCAPTCHA wall's only trace is often a script src
# (captcha-delivery.com, /recaptcha/api.js). <title> and <noscript> stay too: a
# wall's "Just a moment..." lives in its title and "enable JavaScript" often in a
# noscript.
CHALLENGE_PHRASE_MARKERS = (
"just a moment",
"access denied",
"captcha",
"enable javascript and cookies",
)
CHALLENGE_SCAN_BYTES = 200_000 # challenge pages are small; do not lowercase a whole PDF
# A phrase marker only counts on a small page. Walls are tiny -- the ones measured
# for this module extracted 0 to a few dozen words -- while a real article can
# legitimately say "CAPTCHA" or "access denied" in its prose (the GPT-4 TaskRabbit
# story). Reading such an article as blocked would discard real content for a false
# UNVERIFIED(blocked), so past this many extracted words a phrase is just text.
# The structural markers are not bound by it; the URL tokens are. Well above any
# wall, well below an article.
# Known gap, recorded rather than fixed: a wall served inside a site's full template
# (menus not in <nav>) can pass this bar and read as ok.
CHALLENGE_MAX_WORDS = 300
# A <script>/<style> element's contents, for the phrase scan; group 1 is the
# opening tag, which is kept. ``\Z`` also drops the contents of one the
# CHALLENGE_SCAN_BYTES cut left unterminated, so a script straddling the boundary
# cannot leak its text back in. The ``(?=[\s/>])`` lookahead keeps <noscript> (no
# ``<`` directly before "script") and custom elements like <style-guide> out:
# ``\b`` would match those and delete visible text up to the end.
_CODE_ELEMENTS = re.compile(
r"(<(script|style)(?=[\s/>])[^>]*>).*?(?:</\2\s*>|\Z)", re.IGNORECASE | re.DOTALL
)
# A 2xx HTML page with no extractable text at all is a bot wall too: the
# JavaScript/cookie shell publishers serve instead of a 403 (spec section 6.1
# measured Science at "403, 3 words"). A live 50-DOI run showed every observed
Expand Down Expand Up @@ -227,8 +265,17 @@ def classify(status: int | None, kind: Kind, body: bytes) -> Verdict:


def _has_challenge(body: bytes) -> bool:
"""Whether an HTML body is a bot wall by its markers. ``classify`` and
``explain`` both ask this, so a verdict and its note never disagree."""
head = body[:CHALLENGE_SCAN_BYTES].decode("utf-8", errors="replace").lower()
return any(marker in head for marker in CHALLENGE_MARKERS)
if any(marker in head for marker in CHALLENGE_STRUCTURAL_MARKERS):
return True
visible = _CODE_ELEMENTS.sub(r"\1", head)
phrase = any(marker in visible for marker in CHALLENGE_PHRASE_MARKERS)
if not phrase and not any(marker in head for marker in CHALLENGE_TOKEN_MARKERS):
return False
# Parsed only on a phrase or token hit: most pages never pay for it here.
return _html_words(body) < CHALLENGE_MAX_WORDS


def _html_words(body: bytes) -> int:
Expand Down
Loading
Loading