diff --git a/CHANGELOG.md b/CHANGELOG.md index 95fb536..653db07 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,36 @@ All notable changes to this project are documented here. The format follows ## [Unreleased] +### Added +- **`Tavily API:` in `/config`.** The search section of the settings panel has a key + row: press Enter, paste the key, Enter again. It is saved to the `.env` in your + config directory (mode 0600 on macOS/Linux), never to `config.toml` or the project's + `.env`, and it is never shown: the row says only whether a key is saved and where. + Backspace removes it. Saving a key does not turn search on: while + `search.provider` is not `tavily` the row says the key is unused, and with + `tavily` selected and no key it says one is needed. +- **A note on claims too short to pin down.** A claim like "openai won" is supported + by almost any page about any win. A claim with two content words or fewer, and no + linking verb (`is`, `are`, `was`, …) between them, now carries a "short claim" + note on each of its verdict rows, SARIF included. The note tells you to check + that the quoted passage is about the event you meant. It is a rule, not a model, + and it changes no verdict, tier or exit code. + +### Fixed +- **Wikipedia pages no longer read as `UNVERIFIED (blocked)`.** The bot-wall check + matched `captcha` inside MediaWiki's inline configuration script, so every Wikipedia + page was reported blocked at all four ladder steps despite answering 200. Challenge + phrases are now matched outside `' + b"

Istanbul

" + ARTICLE_PROSE.encode() + b"

" +) +# A long real article whose visible text legitimately mentions a CAPTCHA. +CAPTCHA_ARTICLE = ( + b"

GPT-4 and the TaskRabbit worker

" + b"

During testing the model asked a human worker to solve a CAPTCHA for it.

" + + ARTICLE_PROSE.encode() + + b"

" +) +TITLE_ONLY_CHALLENGE = b"Just a moment..." +CF_CHL_IN_SCRIPT = ( + b"" + b"

Checking your browser before accessing the site.

" +) +# Cloudflare's underscore forms of the same token: the challenge options object in +# an inline script, and the challenge token in a form action. +CF_CHL_OPT_SCRIPT = b"" +CF_CHL_FORM_TOKEN = ( + b'
' + b"

One more step

" +) +# Real articles that still carry a Cloudflare URL token after the challenge was +# passed: the clearance redirect echoed in og:url, and an outbound link copied +# from a challenged URL. Long pages, so the token must not read as a wall. +CF_CHL_RT_TOKEN_ARTICLE = ( + b'' + b"

The strait

" + ARTICLE_PROSE.encode() + b"

" + b"" +) +CF_CHL_JSCHL_LINK_ARTICLE = ( + b"

The strait

" + ARTICLE_PROSE.encode() + b"

" + b'

See the source.

' + b"
" +) +# Walls whose only marker trace is a script's src attribute (DataDome, reCAPTCHA): +# the opening tag must survive the script-content removal. +DATADOME_WALL = ( + b'' + b"

Please enable JS and disable any ad blocker

" +) +RECAPTCHA_SRC_WALL = ( + b'' + b"

Please verify you are a human

" +) +# A custom element whose name starts with "style" is not a

Hello there

", "ok"), (503, "html", CHALLENGE, "blocked"), (503, "html", b"

maintenance

", "retryable"), (401, "html", b"", "blocked"), @@ -287,6 +361,53 @@ def test_classify(status: int | None, kind: fx.Kind, body: bytes, expected: str) assert fx.classify(status, kind, body) == expected +def test_article_prose_is_past_the_challenge_word_limit() -> None: + assert fx._html_words(WIKIPEDIA_PAGE) >= fx.CHALLENGE_MAX_WORDS + assert fx._html_words(CAPTCHA_ARTICLE) >= fx.CHALLENGE_MAX_WORDS + assert fx._html_words(CF_CHL_RT_TOKEN_ARTICLE) >= fx.CHALLENGE_MAX_WORDS + assert fx._html_words(CF_CHL_JSCHL_LINK_ARTICLE) >= fx.CHALLENGE_MAX_WORDS + + +def test_script_cut_by_the_scan_boundary_does_not_leak_a_marker() -> None: + """A " + assert body.index(b"captcha") < fx.CHALLENGE_SCAN_BYTES < body.index(b"") + assert not fx._has_challenge(body) + assert fx.classify(200, "html", body) == "ok" + + +def test_unterminated_script_at_the_end_is_ignored() -> None: + body = b"

Short page.