Repository navigation
Evidence search for claims with no source, and a local fallback judge - #9
Merged
Merged
Conversation
A pasted post or plain text with no link, marker or bibliography used to end in a parse error. It is now searched: up to five check-worthy sentences go to Tavily or a SearXNG instance, and the pages found are read and scored through the usual fetch and NLI path. Each finding is marked "found by proofpath", and an empty search is reported as NO EVIDENCE FOUND (searched), never as REFUTED. A judge writes the queries and translates non-English claims; when it is limited, the sentence itself is searched and the run says so. The judge now falls back from Groq to a local qwen3.5:9b in Ollama the moment Groq is rate-limited or down. The request that failed is sent again to the local model, the switch is announced (stderr, TUI line, report), and the next run starts on Groq again. Nothing is ever pulled. Also: --no-search, the search section in /config, footer lines, and an AVeriTeC --search eval mode that excludes the claim's own fact-check site. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BS4saoFyA3Ga9HgfRUUVux
The README gains a section on searching claims with no source, a row in the What it reads table, and a note on the judge's fallback to a local qwen3.5:9b. The landing page's reads list and the X hint name the new path. The AVeriTeC score is a marker until the corrected eval runs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BS4saoFyA3Ga9HgfRUUVux
A page the search found through a DOI or arXiv URL was reported as a
GHOST REFERENCE, because the resolver compared a title against a URL.
Found pages now go straight to the web reader, and an entry that is only
an identifier is never called a ghost: it resolves, or it is AMBIGUOUS.
A v5 cache migration drops GHOST resolutions cached under the old rule.
The English detector flagged ordinary English ("MIT", "et al.", "Los
Angeles") as foreign, then missed short Spanish, French and German
sentences. It now ignores capitalised tokens, counts accented lowercase
words and a fixed set of function words, and a judge's translation is
used unless the original is certainly English.
A run that hit the claim cap now exits 1 even when every searched claim
was supported, so a partly searched run cannot look clean. The AVeriTeC
eval measures the product's own search path, and pages proofpath found
on its own are fetched without the contact address.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BS4saoFyA3Ga9HgfRUUVux
The product's own search path reads the pages it finds for 100 AVeriTeC dev claims and scores 0.404 three-way, against a 0.708 majority baseline. That is better than the 0.270 the gold source URLs gave, but the gate of spec §17.1 is not met, so the feature stays experimental. The run from 2026-09-28 used a path the final review replaced and is dropped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BS4saoFyA3Ga9HgfRUUVux
|
The latest updates on your projects. Learn more about Vercel for GitHub. |
The PR checklist asks for every new reported state to be in spec §15; the four from the evidence-search spec were only there. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BS4saoFyA3Ga9HgfRUUVux
The README's state table and the landing page's state list, X row and judge paragraph still described the tool before this branch: no search states, and a judge that went quiet when its provider stopped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BS4saoFyA3Ga9HgfRUUVux
When the judge's API gives out and no local model can take over, the run now says how to add one: install Ollama and pull qwen3.5:9b, or only pull the model when Ollama is already there. It goes out once per run on the switch notice's channels, and never when the user turned the fallback off. Nothing is installed; it is only a line. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BS4saoFyA3Ga9HgfRUUVux
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
Text that cites nothing is searched on the web: Tavily or SearXNG, experimental, opt in by setting a provider. The pages found are read and scored like cited sources.
When Groq is rate-limited or stops answering, the judge switches at once to an installed local
qwen3.5:9band says so.Why
A pasted post or plain claim with no link used to end in a parse error, and there was nothing to check it against (OPEN-ITEMS §17.1). Groq's free tier also runs out, and the judge then went silent.
In more detail:
FOUND BY PROOFPATH (not cited by the author).NO EVIDENCE FOUND (searched), neverREFUTED.checked as:line. With no judge it isUNVERIFIED (language not supported).GHOST REFERENCE. A v5 cache migration drops ghost results cached under the old rule.Checklist
ruff checkandruff format --checkare cleanpytestpasses🤖 Generated with Claude Code
https://claude.ai/code/session_01BS4saoFyA3Ga9HgfRUUVux