A retrieval system over a corpus of research papers, built by reading the
retrieval literature and implementing it — RAPTOR, STORM, PaperQA2's reranking
pass, HyDE — with cost governance in the call path rather than in a policy
document. Every paid call estimates its price before it runs, caches its result,
and stops to ask when the estimate crosses a threshold. --dry-run walks the
entire pipeline and spends nothing.
It runs against ~900 markdown papers in the deployment it was built for. It runs against six synthetic papers in this repository, offline, with no API key, in about twelve seconds.
$ nris search "how does reranking affect recall"
Top papers for: "how does reranking affect recall" [TF-IDF]
1. [Paper 002] Cross-Encoder Reranking of Retrieved Passages
Score: 0.103 (1 matching chunks)
2. [Paper 005] Cost Accounting for Retrieval Augmented Pipelines
Score: 0.079 (1 matching chunks)
3. [Paper 001] Chunk Size and Overlap in Passage Retrieval
Score: 0.045 (1 matching chunks)
A directory of markdown papers. Each file is one paper; the leading digits of the filename are its number, the first heading is its title. From that the system builds:
| Layer | What it gives you | Cost |
|---|---|---|
| TF-IDF index | Unigram + bigram, sublinear TF, over overlapping chunks | $0 |
| Dense index | Sentence-transformer embeddings in ChromaDB | $0, optional |
| Hybrid search | Both, score-normalized and merged 50/50 | $0 |
| Knowledge graph | Concept co-occurrence, BFS paths, hub ranking | $0 |
| RAPTOR tree | Recursive clusters and summaries, multi-resolution search | $0 |
| Parameter table | Numeric values papers report, pulled by regex | $0 |
| Cited answers | Reranked passages, summarized, synthesized with citations | ~$0.05 |
| Surveys | Multi-perspective article on a topic, fully cited | ~$0.50–1.00 |
The first six run offline forever. The last two are the only ones that cost money, and both refuse to start without confirmation above the threshold.
Each entry says what the system does and how closely it follows its source. No paper text appears anywhere in this repository; the papers are cited by ID so you can read them yourself.
nris_raptor_full.py · Sarthi et al., RAPTOR: Recursive Abstractive Processing
for Tree-Organized Retrieval, arXiv:2401.18059
Embeds every chunk, reduces to at most 50 dimensions with PCA, clusters with KMeans, and picks k by silhouette score over k = 2..max rather than fixing it in advance. Each cluster is summarized, the summaries are re-embedded, and the procedure recurses until the level is smaller than the minimum cluster size. Retrieval then runs over all levels at once, so a thematic question matches an upper-level summary and a specific one matches a leaf.
Faithful: the tree construction — PCA, soft-k selection by silhouette,
recursive re-clustering of summaries, collapsed multi-level retrieval.
This implementation's choice: summarization is extractive, taking the leading
sentence of each member chunk. That keeps the entire tree build at $0 and makes
it deterministic and testable, which is what
tests/test_raptor.py exercises. nris_core.py also ships a cheap section-level
variant that skips clustering entirely, for corpora with reliable headings.
nris_synthesis.py · Shao et al., Assisting in Writing Wikipedia-like Articles
From Scratch with Large Language Models, arXiv:2402.14207
Discovers N expert perspectives on a topic, runs a multi-turn conversation per perspective in which the expert asks and a grounded answerer replies from retrieved passages, folds the transcripts into an outline, then writes the article section by section with citations carried through.
Faithful: the four-stage pipeline — perspective discovery, perspective-guided question asking, grounded answering, outline-then-write. This implementation's choice: the grounding source is the local index, so answers cite corpus papers rather than web URLs. A cross-model verification pass is available on top, which re-reads the finished article as a critical reviewer and reports claims it cannot trace to a source.
nris_query.py · Skarlinski et al., Language agents achieve superhuman
synthesis of scientific knowledge, arXiv:2409.13740 (lineage:
arXiv:2312.07559)
After retrieval, each passage goes back to the model in batches of five with the query attached. The model returns a 0–10 relevance score, a 100–200 word summary written for that query, and a one-sentence key finding. Answer generation then reads the summaries, not the raw passages, so the context window carries query-relevant prose instead of whatever happened to be adjacent.
Faithful: the RCS step itself — rerank plus contextual summarization in one pass, top-k by model relevance, answer from summaries with per-claim citation. This implementation's choice: chunking follows the 3000/750 configuration PaperQA2 reports, and results are cached per (query, passage) pair so a repeated question costs nothing. Citation traversal ranks by shared knowledge-graph entities (Jaccard), which gives the neighbourhood of a paper across a local markdown corpus.
nris_query.py · Gao et al., Precise Zero-Shot Dense Retrieval without
Relevance Labels, arXiv:2212.10496
Writes a fabricated answer in the register of a paper and retrieves with it. A question and a passage share few terms; an invented answer and a passage share many. The fabricated text is never shown to anyone — its vocabulary is the whole product.
Faithful: the mechanism, including the paper's own point that factual accuracy of the hypothetical text is irrelevant. This implementation's choice: the expansion is concatenated to the original query and fed to the same hybrid retriever, and it is cached per query, so the one generation call is paid once.
nris_discovery.py · Semantic Scholar Academic Graph API
Searches the configured topics, scores results by citation count, recency, open access and abstract keyword hits, and remembers what it has already seen. Requests are rate-limited to the unauthenticated 1/sec allowance with backoff and retry, and ingestion converts arXiv HTML to markdown, assigns the next paper number, and registers a content hash so the same paper cannot enter twice under two titles.
This is the part that is built in rather than bolted on.
Everything is priced before it runs. Token counts are estimated from text
length, multiplied by the published per-token rates in nris_query.py, and
printed. tests/test_cost_and_cache.py pins the estimator to those constants,
so the number on screen is derivable from the pricing table rather than typed in
by hand.
One gate, and it fails closed. nris_spend.py is the single place the
system decides it may spend. Under the threshold it proceeds silently. Over it,
it asks and requires a literal y or yes. A non-interactive process — cron,
CI, a piped stdin — raises EOFError on the prompt, which is caught and treated
as a refusal. Silence is never consent.
A refusal aborts, it does not warn. When the gate says no, the caller
returns before the API client is constructed.
tests/test_spend_gate.py proves this by installing an object in
sys.modules["anthropic"] that raises on any attribute access, then refusing at
the prompt and asserting the call still completes.
Repetition is free. RCS results are cached on a SHA-256 of (query, passage). Asking the same question over the same corpus twice makes zero requests the second time, and the tests assert the call count, not just the answer.
--dry-run is a real run. It performs retrieval, batching, ranking and
context assembly, reports what the call would have cost, and never opens a
socket.
Set the threshold with NRIS_CONFIRM_THRESHOLD (default $0.10), pre-approve
with --yes, and re-price for a different model with
NRIS_INPUT_COST_PER_MTOK / NRIS_OUTPUT_COST_PER_MTOK.
| Command | Typical cost |
|---|---|
search, graph, raptor, extract, stats, paths |
$0 |
discover |
$0 (free Semantic Scholar tier) |
ask |
~$0.05, and $0 on a repeat |
ask --hyde |
~$0.07 |
implement |
~$0.20–0.50 |
synthesize |
~$0.50–1.00 |
git clone <this repo> && cd nris
pip install -r requirements.txt # numpy, scikit-learn, joblib, pytest
export NRIS_CORPUS_DIR=/path/to/your/markdown/papers
python3 nris.py paths # confirm where everything will land
python3 nris.py build # TF-IDF index ($0)
python3 nris.py search "your question" # retrieval ($0)
python3 nris.py graph build # knowledge graph ($0)
python3 nris.py ask --dry-run "your question" # price the paid path ($0)
export ANTHROPIC_API_KEY=... # only for `ask`, `synthesize`, `implement`
python3 nris.py ask "your question" # cited answer (~$0.05)Where things live. NRIS_HOME (default ~/.nris) holds every derived
artifact — index, cache, graph, RAPTOR tree, generated code. The repository holds
source and configuration only, and a test asserts that building an index leaves
the repository byte-identical.
Making it yours. Three JSON files in config/ carry all the domain
knowledge, and none of it is in the code:
vocabulary.json— the concept terms the knowledge graph tracksparameters.json— regexes for the numeric values your field reportsdiscovery_queries.json— the topics the Semantic Scholar scan watches
Replace them and the system follows.
tests/test_graph_and_extraction.py swaps the vocabulary mid-test and asserts
the resulting graph changes to match.
An offline test suite. No API key, no network, no model download, no corpus of
your own required — it ships six short synthetic papers in
tests/fixtures/corpus/, written for the purpose, and indexes those.
$ python3 -m pytest
........................................................................ [ 42%]
........................................................................ [ 84%]
........................... [100%]
171 passed in 11.93s
| File | Tests | What it covers |
|---|---|---|
test_chunking.py |
13 | Overlap arithmetic, coverage, section attribution, sentence-boundary preference |
test_retrieval.py |
24 | Index build, six queries returning six different papers, save/load fidelity, hybrid fallback, query routing |
test_cost_and_cache.py |
19 | Estimator derived from the pricing constants, cache key determinism and delimiter safety, corrupt-entry recovery |
test_spend_gate.py |
28 | Refusal in every form, fail-closed on EOF, abort-before-client across RCS, HyDE, synthesis and Paper2Code |
test_raptor.py |
23 | Silhouette recovering a known k on synthetic blobs, PCA above the dimension cap, tree structure, save/load, level-restricted search |
test_rcs_pipeline.py |
16 | Batching, prompt contents, cost from the usage block, JSON parsing through markdown fences and prose, degradation on a failed batch |
test_graph_and_extraction.py |
33 | Graph construction and traversal, vocabulary swap, parameter regexes, source classification, content-hash deduplication |
test_paths_and_cli.py |
15 | Environment-driven relocation, repository stays clean, CLI build/search/ask-dry-run end to end |
Two guarantees are enforced by tests/conftest.py for every test in the suite.
socket.socket is replaced with a class that raises on construction, so a test
that reaches an API fails loudly instead of quietly billing someone. And every
NRIS_* path is redirected into a per-test temporary directory before any module
resolves a path, so the suite cannot touch a real index, cache or corpus.
The paid pipelines are exercised against a RecordingLLM that returns canned
responses with a usage block, which is what lets batching, parsing and cost
accounting be asserted precisely.
nris.py Unified CLI
nris_core.py Corpus loading, chunking, TF-IDF, dense index, section RAPTOR
nris_query.py Retrieval, RCS reranking, HyDE, cited answers, citation traversal
nris_raptor_full.py Clustering RAPTOR: PCA, KMeans, silhouette, recursive summaries
nris_synthesis.py STORM survey pipeline, cross-model verification
nris_graph.py Concept co-occurrence graph, BFS paths, hubs
nris_discovery.py Semantic Scholar search, scoring, rate-limited ingestion
nris_extract.py Regex parameter extraction
nris_llm_extract.py Model-based entity extraction
nris_paper2code.py Paper to runnable implementation
nris_contradict.py Cross-paper contradiction detection
nris_templates.py Source-type classification and extraction templates
nris_tracker.py Usage and graph-growth telemetry
paper_registry.py Append-only registry, content-hash deduplication
nris_paths.py Every filesystem location, in one place
nris_config.py Loads the three domain config files
nris_spend.py The spend gate
config/ vocabulary.json, parameters.json, discovery_queries.json
tests/ 171 tests and a six-paper synthetic corpus
Heavy imports — sentence_transformers, chromadb, anthropic, joblib — are
deferred to the function that needs them, so nris.py search starts fast and
the offline layer works with none of them installed.
This is one component of a larger system, which remains in development. The parts that are not published are inseparable from live credentials and private data, which is the whole of the reason. What is here is the part that stands on its own: the retrieval engine, the techniques, the cost governance, and the tests that hold them.
Read access to more can be arranged for a serious conversation.
MIT — see LICENSE.