Local RAG Experiment

A from-scratch, fully local Retrieval-Augmented Generation (RAG) system over a Wikipedia subset. Built end to end on Apple Silicon to explore the practical trade-offs of small-model RAG: chunking, embeddings, vector search, prompt construction, and on-device generation. Everything runs offline after the initial model and dataset downloads. No API keys, no cloud.

Why RAG?

A language model only knows what was baked into its weights at training time. RAG bolts on a retrieval step so the model can answer from a specific corpus you control: chunk the documents, embed them, and at query time pull the most relevant chunks into the prompt as context. The interesting question this repo probes is whether that grounding actually happens. With a small 500-article corpus and a base (not instruction-tuned) model, it often does not: in the first five-probe eval, only 1 of 5 answers was genuinely grounded in the retrieved text. The other correct answers came from the model's own parametric knowledge, which is exactly the behaviour RAG is meant to replace. A second run then fixed a real defect in the pipeline (a third of every chunk was being silently dropped before embedding), which raised grounding to 3 of 5 and left the error count unchanged at 1 of 5. Documenting both gaps, and the fact that fixing retrieval did not fix answers, is the point of the project.

Hardware & Stack

• Hardware: Apple M4 MacBook Pro (arm64)
• LLM runtime: MLX via mlx-lm
• LLM: Phi-2 (microsoft/phi-2), a small base model, not instruction-tuned
• Embeddings: sentence-transformers/all-MiniLM-L6-v2 (384-dim)
• Vector store: ChromaDB (local, persistent)
• Dataset: HuggingFace wikimedia/wikipedia, config 20231101.simple
• Language: Python 3.11

What This Demonstrates

• A complete RAG pipeline built from parts: streaming ingest, word-window chunking, sentence-transformer embeddings, persistent vector search, and retrieval-augmented prompting.
• Running an LLM natively on Apple Silicon with MLX, no GPU or cloud.
• An honest evaluation that surfaces the failure modes of small-corpus, base-model RAG rather than hiding them.

How RAG Works (5 steps)

1. Chunk: Split each source document into overlapping word windows (here, 175 words with 50-word overlap) so each chunk fits the embedder and stays semantically coherent. Chunk size matters more than it looks: see the truncation finding below.
2. Embed: Run every chunk through a sentence-transformer to produce a dense vector that captures meaning rather than surface tokens.
3. Index: Store the chunks and vectors in a vector database (ChromaDB), keyed for fast nearest-neighbor lookup.
4. Retrieve: Embed the user's question with the same model and pull the top-k most similar chunks from the index.
5. Generate: Feed those chunks plus the question into a local LLM, instructing it to ground its answer in the provided context.

Installation

python3.11 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

Requirements:

• Python 3.11 (arm64 build on Apple Silicon)
• ~6 GB free disk: the Phi-2 weights are ~5 GB, plus ~90 MB for the MiniLM embedder and ~25 MB for the Chroma index over 500 articles
• A working HuggingFace cache (~/.cache/huggingface). Phi-2 is pulled on first run if not already present.

Usage

Run in this order:

# 1. Build the index (streams 500 wiki articles, then chunk, embed, store)
python src/ingest.py

# 2. Ask one-off questions
python src/query.py "Your question here"

# 3. Run the canned evaluation suite (writes data/eval_results.txt)
python src/evaluate_rag.py

# 4. Regenerate the results chart from the committed judgments
python src/plot_results.py

Results

Five canned probes from src/evaluate_rag.py, run twice: once at the original 300-word chunk size, and once at 175 words after discovering that the embedder was silently truncating most chunks (see finding 2). Everything else is identical between runs. Grading is by hand from the committed transcripts, eval_results_300w.txt and eval_results.txt, recorded in data/eval_summary.csv.

#	Question	Retrieved sources (300w → 175w)	Outcome 300w	Outcome 175w
1	Who was Albert Einstein?	Physics, Physics, String theory → Physics ×3	correct, ungrounded	grounded, with an error
2	What is the capital of France?	France ×3 → France ×3	correct, ungrounded	correct, ungrounded
3	What is photosynthesis?	Plant, Plant, Tree → Plant, Plant, Life	grounded, correct	grounded, correct
4	When did World War II end?	Cold War, September, Japan → Cold War, September, Italy	correct, ungrounded	wrong
5	What is the Great Wall of China?	Beijing, Wall, China → Beijing, Beijing, Wall	wrong	grounded, correct

Grounded answers went from 1/5 to 3/5. The number of wrong answers stayed at 1/5 — the identity of the failing probe changed, the count did not.

Probe by probe, at 175 words: probe 3 stayed a clean win with visibly better context. Probe 5 flipped from wrong to right, because retrieval finally surfaced the Beijing chunk that names the Great Wall instead of a generic definition of "wall". Probe 1 became grounded but picked up a misattribution, appending "matter emits radiation" to Einstein when the very next sentence in the same chunk credits it to Max Planck. Probe 2 was untouched: "Paris" now appears in the retrieved text, but only as a tourist destination, never as the capital, so the answer is still parametric. And probe 4 regressed outright, which is the most interesting result in the table (see finding 3).

Regenerate this chart with python src/plot_results.py.

Key Findings

1. Ungrounded generation is the dominant failure. In the 300-word run, three of five answers were correct only because Phi-2 already knew the fact. A base model does not reliably defer to retrieved context, so a right answer is not evidence that retrieval worked. Shrinking chunks cut this to one of five, but did not convert those into correct answers, which is finding 3.
2. A third of every chunk was never being embedded. Chunks were 300 words, about 380 tokens, but all-MiniLM-L6-v2 truncates at 256. 79% of indexed chunks exceeded that ceiling and 28% of all token content never reached the embedding model, even though the full text was still stored and shown to the LLM as context. Retrieval was therefore matching against a systematically clipped view of the corpus. Re-chunking at 175 words dropped truncation to 0.9% of tokens and measurably improved retrieval: probe 1 went from a tangential String-theory chunk to three chunks that name Einstein, and probe 5 from a generic "wall" definition to the Beijing chunk describing the Great Wall. This was a real defect, not a tuning preference, and it had been sitting unexamined behind the "weak retrieval" explanation.
3. Better retrieval did not produce better answers. This is the most uncomfortable result here. Grounding tripled and the error count stayed at one, because probe 4 traded a lucky-correct answer for a confidently wrong one, cancelling out the probe 5 win. At 300 words the model ignored its context and answered "September 2 1945" from memory, which is right. At 175 words the retrieved context contained, verbatim, "September 2 1945: Japan signs the final surrender at the end of World War II" — and the model instead grounded on a neighbouring chunk about Italy's armistice and answered "September 8, 1943". The right answer was in the prompt and it picked the wrong sentence. Fixing retrieval moved the bottleneck to evidence selection, which a base model is poorly equipped for.
4. Grounding in the wrong chunk is the persistent failure mode. It simply moved: at 300 words it produced the bad Great Wall answer, at 175 words the bad WWII date. Retrieval quality changes which probe it bites, not whether it happens.

Known Limitations

• Tiny eval set. Five hand-graded probes are illustrative, not a benchmark. There is no automated scoring, so grading is subjective and not regression-safe.
• Base model, not instruction-tuned. Phi-2 does not reliably follow the "answer only from context" instruction. An instruction-tuned model would defer to retrieval far more often (see below).
• Small corpus. 500 articles means thin coverage; many questions have no answer-bearing chunk to retrieve.
• Re-ingest is not idempotent. ingest.py uses collection.add, which raises on duplicate IDs. Delete db/ before re-running, or switch to upsert.
• Eval output is overwritten each run. The 300-word transcript was archived by hand before the 175-word run; there is no automatic run history, so comparing more than two configurations would need a real harness.
• Chunking cuts across facts. Word-window chunking is dependency-light but can split a fact across a chunk boundary.
• Embedding truncation is reduced, not eliminated. At the current 175-word chunk size, 4.2% of chunks still exceed the embedder's 256-token ceiling and 0.9% of total token content is dropped, down from 79% and 28% at 300 words. The remainder is not prose: it is leftover wikitable markup (style="width:115px;" and similar, in articles like Belgium, Red, and Taiwan) that tokenizes at roughly 3.8 tokens per word instead of the corpus average of 1.27. Stripping markup at ingest would close the gap; word-count chunking cannot, because the problem is token density rather than length.
• Chunk size was tuned on one 5-probe eval. 175 words was chosen from the measured token distribution, then validated on the same five probes used to report results. With an n of 5 and hand grading, the before/after difference is directional evidence, not a benchmark result.

Potential Improvements

1. Instruction-tuned LLM: swap Phi-2 for Mistral-Instruct, Llama-3-Instruct, or Phi-3-mini so the model actually follows "answer only from context". Now the highest-value change by some distance: finding 3 shows retrieval is no longer the binding constraint, evidence selection is.
2. Reranking: run a cross-encoder (e.g. bge-reranker) over the top-k retrievals to push the most relevant chunk to position 1. Probe 4 would be the test case, since the answer-bearing chunk was retrieved but not the one the model used.
3. Strip wikitable markup at ingest: would close the residual 4.2% truncation noted above, and stop table syntax competing with prose for embedding space.
4. Hybrid search: combine BM25 (lexical) with semantic search to catch cases where the question shares rare terms with the source.
5. Larger corpus: ingest the full simple-Wikipedia dump (~200k articles) for serious coverage.
6. A real eval harness: automated scoring and per-run history, so chunk-size changes like this one can be compared over more than five probes.

Project Structure

rag-experiment/
├── src/
│   ├── ingest.py         # stream, chunk, embed, persist to ChromaDB
│   ├── query.py          # CLI: question, retrieval, Phi-2 answer
│   ├── evaluate_rag.py   # batch eval over the fixed probe set
│   └── plot_results.py   # render data/eval_summary.png from the CSV
├── tests/
│   └── test_ingest.py    # unit tests for chunk_text (run: pytest)
├── data/
│   ├── eval_results.txt       # transcript, current 175-word run (committed)
│   ├── eval_results_300w.txt  # transcript, original 300-word run (committed)
│   ├── eval_summary.csv       # hand-graded outcome per probe, both runs
│   └── eval_summary.png       # results chart (committed)
├── db/                   # ChromaDB persistent store (gitignored)
├── requirements.txt
├── .python-version
├── README.md
└── LICENSE

License

Released under the MIT License.

Acknowledgements

Built with assistance from Claude (Anthropic) for code generation and project scaffolding. All system design, evaluation, and analysis are my own.