safe_store: The Local Multi-Modal Vector, Graph & Semantic Engine

safe_store is an ultra-fast, local, and sovereign knowledge engine for Python. It transforms unstructured documents (PDF, DOCX, HTML, Markdown, Code) and structured datasets (CSV, Excel XLSX, SQLite) into an interconnected, queryable knowledge base combining:

1. 🧠 Dense Semantic Vector Search: Embeddings powered by Sentence-Transformers, Ollama, OpenAI, Cohere, Lollms, or TF-IDF.
2. ⚡ Sparse Lexical Search (BM25): Native SQLite FTS5 full-text indexing for exact technical identifiers, part numbers, and error codes.
3. 📖 Full Document & Context Window Retrieval: Query entire documents aggregated from chunk hits, retrieve surrounding chunk neighborhoods with window expansion, or paginate through document content.
4. 🧩 Overlapping Chunk Reconstruction & Chronological Fusion: Automatically fuses adjacent overlapping chunks in chronological order, deduplicating repetitive boundary seams, bridging gaps with ..., and uniting metadata into a single context header.
5. 🚀 Multi-User / Multi-Process Shared Server & Dynamic Batching: Eliminates OOM crashes by running a persistent shared SentenceTransformer model daemon across processes with dynamic micro-batching and an explicit shutdown command.
6. 🤖 Tool-Agnostic LLM Callable Generator: Inject ANY LLM or generation tool (OpenAI, Anthropic, Ollama, LangChain, local models, or lambdas) via a clean callable signature.
7. 🕸️ Dynamic & Ontology Knowledge Graph: Open-ended concept/entity extraction from text files or strict TBox/OWL schema mapping, with live chunk extraction reporting.
8. ⚖️ OWL/RDFS Semantic Reasoning & Exclusivity Engine: Automatic deductive inference (inverses, symmetry, transitivity, subclasses, owl:sameAs) and contradiction verification (owl:disjointWith).
9. 🔍 W3C SPARQL 1.1 Query & Update Engine: Native TBox/ABox ontology management, declarative tabular mapping, and full SPARQL (SELECT, ASK, CONSTRUCT, DESCRIBE, and INSERT/DELETE DATA updates).
10. 📐 TBox / ABox Graph Scope Separation & AI CSV Mapping: Visualize ontology schema classes, factual instances, or unified graphs. Ingest CSVs via manual mapping or AI rule synthesis with user verification.
11. 🧠 LLM Cognitive Memory & Thought Reorganization: Episodic memory logging, associative semantic traversal, grounded text chunk evidence linking, and native function-calling tool dispatching.
12. 🔀 Tri-Modal Reciprocal Rank Fusion (RRF): Merges dense similarity, lexical BM25, and symbolic graph traversals into unified results with universal 0–100 relevance grades.
13. 🔍 Database Diagnostics & Introspection (store.info()): Instant inspection of vectorizers, chunking parameters, document chunk counts, ontology schemas, and graph topology metrics.
14. 📊 State-of-the-Art Semantic Datalake & Point Cloud Engine: 2D/3D UMAP manifold projections (with cosine metric, plus PCA & t-SNE), persistent SQLite caching, streaming lazy loading (IncrementalPCA), and interactive HTML visualizer exports.
15. 🎨 Semantic Document Clustering & Thematic Synthesis: Groups documents using vector centroids and synthesizes thematic titles, descriptions, and topic tags with auto-K estimation.
16. 💬 Sourced Agentic Document Chat & Q&A: Talk to your documents using semantic search, full text, and SPARQL logic, with interactive source citation pills and previews.
17. 🔐 Total Zero-Leakage Local Encryption: End-to-end AES-128/HMAC encryption for chunk text, document paths/names (blind indexed), metadata, and knowledge graph nodes/edges at rest.
18. 📑 DocMetadataMode.PARSED Structural Metadata Engine: Zero-LLM, structure-aware per-chunk metadata with hierarchical breadcrumbs parsed from Markdown headings (#/##/###), YAML front-matter, or the Python AST — every chunk knows its path (chapter/section/subsection), title, authors, and abstract.
19. 🖥️ SafeStore Studio Desktop & Web App: Interactive UI built on NiceGUI and pywebview with LLM Setup, Sourced Chat, VectorDB editing, 2D/3D Datalake, and SPARQL console.
20. 🔄 Rechunking & Re-vectorization: store.rechunk_and_vectorize(...) atomically reconstructs every document, applies a new chunking (size / overlap / strategy), and re-embeds with an optionally new vectorizer — one call migrates an entire knowledge base.

---

📦 Installation

# Core package
pip install safe_store

# With Desktop UI & Studio support
pip install "safe_store[ui]"

---

🖥️ SafeStore Studio (Visual VectorDB & Graph RAG Desktop App)

Launch the visual desktop editor directly from the command line:

# Launch with native desktop window (NiceGUI + pywebview)
safe-store-studio my_knowledge.db

# Or launch in web browser mode
safe-store-studio my_knowledge.db --browser --port 8080

# Or via python module syntax
python -m safe_store my_knowledge.db

Features in SafeStore Studio:

• File & Document Manager: Inspect indexed documents, add custom files (.pdf, .docx, .md, .txt, .csv), browse chunks, and review reconstructed full text.
• Semantic Datalake Explorer: Interactive 2D or 3D PCA/t-SNE scatter plots powered by Plotly with real-time hover chunk inspection.
• Knowledge Graph & SPARQL Console: Inspect extracted entities and relationships, review class hierarchies, and execute live SPARQL 1.1 queries (SELECT, ASK, CONSTRUCT).
• RAG Search Studio: Compare dense, BM25, and hybrid queries with live threshold sliders and toggle chronological chunk reconstruction.
• Database Switcher: Dynamically switch or open different .db files without restarting.

---

🌟 Core Architecture & Pillars

                               ┌────────────────────────────────────────┐
                               │           User Natural Query           │
                               └──────────────────┬─────────────────────┘
                                                  │
                ┌─────────────────────────────────┼─────────────────────────────────┐
                ▼                                 ▼                                 ▼
    ┌───────────────────────┐         ┌───────────────────────┐         ┌───────────────────────┐
    │  Dense Vector Search  │         │   Sparse BM25 Search  │         │  Symbolic Graph Query │
    │  (Semantic Context)   │         │ (Exact IDs/SKUs/Names)│         │ (TBox/ABox/SPARQL/Hop)│
    └───────────┬───────────┘         └───────────┬───────────┘         └───────────┬───────────┘
                │                                 │                                 │
                │        [Candidate Set 1]        │        [Candidate Set 2]        │ [Candidate Set 3]
                └─────────────────────────────────┼─────────────────────────────────┘
                                                  ▼
                               ┌─────────────────────────────────────┐
                               │  Reciprocal Rank Fusion (RRF / WCS) │
                               │  Score = Σ (w_i / (k + rank_i))     │
                               └──────────────────┬──────────────────┘
                                                  ▼
                               ┌─────────────────────────────────────┐
                               │  Enriched Context + Provenance Lineage│
                               └──────────────────┬──────────────────┘
                                                  ▼
                               ┌─────────────────────────────────────┐
                               │       LLM Response Generation       │
                               └─────────────────────────────────────┘

---

🚀 Quick Start

0. Instant Database Introspection & Diagnostics (store.info())

You can inspect any database state, vectorizer configuration, per-document chunk counts, ontology schemas, and knowledge graph metrics with a single method call:

import safe_store

store = safe_store.SafeStore("knowledge.db")

# 1. Print formatted diagnostics panel to console
store.info()

# 2. Or retrieve structured dictionary for APIs and dashboards
db_info = store.get_database_info()
print(f"Total Docs: {db_info['documents']['total_documents']}")

---

0.2 Document-Level Query Aggregation (group_by_document=True)

In classic chunk retrieval, a single query that matches several passages of the same document returns multiple rows for the same file. The caller then has to re-group, sort, dedupe, and fuse them before sending context to an LLM.

safe_store collapses this for you: pass group_by_document=True and every result row becomes one entry per document, containing:

• a unified metadata header (filename, path, document metadata);
• the matched chunks ordered by chunk_seq, with non-contiguous spans bridged by ... (hidden content) ...;
• a chunks_found counter;
• a compact chunks list with each chunk's chunk_id, chunk_seq, and relevance_score;
• a relevance_score equal to max(chunk relevance) — so legacy callers that read results[0]['relevance_score'] on a chunk-level result keep working unchanged (one document behaves like its best-matching chunk).

Usage

import safe_store

store = safe_store.SafeStore("docs.db")

# Dense
docs = store.query(
    "how to configure TLS and reload supervisor",
    top_k=3,
    group_by_document=True,
)

# Hybrid (dense + BM25 + RRF)
docs = store.hybrid_query(
    "ERR-4091 supervisor daemon restart",
    top_k=3,
    group_by_document=True,
)

for doc in docs:
    print(f"{doc['document_title']}  (relevance: {doc['relevance_score']:.1f}%)")
    print(f"Matched chunks: {doc['chunks_found']}  -> seq {doc['chunk_seqs']}")
    for c in doc["chunks"]:
        print(f"  • chunk_seq={c['chunk_seq']:>3}  id={c['chunk_id']:>5}  rel={c['relevance_score']:.1f}%")
    print(doc["rag_context"][:200], "...\n")

Re-group any existing result list

You can also apply grouping retroactively to any chunk-level result list (e.g. produced by bm25.search(...) or query() without the flag):

raw_hits = store.query("tls certificates", top_k=20)
grouped  = store.group_results_by_document(raw_hits, top_k_docs=3)

Output schema (per document)

Key	Type	Description
doc_id	int	Document identifier.
file_path	str	Full path of the source document.
document_title	str	Path(file_path).name.
document_metadata	dict | None	Document-level metadata (LLM-synthesized, PARSED-extracted, or None).
relevance_score	float	max(chunk_relevance) — backward-compatible drop-in for legacy code.
similarity_percent	float	Alias of relevance_score (kept for compatibility).
similarity_score	float	Raw cosine score of the top-scoring chunk.
chunks_found	int	Number of matched chunks in this document.
chunks	list[dict]	Ordered by chunk_seq; each entry has chunk_id, chunk_seq, relevance_score, chunk_text.
chunk_seqs	list[int]	Convenience [c['chunk_seq'] for c in chunks].
chunk_text	str	Unified header + fused content — non-contiguous spans separated by ... (hidden content) ....
combined_text	str	Fused content only (no header).
rag_context	str	Alias of chunk_text — ready to paste into an LLM prompt.
head_snippet	str	First ~500 chars of the document, whitespace-collapsed — LLM-less title/authors/abstract hint (see below).
metadata_title	str | None	Shortcut for document_metadata.get("title") when present.
chunk_id / chunk_seq	int	Identity of the top-scoring chunk (legacy compatibility).

Guaranteed Header & Head Snippet Fallback

Every grouped document always carries a --- Document: ... --- header — even when the document was indexed without metadata=. This makes chunk/documents self-describing out of the box, so downstream UIs and RAG prompts never receive a bare body of text with no provenance.

The header is assembled from three independent sources, in this priority order:

Priority	Source	When present	Emitted as
1	Document-level metadata (documents.metadata column)	store.add_document(..., metadata={...})	header + Title:, Authors: etc. lines
2	PARSED chunk-level metadata (chunks.tags._metadata)	store.add_document(..., metadata_mode="parsed")	merged into the same header (title, authors, abstract, path)
3	Filename + path (always available)	any indexed document	--- Document: <filename> --- + Path: ...
★	Head snippet (top of document)	any document with stored full_text	Head: <first 500 chars> ...

The head snippet is purely heuristic and LLM-less: it takes the first ~500 characters of the document, collapses runs of whitespace to a single line, and injects it into the header. This lets most scientific / technical documents surface their title, authors, and abstract even when no structured metadata exists — zero LLM calls, zero indexing cost.

Example output for a document indexed with no metadata:

--- Document: SXX-1_V2_GHZ_RF_front-end_circuits_in_CMOS_SIMOX.pdf ---
Path: \\fileserver\papers\SXX-1_V2_GHZ_RF_front-end_circuits_in_CMOS_SIMOX.pdf
Head: 0.5-1V 2GHz RF Front-End Circuits in CMOS-SIMOX  Jane Doe, John Smith  Abstract — This paper presents a fully integrated RF front-end ...
Matched Chunks: 3
------------------------

<chunk 0 text>
... (hidden content) ...
<chunk 1 text>

The same guarantees apply to reconstruct_overlapping_chunks=True (see §0.1) — both retrieval paths return document_title, head_snippet, and a populated header.

💡 Tip — get richer metadata without an LLM: pass metadata_mode="parsed" at ingestion to have safe_store extract structural breadcrumbs (title, chapter, section, subsection, authors, abstract) directly from the document. For documents already indexed, call store.rechunk_and_vectorize(metadata_mode="parsed") once to upgrade them in place.

Customizing the gap marker

grouped = store.group_results_by_document(
    raw_hits,
    gap_marker="\n[... omitted from source ...]\n",
)

Comparison with reconstruct_overlapping_chunks

Aspect	reconstruct_overlapping_chunks=True	group_by_document=True
Groups by document	✅	✅
Sorts chunks chronologically	✅	✅
Deduplicates overlapping boundary text	✅	❌ (keeps chunks verbatim)
Bridges non-contiguous spans with ...	✅	✅
Exposes chunks_found + per-chunk list	❌	✅
Backward-compatible relevance_score = max(...)	❌ (max-like, but header-formatted differently)	✅ (guaranteed)
Preserves original chunk text exactly	❌ (rewrites boundaries)	✅

Use reconstruct_overlapping_chunks when you want the most natural, prose-like document text for LLM consumption. Use group_by_document when you want a precise per-document audit trail (which chunks matched, at which sequence position, at what score) with a fused block that preserves chunk boundaries verbatim.

---

0.1 Overlapping Chunk Reconstruction & Anti-Hallucination Fusion

In traditional vector retrieval, adjacent chunks from the same document often score in reverse order (e.g., chunk 3 before chunk 2), resulting in disjointed narrative flow, duplicated boundary phrases, and redundant metadata headers.

safe_store can automatically reconstruct contiguous chunks in their true chronological order, eliminate the overlapping boundary stutter, bridge non-contiguous passages with ..., and prepend a single metadata header per document:

import safe_store

store = safe_store.SafeStore("docs.db")

# Option A: Built into query() or hybrid_query()
results = store.query(
    "how to configure TLS encryption and supervisor daemon",
    top_k=4,
    reconstruct_overlapping_chunks=True
)

for doc in results:
    print(f"Document: {doc['document_title']} (Peak Relevance: {doc['relevance_score']:.1f}%)")
    print(f"Fused Chunk Seqs: {doc['chunk_seqs']}")
    print(f"Content:\n{doc['chunk_text']}\n")

# Option B: Run on any existing result list
reconstructed = store.reconstruct_overlapping_chunks(raw_results, add_metadata=True)

🏷️ Always-on header + head snippet: since the reconstruction path is used as a RAG-facing view, the returned chunk_text always begins with a --- Document: ... --- header — falling back to the filename when no metadata exists — and adds a Head: ... snippet (first ~500 chars of the document, LLM-less) when structured metadata is unavailable. The head_snippet field is also exposed verbatim on each result so callers can render it separately. See §0.2 — Guaranteed Header & Head Snippet Fallback for the exact priority order.

---

0.1 Overlapping Chunk Reconstruction & Anti-Hallucination Fusion

In traditional vector retrieval, adjacent chunks from the same document often score in reverse order (e.g., chunk 3 before chunk 2), resulting in disjointed narrative flow, duplicated boundary phrases, and redundant metadata headers.

safe_store can automatically reconstruct contiguous chunks in their true chronological order, eliminate the overlapping boundary stutter, bridge non-contiguous passages with ..., and prepend a single metadata header per document:

import safe_store

store = safe_store.SafeStore("docs.db")

# Option A: Built into query() or hybrid_query()
results = store.query(
    "how to configure TLS encryption and supervisor daemon",
    top_k=4,
    reconstruct_overlapping_chunks=True
)

for doc in results:
    print(f"Document: {doc['document_title']} (Peak Relevance: {doc['relevance_score']:.1f}%)")
    print(f"Fused Chunk Seqs: {doc['chunk_seqs']}")
    print(f"Content:\n{doc['chunk_text']}\n")

# Option B: Run on any existing result list
reconstructed = store.reconstruct_overlapping_chunks(raw_results, add_metadata=True)

db_info = store.get_database_info() print(f"Total Docs: {db_info['documents']['total_documents']}") for doc in db_info['documents']['list']: print(f" • {doc['document_title']}: {doc['chunk_count']} chunks")

print(f"Knowledge Graph: {db_info['knowledge_graph']['total_nodes']} nodes, {db_info['knowledge_graph']['total_relationships']} edges")

---

### 1. Tri-Modal Hybrid Retrieval (Dense Vectors + BM25 Lexical + RRF)

Combining dense embeddings with sparse BM25 guarantees precision for both fuzzy conceptual questions and exact code/identifier queries.

```python
import safe_store

# Shared Server Mode: Only 1 copy of SentenceTransformer in memory/VRAM across all workers!
store = safe_store.SafeStore(
    db_path="hybrid_kb.db",
    vectorizer_name="st",
    vectorizer_config={
        "model": "all-MiniLM-L6-v2",
        "use_shared_server": True, # Enables persistent daemon and dynamic micro-batching
        "port": 8765
    },
    chunk_size=128,
    chunk_overlap=16
)

# Shutdown shared daemon when terminating multi-user service:
# SafeStore.shutdown_shared_vectorizer(port=8765)

Inspect database summary and diagnostics anytime:

store.info()

---

### 1.0 Custom LLM Generator Callable (Tool-Agnostic Generation)

You can pass **any** custom generator function or lambda to SafeStore. SafeStore uses it for knowledge graph extraction, entity fusion, and natural language SPARQL query synthesis:

```python
from safe_store import SafeStore

# Option A: Full standard signature (supports system prompt and json_mode)
def my_custom_llm(prompt: str, system_prompt: str = None, json_mode: bool = False, **kwargs) -> str:
    # Call OpenAI, Ollama, Anthropic, vLLM, or internal microservices
    return openai_client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "system", "content": system_prompt or ""}, {"role": "user", "content": prompt}],
        response_format={"type": "json_object"} if json_mode else None
    ).choices[0].message.content

store = SafeStore("kb.db", llm_generator=my_custom_llm)

# Option B: Minimal 1-liner lambda
store_fast = SafeStore("kb.db", llm_generator=lambda prompt: my_local_model(prompt))

# Option C: Use with LangChain
# store_lc = SafeStore("kb.db", llm_generator=lambda prompt, **kw: langchain_chat.invoke(prompt).content)

---

1.1 Semantic Document Clustering & Thematic Groups

Group documents by semantic similarity and extract themes:

import safe_store

store = safe_store.SafeStore("knowledge.db")

# Cluster documents and synthesize theme titles, descriptions, and tags
clusters = store.cluster_documents(n_clusters='auto', method='kmeans')

# Export database cleanly via online backup
store.export_db_file("backup.db")

# Import database
restored = SafeStore.import_db_file("backup.db", "restored.db")

---

1.1 Full Document & Neighborhood Context Window Retrieval

When an LLM needs complete document context or the continuous paragraph surrounding a chunk match:

with store:
    # 1. Full Document Retrieval: Discovers matching chunks, aggregates scores on 0-100 grade,
    #    and excludes documents under the relevance threshold (e.g. min_relevance_percent=50.0)
    full_docs = store.query_full_documents(
        query_text="memory leak troubleshooting",
        top_k_docs=1,
        search_mode='hybrid',
        min_relevance_percent=50.0 # Prevents retrieving irrelevant docs
    )
    if full_docs:
        print(f"Top Document: {full_docs[0]['document_title']} (Relevance: {full_docs[0]['relevance_score']:.1f}%)")
        print(f"Full Text:\n{full_docs[0]['full_text']}\n")
    else:
        print("No document exceeded the 50% relevance threshold.")

    # 2. Window Expansion Retrieval: Expands matching chunks by window_before / window_after chunks

---

1.2 Unstructured File Ingestion & Dynamic Knowledge Graph Construction

safe_store allows you to extract rich knowledge graphs directly from unstructured files (Markdown, PDF, DOCX, Text) with live per-chunk extraction reporting:

from safe_store import SafeStore, GraphStore

store = SafeStore("project_kb.db", vectorizer_name="st")

with store:
    # 1. Ingest unstructured documents
    store.add_document("architecture_notes.md", metadata={"source": "Design Team"})
    store.add_document("incident_report.pdf", metadata={"source": "SRE"})

    # 2. Initialize GraphStore
    # When no ontology is supplied, it operates in dynamic extraction mode (concepts, tools, entities, relations)
    graph = GraphStore(store=store, llm_executor_callback=my_llm_callback)

    # 3. Build graph across all documents with real-time progress & node/edge reporting
    stats = graph.build_graph_for_all_documents()
    print(f"Graph build finished: {stats['nodes_created']} nodes, {stats['relationships_created']} relationships.")

    # 4. Inspect graph statistics
    graph_info = graph.get_graph_info()
    print(f"Nodes by Label: {graph_info['nodes_by_label']}")
    print(f"Edges by Type:  {graph_info['relationships_by_type']}")

    # 2. Window Expansion Retrieval: Expands matching chunks by window_before / window_after chunks
    windows = store.query_document_content_window(
        query_text="ERR-4091 supervisor daemon",
        top_k_hits=1,
        window_before=1,
        window_after=1,
        min_relevance_percent=40.0
    )
    if windows:
        print(f"Stitched Window Text:\n{windows[0]['stitched_window_text']}\n")

    # 3. Document Chunk Pagination: Browse chunks page by page with sequence tracking
    page_data = store.get_document_content_paginated("incident_001", page=1, page_size=5)
    print(f"Page {page_data['page']} of {page_data['total_pages']} (Total Chunks: {page_data['total_chunks']})")
    print(f"Stitched Page Text:\n{page_data['stitched_text']}")

with store: # Index unstructured technical documents store.add_text( unique_id="incident_001", text="Production node crashed due to OOMKilled condition in supervisor daemon. " "Error code ERR-4091 was emitted by telemetry controller.", metadata={"service": "Telemetry", "severity": "Critical"} ) store.add_text( unique_id="manual_001", text="Troubleshooting Guide: When encountering error code ERR-4091, replace the " "memory buffer chip and execute supervisor restart.", metadata={"doc_type": "Runbook"} )

# Hybrid Query: Score-Calibrated Fusion of Dense Semantic Similarity with BM25 Sparse Lexical Score
results = store.hybrid_query(
    query_text="troubleshooting memory failure ERR-4091",
    top_k=2,
    dense_weight=0.5,
    bm25_weight=0.5,
    rrf_k=60,
    min_relevance_percent=40.0 # Standard 0-100 threshold filter
)

for r in results:
    print(f"[{r['file_path']}] (Relevance: {r['relevance_score']:.1f}% | Raw RRF: {r['raw_rrf_score']:.5f})")
    print(f"Content: {r['chunk_text']}\n")

---

### 2. LLM Cognitive Memory & SPARQL 1.1 Reorganization

Empower LLM agents to reorganize thoughts, record episodic memory events, and traverse associative concept graphs grounded in physical document chunks:

```python
from safe_store import SafeStore, GraphStore

store = SafeStore(db_path="agent_memory.db", vectorizer_name="st")
graph = GraphStore(store=store)

# 1. LLM Reorganizes Knowledge Graph via SPARQL 1.1 UPDATE
graph.execute_sparql_update("""
PREFIX ont: <http://example.org/ontology/>
PREFIX ex: <http://example.org/>
INSERT DATA {
    ex:Alice a ont:Architect ;
             ont:name "Alice Smith" ;
             ont:leadsProject ex:ProjectPhoenix .
    ex:ProjectPhoenix a ont:Project ;
                      ont:status "Active" .
}
""")

# 2. Record an Episodic Event with Chunk Grounding
episode_id = graph.memory.record_episode(
    title="Architecture Design Review",
    description="Alice presented the decentralized ledger protocol for Project Phoenix.",
    participants=["Alice Smith"],
    outcome="Approved",
    source_chunk_ids=[1] # Grounded in chunk #1
)

# 3. Associative Recall: Traverse Semantic Neighborhoods & Evidence
memory_view = graph.memory.recall_associative("Alice Smith", max_hops=2)
print("Associated Entities:", [e['properties']['name'] for e in memory_view['associated_entities']])
print("Source Chunk Evidence:", memory_view['grounded_chunks'][0]['chunk_text'])

# 4. Expose Standard Function-Calling Tools to LLM Agents
llm_tools = graph.get_tool_definitions()
# Pass llm_tools directly to OpenAI, Anthropic, Ollama, or Lollms tool definitions!

---

3. W3C SPARQL 1.1 Knowledge Graph Engine

safe_store provides a full, standards-compliant SPARQL 1.1 engine supporting SELECT, ASK, CONSTRUCT, and DESCRIBE queries across multi-hop relational graphs.

from safe_store import SafeStore, GraphStore

store = SafeStore(db_path="enterprise_kg.db", vectorizer_name="st")
graph = GraphStore(store=store)

# Create Graph Entities and Relationships
alice_id = graph.add_node("Person", {"name": "Alice Smith", "role": "Lead Architect"})
bob_id = graph.add_node("Person", {"name": "Bob Jones", "role": "Data Scientist"})
acme_id = graph.add_node("Company", {"name": "Acme Robotics", "industry": "AI"}) # Company is an Organization and Agent

graph.add_relationship(alice_id, acme_id, "worksFor") # Automatically infers 'Acme employs Alice'!

# 0. Deductive Reasoning & Exclusivity Check
report = graph.check_consistency()
print("Consistency:", report["is_consistent"])

# Inferred SPARQL Query (employs was never explicitly added)
res = graph.query_sparql("""
PREFIX ont: <http://example.org/ontology/>
SELECT ?c ?p WHERE { ?c ont:employs ?p }
""", enable_reasoning=True)

# 1. SPARQL SELECT: Multi-Hop Relational Traversal
sparql_select = """
PREFIX ex: <http://example.org/>
PREFIX ont: <http://example.org/ontology/>
SELECT ?personName ?cityName WHERE {
    ?person ont:worksFor ?company ;
            ont:hasName ?personName .
    ?company ont:locatedIn ?city .
    ?city ont:hasName ?cityName .
}
"""
results = graph.query_sparql(sparql_select)
for b in results["results"]["bindings"]:
    print(f"Person: {b['personName']['value']} works in City: {b['cityName']['value']}")

# 2. SPARQL ASK: Boolean Verification
sparql_ask = """
PREFIX ont: <http://example.org/ontology/>
ASK {
    ?person ont:worksFor ?company .
    ?company ont:hasName "Acme Robotics" .
}
"""
is_valid = graph.query_sparql(sparql_ask)
print(f"Acme Robotics employs personnel: {is_valid['boolean']}")

# 3. SPARQL CONSTRUCT: Subgraph Transformation
sparql_construct = """
PREFIX ont: <http://example.org/ontology/>
PREFIX foaf: <http://xmlns.com/foaf/0.1/>
CONSTRUCT {
    ?person foaf:workplaceHomepage ?company .
}
WHERE {
    ?person ont:worksFor ?company .
}
"""
subgraph = graph.query_sparql(sparql_construct)
for triple in subgraph["triples"]:
    print(f"Constructed: {triple['subject']['value']} -> {triple['predicate']['value']} -> {triple['object']['value']}")

---

3. TBox (Ontology) & Declarative Tabular-to-Graph Mapping (CSV / XLSX / SQLite)

Convert structured business tables directly into grounded RDF knowledge graphs matching an explicit RDFS/OWL ontology (TBox).

from safe_store import SafeStore, TBoxManager, TabularMapper

store = SafeStore(db_path="supply_chain.db")

# 1. Load TBox Ontology (Turtle format)
tbox = TBoxManager()
tbox.load_ontology("""
@prefix owl: <http://www.w3.org/2002/07/owl#> .
@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
@prefix ex: <http://example.org/ontology/> .

ex:Product a owl:Class .
ex:Supplier a owl:Class .

ex:suppliedBy a owl:ObjectProperty ;
    rdfs:domain ex:Product ;
    rdfs:range ex:Supplier .

ex:hasPrice a owl:DatatypeProperty ;
    rdfs:domain ex:Product .
""", format="turtle")

# 2. Declarative Mapping Configuration
mapping_rules = {
    "entity_mappings": [
        {
            "class": "http://example.org/ontology/Product",
            "subject_template": "http://example.org/product/{sku}",
            "properties": {
                "product_name": "http://example.org/ontology/hasName",
                "unit_price": "http://example.org/ontology/hasPrice"
            }
        },
        {
            "class": "http://example.org/ontology/Supplier",
            "subject_template": "http://example.org/supplier/{supplier_id}",
            "properties": {
                "supplier_name": "http://example.org/ontology/hasName"
            }
        }
    ],
    "relationship_mappings": [
        {
            "predicate": "http://example.org/ontology/suppliedBy",
            "source_template": "http://example.org/product/{sku}",
            "target_template": "http://example.org/supplier/{supplier_id}"
        }
    ]
}

# 3. Ingest CSV or Excel Sheet directly into ABox Graph
mapper = TabularMapper(store=store, tbox=tbox)
summary = mapper.map_csv("inventory.csv", mapping_rules=mapping_rules)
# Alternatively: mapper.map_excel("inventory.xlsx", mapping_rules=mapping_rules, sheet_name="Q3_Stock")
# Alternatively: mapper.map_sqlite_table("legacy.db", "products", mapping_rules=mapping_rules)

print(f"Mapped {summary['records_processed']} records into {summary['triples_generated']} RDF triples.")

---

4. Declarative Tabular-to-Graph Mapping (CSV / XLSX / SQLite)

Instant, Zero-LLM Knowledge Graph Construction from Structured Data

When your data already lives in structured tables (CSV exports, Excel sheets, or legacy SQLite databases), you don't need slow LLM-based extraction. safe_store provides a declarative mapping engine that transforms tabular records into grounded RDF knowledge graphs in milliseconds—no tokens consumed, no API latency.

How It Works

1. Define Your Ontology (TBox): Load an RDFS/OWL schema that describes your domain classes and relationships.
2. Write Mapping Rules: Declare how table columns map to entity classes, properties, and relationships using simple templates.
3. Bulk Ingest: Point the mapper at your file or database table. It performs batch transactional insertion directly into the graph store.

Complete Working Example

from safe_store import SafeStore, TBoxManager, TabularMapper

# 1. Initialize the store (no LLM required for mapping!)
store = SafeStore(db_path="supply_chain.db")

# 2. Load your domain ontology (Turtle format)
tbox = TBoxManager()
tbox.load_ontology("""
@prefix owl: <http://www.w3.org/2002/07/owl#> .
@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
@prefix ex: <http://example.org/ontology/> .

ex:Product a owl:Class .
ex:Supplier a owl:Class .

ex:suppliedBy a owl:ObjectProperty ;
    rdfs:domain ex:Product ;
    rdfs:range ex:Supplier .

ex:hasPrice a owl:DatatypeProperty ;
    rdfs:domain ex:Product .
ex:hasName a owl:DatatypeProperty .
""", format="turtle")

# 3. Define declarative mapping rules
mapping_rules = {
    "entity_mappings": [
        {
            "class": "http://example.org/ontology/Product",
            "subject_template": "http://example.org/product/{sku}",
            "properties": {
                "product_name": "http://example.org/ontology/hasName",
                "unit_price": "http://example.org/ontology/hasPrice"
            }
        },
        {
            "class": "http://example.org/ontology/Supplier",
            "subject_template": "http://example.org/supplier/{supplier_id}",
            "properties": {
                "supplier_name": "http://example.org/ontology/hasName"
            }
        }
    ],
    "relationship_mappings": [
        {
            "predicate": "http://example.org/ontology/suppliedBy",
            "source_template": "http://example.org/product/{sku}",
            "target_template": "http://example.org/supplier/{supplier_id}"
        }
    ]
}

# 4. Map your structured data instantly
mapper = TabularMapper(store=store, tbox=tbox)

# From CSV
summary = mapper.map_csv("inventory.csv", mapping_rules=mapping_rules)
print(f"CSV: {summary['records_processed']} rows -> {summary['entities_created']} entities")

# From Excel (specific sheet)
summary = mapper.map_excel("inventory.xlsx", mapping_rules=mapping_rules, sheet_name="Q3_Stock")

# From SQLite table
summary = mapper.map_sqlite_table("legacy.db", "products", mapping_rules=mapping_rules)

Input Format Examples

CSV Input (inventory.csv):

sku,product_name,unit_price,supplier_id,supplier_name
WIDGET-001,Industrial Widget,49.99,SUP-ACME,Acme Manufacturing
GADGET-002,Smart Gadget,199.99,SUP-TECH,TechParts Ltd

Mapping Templates:

• {sku} → Replaced with value from the sku column
• {supplier_id} → Replaced with value from the supplier_id column
• Creates URIs like http://example.org/product/WIDGET-001

Why Use Tabular Mapping?

Feature	LLM Extraction	Tabular Mapping
Speed	Slow (seconds per chunk)	Instant (thousands of rows/sec)
Cost	High (API tokens)	Zero (local computation)
Accuracy	Probabilistic	Deterministic
Ontology Compliance	Prompt-dependent	Schema-enforced
Use Case	Unstructured text	Structured tables

Integration with SPARQL

Once mapped, your tabular data is immediately queryable via the full SPARQL 1.1 engine:

from safe_store import GraphStore

graph = GraphStore(store=store)

# Find all products supplied by Acme
results = graph.query_sparql("""
PREFIX ex: <http://example.org/ontology/>
SELECT ?productName ?price WHERE {
    ?product a ex:Product ;
             ex:hasName ?productName ;
             ex:hasPrice ?price ;
             ex:suppliedBy ?supplier .
    ?supplier ex:hasName "Acme Manufacturing" .
}
""")

for binding in results["results"]["bindings"]:
    print(f"Product: {binding['productName']['value']}, Price: {binding['price']['value']}")

---

5. Tri-Modal Unified Graph Retrieval (query_graph_hybrid)

Execute multi-channel queries combining Graph Subgraph Exploration, Dense Vectors, and Sparse BM25 Lexical search in a single call.

from safe_store import SafeStore, GraphStore

store = SafeStore(db_path="enterprise_kb.db", vectorizer_name="st")
graph = GraphStore(store=store)

# Unified retrieval: discovers related subgraph entities + BM25 hits + semantic vector chunks
response = graph.query_graph_hybrid(
    query_text="What microservices depend on AuthEngine and what database tables do they use?",
    top_k=5,
    dense_weight=0.4,
    bm25_weight=0.3,
    graph_weight=0.3
)

print(f"Retrieved {len(response['ranked_chunks'])} fused context chunks.")
print(f"Identified Subgraph Nodes: {len(response['subgraph']['nodes'])}")
print(f"Identified Subgraph Edges: {len(response['subgraph']['relationships'])}")

---

🔄 Database Portability & Re-Vectorization

safe_store allows you to migrate your entire knowledge base between different embedding models and export/import your database for backup or transfer.

Re-Vectorizing a Database

If you want to switch from one vectorizer (e.g., Sentence-Transformers) to another (e.g., OpenAI or Ollama), you can re-vectorize the entire database in-place. This will decrypt chunks, re-embed them using the new model, and update the database metadata atomically.

import safe_store

store = safe_store.SafeStore("my_knowledge.db", vectorizer_name="st")

# Re-vectorize using OpenAI's text-embedding-3-small
store.revectorize_database(
    new_vectorizer_name="openai",
    new_vectorizer_config={"model": "text-embedding-3-small"}
)

print("Database successfully migrated to OpenAI embeddings.")

Exporting and Importing Databases

You can export the entire state of your database (documents, chunks, vectors, graphs, FTS indices) to a portable JSON file. This is useful for backups, sharing datasets, or migrating between machines.

import safe_store

# 1. Export the database
store = safe_store.SafeStore("my_knowledge.db", vectorizer_name="st")
store.export_database("backup.json", decrypt=False) # Set decrypt=True to export plaintext

# 2. Import the database on another machine or into a new file
new_store = safe_store.SafeStore.import_database(
    input_path="backup.json",
    db_path="restored_knowledge.db",
    vectorizer_name="st" # Must match the exported vectorizer or be re-vectorized after import
)

Handling Encrypted Databases

If the database is encrypted, you can export it securely (keeping the encrypted blobs) or decrypt it during export. When importing, you must provide the decryption_key if the data was exported in its encrypted state.

# Export encrypted data (requires key to read, but keeps it encrypted in JSON)
store = safe_store.SafeStore("secure.db", encryption_key="secret123")
store.export_database("secure_backup.json", decrypt=False)

# Import encrypted data into a new encrypted store
new_store = safe_store.SafeStore.import_database(
    input_path="secure_backup.json",
    db_path="restored_secure.db",
    decryption_key="secret123", # Required to read the encrypted JSON blobs
    encryption_key="newsecret456" # Optional: re-encrypt with a new key
)

---

🔄 Rechunking & Re-vectorizing a Database

safe_store offers two database-wide migration operations:

Operation	What changes	Preserves chunk layout?	Use when…
store.revectorize_database(...)	Embeddings only	✅ Yes	You switched embedding models but kept the same chunk boundaries.
store.rechunk_and_vectorize(...)	Chunk boundaries and embeddings	❌ No — chunks are rebuilt	You changed chunk_size, chunk_overlap or chunking_strategy, or you want structural metadata merged into every chunk.

store.rechunk_and_vectorize(...)

Rebuilds the chunk layout and re-embeds an existing indexed database in place. Every document is reconstructed from the database first, re-chunked with the supplied parameters, and re-embedded — atomically, per document.

store.rechunk_and_vectorize(
    new_chunk_size=2048,
    new_chunk_overlap=64,
    new_chunking_strategy="token",
    new_vectorizer_name="sentense_transformer",
    new_vectorizer_config={"model": "all-MiniLM-L6-v2"},
    vectorize_with_metadata=True,
    metadata_mode="parsed",
    batch_size=50,
)

Signature

SafeStore.rechunk_and_vectorize(
    new_chunk_size: Optional[int] = None,
    new_chunk_overlap: Optional[int] = None,
    new_chunking_strategy: Optional[str] = None,
    new_vectorizer_name: Optional[str] = None,
    new_vectorizer_config: Optional[Dict[str, Any]] = None,
    vectorize_with_metadata: Optional[bool] = None,
    metadata_mode: Optional[Union[str, DocMetadataMode]] = None,
    batch_size: int = 50,
) -> Dict[str, Any]

Parameters

Parameter	Type	Default	Description
new_chunk_size	int	store's chunk_size (2048 for new stores)	Target chunk size (tokens/characters, strategy-dependent).
new_chunk_overlap	int	store's chunk_overlap	Overlap between adjacent chunks. Must be strictly < new_chunk_size.
new_chunking_strategy	str	store's chunking_strategy	One of character, token, paragraph, semantic, recursive, structure, markdown, contextual, late.
new_vectorizer_name	str	store's current vectorizer	Vectorizer family (sentense_transformer, openai, ollama, cohere, grepper, lollms, …).
new_vectorizer_config	dict	store's current config	Model-specific configuration for the new vectorizer.
vectorize_with_metadata	bool	True	Prepend document-level metadata to each chunk's vectorized text.
metadata_mode	str / DocMetadataMode	None	"parsed" enriches chunks with structural metadata (breadcrumbs, headings, tables).
batch_size	int	50	Batch size for the embedding pass.

⚠️ At least one parameter must be supplied. Calling rechunk_and_vectorize() with no arguments raises ConfigurationError — it would change nothing but waste a full re-embedding pass over the entire database.

Execution order

The data-safety contract is the reconstruct-before-delete invariant:

1. Validate — no parameters supplied → ConfigurationError; overlap >= size → ConfigurationError; unknown strategy → ConfigurationError.
2. Resolve the effective parameters (fall back to the store's current values).
3. Resolve the vectorizer — if new_vectorizer_name / new_vectorizer_config were given, a new instance is created (or a new vectorizer is picked up from the manager).
4. Reconstruct every document's plain text (reconstruct_document_text(doc_id)). This happens before ANY delete — a reconstruction failure cannot lose data.
5. Clear all existing chunks and vectors in one transaction (the chunks table and the FTS index chunks_fts).
6. Per document: clean → chunk → batch-embed → insert chunks + vectors. Each document is committed independently, so a failure rolls back only that document and increments documents_failed.
7. Update the store-level configuration (chunk_size, chunk_overlap, chunking_strategy) and — if the vectorizer changed — the vectorizer_info metadata, atomically.
8. Return a summary dict (below).

Return value

{
    "documents_processed": 42,
    "documents_failed": 0,
    "chunks_created": 1873,
    "chunk_size": 2048,
    "chunk_overlap": 64,
    "chunking_strategy": "token",
    "vectorizer_name": "sentense_transformer",
}

Behaviour notes

• Default chunk size is now 2048 (previously 384). This default only applies to newly created stores; existing databases load their stored store_config['chunk_size'] unchanged on reopen.
• Changing the vectorizer changes the tokenizer, so the same numeric chunk_size can produce a different number of chunks before and after migration.
• Encrypted databases: chunks are re-encrypted with the store's active encryptor. Migrating a database to a different encryption key is not supported — instantiate SafeStore with the same encryptor first.
• metadata_mode="parsed" requires plaintext; it is silently disabled on encrypted stores.
• LLM-backed strategies (semantic, contextual, late) require an llm_generator callable passed to SafeStore.__init__; otherwise those documents are counted under documents_failed.
• Concurrency: the operation holds the store's instance lock — concurrent add_text / query calls block until it completes.
• Idempotent: calling twice with the same parameters produces identical chunk counts and vectors.

---

🔐 Zero-Leakage Encryption at Rest

safe_store provides transparent, chunk-level authenticated encryption using Fernet (AES-128-CBC with HMAC-SHA256). User-supplied passwords are hardened via PBKDF2-HMAC-SHA256 (600,000 iterations) before key derivation.

What Is Protected

Data	Encrypted?	Notes
Chunk text	✅ Yes	Decrypted transparently during query()
Document metadata	✅ Yes	JSON blob is encrypted at rest
Document full_text	✅ Yes	Stored in documents table
Vector embeddings	❌ No	Required for similarity search
Graph nodes/edges	❌ No	Structural knowledge graph data
File paths / timestamps	❌ No	Operational metadata

Basic Usage

import safe_store

# 1. Create a zero-leakage encrypted store
store = safe_store.SafeStore(
    db_path="classified.db",
    encryption_key="my-super-secure-passphrase",
    vectorizer_name="st"
)

with store:
    # Path, document text, metadata, and graph elements are all encrypted!
    store.add_text(
        unique_id="/classified/merger_contract.pdf",
        text="Acme Corp will acquire Beta Labs for $500M in Q4.",
        metadata={"classification": "Top Secret"}
    )

    # Knowledge graph elements are encrypted at rest too!
    store.graph.add_node("ClassifiedDeal", {"name": "Project Titan", "target": "Beta Labs"})

    # Query decrypts transparently in memory
    results = store.query("merger acquisition details", top_k=1)
    print(results[0]["document_title"])  # Decrypted: merger_contract.pdf
    print(results[0]["chunk_text"])

Opening Without a Key (Graceful Degradation)

If the database is opened without providing the encryption key, queries still function but return encrypted placeholders instead of plaintext. This prevents accidental crashes while signalling that the data is protected.

# Re-open the same database WITHOUT the key
unauth_store = safe_store.SafeStore("classified.db", encryption_key=None)

with unauth_store:
    res = unauth_store.query("liability clauses", top_k=1)
    print(res[0]["chunk_text"])
    # >>> "[Encrypted Chunk - Key Unavailable]"

Wrong Key Detection

Supplying an incorrect key is detected immediately during decryption (via Fernet's HMAC verification). The library distinguishes between "no key provided" and "wrong key provided":

# Re-open with an INCORRECT key
wrong_store = safe_store.SafeStore(
    "classified.db",
    encryption_key="this-is-definitely-wrong"
)

with wrong_store:
    res = wrong_store.query("liability clauses", top_k=1)
    print(res[0]["chunk_text"])
    # >>> "[Encrypted Chunk - Decryption Failed]"

Verifying Encryption Programmatically

You can inspect the database directly to confirm that encryption flags are set correctly on every chunk and document:

import sqlite3

store = safe_store.SafeStore(
    "audit.db",
    encryption_key="audit-key",
    vectorizer_name="st"
)

with store:
    store.add_text("sensitive_unique_42", "Payload data here.", metadata={"owner": "Alice"})

# Verify raw DB state
conn = sqlite3.connect("audit.db")
cursor = conn.cursor()
cursor.execute("SELECT is_encrypted FROM chunks WHERE doc_id = 1")
flags = cursor.fetchall()
assert all(flag[0] == 1 for flag in flags), "Not all chunks are encrypted!"
conn.close()

Metadata Encryption

When encryption is enabled, the metadata dictionary is also encrypted as a single JSON blob. This is transparent during queries:

with store:
    store.add_text(
        unique_id="report_001",
        text="Q3 Financial Analysis...",
        metadata={"department": "Finance", "clearance": "Restricted"}
    )
    
    # The metadata is decrypted and prepended as context in query results
    results = store.query("Q3 analysis", top_k=1)
    print(results[0]["document_metadata"])
    # >>> {'department': 'Finance', 'clearance': 'Restricted'}

Security Considerations

• Fixed Salt: This implementation uses a fixed salt for PBKDF2 derivation. This means the same password always yields the same key, which is a deliberate trade-off for portability (a single .db file can be moved between machines without external salt storage). For higher security requirements, consider wrapping the database file with OS-level full-disk encryption.
• Vectors Remain Plaintext: Vector embeddings are stored as raw BLOBs to allow cosine-similarity search without decrypting the entire dataset. If your threat model requires vectors to be secret, encrypt the underlying filesystem.
• Memory Safety: Decryption occurs in-memory during query(). Plaintext chunks exist only for the duration of the result formatting and are not cached outside of the SQLite connection scope.

Complete Example: Encrypted Document Lifecycle

import safe_store
from pathlib import Path
import shutil

DB_FILE = "encrypted_lifecycle.db"
KEY = "correct-horse-battery-staple"

# Cleanup from previous runs
for p in [DB_FILE, f"{DB_FILE}.lock", f"{DB_FILE}-wal", f"{DB_FILE}-shm"]:
    Path(p).unlink(missing_ok=True)

# Phase 1: Write encrypted data
writer = safe_store.SafeStore(
    db_path=DB_FILE,
    vectorizer_name="st",
    vectorizer_config={"model": "all-MiniLM-L6-v2"},
    encryption_key=KEY
)

doc = Path("secret_notes.txt")
doc.write_text("Project Phoenix launch is Q4. Key personnel: Alice, Bob.")

with writer:
    writer.add_document(doc, metadata={"sensitivity": "high"})
    print("Document encrypted and stored.")

# Phase 2: Read with correct key
reader = safe_store.SafeStore(DB_FILE, encryption_key=KEY)
with reader:
    results = reader.query("Project Phoenix", top_k=1)
    assert "Project Phoenix" in results[0]["chunk_text"]
    print("Decryption successful with correct key.")

# Phase 3: Read without key (placeholder)
no_key = safe_store.SafeStore(DB_FILE, encryption_key=None)
with no_key:
    res = no_key.query("Project Phoenix", top_k=1)
    assert res[0]["chunk_text"] == "[Encrypted Chunk - Key Unavailable]"
    print("Confirmed: no key returns placeholder.")

# Phase 4: Read with wrong key (tamper detection)
bad_key = safe_store.SafeStore(DB_FILE, encryption_key="wrong-key")
with bad_key:
    res = bad_key.query("Project Phoenix", top_k=1)
    assert res[0]["chunk_text"] == "[Encrypted Chunk - Decryption Failed]"
    print("Confirmed: wrong key is rejected via HMAC.")

# Cleanup
doc.unlink(missing_ok=True)
for p in [DB_FILE, f"{DB_FILE}.lock", f"{DB_FILE}-wal", f"{DB_FILE}-shm"]:
    Path(p).unlink(missing_ok=True)
print("Encrypted lifecycle demo complete.")

---

📊 Semantic Datalake & Point Cloud Visualization

safe_store includes a powerful Semantic Datalake Engine that allows you to visualize your entire knowledge base as an interactive 2D or 3D point cloud. This is essential for understanding data clustering, identifying outliers, and auditing the quality of your embeddings.

1. Programmatic Projections (UMAP, PCA, t-SNE)

You can reduce high-dimensional vector embeddings to 2D or 3D coordinates using state-of-the-art UMAP (Uniform Manifold Approximation and Projection) with cosine distance, or classic PCA / t-SNE.

import safe_store

store = safe_store.SafeStore("my_knowledge.db", vectorizer_name="st")

with store:
    # Get 2D state-of-the-art UMAP projection (default)
    points_2d = store.get_datalake_view(
        method='umap', 
        n_components=2, 
        output_format='dict'
    )

    for p in points_2d:
        print(f"Doc: {p['document_title']} | X: {p['x']:.2f}, Y: {p['y']:.2f}")

    # Get 3D UMAP projection
    points_3d = store.get_datalake_view(
        method='umap', 
        n_components=3, 
        output_format='dict'
    )

2. Persistent Caching for Instant Visualization

Projecting 100,000+ vectors can take several seconds. safe_store automatically caches the projection results inside the SQLite database. The next time you call get_datalake_view with the same parameters, it returns instantly.

# First call: Computes UMAP and caches results in SQLite
store.get_datalake_view(method='umap', use_cache=True)

# Second call: Returns instantly from SQLite cache
store.get_datalake_view(method='umap', use_cache=True)

Note: Cache is automatically invalidated whenever a document is added or deleted.

3. Lazy Streaming for Massive Datasets

For extremely large databases that might not fit into RAM, use the lazy streaming generator. It uses IncrementalPCA to process vectors in batches.

# Stream points in batches of 500
stream = store.stream_datalake_chunks(batch_size=500, method='incremental_pca')

for point in stream:
    # Process point-by-point without loading the whole matrix
    print(point['x'], point['y'])

4. Interactive HTML Visualizer Export

The most powerful feature is the ability to export a standalone, interactive HTML file that you can share with others. It includes a Plotly-powered 3D canvas, search filtering, and a chunk inspector.

store.export_datalake_html(
    output_file="my_datalake.html",
    title="Enterprise Knowledge Base Audit",
    method='umap',
    n_components=3
)

Features of the exported HTML:

• Interactive 3D/2D Canvas: Rotate, zoom, and pan through your data.
• Hover Inspection: See the actual text content and metadata of any point.
• Real-time Filtering: Filter points by document title or metadata keywords.
• Zero Dependencies: The exported file works offline in any modern browser.

---

🎯 Supported Vectorization Backends

Backend	Identifier	Typical Model / Target	Local / Remote
Sentence-Transformers	"st"	all-MiniLM-L6-v2, all-mpnet-base-v2	Local (PyTorch / HuggingFace)
Ollama	"ollama"	nomic-embed-text, qwen3-embedding	Local (Ollama Server)
OpenAI	"openai"	text-embedding-3-small, text-embedding-3-large	Remote API
Cohere	"cohere"	embed-english-v3.0, embed-multilingual-v3.0	Remote API
Lollms	"lollms"	Any OpenAI-compatible local/remote endpoint	Local / Remote
TF-IDF	"tfidf" / "tf_idf"	Data-dependent sparse baseline	Local (Scikit-Learn)
Grepper	"grepper"	Lightweight inverted index with markdown trees	Local (Zero-ML)

---

📑 Supported Document & File Formats

safe_store parses structured, unstructured, and source files out-of-the-box:

• Unstructured Documents: .pdf, .docx, .pptx, .html, .htm, .txt, .md, .rst, .msg, .rtf
• Data & Tables: .csv, .tsv, .json, .xlsx, .xls, .xml, .sql
• Source Code: .py, .js, .ts, .tsx, .jsx, .c, .cpp, .h, .cs, .java, .go, .rs, .php, .rb, .swift, .kt, .sh, .ps1, .lua, .sql

---

🔍 W3C SPARQL 1.1 Query Forms Cheat Sheet

safe_store natively executes all four standard W3C SPARQL 1.1 query forms across your knowledge graph:

Query Form	Purpose	Return Type	Typical Use Case
SELECT	Tabular projections across graph patterns	{"head": {"vars": [...]}, "results": {"bindings": [...]}}	Relational multi-hop traversals, aggregations (COUNT, GROUP BY), and filtered lookups.
ASK	Boolean existence test	{"boolean": True / False}	Fast sanity checking and compliance verification without retrieving payloads.
CONSTRUCT	Subgraph transformation & inference	{"triples": [{"subject": ..., "predicate": ..., "object": ...}]}	Transforming schemas, creating direct shortcut edges, or exporting custom RDF subgraphs.
DESCRIBE	Resource neighborhood extraction	{"triples": [...]}	Pulling all known incoming and outgoing triples associated with an entity.

---

📊 Performance Benchmarks

Typical benchmarks measured on consumer hardware (Intel i7 / 16GB RAM / SSD):

Operation	Scale / Dataset	Elapsed Time	Mode
Dense Vector Query	50,000 Chunks	~15 ms	NumPy Cosine Dot Product
BM25 Lexical Search	100,000 Chunks	~4 ms	SQLite FTS5 (Porter Stemmed)
W3C SPARQL Relational Join	20,000 Triples (2-hop)	~8 ms	RDFLib + In-Memory Quad Index
Tabular Mapping	10,000 CSV Rows	~1.2 s	Batch Transactional Insertion
Document Ingestion (ST)	1 MB Text (~300 pages)	~3.5 s	Parsing + Token Chunking + Embedding

---

6. The 8 RAG Chunking Strategies (Beyond the Basics)

Retrieval quality is decided at cut time. safe_store implements a complete suite of 8 distinct chunking strategies:

 1. Fixed-Size [====][====][====]  -> Slices at fixed intervals (fast, baseline)
 2. Overlap    [====--]            -> Rescues broken sentences across boundaries
                  [--====--]
 3. Recursive  Document            -> Splits paragraphs -> sentences -> words
               ├── Para 1
               └── Para 2 -> S1, S2
 4. Semantic   ───📉───📉───       -> Cuts at cosine similarity valleys (topic shifts)
 5. Contextual [Prefix] + [Chunk]  -> Prepends full-document situating context (Anthropic)
 6. Structure  # H1 > ## H2        -> Injects section breadcrumb paths [H1 > H2]
 7. Late       Tokens ──[Transformer]──> Contextual Embeddings ──[Mean Pool]──> Vectors
 8. Graph      Entities & Relations-> Tri-Tier Multi-Hop Graph Traversal

Strategy	Flag	Ideal For	Mechanics & Key Benefit
Token Window	'token' (Default)	Standard RAG	Slices by tokenizer limits (tiktoken/HF) with offset mapping preserving all \n line breaks.
Recursive Tree	'recursive'	General Docs & Code	Hierarchically splits by \n\n $\rightarrow$ # Headers $\rightarrow$ \n $\rightarrow$ sentences $\rightarrow$ words. Best all-around balance.
Structure-Aware	'structure' / 'markdown'	Technical Manuals & Specs	Parses Markdown # H1 $\rightarrow$ ## H2 $\rightarrow$ ### H3 stacks, attaching lineage breadcrumbs [H1 > H2].
Semantic Valley	'semantic'	Long Essays & Narrative	Embeds sentences and cuts where adjacent cosine similarity drops below threshold (topic boundary).
Contextual Retrieval	'contextual'	Complex Knowledge Bases	Injects full-document situating summaries before storage (Anthropic pattern), eliminating ambiguous pronouns.
Late Chunking	'late'	Dense Technical Context	Passes the entire document through the transformer first, then mean-pools chunk token representations (Jina AI pattern).
Paragraph	'paragraph'	Articles & Prose	Groups double-newline paragraph blocks up to chunk_size without mid-thought cuts.
Fixed Character	'character'	Raw Log Streams	Fast character slicing with sliding window overlap.

Strategy Implementation Examples

from safe_store import SafeStore

# Strategy A: Structure-Aware Markdown with Breadcrumbs
store_md = SafeStore(
    "manual.db",
    vectorizer_name="st",
    chunk_size=200,
    chunking_strategy="structure" # Injects [Section: Architecture > Storage > WAL] into chunks
)

# Strategy B: Semantic Chunking (Topic Shift Detection)
store_sem = SafeStore(
    "research.db",
    vectorizer_name="st",
    chunk_size=300,
    chunking_strategy="semantic", # Splits at cosine similarity valleys
    chunking_kwargs={"similarity_threshold": 0.65}
)

# Strategy C: Contextual Retrieval (Anthropic Pattern)
def my_context_enricher(full_doc: str, chunk: str) -> str:
    # Optional LLM or heuristic summary
    return f"From document '{full_doc[:40]}...': Topic covers database storage engine."

store_ctx = SafeStore(
    "enterprise.db",
    vectorizer_name="st",
    chunk_size=256,
    chunking_strategy="contextual",
    context_enricher=my_context_enricher
)

# Strategy D: Context Expansion Windowing
store_exp = SafeStore(
    "logs.db",
    vectorizer_name="st",
    chunk_size=128,
    expand_before=30, # Injects 30 tokens of preceding context into LLM prompt
    expand_after=30   # Injects 30 tokens of succeeding context into LLM prompt
)

---

📑 Parsed Metadata Mode (DocMetadataMode.PARSED) — Zero-LLM Structural Metadata

Not every knowledge base needs (or can afford) an LLM reading the full document to synthesize metadata. safe_store supports three metadata generation modes, selectable per document at ingestion time:

Mode	Flag	LLM Calls	Content	Ideal For
LLM (default)	"llm" / True	Yes, once per doc	LLM-synthesized title, summary, keywords	Rich semantic annotations when an LLM is available
PARSED	"parsed"	Zero	Structural breadcrumb path, title, authors, abstract extracted from the document itself	Fast/offline ingestion, privacy-sensitive docs, code repositories, large batch indexing
TITLE_ONLY	"title_only" / False	Zero	Document title only	Minimal overhead, quick indexing

How It Works

Since all ingested formats (PDF, DOCX, HTML…) are normalized to Markdown internally, the parsers operate on a uniform representation:

• Markdown / converted documents — MarkdownMetadataParser walks the heading tree (# → chapter, ## → section, ### → subsection…), tracking the character span of every node. It also extracts the title (first # heading), authors (Authors: line or YAML front-matter), and abstract (the "Abstract" section) when present. Code-fenced blocks are never mistaken for headings.
• Python source files — ASTMetadataParser parses the module with ast, mapping ClassDef/FunctionDef/AsyncFunctionDef line spans into a module → class → function hierarchy. The module docstring becomes the title.

Each chunk is located in the full document via a forward-cursor scan, and its metadata is enriched with the breadcrumb of the structural element containing it.

Usage

from safe_store import SafeStore, DocMetadataMode
# Or from safe_store.indexing.chunking import DocMetadataMode

store = SafeStore("research.db", vectorizer_name="st")

with store:
    # Markdown document — string or enum, both accepted
    store.add_document("quantum_paper.pdf", metadata_mode="parsed")
    store.add_document("notes.md", metadata_mode=DocMetadataMode.PARSED)

    # Plain text works too
    store.add_text(
        unique_id="report.md",
        text="# Q3 Report\n\n## 1. Revenue\n\nStrong growth in all segments.\n",
        metadata_mode="parsed"
    )

    # Every other add_* entry point accepts the same parameter
    store.add_file("module.py", metadata_mode=DocMetadataMode.PARSED)

    results = store.query("revenue growth", top_k=3)
    for r in results:
        print(r["chunk_tags"])  # contains the parsed breadcrumb metadata

Metadata Output Format

For a Markdown document, each chunk is tagged with its full position in the document tree:

{
    "path": "Quantum Paper / 1. Background / 1.1 Motivation",  # breadcrumb
    "title": "Quantum Paper",       # doc-level: first '#' heading
    "authors": "Jane Doe, John Smith",   # when detected
    "abstract": "Quantum computers exploit superposition...",  # when detected
    "chapter": "Quantum Paper",
    "section": "1. Background",
    "subsection": "1.1 Motivation"
}

For Python files, the breadcrumb follows the code structure:

{
    "path": "module / MyClass / my_method",
    "title": "Sample module docstring..."
}

Why Use PARSED Mode?

• 🚀 Speed: Orders of magnitude faster than LLM summarization — pure local computation.
• 💸 Zero cost: No API tokens, no LLM latency, no rate limits.
• 🔐 Privacy: No document content ever leaves your machine.
• 🎯 Deterministic: The same document always produces the same metadata — no hallucinated summaries.
• 🧭 Provenance: Every chunk knows exactly where it lives in the document hierarchy, enabling section-scoped filtering and richer citations in RAG responses.

Programmatic Access

The parsers can also be used standalone (safe_store/indexing/chunking.py):

from safe_store.indexing.chunking import build_metadata_parser

parser = build_metadata_parser(source_text, source_path="docs/paper.md")  # .py → AST, else Markdown
meta = parser.enrich(start=chunk_offset, end=chunk_offset + len(chunk))
print(meta["path"])

---

🗺️ Roadmap

• SQLite-backed dense vector database with auto-configuration persistence
• Multi-backend vectorizer hub (ST, Ollama, OpenAI, Cohere, Lollms, TF-IDF, Grepper)
• W3C SPARQL 1.1 Engine (SELECT, ASK, CONSTRUCT, DESCRIBE)
• TBox & ABox Ontology Management (OWL / RDFS introspection)
• Declarative Tabular Mapping for CSV, XLSX, and SQLite tables
• Tri-Modal Hybrid Retrieval Engine (BM25 FTS5 + Dense Vectors + RRF)
• Semantic Datalake Point Cloud Engine (2D/3D PCA, t-SNE, persistent caching, lazy streaming, and HTML visualizer)
• AES-128/HMAC Authenticated Encryption at Rest
• Multi-Modal Image Vector Database using SigLIP / CLIP embeddings
• Web-based Visual Knowledge Graph Studio & Inspector

---

🤝 Contributing & License

Contributions are welcome! Please open an issue or submit a pull request on GitHub.

Licensed under the Apache 2.0 License.