feat: Voyage 4 mixed-model embedding (lite on store, large on recall)

[jack-arturo]

Summary

Voyage 4's shared embedding space allows mixing models within the same vector collection — vectors from voyage-4-lite and voyage-4-large are directly comparable via cosine similarity. This opens up a cost optimization where we use a cheaper/faster model on store and a more accurate model on recall.

Concept

Operation	Model	Why
Store (memory creation)	voyage-4-lite	Cheapest, fastest, lowest latency. Memories are written constantly.
Recall (query time)	voyage-4-large	Most accurate query understanding. Recalls happen less frequently.

This works because all Voyage 4 family models (voyage-4-large, voyage-4, voyage-4-lite, voyage-4-nano) produce vectors in the same geometric space. A query vector from voyage-4-large can match against document vectors from voyage-4-lite with minimal quality loss — per Voyage's claims.

Why this matters for AutoMem

• Cost optimization: Store is high-volume (every memory), recall is lower-volume (user queries). Using lite for the hot path saves money.
• Latency: Faster embedding on the store path means lower write latency.
• Upgradable without re-embedding: Users can start with voyage-4-lite everywhere, then upgrade recall to voyage-4-large without touching stored vectors.
• Product differentiator: No other memory service offers this. Could be a unique selling point.

Validation needed

Before making this the default, we need to benchmark on our test suite:

1. Baseline: Voyage 4 single-model

• Run LoCoMo benchmark with voyage-4 on both store and recall
• Compare against current text-embedding-3-large baseline (90.53%)

2. Mixed-model test

• Store all memories with voyage-4-lite
• Recall with voyage-4-large
• Compare accuracy vs single-model baseline
• Measure quality degradation (if any)

3. Cost/performance matrix

Config	Store model	Recall model	LoCoMo score	Cost/1M tokens (store)	Cost/1M tokens (recall)
Current default	text-embedding-3-large	text-embedding-3-large	90.53%	$0.13	$0.13
Voyage single	voyage-4	voyage-4	?	$0.06	$0.06
Voyage mixed	voyage-4-lite	voyage-4-large	?	?	?
Voyage lite-only	voyage-4-lite	voyage-4-lite	?	?	?

Implementation

Phase 1: Config support

Add new env vars:

VOYAGE_STORE_MODEL=voyage-4-lite
VOYAGE_RECALL_MODEL=voyage-4-large

When both are set, the Voyage provider uses the store model for generate_embedding() calls from the store path, and the recall model for embeddings generated during recall/search.

Phase 2: Code changes

• VoyageEmbeddingProvider needs to support two model handles
• Store path (/store endpoint) calls with store model
• Recall path (/recall endpoint) calls with recall model
• Fallback: if only VOYAGE_MODEL is set (current behavior), use same model for both

Phase 3: If benchmarks validate

• Make Voyage the default EMBEDDING_PROVIDER
• Default to mixed-model config
• Update docs and migration guide

Related

• feat(embedding): add Voyage AI embedding provider #67 — Voyage AI provider (merged in v0.11.0)
• Change default embedding model to text-embedding-3-small (768d) #84 — Discussion on dropping to text-embedding-3-small as default
• Voyage 4 announcement: https://blog.voyageai.com/2026/01/15/voyage-4/

[coderabbitai]

🔗 Related PRs

#56 - feat(embedding): add Ollama embedding provider and expand docs [merged]
#67 - feat(embedding): add Voyage AI embedding provider [merged]

---

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

• Create Plan

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

[jack-arturo]

Implementation Detail

Provider changes (automem/embedding/voyage.py)

VoyageEmbeddingProvider currently holds a single self.model. The change:

class VoyageEmbeddingProvider(EmbeddingProvider):
    def __init__(
        self,
        api_key=None,
        model="voyage-4",           # used when store/recall not split
        store_model=None,           # override for store path
        recall_model=None,          # override for recall path
        dimension=1024,
        ...
    ):
        self._store_model = store_model or model
        self._recall_model = recall_model or model
        self.model = model  # keep for provider_name()

Add two public methods that callers use instead of generate_embedding() directly:

def embed_for_store(self, text: str) -> List[float]:
    """Embed a document for storage."""
    return self._make_request([text], model_override=self._store_model)[0]

def embed_for_recall(self, text: str) -> List[float]:
    """Embed a query for recall."""
    return self._make_request([text], model_override=self._recall_model)[0]

_make_request gets a model_override param that swaps payload["model"].

generate_embedding() stays as-is for backward compat (uses self._store_model by default, since most callers are on the store path).

Provider init (automem/embedding/provider_init.py)

voyage_store_model = os.getenv("VOYAGE_STORE_MODEL")
voyage_recall_model = os.getenv("VOYAGE_RECALL_MODEL")
voyage_model = os.getenv("VOYAGE_MODEL", "voyage-4")

state.embedding_provider = VoyageEmbeddingProvider(
    api_key=api_key,
    model=voyage_model,
    store_model=voyage_store_model,
    recall_model=voyage_recall_model,
    dimension=vector_size,
)

Store path (automem/api/memory.py or app.py)

Where we call embedding_provider.generate_embedding(content) on store, switch to embed_for_store(content). This is ~2-3 call sites.

Recall path (automem/api/recall.py, search modules)

Where we embed the query string, switch to embed_for_recall(query). Also ~2-3 call sites.

Base class (automem/embedding/provider.py)

Add embed_for_store and embed_for_recall to EmbeddingProvider ABC with default implementations that just call generate_embedding(). This way other providers (OpenAI, FastEmbed, etc.) work unchanged.

Config (automem/config.py)

VOYAGE_STORE_MODEL = os.getenv("VOYAGE_STORE_MODEL")
VOYAGE_RECALL_MODEL = os.getenv("VOYAGE_RECALL_MODEL")

Total scope

~6-8 files touched, ~80-120 lines of actual code. No schema changes, no migration needed.

---

Test Strategy

Unit tests (tests/test_voyage_mixed_model.py)

1. Provider instantiation — verify _store_model / _recall_model are set correctly from various env var combos (both set, one set, neither set)
2. Model selection — mock _make_request, call embed_for_store() and embed_for_recall(), assert correct model in payload
3. Backward compat — generate_embedding() still works when no split models configured
4. Dimension validation — both models must use same dimension (enforced in constructor)

Integration tests (tests/test_integration.py)

1. Round-trip with split models — store memory with lite, recall with large, verify the memory is found
2. Cosine similarity sanity check — embed same text with both models, verify cosine sim > 0.9 (proves shared space claim)
3. Batch operations — generate_embeddings_batch() uses store model by default

Benchmark tests (the real validation)

Run the 4-config matrix from the issue body against LoCoMo. The key comparison:

Config	Expected outcome
voyage-4 / voyage-4	Baseline — this is what we compare against
voyage-4-lite / voyage-4-large	Should be within ~1-2% of baseline if Voyage's claims hold
voyage-4-lite / voyage-4-lite	Lower bound — shows how much quality lite alone loses
voyage-4-large / voyage-4-large	Upper bound — best possible quality

Ship criteria: lite-store/large-recall scores within 2% of large/large on LoCoMo. If the gap is >2%, try voyage-4 store / voyage-4-large recall as middle ground.

Timing

This is post-baseline, post-bug-fix-sprint work. Depends on Phase 1 baseline results from the recall quality sprint plan.