feat: resolve ingest conflicts and complete storage, Docker, and API fixes - #169
Merged
Conversation
… path
Two `localrag ingest` processes writing the same `CHROMA_PERSIST_PATH` silently
lost writes. Chroma's embedded `PersistentClient` keeps HNSW segments in
per-process memory with no cross-process invalidation, so the loser's upsert
failed inside Chroma ("Error creating hnsw segment reader: Nothing found on
disk") while the CLI still reported `status=ok` and exit code 0.
`VectorStore._write_lock` is a `threading.RLock` and only ever serialized
writers inside one process. This adds the missing cross-process boundary:
`localrag/storage/persist_lock.py::ingest_lock` takes an advisory, non-blocking
`flock` on `<persist_path>/.ingest.lock`. This enforces the contract ADR 035
already states ("multi-process writers require an external ownership/locking
boundary and are not supported by this contract") rather than setting a new one,
so no new ADR accompanies it.
The lock sits at the ingest use-case boundary (`IngestionService.ingest_paths`
and `rebuild_collection`), not in `VectorStore`. `api/dependencies.py` and
`application/runtime.py` each cache a `VectorStore` for the whole process
lifetime, so a store-scoped lock would let a running API hold it forever and
deadlock every CLI ingest. Held only for the duration of a write; read and query
paths stay unlocked. Rebuild takes it too because it deletes and re-embeds every
source.
Acquisition fails fast instead of queueing — an ingest can run for minutes, so a
second caller is better told to retry than left hanging. Contention surfaces as
`ConcurrentIngestError`, mapped to `409 Conflict` over HTTP and to a stderr
message plus exit code 1 on the CLI. Nested acquisition is counted in-process
because `flock` is per-fd and an inner release would otherwise drop the outer
lock. An unwritable persist directory or a filesystem without `flock` degrades
to a logged warning rather than blocking ingest.
Closes #160
Claude-Session: https://claude.ai/code/session_01BXupLJw7GHrgJg3eVaAraq
n0nuser
force-pushed
the
fix/160-ingest-lock
branch
from
August 6, 2026 14:24
7303c48 to
c7fa124
Compare
n0nuser
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #160
Closes #168
Closes #158
Closes #155
Summary
This feature PR rebases and resolves the concurrent-ingest conflict from the
original PR, then completes the related storage, Docker, and API follow-ups.
Changes
lock for every write entry point.
collection, and invalidate collection-bound retrieval caches.
LOCALRAG_BUILD_SHA, expose authenticated/build-info, and addtask docker-checkfor stale-stack detection.collectionselection to JSON, contexts, and SSEquery paths while retaining the configured collection as the default.
Verification
were skipped by their existing prerequisites.
threshold.
build-infomatched the Gitrevision.