fix(encoder): the provider-unavailable embed skip warns, and then throttles itself - #216
Merged
acidkill merged 1 commit intoSep 7, 2026
Conversation
…ottles itself The inline embedding path has two failure branches that end identically: the neuron is saved keyword-only and `smem reindex` is the way to back-fill the vector. The TimeoutError branch logged that at WARNING with the hint. The provider-unavailable branch logged at DEBUG, with no hint — invisible under any default configuration, for what is the more common of the two failures. The operator's first sign of trouble was recall quietly getting worse. Raising it to WARNING alone would trade a silent failure for a loud one. The write path encodes in a loop — `smem train`, `smem train-db` and `smem_remember_batch` all run many neurons through it per invocation — so one WARNING with a traceback per neuron turns a provider outage into a log flood of roughly a kilobyte per record. So the branch warns on the first occurrence and every hundredth after that, with the traceback and the reindex hint, and keeps the ones in between at DEBUG with a running count. A down provider is now impossible to miss and costs a bounded number of log records rather than one per neuron. The counter is process-wide and is not reset when the provider recovers, so a second outage in the same long-lived process warns on its hundredth occurrence rather than its first. That is a deliberate simplification: the alternative is state that has to know what "recovered" means, and the every-hundredth rung still surfaces the outage. tests/unit/test_encoder_provider_unavailable.py covers the level, the hint, the first-occurrence warning and the throttle interval. It pins the fail-soft contract from the encoder's side rather than the storage's: the stub storage asserts that update_neuron is never called when the embed failed, so the branch cannot start writing rows it has no vector for.
acidkill
approved these changes
Sep 7, 2026
acidkill
left a comment
Owner
There was a problem hiding this comment.
Reviewed the full diff. Provider-unavailable now warns like the timeout branch, throttled to first + every 100th so a down provider cannot flood the log.
This was referenced Sep 7, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
WARNINGwith thesmem reindexhint, matching theTimeoutErrorbranch beside it, instead of atDEBUGwith nothing.DEBUGwith a running count.Why
_embed_created_neuronshas two failure branches that end in the same place: the neuron is saved keyword-only andsmem reindexback-fills the vector later. TheTimeoutErrorbranch says so atWARNING. The provider-unavailable branch — the more common of the two, since a provider being down outlasts any single request — said nothing atDEBUG, invisible under any default configuration. The first sign of trouble was recall quietly getting worse.Raising it to
WARNINGand stopping there would trade a silent failure for a loud one, because the write path encodes in a loop:smem train,smem train-dbandsmem_remember_batchall push many neurons through this branch per invocation, and eachWARNINGcarries a traceback.Measured over 200 consecutive failures against a provider that raises, counting records and formatted bytes:
WARNINGWARNINGDEBUGmaintodayWARNINGwith no throttleThe middle row is the version of this fix I am not proposing. Under a default configuration the operator sees the first column: nothing at all today, roughly 171 KiB of tracebacks under the naive fix, and 2.7 KiB here — visible, and bounded.
Changes
engine/encoder.py: a module-level counter for the branch;WARNINGwithexc_infoand the reindex hint on occurrence 1 and every 100th,DEBUGwith the running count otherwise. Both messages carry the number of neurons affected and the occurrence number.tests/unit/test_encoder_provider_unavailable.py(new): the level and hint, that three failures produce oneWARNINGand twoDEBUG, and that the warning recurs at occurrence 100.A simplification worth naming
The counter is process-wide and is never reset. If a provider goes down, recovers, and goes down again inside one long-lived process, the second outage warns at its hundredth occurrence rather than its first. Resetting it properly means teaching the encoder what "recovered" means, which is more state than this fix should own, and the every-hundredth rung still surfaces the outage. Happy to revisit if you would rather it tracked recovery.
Test plan
pytest tests/unit/test_encoder_provider_unavailable.py— 3 passed.engine/encoder.pyreverted tomainand the tests kept, all three fail — they measure this branch rather than themselves.pytest tests/ -m "not stress" -n 4against a live SurrealDB v3.2.0 — 7286 passed, 48 skipped, 1 xfailed, which ismain's 7283 plus the three tests added here. Two tests intests/unit/test_dashboard_brains_scope.pyfail on this branch and onmainalike: they want a live database and collide with one another under-n. Both pass when that file is run on its own.ruff check src/ tests/clean;ruff format --check src/ tests/reports 740 files already formatted.mypy src/ --ignore-missing-imports— success, no issues found in 354 source files.main.CHANGELOG.mduntouched — left to the release entry, as with fix(storage): bind datetimes in time comparisons so they select by value #191–fix(memory): refresh content-derived fields on compress, restore, and refine #193.Verified by
@RobertSigmundsson