Skip to content

fix(smem-hygiene): remember the gate's refusals, not only its saves - #213

Merged
acidkill merged 1 commit into
acidkill:mainfrom
RobertSigmundsson:fix/write-gate-rejected-fragments-reevaluated-every-stop
Sep 7, 2026
Merged

acidkill merged 1 commit into
acidkill:mainfrom
RobertSigmundsson:fix/write-gate-rejected-fragments-reevaluated-every-stop

Conversation

@RobertSigmundsson

Copy link
Copy Markdown
Contributor

Summary

  • The auto-capture hooks now record the write gate's refusals in the idempotency ledger, not only its acceptances, so a rejected fragment is judged once per session instead of on every Stop.
  • PreCompact honours a refusal Stop already recorded, and records its own.
  • Adds ten tests, including three that count gate invocations rather than trusting a save count.

Why

The ledger from #115 remembered what the gate accepted. A refusal left no trace at all, so the next Stop in the same session re-extracted the same fragment, re-encoded it, and put it to the gate again — which, being deterministic, refused it again. PreCompact then repeated the exercise from the top. Nothing crashes and nothing corrupts; the gate simply does the same homework on the same content forever, which is a shame in a mechanism that exists to stop precisely that.

Measured on one live brain over twenty-four hours on 2026-08-08: 499 auto decisions carried just 134 distinct contents, one fragment having been judged thirty-six times. That inflates the gate's denominator about 3.7-fold and makes its observed accept rate look several times worse than it is — 0.39% counted over rows against roughly 1.4% over distinct content. The wasted encoding is the smaller half of the cost; the misleading acceptance statistic is the half that gets acted upon.

Changes

  • hooks/capture_state.py: vocabulary for recording a refusal, keyed so that it is scoped to the threshold that produced it. A refusal recorded under one threshold does not suppress a re-judgement after the threshold changes, whilst an acceptance survives such a change.
  • hooks/stop.py: the main capture loop records the refusal alongside the outcomes it already recorded.
  • hooks/pre_compact.py: honours a refusal recorded by Stop rather than re-deciding it, and records its own refusals the same way.
  • tests/unit/test_capture_state.py (+6) and tests/unit/test_hook_capture_idempotency.py (+4).

One rough edge, disclosed rather than hidden

When every detected fragment was duplicate-filtered, the run reports the existing duplicate content already captured status and does not fall through to a session summary. Reusing that message is slightly imprecise — the two situations are not identical — but replacing an accurate duplicate status with a brand-new summary save would be the worse outcome, and a separate counter looked like more machinery than the distinction earns. Happy to split it if you would rather the two were distinguishable.

The module docstring in test_hook_capture_idempotency.py also loses a stale absolute path from a developer's machine. The file is being edited here anyway, so it seemed a good moment.

Test plan

  • pytest tests/unit/test_hook_capture_idempotency.py tests/unit/test_capture_state.py — 43 passed.
  • Differential control, and the reason it is reported in two layers: an earlier shape of test_pre_compact_honours_refusal_recorded_by_stop passed with the fix entirely removed, because PreCompact never reached the gate on that input and the assertion could not distinguish that from success. The tests now count gate invocations and assert the gate was called before asserting it was not called again.
    • Reverting only hooks/pre_compact.py: 2 failed, 14 passed — test_pre_compact_honours_refusal_recorded_by_stop and test_pre_compact_records_its_own_refusal.
    • Reverting hooks/pre_compact.py and hooks/stop.py: 3 failed, 40 passed — the two above plus test_stop_hook_does_not_rejudge_rejected_content.
    • Each test is therefore pinned to the file whose hunk it covers.
  • Line coverage of the change itself, rather than of the modules: every executable line this PR adds to capture_state.py, pre_compact.py and stop.py is executed by the tests above — 12 statements, 0 uncovered. (The modules' own totals are low because they are large hook entry points with a great deal of unrelated code.)
  • pytest tests/ -m "not stress" -n 4 against a live SurrealDB v3.2.0 — 7293 passed, 48 skipped, 1 xfailed, which is main's 7283 plus exactly the ten tests added here. Two tests in tests/unit/test_dashboard_brains_scope.py fail on this branch and on main alike: they want a live database and collide with one another under -n. Both pass when that file is run on its own.
  • ruff check src/ tests/ clean; ruff format --check src/ tests/ reports 739 files already formatted.
  • mypy src/ --ignore-missing-imports — success, no issues found in 354 source files.
  • Coverage under the CI gate: 72.44%, against 72.36% on main.
  • CHANGELOG.md untouched — left to the release entry, as with fix(storage): bind datetimes in time comparisons so they select by value #191fix(memory): refresh content-derived fields on compress, restore, and refine #193.

Related issues

None filed. #115 introduced the ledger this extends; the refusal half was simply never part of it.

Verified by

@RobertSigmundsson

The auto-capture hooks recorded a fragment in the idempotency ledger only
when the write gate accepted it. A refusal left no trace, so the next Stop
in the same session re-extracted the same fragment, re-encoded it and asked
the gate again — and the gate, being deterministic, refused it again. A
session that produced one rejected fragment paid for it on every subsequent
Stop, and PreCompact repeated the whole exercise from the beginning.

Refusals are now recorded the same way accepted content is. capture_state
gains the vocabulary for it, stop.py records the refusal in its main loop,
and pre_compact.py honours a refusal that Stop already recorded instead of
re-deciding it.

There is one deliberate rough edge, called out rather than hidden: when
every detected fragment was duplicate-filtered, the run reports the
"duplicate content already captured" status and does not fall through to a
session summary. Replacing an accurate duplicate status with a brand-new
summary save would be worse than the slight imprecision of reusing the
message, and a separate counter for the two cases seemed more machinery
than the distinction is worth. Say the word if you would rather have it
split.

The tests are written to fail without the fix, which the first version of
them did not manage. Both refusal tests now count gate invocations and
force the path through the gate: an earlier shape of
test_pre_compact_honours_refusal_recorded_by_stop passed with the fix
entirely removed, because PreCompact never reached the gate on that input
and the assertion could not tell that apart from success. It now asserts
the gate was actually called before asserting it was not called again.

Also drops a stale absolute path from this module's docstring, since the
file is being edited anyway.

@acidkill acidkill left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the full diff. Rejection keys namespaced by threshold are the right shape — remembered refusals without silencing content if auto_capture_min_score is later lowered; shared seen-set across Stop and PreCompact kept consistent.

@acidkill
acidkill merged commit 82d4391 into acidkill:main Sep 7, 2026
9 checks passed
@acidkill acidkill mentioned this pull request Sep 7, 2026
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants