fix(tests): the mock embedding provider must not depend on the hash seed - #223
Merged
acidkill merged 1 commit intoSep 7, 2026
Conversation
MockEmbeddingProvider derives its vector from the built-in hash(), which Python salts per interpreter. Two different texts therefore collide modulo 1000 about once in a thousand runs, and test_different_texts_differ fails with two identical vectors on a change that never touched embeddings. blake2b gives the same digest in every process, so the mock is deterministic and a collision between two texts either exists or never does. A new test pins the vectors for "text a" and "text b"; on a salted hash it fails under every seed but the one that produced the constants.
acidkill
approved these changes
Sep 7, 2026
acidkill
left a comment
Owner
There was a problem hiding this comment.
Reviewed the full diff. blake2b with pinned constants is the right fix — deterministic across processes, and the regression test fails under a salted hash() regardless of seed. Green CI.
This was referenced Sep 7, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
MockEmbeddingProviderderives its vector from the built-inhash(), which CPython salts perinterpreter, so the mock returns different vectors from run to run.
test_different_texts_differthen fails on two byte-identical vectors.blake2bgives the same digest in every process, so a collision between two texts is now aproperty of the texts rather than of the seed the job happened to draw.
Why
Test (Python 3.11)failed on a pull request whose diff touchedunified_config.pyand aconsolidation test, several directories away from embeddings:
A mock written for determinism should be deterministic; a failure whose cause is the hash seed
teaches contributors to re-run the job rather than read it, which is a habit worth avoiding.
Changes
tests/unit/test_embedding_provider.py:MockEmbeddingProvider.embedderiveshfromhashlib.blake2b(text.encode("utf-8"), digest_size=8)instead ofhash(text).test_embedding_does_not_depend_on_the_hash_seed, which pins the vectorsfor
"text a"and"text b". On a salted hash it fails under every seed but the one thatproduced the constants, so the old behaviour cannot return unnoticed.
Test plan
PYTHONHASHSEED=99 pytest tests/unit/test_embedding_provider.pyonmain—1 failed, 82 passed, the failure beingtest_different_texts_differwith two identical vectors.Seed 99 is simply one that collides; most seeds pass, which is the bug.
84 passed. Also green under seeds 0, 1, 7, 12345 and424242, each
84 passed.test_embedding_does_not_depend_on_the_hash_seedfails under all four seeds tried, and seed 99 additionally fails
test_different_texts_differ.ruff check src/ tests/clean;ruff format --check src/ tests/reports 739 files alreadyformatted.
mypy src/ --ignore-missing-imports— success, no issues found in 354 source files.pytest tests/ --timeout=120 -m "not stress"against a live SurrealDB v3.2.0 on a freshlycreated namespace, serial as the
Integration (SurrealDB)job runs it — 7286 passed, 48skipped, 1 xfailed, which is
main's 7285 measured in the same environment plus the onetest added here.
CHANGELOG.mduntouched — left to the release entry.Related issues
Reported in #226.
Verified by @RobertSigmundsson