chore: purge every trace of the SQLite engine - #361
Conversation
Storage is per-host PostgreSQL via scitex_dev.store; scitex-python must
stop naming, advertising or shipping the SQLite engine. Only docs/adr/ is
exempt fleet-wide, and it carried no occurrence.
Content:
- README.md / docs/05_ADDITIONAL_MODULES.md: the scitex.db section now
documents the PostgreSQL client only. The example is rewritten against
the real PostgreSQL signature (dbname/user/host/port) and methods that
actually exist (execute, save_array, load_array, get_summaries). The
advertised delete_duplicates is dropped: it is engine-specific and
deprecated upstream.
- docs/sphinx/{core_concepts.rst,modules/io.rst}: provenance hashes are
recorded "to the Clew store", not to a named engine.
- docs/guides/CROSSREF_API_CONFIGURATION.md, .env.d.examples: the local
CrossRef mirror is described by what it is, a local database file.
- pyproject.toml: the clew extra note no longer enumerates the engine.
- scripts/maintenance/_pypi_packages.py: drop the stdlib-module entry.
It only suppressed a PyPI mapping that does not exist, so behaviour is
unchanged and an import of it would now be visible instead of silently
classified.
Removed artefacts:
- .scitex/clew/runtime/db.sqlite and
examples/_legacy/scitex/clew/multi_parent/scitex/clew.db: two engine
database files committed as run leftovers. The second is already
covered by .gitignore (**/scitex/clew.db). Nothing references either.
- examples/_legacy/scitex/db/_sqlite3/**: twelve zero-byte placeholder
example files whose paths were the only thing naming the engine.
- examples/_legacy/notebooks/19_scitex_db.ipynb: a legacy notebook that
existed to demo the engine class; its two index entries in
00_SCITEX_MASTER_INDEX.ipynb are removed with it so no link dangles.
- examples/_legacy/notebooks/27_scitex_scholar.ipynb: the hand-rolled
direct-database impact-factor lookups are removed; Scholar is already
constructed with enrich_by_default=True, which is what supplies that
metadata. Those cells were non-executable legacy stubs already.
Tests:
- tests/integration/test_integration.py: drop
`assert hasattr(stx.db, "SQLite3")` from test_db_delegates. It asserted
a mechanism that must not be advertised; the PostgreSQL and check_health
assertions in the same test remain.
- tests/integration/test_cross_package_imports.py: drop
"scitex_io._load_modules._sqlite3" from the auto-generated list. That
module ships in neither the scitex-io checkout nor the installed
package, so the entry was already dead.
Note the enumeration: `git grep -I` hides binary matches and no content
grep can see a path, so this was verified with three passes — content
without -I, `git ls-files | rg -i`, and a full untracked rg. All zero.
…ective PR #361 (chore/purge-sqlite) swept this repo for the same directive a few hours earlier. Both branches reach 0 on the agreed measurement, so one has to go; this commit takes everything #361 did better so the survivor is a strict superset, and #361 is closed. From #361: - Delete examples/_legacy/scitex/db/_sqlite3/ (12 files). They are EMPTY placeholders, so the content grep never saw them -- only their paths carry the retired engine's name. Path-level traces count too. - Better wording: docs now name the Clew store (the actual SciTeX provenance store, per the 'Provenance Tracking (Clew)' section) instead of a generic 'local database'; the schematic box is a 'verification store'; the CrossRef entries say 'database file'. Kept from this branch, which #361 lacks: - .gitignore gains .scitex/clew/, so the deleted runtime artefact cannot reappear on the next run. Verified with git check-ignore. - The README example passes password from the environment and INSERTs the row that save_array(ids=1) then targets.
|
Closing in favour of #362, which now carries everything this PR did plus two things it was missing. Two PRs were opened against the same SQLite-eradication directive a few hours apart; this is the deduplication, not a rejection — the work here was good and most of it is now in #362. Both branches reach the same measurement. Using What #362 took from this PR — verified by diffing the two branches, not assumed:
What #362 has that this PR does not:
CI is a wash, and the one red check belongs to neither of us. Worth recording for whoever picks up the py3.13 segfault: The parallel duplication also exists in scitex-template (#64 vs #65) and still needs resolving there. |
Storage is per-host PostgreSQL via
scitex_dev.store. Under the 2026-08-29 ruling the SQLite engine is abolished fleet-wide and its name appearing anywhere in this repo is itself the bug.scitex-dbstays exempt as a database library;scitex-pythondoes not, so the umbrella stops advertising and shipping the engine. Onlydocs/adr/is exempt, and it carried no occurrence.No SQLite-aware error path was added anywhere. Naming the engine in order to refuse it is still naming it.
Enumeration
git grep -Ihides binary matches, and no content grep can see a path. Three passes were needed, and the last two found things the first could not:git grep -n -i sqlite(content, binaries included)git ls-files | rg -i sqlite(tracked paths)rg -i --hidden --no-ignore(incl. untracked)The
-Iin the prescribedgit grep -Inhid two committed engine database files whose content matches; the path pass found twelve zero-byte example files whose names were the only occurrence.Prose and config
README.md,docs/05_ADDITIONAL_MODULES.md— thescitex.dbsection now documents the PostgreSQL client only. The example is rewritten against the real signature (dbname/user/host/port) and methods that exist (execute,save_array,load_array,get_summaries); the old snippet calledto_df/check_healthas instance methods, which the client does not define.delete_duplicatesis dropped from the advertised surface: upstream marks it deprecated and engine-specific.docs/sphinx/core_concepts.rst,docs/sphinx/modules/io.rst— provenance hashes are recorded "to the Clew store", not to a named engine.docs/guides/CROSSREF_API_CONFIGURATION.md,.env.d.examples/00_crossref-local.env— the local CrossRef mirror is described by what it is.pyproject.toml— theclewextra note no longer enumerates the engine module..gitignore(x2) —db.sqlite3,db.sqlite3-journal,**/*.sqlitedeleted outright.scripts/maintenance/_pypi_packages.py— the stdlib-module entry is removed. It only suppressed a PyPI mapping that does not exist, soget_package_namestill returnsNone; behaviour is unchanged and such an import would now be visible rather than silently classified.Artefacts removed
.scitex/clew/runtime/db.sqliteandexamples/_legacy/scitex/clew/multi_parent/scitex/clew.db— two engine database files committed as run leftovers. The second is already covered by.gitignore(**/scitex/clew.db) and should never have landed. Nothing in the tree references either.examples/_legacy/scitex/db/_sqlite3/**— twelve zero-byte placeholder example files. Their paths were the only thing naming the engine.examples/_legacy/notebooks/19_scitex_db.ipynb— a legacy notebook that existed to demo the engine class. Its two index entries in00_SCITEX_MASTER_INDEX.ipynbare removed with it, so no link dangles.examples/_legacy/notebooks/27_scitex_scholar.ipynb— the hand-rolled direct-database impact-factor lookups are removed.Scholar(enrich_by_default=True), already constructed two cells earlier, is what supplies that metadata; the removed cells were non-executable legacy stubs (if ...:with a# Condition metcomment for a body).Test changes
tests/integration/test_integration.py::test_db_delegates— droppedassert hasattr(stx.db, "SQLite3"). It asserted a mechanism that must not be advertised. ThePostgreSQLandcheck_healthassertions in the same test are untouched, so the test still covers the delegation.tests/integration/test_cross_package_imports.py— dropped"scitex_io._load_modules._sqlite3"from the auto-generated list. That module ships in neither thescitex-iocheckout nor the installed package, so the entry was already dead. See the finding below.No test was deleted outright.
Finding for
scitex-io(out of scope here)scitex_io._load_modules._sqlite3does not exist in/home/ywatanabe/proj/scitex-io/src/scitex_io/_load_modules/nor in the installedscitex_io. The entry here was a stale auto-generated reference. The block is regenerated byscitex-dev ecosystem write-integration-tests, so if that generator still reads a snapshot naming the module it will reintroduce the string on the next regeneration.Test evidence
Worktree source confirmed to shadow the installed copy before trusting anything:
Both runs use the same interpreter and the same command shape, one per worktree:
The suite is not green, and was not green before this change. The two
FAILEDlists are byte-identical (33 lines,diffexit 0), so no failure is attributable to this change. The one-skip delta is exactly the removedscitex_io._load_modules._sqlite3parametrization, which was being skipped because the module does not exist.The failures are optional peers absent from this container (
stx.types,stx.path,stx.compat,stx.etc,stx.gists,stx.db, stats, clew).scitex_dbis not installed here at all, which is whytest_db_delegatesfails on both sides — the assertion this PR removed was never the reason it failed.