Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .env.d.examples/00_crossref-local.env
Original file line number Diff line number Diff line change
Expand Up @@ -11,5 +11,5 @@ CROSSREF_LOCAL_API_URL=http://localhost:31291
CROSSREF_LOCAL_HOST=0.0.0.0
CROSSREF_LOCAL_PORT=31291

# Path to SQLite database (for db mode)
# Path to the local database file (for db mode)
# CROSSREF_LOCAL_DB=/path/to/crossref.db
5 changes: 1 addition & 4 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -144,8 +144,6 @@ cover/
# Django stuff:
# *.log
local_settings.py
db.sqlite3
db.sqlite3-journal

# Flask stuff:
instance/
Expand Down Expand Up @@ -389,8 +387,6 @@ cover/
# Django stuff:
*.log
local_settings.py
db.sqlite3
db.sqlite3-journal

# Flask stuff:
instance/
Expand Down Expand Up @@ -928,6 +924,7 @@ tests/results/
mgmt
.jupyter_ystore.db
**/scitex/clew.db
.scitex/clew/
docs/visualization/FROM_USER.md
/.claude-worktree-symlinks
# Claude worktree symlink
Expand Down
Binary file removed .scitex/clew/runtime/db.sqlite
Binary file not shown.
29 changes: 18 additions & 11 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -526,22 +526,29 @@ Extends stdlib `logging` with SUCCESS/FAIL levels, a 30+ class exception tree (`
</details>

<details>
<summary><strong><code>scitex.db</code> -- SQLite3 / PostgreSQL with ndarray BLOB Storage</strong></summary>
<summary><strong><code>scitex.db</code> -- PostgreSQL with ndarray BLOB Storage</strong></summary>

```python
import os
import scitex as stx, numpy as np

db = stx.db.SQLite3("experiments.db")
with db: # context-manager transaction
db.execute("CREATE TABLE IF NOT EXISTS runs (id TEXT, acc REAL)")
db.save_array("weights_epoch_87", np.random.rand(1024, 1024)) # compressed BLOB

df = db.to_df("runs") # pandas round-trip
w = db.load_array("weights_epoch_87") # typed ndarray back
db.check_health() # integrity + schema drift
stx.db.delete_duplicates(conn, "runs", columns=["id"])
db = stx.db.PostgreSQL(
dbname="experiments",
user="researcher",
password=os.environ["PGPASSWORD"],
host="localhost",
)
with db: # closes the connection on exit
db.execute(
"CREATE TABLE IF NOT EXISTS runs (id SERIAL PRIMARY KEY, acc REAL, weights BYTEA)"
)
db.execute("INSERT INTO runs (acc) VALUES (0.87)")
db.save_array("runs", np.random.rand(1024, 1024), column="weights", ids=1) # BLOB

df = db.get_rows("runs") # pandas round-trip
w = db.load_array("runs", "weights", ids=1) # typed ndarray back
```
SQLite / PostgreSQL clients with first-class compressed-ndarray BLOBs, dataframe round-trips, health checks, and duplicate removal. Drop-in replacement for hand-rolling `pickle → BLOB` storage or SQLAlchemy Core when you don't need an ORM.
A PostgreSQL client with first-class compressed-ndarray BLOBs and dataframe round-trips. Drop-in replacement for hand-rolling `pickle → BLOB` storage or SQLAlchemy Core when you don't need an ORM.
</details>

<details>
Expand Down
2 changes: 1 addition & 1 deletion docs/05_ADDITIONAL_MODULES.md
Original file line number Diff line number Diff line change
Expand Up @@ -118,7 +118,7 @@ Lower-level SciTeX utilities re-exported under the umbrella:
| `stx.dict` | `DotDict` + safe merge / flatten | `DotDict`, `safe_merge`, `flatten` |
| `stx.logging` | stdlib-logging + SUCCESS/FAIL + `SciTeXError` | `getLogger`, `warn_deprecated`, `Tee` |
| `stx.types` | Union type aliases + predicates | `ArrayLike`, `ColorLike`, `is_array_like` |
| `stx.db` | SQLite3 / PostgreSQL wrapper w/ ndarray BLOBs | `SQLite3`, `PostgreSQL`, `delete_duplicates` |
| `stx.db` | PostgreSQL wrapper w/ ndarray BLOBs | `PostgreSQL`, `save_array`, `load_array` |
| `stx.audit` | Unified security scan (bandit / shellcheck / pip-audit) | `audit()` |
| `stx.browser` | Playwright helpers for scraping | `save_as_pdf`, `click_with_fallbacks_async` |
| `stx.compat` | Deprecation shims | `@deprecated`, `notify` legacy alias |
Expand Down
2 changes: 1 addition & 1 deletion docs/guides/CROSSREF_API_CONFIGURATION.md
Original file line number Diff line number Diff line change
Expand Up @@ -146,7 +146,7 @@ else:

- **Total papers**: 167,008,748
- **Database size**: 1.2TB (1,197,367 MB)
- **Format**: SQLite with indexed fields
- **Format**: Local database file with indexed fields
- **Source**: CrossRef official database

## Best Practices
Expand Down
4 changes: 2 additions & 2 deletions docs/sphinx/core_concepts.rst
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ The Session Model
2. **Logging**: All stdout/stderr captured to ``script.log``
3. **Config injection**: YAML files from ``./config/`` merged and injected
4. **CLI generation**: Function parameters become ``--flags``
5. **Provenance**: File hashes recorded to SQLite for reproducibility
5. **Provenance**: File hashes recorded to the Clew store for reproducibility

.. code-block:: python

Expand Down Expand Up @@ -124,7 +124,7 @@ Provenance Tracking (Clew)
--------------------------

Inside ``@stx.session``, every ``stx.io.save`` and ``stx.io.load`` call
records the file's SHA-256 hash to a local SQLite database. This enables:
records the file's SHA-256 hash to the local Clew store. This enables:

- **Verification**: Check if output files have been modified since creation
- **DAG reconstruction**: Trace which inputs produced which outputs
Expand Down
2 changes: 1 addition & 1 deletion docs/sphinx/modules/io.rst
Original file line number Diff line number Diff line change
Expand Up @@ -94,7 +94,7 @@ Provenance Tracking
-------------------

Inside ``@stx.session``, every ``save`` and ``load`` records file hashes
to a local SQLite database for reproducibility verification.
to the local Clew store for reproducibility verification.

.. code-block:: python

Expand Down
7 changes: 0 additions & 7 deletions examples/_legacy/notebooks/00_SCITEX_MASTER_INDEX.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -202,12 +202,6 @@
"- PDF downloads and bibliography management\n",
"- BibTeX generation with enriched metadata\n",
"\n",
"### Database Operations\n",
"**🗄️ [19_scitex_db.ipynb](19_scitex_db.ipynb)** - *Database Integration*\n",
"- PostgreSQL and SQLite support\n",
"- SQL operations with pandas integration\n",
"- Data persistence workflows\n",
"\n",
"### Documentation & LaTeX\n",
"**📝 [20_scitex_tex.ipynb](20_scitex_tex.ipynb)** - *LaTeX Integration*\n",
"- LaTeX rendering and preview\n",
Expand Down Expand Up @@ -310,7 +304,6 @@
"\n",
"### Research Tools\n",
"- **scitex.scholar** → [16_scitex_scholar.ipynb](16_scitex_scholar.ipynb)\n",
"- **scitex.db** → [19_scitex_db.ipynb](19_scitex_db.ipynb)\n",
"- **scitex.tex** → [20_scitex_tex.ipynb](20_scitex_tex.ipynb)\n",
"- **scitex.decorators** → [21_scitex_decorators.ipynb](21_scitex_decorators.ipynb)\n",
"- **scitex.repro** → [22_scitex_repro.ipynb](22_scitex_repro.ipynb)\n",
Expand Down
Loading
Loading