Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 20 additions & 2 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,22 @@ the project follows [Semantic Versioning](https://semver.org/).

## [Unreleased]

### Added

- Documentation: a chain guide (`docs/the-chain.md`: the three verbs,
file format, migrate's gates, CONCURRENTLY replay semantics, the
data-backfill workflow, the hand-authored `0000_extensions.sql`
pattern, and an honest-limitations list) and an alembic migration
guide (`docs/migrating-from-alembic.md`: parity check first, the
empty-chain baseline, dual-run switchover, and a scratch-DB
equivalence proof; no automated revision conversion, stated up
front). The README now documents all six verbs: the full exit-code
table, a per-verb flags reference, the inherited-database story, and
schema-scoping facts. The front page leads with the `create_all`
retirement story (the tutorial lifespan against the one-line
`aensure_schema(..., mode="check")` boot guard), and the PyPI
keywords now include `fastapi` and `sqlmodel`.

## [0.5.0] - 2026-09-02

### Added
Expand Down Expand Up @@ -66,8 +82,10 @@ the project follows [Semantic Versioning](https://semver.org/).
### Known limitations

- Generated chain files replay per-op on their `-- op N [label]`
delimiters; hand-edits bypass that tokenization (pinned chain spec
§7). A label-less body containing CONCURRENTLY routes whole to the
delimiters; hand-edits bypass that tokenization (a deliberate
trade-off: generated files carry the op labels the replay splits on;
hand-edits cannot rely on them). A label-less body containing
CONCURRENTLY routes whole to the
autocommit lane statement-by-statement, and lines starting `--` are
stripped from per-op parsing — dollar-quoted bodies containing `--`
lines are only safe in the whole-text fast path.
Expand Down
211 changes: 146 additions & 65 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,22 +4,35 @@
[![Python](https://img.shields.io/pypi/pyversions/sqlpush?style=for-the-badge)](https://pypi.org/project/sqlpush/)
[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg?style=for-the-badge)](LICENSE)

**Prisma `db push` for SQLAlchemy.** Apply your models (SQLAlchemy,
SQLModel, anything built on `MetaData`) to a live PostgreSQL / TimescaleDB
database directly, no migration files. sqlpush
diffs your models against the real schema, classifies every operation by
risk (safe / risky / destructive), and applies the plan atomically. Drift
checks exit with codes your CI can gate on.
**Prisma `db push` for SQLAlchemy.** Your models are the migration.
sqlpush diffs them (SQLAlchemy, SQLModel, anything built on `MetaData`)
against the live PostgreSQL / TimescaleDB database, classifies every
operation by risk (safe / risky / destructive), and applies the plan
atomically. No migration files to write, no `upgrade` step to forget.
Drift checks exit with codes your CI can gate on.

```console
sqlpush diff "myapp.models:metadata" # see the SQL, ordered by risk
sqlpush check "myapp.models:metadata" # CI gate: exit 0/2/3
sqlpush push "myapp.models:metadata" # apply (destructive gated)
```

If you've ever run `Base.metadata.create_all()` in production and known it
was wrong, then sighed at the migration-script treadmill when you reached
for alembic: sqlpush is for you.
If you've run `Base.metadata.create_all()` in production and known it
was wrong, sqlpush is for you.

## Install

```console
pip install sqlpush
```

Or from source:

```console
git clone https://github.com/juanmicl/sqlpush && cd sqlpush && uv sync
```

Python 3.10 or newer. PostgreSQL only.

## Why

Expand All @@ -28,35 +41,27 @@ re-encode what the models say, drift from them, and pile up forever.
sqlpush closes the loop the way Prisma's `db push` does for its schema
language, but for the SQLAlchemy ecosystem (SQLModel included):

- **No migration files, ever.** The diff *is* the migration: computed fresh
- **No migration files required.** The diff *is* the migration: computed fresh
from models vs. live database on every run, via alembic's autogenerate
engine used as a library.
engine used as a library. Files exist as a second workflow when you want
them (see below).
- **Risk-aware by default.** Every operation is classified `safe` /
`risky` / `destructive`. Destructive ops (drops) are **blocked until
`--allow-destructive`**: nothing executes at all while any is present.
- **Drift detection built for CI.** `check` plans once and exits `0` clean /
`2` drift / `3` destructive drift, scriptable without parsing output.
`--json` emits a stable versioned contract.
- **Safe under concurrency.** An advisory lock (keyed to the database, not
the DSN) coordinates workers: one pusher at a time, losers wait bounded
- **Drift detection built for CI.** `check` plans once and reports
through its exit code, no output parsing; `--json` emits a stable
versioned contract.
- **Safe under concurrency.** An advisory lock coordinates workers: one
pusher at a time, losers wait bounded
and re-verify, so deploy pipelines can race without corrupting anything.
- **Hypertables without hand-written SQL.** Decorate a model with
`@hypertable` and the `create_hypertable` directive is planned
state-aware: idempotent pushes, clean checks, no false drift.

PostgreSQL only, by design.
If you know alembic: sqlpush is its autogenerate engine, productized
into apply and check verbs, with no revision scripts to maintain.

## Install

```console
pip install sqlpush
```

Or from source:

```console
git clone https://github.com/juanmicl/sqlpush && cd sqlpush && uv sync
```
PostgreSQL only, by design.

## The 30-second tour

Expand All @@ -68,14 +73,13 @@ $ export DATABASE_URL="postgresql+psycopg://user:pass@host:5432/db"

$ sqlpush diff "myapp.models:metadata"
-- safe

CREATE TABLE hero (
id SERIAL NOT NULL PRIMARY KEY,
name VARCHAR(50) NOT NULL
id SERIAL NOT NULL,
name VARCHAR(50) NOT NULL,
PRIMARY KEY (id)
);

-- risky

CREATE INDEX ix_hero_name ON hero (name);
```

Expand All @@ -95,22 +99,26 @@ $ echo $?
In CI, check drift and fail loudly (see exit codes below). Limit scope with
repeated `--schema` / `--exclude` options.

## Exit codes
## FastAPI: retire `create_all()`

| verb | 0 | 1 | 2 | 3 |
| --- | --- | --- | --- | --- |
| `diff` | always | | | |
| `check` | clean | | drift | destructive drift |
| `push` | applied | destructive blocked | error (incl. partial failure) | |
Most FastAPI + SQLModel apps ship the lifespan the tutorials teach:

`push --safe-only` runs only safe operations and skips the rest
informationally (exit `0`). Indexes on existing tables build
`CONCURRENTLY` by default (opt out with `--no-concurrently`); a failed
`CREATE INDEX CONCURRENTLY` marks the run as partial failure (exit `2`)
instead of silently half-applying, and leaves an INVALID index — drop it
(`DROP INDEX CONCURRENTLY`) and re-push.
```python
@asynccontextmanager
async def lifespan(app):
async with engine.begin() as conn:
await conn.run_sync(SQLModel.metadata.create_all)
yield
```

`create_all` creates tables that are missing. That is all it ever
does. Add a column to a model and the database never hears about it;
an index on an existing table, a type change, a drop: nothing.
Production drifts from the models in silence, so every real change
still rides the alembic treadmill: autogenerate, review, upgrade, and
two histories to keep in agreement forever.

## FastAPI / SQLModel: replace `create_all`
The sqlpush lifespan is one line:

```python
from contextlib import asynccontextmanager
Expand All @@ -123,7 +131,81 @@ async def lifespan(app):
yield
```

Push in the deploy pipeline, check at startup.
`mode="check"` verifies the models against the database at startup and
raises when they disagree: the app refuses to boot against a schema it
does not match, which beats failing on the first query at 3am. The
schema change itself comes from wherever you put it: `sqlpush push` in
the deploy pipeline (destructive ops gated), or
`aensure_schema(..., mode="push")` when you want the API to apply it.

asyncpg URLs work too: a DSN or `AsyncEngine` spelling
`postgresql+asyncpg` is translated to the psycopg driver automatically,
and asyncpg is never required in the sqlpush process.

Push in the deploy pipeline, check at boot.

## An inherited database

The first `check` against a database with history often reports drift:
hand-built indexes, audit tables, a column someone added by hand. If
any of the drift looks destructive, `check` exits `3` and `push`
blocks. That is the tool refusing to silently drop your legacy
objects. Two escape hatches: `--exclude` accepts objects you choose to
keep (fnmatch patterns, repeatable), and `--allow-destructive` accepts
the drops when you really do want them.

## When you want files: the chain

Most changes never need a file. When one does, sqlpush has a second
workflow built on the same diff engine: the chain. `revision` writes
the next numbered SQL file from your models against a reference DB,
`migrate` replays pending files with gates and checksums, and `stamp`
adopts an existing database without executing anything.

The files are plain SQL you can review, edit before first apply, and
run under `psql`. Schema change and data backfill ship as one file.
The [chain guide](docs/the-chain.md) covers the format, the gates and
the workflows.

## Exit codes

| verb | 0 | 1 | 2 | 3 |
| --- | --- | --- | --- | --- |
| `diff` | always | | | |
| `check` | clean | | drift | destructive drift |
| `push` | applied | destructive blocked | error (incl. partial failure) | |
| `revision` | file written | error (empty drift refuses) | | |
| `migrate` | clean | blocked or partial failure | | |
| `stamp` | registered | blocked or refused | | |

Failures print a typed error on stderr, never a traceback.

`push --safe-only` runs only safe operations and skips the rest
informationally (exit `0`). Indexes on existing tables build
`CONCURRENTLY` by default (opt out with `--no-concurrently`); a failed
`CREATE INDEX CONCURRENTLY` marks the run as partial failure (exit `2`)
instead of silently half-applying, and leaves an INVALID index: drop it
(`DROP INDEX CONCURRENTLY`) and re-push. `stamp` refuses a file whose
checksum no longer matches the registry; `--force` accepts the new
content.

The knobs, per verb:

| verb | flags |
| --- | --- |
| `push` | `--allow-destructive` `--safe-only` `--no-lock` `--lock-timeout` `--advisory-wait` `--no-concurrently` `--statement-timeout` |
| `revision` | `--ref-dsn` (required) `-m/--message` `--no-concurrently` `--dir` |
| `migrate` | `--allow-destructive` `--advisory-wait` `--lock-timeout` `--statement-timeout` `--dir` |
| `stamp` | `--force` `--dir` |

Every verb except `revision` takes `--dsn` (or `$DATABASE_URL`).
`revision` requires `--ref-dsn`, with no env fallback: the reference
DB is a different database from the push target. `diff`, `check`,
`push` and `revision` also take repeatable `--schema` / `--exclude`.
Timeouts are seconds; a `lock_timeout` bounds how long a statement
waits on a lock before failing, `statement_timeout` bounds each
statement's runtime, and an exhausted `advisory-wait` raises instead
of hanging on a stuck lock holder.

## How it works

Expand All @@ -138,26 +220,31 @@ flowchart LR
apply --> report["report"]
```

- **Diff engine** scopes reflection to your target schemas (default: the
session's real `search_path`) and prunes system catalogs (TimescaleDB
internals included) before reflection even starts.
- **Diff engine** scopes reflection to your target schemas and prunes
system catalogs (TimescaleDB internals included) before reflection
even starts.
- **Classifier** maps each operation to a risk class; unknown operations
are `risky`, never silently safe.
- **Executor** splits the plan: existing-table indexes render
`CREATE INDEX CONCURRENTLY` and run one-per-transaction on autocommit
(`--no-concurrently` opts out; indexes on tables the same plan creates
stay in the atomic transaction), everything else applies in a single
atomic transaction with a bounded `lock_timeout`.
- **Executor** splits the plan: concurrent index builds run one per
transaction on autocommit, everything else applies in a single atomic
transaction with a bounded `lock_timeout`.
- **Typed errors**: only `SqlpushError` / `ConnectFailed` /
`MetadataImportError` escape the API, never raw driver exceptions.

Scoping: `--schema` restricts the diff to named schemas (default: the
session's real `search_path`). Extension-owned schemas never enter
scope automatically, and schemas you pass explicitly are never
filtered. The chain's registry table (`public.sqlpush_versions`)
always lives in `public` and is pruned from every diff, so `check`
after `migrate` is clean. `alembic_version` gets the same treatment.

## Comparison

An honest view of the neighborhood (stars as of 2026-08):

| | migration files | source of truth | risk gate | CI drift exit codes | TimescaleDB |
| --- | --- | --- | --- | --- | --- |
| **sqlpush** | none (the diff is the migration) | SQLAlchemy `MetaData` | classified safe/risky/destructive, destructive blocked by default | `check` 0/2/3 | `@hypertable` directives |
| **sqlpush** | optional: push needs none; the chain has reviewable, checksummed files | SQLAlchemy `MetaData` | classified safe/risky/destructive, destructive blocked by default | `check` 0/2/3 | `@hypertable` directives |
| [alembic](https://github.com/sqlalchemy/alembic) (4.4k★) | yes | migration scripts (autogenerate assists) | no | no | no |
| [atlas](https://github.com/ariga/atlas) (8.7k★) | optional (HCL) | HCL / SQL (ORMs via providers) | lint policies | yes | no |
| [prisma `db push`](https://www.prisma.io/docs/orm/reference/prisma-cli-reference) (47k★) | none | Prisma schema (Node/TS) | no | no | no |
Expand All @@ -167,8 +254,10 @@ sqlpush is narrower than atlas and younger than alembic, deliberately.
It is one tool for one job: keep a PostgreSQL schema in lockstep with
SQLAlchemy models, safely enough to run from CI.

Coming from [migra](https://github.com/djrobstep/migra) (now
deprecated)? There is a [migration guide](docs/migrating-from-migra.md).
Guides: [the chain](docs/the-chain.md) (file format, gates, backfills),
[migrating from alembic](docs/migrating-from-alembic.md), and
[migrating from migra](docs/migrating-from-migra.md) (deprecated).
Changes land in the [CHANGELOG](CHANGELOG.md).

## Design notes

Expand All @@ -179,11 +268,3 @@ deprecated)? There is a [migration guide](docs/migrating-from-migra.md).
- `--json` output is a versioned contract (`"version": 1`) meant for
tooling; additive changes only within a version (operations now carry
a `concurrent` boolean).

## Roadmap (0.1.x)

- jsonschema-validated `--json` output

## License

[MIT](LICENSE) · © 2026 Juan Miguel Contreras
Loading
Loading