CRDT for documents

[gearnode]

Summary

• add a Probo-owned, no-CGO Go API backed by official Rust Automerge 0.10 through WASI/wazero
• ship a pure-Go Automerge engine as the default, with the Rust/WASM engine retained as a differential oracle
• reach complete V2 interoperability parity: every interop-required upstream Rust test is covered and the zero-pending audit gate passes
• write compacted document-chunk snapshots that are byte-identical to Rust/JS save(), replacing the change-stream save
• beat native Rust on create, map, text editing, and repeated save; load is faster than the WASM oracle though still behind native Rust
• persist draft snapshots in PostgreSQL with row-locked CRDT merging and authenticated WebSocket sync
• bind Tiptap rich text to Automerge and safely migrate existing ProseMirror JSON
• materialize each CRDT update to validated ProseMirror JSON for publishing, PDFs, MCP, and reloads
• expose revision-guarded live-document tools and stable cursors for Go agents
• reconnect peers while retaining unsynchronized local changes
• persist short-lived presence and render remote collaborator cursors across server instances
• support paragraphs, headings, lists, blockquotes, code blocks, common marks, hard breaks, and horizontal rules
• ABI/checksum-validate the engine, fuzz malformed documents, and audit Rust advisories/licenses/sources
• keep table documents on the legacy path pending a lossless table CRDT representation

Interoperability parity

AUTOMERGE_REQUIRE_FULL_INTEROP=1 passes: 312 interop-required upstream tests covered, 0 pending.

Capabilities added to close the final gaps:

• Transaction isolation — isolate pins reads and writes to a historical frontier and branches using derived concurrency actors (matching Rust's with_concurrency scheme); merges stay hidden until integrate, and repeated isolate/integrate cycles reproduce the reference exactly.
• Compacted snapshot save — Save writes a single document chunk (frontier changes plus indexed ancestry, deletes folded into successor lists, columns DEFLATE-compressed above the reference threshold), byte-identical to the reference for the same history; Save(ctx, NoCompress()) and Save(ctx, DiscardOrphans()) cover the option combinations, and the save falls back to the faithful change stream when a history cannot be compacted.
• Orphan retention — saves append retained orphan changes and loads fall back to a dependency-tolerant path that applies what it can and queues the rest, while still rejecting a bare orphan with no base.
• Isolation-aware incremental diff — diffs chain through the isolation frontiers recorded in a window so the patch stream matches the reference across isolate/integrate; materialization now skips losing conflict alternatives at a map key.
• V2 sync internals — the empty message round-trips to the reference wire bytes, and Bloom false-positive recovery converges on both engines (the native engine compares heads exactly rather than using Bloom filters, so it is immune to false positives by construction).

Two behaviours are documented rather than silently asserted: observe_counter_change_application is covered as native-matches-reference because the pinned reference collapses the applied counter change into a single put, and the dangling-mark error path (marking past the end of the text) is now differentially gated across randomized scenarios. See PARITY_PLAN.md.

Legacy V1 sync, JavaScript binding ergonomics, and Rust-internal rustdoc examples are intentionally out of scope and classified as api-convenience or language-specific (44 api-convenience covered, 233 pending, 123 language-specific).

Robustness fixes

Two production-reported defects were root-caused and fixed with regression gates:

• Snapshot load dropped ancestry — a document chunk only stores hashes for the frontier, so every non-head change decoded without a hash or bytes and never entered the change graph; the document read fine but ChangesSince aborted with "cannot compute changes from unknown heads", wedging collaboration. The decoder now rebuilds each change's identity (dependencies, hash, and bytes) while loading, verified byte-identical to the reference. No stored data was lost, so affected documents recover on next load.
• Sync could fail to quiesce — an orphaned change and a stale read-only need could each keep the protocol from settling; both now converge, covered by a model-based sync chaos test.

ChangesSince also degrades to the consistent reachable prefix instead of failing the whole read, so a single unreachable change can never again take a document offline.

Performance

Several quadratic and rebuild-on-every-call paths were removed. Sequence reads, per-keystroke splices, and map-property reads are backed by incremental indexes (insertion order, materialized elements/values, cumulative UTF-16 offsets, insert-order positions, and a map-property index) that extend in place on append and rebuild lazily otherwise. The compacted save is cached against a mutation revision so repeated saves of an unchanged document are O(1). Load decode buffers are presized to the counts they are about to hold.

Native Go vs native Rust (median of 3; Rust/Go above 1 means Go is faster):

Workload	Size	Native Go	Native Rust	Rust/Go
create	—	1.47 µs	1.38 µs	0.94x
map	100	0.24 ms	0.96 ms	3.97x
map	1000	2.64 ms	10.15 ms	3.85x
text	100	0.10 ms	0.70 ms	6.88x
text	1000	1.16 ms	6.88 ms	5.93x
load	10000	6.23 ms	2.45 ms	0.39x
save	10000 (unchanged)	60 ns	29.8 µs	496x

Highlights from this pass: text typing became linear in document size (a thousand inserts fell from 7.5 ms to 1.16 ms, ~5.9x faster than Rust), repeated save fell from 3.9 ms to tens of nanoseconds (the collaboration snapshot path), and load fell from 11 ms to 6.2 ms with memory halved from 27 MB to 14 MB.

The save row is the cached, unchanged-document case (the server snapshot pattern); the first save after an edit rebuilds and re-encodes the document. load is faster than the WASM oracle but still ~2.5x slower than native Rust — the remaining gap is per-operation decode allocations and change-identity reconstruction for document chunks, and is the main open optimization.

Testing

• Go race tests and vet across Automerge, coredata, probo, and console API (4,889 tests; database-backed integration tests skipped without PostgreSQL)
• zero-pending V2 interoperability audit gate
• randomized/property differential testing for marks (including the out-of-range dangling-mark error path), byte-identity gates for concurrent edits and snapshot encoding, and a model-based sync chaos test
• malformed-input fuzzing across load, core operations, decode, sync messages, and rendering
• official Automerge JavaScript interoperability suite
• official Rust benchmark-battery fixture replay (Rust to Go to Rust)
• native and WASM engine benchmarks, plus Go/native-Rust comparison
• Rust formatting, clippy with warnings denied, and cargo-deny audit
• UI and console TypeScript checks, rich-editor Vitest collaboration tests, production console build, and ESLint on changed TypeScript files

  

---

Summary by cubic

Live, cross‑instance CRDT collaboration for documents using Automerge, with incremental sync, presence, and a console editor that collaborates in real time. A pure‑Go Automerge 0.10 engine is now the default (with a pinned Rust/WASM oracle); previously editors saved isolated ProseMirror JSON, now they sync live while the JSON path remains the publishing projection.

Tests +5646 -0

• Differential parity and interop suites for decode/encode/materialize/diff (including incremental), sync (v1/v2, read‑only), rich‑text spans/marks/blocks, rollback; UI collaboration tests, fuzzing, and JS oracle fixtures.

Service +3976 -0

• New owned pkg/automerge (pure‑Go default, Rust/WASM oracle): stable cursors; current‑state and diff with incremental cursors; rich‑text spans/marks/blocks; persisted sync state.
• Collaboration codec/transport for @automerge/automerge-repo: incremental sync with in‑flight ack tracking and topological apply; preserved remote hashes; CBOR presence; in‑memory rooms; append‑only log with compaction; PostgreSQL NOTIFY fanout; graceful shutdown.

App: console +613 -3

• Bind RichEditor via createRichEditorAutomergeDocument over authenticated WebSocket with reconnect/backoff; pause edits during catch‑up; render remote cursors.
• Enable dev WS proxy.

Package: ui +1884 -145

• Collaborative RichEditor via @automerge/prosemirror: custom sync plugin with structural span updates, presence, table and divider support, and collaborative undo.
• Keep slash commands out of CRDT history and hide placeholder during commands; export createRichEditorAutomergeDocument and supportsRichEditorCollaboration.

Package: automerge-conformance +4517 -0

• JS oracle, pinned parity inventory generator, collaboration fixtures, and mappings.

Package: automerge-benchmark +2588 -0

• Go/Rust harnesses and compare script; official upstream fixtures runner.

Package: i18n +5 -2

• humanizeSeconds returns session/persistent strings for non‑expiring values (optional storageType).

Package: coredata +1 -1

• Test script tolerates no tests.

Package: helpers +1 -1

• Test script tolerates no tests.

Package: hooks +1 -1

• Test script tolerates no tests.

Package: relay +1 -1

• Test script tolerates no tests.

Package: routes +1 -1

• Test script tolerates no tests.

Other +592 -138

• CI: automerge-battery and test-js workflows; Make targets for parity/fuzz/bench/fixtures; ignore WASM artifacts; trufflehog excludes.
• Deps: add github.com/tetratelabs/wazero, github.com/coder/websocket, github.com/rivo/uniseg, github.com/fxamacker/cbor/v2; lockfile updates; benchmark script.

^{Written for commit 23ad6ad. Summary will update on new commits.}

[gearnode]

Closing this in favor of a review stack — same change, split into 6 sequential PRs for easier review:

1. CRDT for documents (1/6): bring up Automerge-backed collaboration (prototype) #1749 — bring up Automerge-backed collaboration (prototype)
2. CRDT for documents (2/6): Automerge engine core + full upstream interop parity #1750 — Automerge engine core + full upstream interop parity
3. CRDT for documents (3/6): fuzzing, ProseMirror render parity, internal refactor #1751 — fuzzing, ProseMirror render parity, internal refactor
4. CRDT for documents (4/6): public API cleanup + save/load performance #1752 — public API cleanup + save/load performance
5. CRDT for documents (5/6): migrate wire protocol to automerge-repo + cross-instance fan-out #1753 — migrate wire protocol to automerge-repo + cross-instance fan-out
6. CRDT for documents (6/6): cut the frontend over, remove the legacy protocol #1754 — cut the frontend over, remove the legacy protocol

Each PR is based on the previous stage's branch; merge in order top to bottom.