CRDT for documents (1/6): bring up Automerge-backed collaboration (prototype) - #1749
Closed
gearnode wants to merge 45 commits into
Closed
CRDT for documents (1/6): bring up Automerge-backed collaboration (prototype)#1749gearnode wants to merge 45 commits into
gearnode wants to merge 45 commits into
Conversation
Own the Go API and WASI adapter while using the official Rust engine as the initial correctness oracle. This avoids CGO and keeps backend replacement under our control. Add convergence and JavaScript interoperability checks so a future native Go engine can be evaluated independently before adoption. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Provide peer-specific sync state and message exchange through the owned Go API. Persistable sync state allows WebSocket sessions to reconnect without coupling callers to the Rust reference backend. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Store version-scoped Automerge snapshots in PostgreSQL and expose an authenticated WebSocket sync endpoint. Row locking and CRDT merging keep concurrent writers convergent across server instances. Add stable text cursors and a service operation so Go agents can edit the same live document without replacing the frontend state. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Seed supported ProseMirror documents into Automerge rich text and bind Tiptap transactions to the live CRDT over the authenticated WebSocket. Remote changes now appear in open editors while the existing JSON save path remains the publication and PDF projection. Fall back safely for schema nodes that cannot round-trip yet, and disable the collaborative editor after a connection loss to prevent divergence. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Render Automerge rich-text spans to validated ProseMirror JSON inside the same transaction that persists CRDT state. Agent-only edits now update publishing, PDF, MCP, and reload paths without requiring an open browser. Expose revision-guarded read and edit tools so Go agents retry safely when a customer changes the document concurrently. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Keep unsynchronized local CRDT changes across WebSocket failures and retry with bounded exponential backoff. Editing pauses while disconnected and resumes after the peer catches up with PostgreSQL state. Verify the embedded engine checksum, disable unsupported collaborative undo, and fuzz malformed Automerge documents to harden the boundary. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Persist short-lived selection presence in PostgreSQL so remote cursors work across server instances and disappear after disconnects or expiry. The editor relays throttled heartbeats and renders deterministic collaborator selections without exposing names. Also map hard breaks and horizontal rules through Automerge rich text and the server-side ProseMirror projection. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Check advisories, licenses, duplicate crates, wildcard requirements, and dependency sources with the pinned Rust toolchain. The policy allows only licenses present in the locked WASM graph and documents the single unavoidable syn version split. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Start the clean-room Go engine with bounded LEB128, RLE, column, change, compressed-change, and document decoders. Validate chunk checksums and decompression limits before exposing any operation state. Cross-check JavaScript-generated change metadata and official WASM document snapshots while keeping the reference backend as production default. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Decode v0.10 batch operations, booleans, deltas, scalars, object IDs, sequence keys, and predecessor groups into a bounded Go operation graph. Apply causal changes, resolve deterministic concurrent insert ordering, track heads, and match JavaScript materialization for concurrent text histories applied in either order. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Parse and reproduce v1 and v2 sync wire messages with bounded heads, needs, bloom filters, changes, and capability flags. Cross-check byte-for-byte output against JavaScript-generated messages while the native peer state machine remains gated. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Add a clean-room native parser for Automerge 0.10 document, change, and compressed-change chunks. Validate checksums, canonical encodings, column structure, operation identifiers, and dependency graph invariants. Keep the implementation internal and leave the reference WASM backend as the production default while native materialization remains incomplete. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Replace the first container prototype with the independently tested decoder that reconstructs snapshot changes and operations, preserves unknown data, validates ownership and causal frontiers, and fuzzes malformed histories. Retain the native materializer and sync wire codec on the stronger typed representation. Production still uses the WASM oracle until mutation, encoding, and peer-state parity are complete. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Serialize typed Go operations into official change chunks with deterministic actor tables, scalar metadata, predecessor groups, checksums, and implicit operation IDs. Load the native output in official Automerge JavaScript and verify materialized content, metadata, and hashes before enabling mutation APIs. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Implement native map and text mutations, UTF-16 splices, canonical cursors, commits, saves, snapshot extension, deterministic merges, heads, persisted sync state, and bidirectional synchronization with the reference backend. Keep WASM as the default while rich-text marks, full cursor deletion semantics, compaction, and exhaustive sync transcripts remain under differential validation. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Complete native snapshot loading, rich-text span hydration, canonical cursor behavior, causal partial-change merging, actor reuse, and bidirectional reference sync. Keep explicit reference constructors as the independent oracle. Randomized text histories, repeated mixed-peer transcripts, overlapping rich-text marks, official JavaScript change loading, race tests, and 319k malformed-input fuzz executions now pass with native New and Load. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Apply the repository whitespace policy across the collaboration code and correct staticcheck and errcheck findings for error text, string iteration, and DEFLATE reader cleanup. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Track native sync messages in flight and stop generating until the remote peer responds. This lets the server drain one outbound message and quiesce instead of closing and reconnecting in a loop. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Order every change in a sync payload by its causal dependencies before applying it. ProseMirror can emit a reconciliation child before its parent when Enter splits a block, which previously closed the WebSocket and triggered reconnect loops. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Store imported change chunks byte-for-byte instead of re-encoding them under new hashes. Queue causally incomplete changes across messages, request their missing dependencies, and apply the queue once each parent arrives. This fixes the follow-up edit loop after a successful block split and preserves unknown columns and compressed changes during persistence. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Show the block and slash menus during collaboration while filtering only the unsupported table action. Headings, lists, code blocks, blockquotes, Mermaid, and dividers are available again without enabling lossy table serialization. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Encode table and row containers as explicit Automerge block markers so row boundaries survive sync. Cells retain independent text and colspan, rowspan, and colwidth attributes while TableKit menus and commands remain enabled. Add seed migration, native Go hydration, server-side ProseMirror projection, cell-edit and row-insert plugin tests, and concurrent row convergence coverage. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Keep Tiptap's local history extension enabled with Automerge. Local table and text edits are encoded as inverse CRDT changes on undo, while remote sync transactions remain excluded from the local history stack. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Mark horizontal rules as explicit block markers instead of inline embeds. This closes the active paragraph before insertion, preventing the Automerge ProseMirror adapter from producing a null content match. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Round-trip the code-block language attribute through Automerge and ProseMirror. Mermaid blocks now retain language=mermaid after local edits, remote sync, reload, and server-side projection instead of degrading to ordinary code blocks. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Compute the causal change set missing from each peer's acknowledged heads and send exact raw change chunks. Full snapshots are now reserved for unknown baselines, while explicit need requests return only requested changes. Add a regression proving steady-state frames contain one new change and remain smaller than the accumulated document. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Share one Automerge document per active version within a server process and wake every peer immediately after applying an edit. Same-instance collaboration no longer waits for the 500 ms PostgreSQL reconciliation tick. Keep row-locked persistence and periodic refresh as durability and cross-instance fallbacks, with reference-counted room teardown after the last peer leaves. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Store exact Automerge change chunks with monotonic change revisions instead of rewriting the snapshot row for every edit. Rooms load the compacted snapshot once and replay bounded change batches. Acknowledge and broadcast edits before durable persistence, retain periodic snapshot reconciliation, and expose incremental change APIs for notification and compaction workers. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Roll every 500 appended changes into the version snapshot and delete covered change rows in the same transaction. Active peers retain their acknowledged raw changes, while PostgreSQL replay remains bounded. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Acknowledge and broadcast CRDT edits immediately, then coalesce PostgreSQL persistence and ProseMirror projection in a per-room worker. Flush dirty state before room eviction and surface background persistence failures to connected peers. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Publish only the document-version GID after append-log commits and consume it through one reconnecting LISTEN connection per process. Active rooms fetch missing revisions and broadcast incremental changes immediately across instances. Keep notification payloads bounded, notifications commit-coupled, and periodic reconciliation as the durable recovery path for dropped events. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Publish selection updates to same-instance peers before PostgreSQL persistence, maintain per-room presence snapshots, and broadcast departures immediately. Reduce the browser selection throttle from 150 ms to 50 ms while retaining database reconciliation and heartbeats. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Propagate the API lifecycle context into WebSocket handlers, close upgraded connections immediately before HTTP shutdown waits, stop the PostgreSQL notification listener explicitly, and bound final dirty-room flushes to five seconds. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Treat historical horizontal-rule embeds as closed leaf blocks and place any following text into an implicit paragraph with the same parents. Existing CRDT documents created before structural divider markers now materialize without reconnect loops. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Remove the temporary legacy inline-divider projection and its compatibility test. Test fixtures may be deleted and reseeded; all collaboration data now uses the single structural divider representation. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Encode a new document's zero-head frontier as an empty non-nil byte slice so pgx writes an empty BYTEA value rather than NULL into the required heads column. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Correct the ProseMirror adapter's empty-paragraph insertion order inside the same Automerge transaction. Text typed after a divider now follows its paragraph marker rather than being interpreted as divider content by remote peers. Cover the exact two-transaction user flow: insert divider with a new paragraph, then type and round-trip the result to a second editor. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Reproduce a reference client committing a second edit before the first sync message is acknowledged, then verify the native peer sends and applies the pending change after acknowledgement. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Retain Automerge block identity in the Tiptap schema and bypass the upstream text-offset fast path when a document contains dividers. Full structural span updates keep every later paragraph marker ahead of its text. Cover repeated Enter and typing after a divider, remote round-trip, tables, undo, Mermaid, and ordinary rich-text edits through the custom sync plugin. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Add opt-in, content-free browser diagnostics for editor transactions, Automerge spans, patches, heads, and WebSocket sync frames so the remaining divider synchronization failure can be traced precisely. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Incremental ProseMirror patch conversion can produce invalid document content when remote text lands between empty structural blocks. Rebuild the target document from Automerge spans and apply its minimal valid diff for documents containing horizontal rules. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Store command-menu queries in local ProseMirror plugin state so transient slashes and search text do not enter Automerge history or appear to remote collaborators. Sync only the document operation selected from the menu. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Decorate the active command paragraph so its ordinary empty-editor placeholder does not overlap the local slash query widget. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
This was referenced Aug 21, 2026
Closed
Closed
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of the split of #1657 ("CRDT for documents") into a review stack. Stack index: 1/6.
What this PR does
Establishes a reference Automerge backend, a first (DB-persisted) sync protocol for collaborative documents, wires the editor to it, and adds the first pure-Go Automerge decode/encode engine. Then hardens that prototype: peer-ack ordering, topological sync application, in-memory room broadcast, Postgres LISTEN/NOTIFY fan-out, structural-divider/slash-command edge cases, and sync diagnostics.
Heads up for reviewers
The sync protocol and DB persistence introduced here are replaced and removed in stage 6 (
automerge/6-frontend-cutover-legacy-removal) once the automerge-repo protocol takes over. This isn't dead-on-arrival code — it's the working vertical slice the rest of the stack builds on and eventually migrates off of — but don't be surprised to see it deleted a few PRs later.Stack
automerge/2-engine-core-parity: Automerge engine core + full upstream interop parityautomerge/3-engine-hardening-refactor: fuzzing, ProseMirror render parity, internal refactorautomerge/4-api-cleanup-perf: public API cleanup + save/load performanceautomerge/5-repo-protocol-migration: migrate wire protocol to automerge-repo + cross-instance fan-outautomerge/6-frontend-cutover-legacy-removal: cut the frontend over, remove the legacy protocolMerge order is top to bottom; each PR is based on the previous stage's branch.