CRDT for documents (4/6): public API cleanup + save/load performance - #1752
Closed
gearnode wants to merge 17 commits into
Closed
CRDT for documents (4/6): public API cleanup + save/load performance#1752gearnode wants to merge 17 commits into
gearnode wants to merge 17 commits into
Conversation
The shared CRDT model lived in a catch-all internal/types package, which drifts from the Rust/develer layout of dedicated, domain-named packages. Rename it to internal/opset since the model is precisely the Automerge operation-set vocabulary: actors, operation IDs, operations, changes, scalars, objects and the validated document history. Update the storage and native alias facades to re-export from opset, rename native's facade file from types.go to model.go for parity with storage, and refresh the architecture doc's boundary table and dependency graph. No behavior change: the model is byte-for-byte the same, and parity, storage and native tests stay green. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Collaboration failed permanently for affected documents with "cannot
compute changes from unknown heads". Every request reloaded the stored
snapshot, rebuilt the same broken change graph and failed again.
A document chunk records hashes for the frontier only and names ancestry
by column index, so every non-head change decoded without a hash and
without the bytes its writer hashed. Only hashed changes entered the
graph, which left the heads present and all their ancestors missing.
Reading the document still worked because operations load separately,
which is why the damage stayed invisible until a walk of a head's
ancestry hit an absent change and aborted.
Restore each change's identity while decoding the chunk: resolve
dependency indexes into hashes, then re-encode the change to recover its
hash and bytes, and refuse the document when a rebuilt frontier hash
disagrees with the one recorded. Three encoding details had to match the
writer for the bytes to come out identical:
- startOp was a permissive per-actor lower bound that only had to
locate operations. It is now derived from the operation count, which
is exact because a change's counters are consecutive.
- A snapshot drops delete operations and keeps them only as successor
entries, and it names predecessors nowhere. Both are reconstructed,
with a sequence delete keyed by the element it removes.
- The expand column is shared across a snapshot and dense because
booleans cannot be null, so it leaked onto ordinary operations and
onto marks that expand in neither direction. It is now written only
when some operation actually expands, matching the writer.
No stored data was lost, so affected documents recover on their next
load. The walk in changesSince now stops at a change the peer already
holds instead of descending into ancestry it cannot need, so a single
unavailable change can no longer fail an entire read.
Signed-off-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Go could read the compacted snapshot Rust and JavaScript write from save() but could never write one. Saving concatenated the existing base with every change chunk since, so a Go-authored history was stored as a stream of changes rather than a compacted operation set. That left the snapshot decoder one-sided, with no encoder to test it against, and it is why the decoder's ancestry loss went unnoticed for so long. Write the format properly. A document chunk stores the operation set once in operation-set order rather than per change, reduces each change to a row whose ancestry is an index into the same table, and lets deletes exist only as successor entries on the operations they removed. The encoder inverts predecessors into successors, drops deletes, orders changes so ancestors precede dependents, builds the sorted actor and head tables, and frames the chunk with its checksum. Two things the format keeps that Go was dropping are carried through: a change's extra payload, which a snapshot stores as a scalar while a change stores trailing bytes, and columns a reader did not understand, which are written back to the table they came from. Operation order is supplied by the caller because it depends on sequence state only the engine maintains, which keeps the CRDT ordering rules out of the storage layer. Byte identity is the gate rather than round-tripping: re-encoding a snapshot the reference wrote reproduces that file exactly, for the official fixture and for reference-authored histories covering marks and unmarks, deletes and overwrites, lists, counters, nested objects and a merged multi-actor history. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
The file was used to learn the snapshot operation order and has no value now that the byte-identity tests cover it. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Saving concatenated the base with every change chunk since, so a Go history grew with each commit and never took the compacted shape the other implementations write. SaveDocument now writes the whole history as one document chunk. The operation-set order the format needs comes from the engine, which is the only place that knows it: the root map first, then every object by identifier, with a map's operations grouped by property and a sequence's in the order a reader sees, marks and tombstones included because insertions anchor to them. Deletes are left out, since a snapshot keeps them only as successors of what they removed. A change with no extra payload is stored as an empty byte string rather than null, which is what the reference writes and the last difference between the two encoders. The gate is byte identity against the reference for the same history, covering linear text, map puts and a delete, a counter increment, marks and unmarks, and text with a deletion. Each snapshot is also reloaded to confirm it carries the full history and is accepted by the reference. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Note that snapshot writing exists and how it is gated, that Save still writes the change stream, and the two storage differences that remain: the save_no_orphans shim disabling compression where Rust does not, and unknown columns being unpreservable on the dependency-tolerant load path that rebuilds its base from applied changes. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
ChangesSince failed the whole read when any change in the frontier's ancestry was unreachable, and because every collaboration request recomputes it, one unreachable change wedged the document permanently. The decoder fix removed the cause, but a single bad change should never again be able to take a document offline. changesSince now returns the consistent, replayable prefix it can produce rather than nothing: a change is emitted only once all of its ancestors are, so a caller never receives a change whose dependency it was not also given. The second return becomes a completeness signal rather than a hard failure. Sync keeps using it to fall back to sending a whole document, which is unchanged and still optimal, while the incremental read returns the prefix regardless, since a change that cannot be emitted has no bytes to return in any case. A branched regression drops the ancestor of one head and asserts the intact branch still comes back as a dependency-ordered prefix while completeness reports false and the engine method does not error. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Save wrote the history as a stream of change chunks, a Go-specific deviation that grew without bound and left the document-chunk encoder exercised only by tests. It now writes the compacted document chunk the Rust and JavaScript save() produce, so a long history is stored in a fraction of the space and Go finally matches the reference save format. The whole history becomes one document chunk, followed by any retained orphan changes as trailing change chunks so nothing an orphan carried is lost. Columns above a size threshold are DEFLATE-compressed, which the decoder already reverses. When a history cannot be compacted, while isolated or when the change graph is inconsistent, Save falls back to the faithful change stream that preserves the loaded bytes verbatim, so a document is never rewritten in a lossy way. The incremental cursor is left at the end exactly as before, so SaveIncremental is unaffected. Because Save now compacts, the separate SaveDocument method it briefly had is redundant and is removed; its parity gate now covers Save. The conformance merge check no longer asserts the saved bytes embed a change verbatim, which was the old stream behavior, and instead requires the reloaded snapshot to reproduce the same frontier. Compressed columns are not byte-identical to the reference because the DEFLATE implementations differ, so the byte-identity tests stay on histories below the threshold; above it both files load equally. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Marking text with an out-of-range end boundary leaves a dangling begin, and the spans it produced still diverged from the reference on about three percent of randomized error-path scenarios. The cause was not span computation but authoring: SplitBlock built its insertion anchor directly from the preceding element, while Splice resolved the anchor against neighbouring mark boundaries first. A block marker lives in the same unified rich-text sequence as text, so a block inserted next to a dangling begin has to land on the same side of it that text would. Anchoring at the raw element instead put the block on the wrong side, and every later insertion anchored after that block inherited the mistake, so the marks the following text carried diverged in both directions: a mark dropped where two dangling begins shared an anchor, or a mark leaking past a block. SplitBlock now resolves its anchor through the same insert query as Splice. TestRustText_DanglingMarkBoundaries grows to eleven delta-debugged reproducers, and a new randomized gate compares marked spans run for run against the reference across two thousand scenarios that include out-of-range boundaries and every expand mode. A wider sweep of twenty-four thousand scenarios across six seeds found no divergence. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Save, SaveNoCompress and SaveWithOptions(bool) encoded two independent
axes, orphan retention and compression, as three methods plus a bare
boolean, and no combination could turn both off. Collapse them into one
variadic entry point with self-documenting options:
document.Save(ctx)
document.Save(ctx, automerge.NoCompress())
document.Save(ctx, automerge.DiscardOrphans())
The engine interface drops to a single Save(ctx, retainOrphans, compress)
that both the native and reference engines implement; the reference maps
the flag combination onto its three save entry points as closely as they
allow. Existing Save(ctx) calls keep working because the option parameter
is variadic.
Signed-off-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Loading had four constructors where two axes belonged on one: Load and
LoadConvertingStrings for the native engine, mirrored again for the
reference. Load and LoadReference now take LoadOption, so the string
migration is a named option:
automerge.Load(ctx, data, actor, automerge.ConvertStringsToText())
NewPureGo and LoadPureGo were aliases from when the native engine was
experimental and New/Load delegated to them. The native engine is the
default now, so New and Load hold the implementation directly and the
aliases, along with their stale "experimental" docs, are removed. Tests
that reached for the PureGo names now call New and Load. NewReference and
LoadReference stay but say plainly they are test oracles, and Close no
longer describes a WASM-only detail.
Signed-off-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
ChangesSince returns []Change while ApplyChanges took [][]byte, so the two halves of the incremental-change API did not compose: the changes one document produced could not be handed straight to another. Apply now takes []Change too. The collaboration service, the only production caller, becomes simpler in the merge path, which now forwards ChangesSince output directly, and the change-log replay wraps its stored bytes in Change values. The raw bytes still travel over the interface to the engine unchanged. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Refer to Save(ctx, NoCompress()) and Save(ctx, DiscardOrphans()) where the plan named the removed SaveNoCompress and SaveWithOptions methods. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Three ergonomic gaps from the API review: ActorID now has a String method that renders lowercase hex, matching Hash, so logs and tests stop hand-encoding it. Object.Text takes a context like every other operation on the type; it was the lone exception, which made call sites inconsistent. Scalar gains a constructor per type (NullScalar, BoolScalar, IntScalar and so on). The bare struct is a tagged union where the type and the value field can disagree; the constructors set them together so that class of mistake cannot be written. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Save rebuilt the entire columnar document on every call, which made the collaboration snapshot path pay a full re-encode per request even when nothing had changed. Rebuilding is the costly part of a save: for a 10k-operation document it was ~3.9ms and 10k allocations, dominated by reconstructing OpID-keyed maps and the operation columns. The engine now keeps a revision counter that advances on every committed change, orphan queue update, and isolate or integrate, and caches the compacted bytes against it. An unchanged document returns the cached bytes, so repeated saves drop from ~3.9ms to ~78ns. The cache stores and returns copies so callers keep ownership of their slices, and it is keyed by the retain-orphans and compress options as well as the revision, so an option change still rebuilds. Every apply path funnels through Commit, Merge, Isolate, Integrate or Rollback, so a single revision bump in each covers sync and incremental load too; bumping is deliberately conservative, since an extra bump only costs one rebuild and never returns stale bytes. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Each splice translated its UTF-16 index to an element by walking the whole visible sequence from the start and recomputing every element's width, and resolved its insertion anchor by scanning the whole insert order. Both are O(n) per splice, so typing n characters was O(n squared): a thousand-character insert took 7.2ms and grew about 53x when the size grew 10x, while Rust grew linearly. Two incremental indexes remove the walks. A per-object cumulative width slice lets a splice find its position by binary search, and a per-object insert-order position map resolves an anchor in constant time. Both are extended in place on the common append-at-end path and rebuilt lazily otherwise; each is guarded by length, and insert order only ever grows, so a stale index is always detected. Typing a thousand characters now takes 1.28ms, about 5.6x faster than before and linear with size, and about 43x faster than the reference on the same workload. The differential, mark, and splice parity suites are unchanged. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Loading a large document spent most of its time in the garbage collector: a 10k-operation document allocated ~40k objects and 27MB, because the decode buffers grew by repeated reallocation. Three buffers are now sized up front. assignOperations gives each change an operation slice of its exact size instead of growing it by append, which was the single largest source at a third of all bytes. The RLE column decoder grows its value slice once per run rather than one item at a time, which matters most for text where an entire column is one long run. NewStateFromDocument presizes the operation and change maps to the counts it is about to insert. Load drops from ~11ms to ~6.4ms and from 27MB to 14MB for the 10k document, and is now about 2.9x faster than the WASM reference. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
This was referenced Aug 21, 2026
Closed
Closed
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of the split of #1657 ("CRDT for documents") into a review stack. Stack index: 4/6. Stacked on #1751.
What this PR does
Public API rounding (Save/Load option folding,
ApplyChangesacceptingChangevalues directly), snapshot compaction on save, incremental-read degradation instead of wedging, linear-time text splicing, and decode-buffer preallocation.Review lens
Small and low-risk relative to the rest of the stack — mostly API ergonomics and perf, easiest PR in the stack to review.
Stack
automerge/1-prototype-bringup(CRDT for documents (1/6): bring up Automerge-backed collaboration (prototype) #1749): bring up Automerge-backed collaboration (prototype)automerge/2-engine-core-parity(CRDT for documents (2/6): Automerge engine core + full upstream interop parity #1750): Automerge engine core + full upstream interop parityautomerge/3-engine-hardening-refactor(CRDT for documents (3/6): fuzzing, ProseMirror render parity, internal refactor #1751): fuzzing, ProseMirror render parity, internal refactorautomerge/5-repo-protocol-migration: migrate wire protocol to automerge-repo + cross-instance fan-outautomerge/6-frontend-cutover-legacy-removal: cut the frontend over, remove the legacy protocolSummary by cubic
Public API cleanup with faster, safer save/load in
automerge. Save now compacts history by default (old: appended change chunks), supports options, and caches bytes; load recreates change identity from snapshots, and incremental reads degrade to a consistent prefix instead of wedging.Service
+1725-409ChangesSincereturns a dependency-ordered prefix and a completeness signal instead of erroring on unreachable ancestors.ApplyChangesaccepts[]Change;Object.Text(ctx)takes a context; scalar constructors (NullScalar,BoolScalar,IntScalar,UintScalar,FloatScalar,StringScalar,BytesScalar,CounterScalar,TimestampScalar,CursorScalar) prevent mismatched tag/value pairs;Load/LoadReferencenow takeLoadOption(e.g.,ConvertStringsToText()).internal/typestointernal/opset; native/reference facades and docs updated.SaveNoCompress(ctx)withSave(ctx, NoCompress()).SaveWithOptions(ctx, false)withSave(ctx, DiscardOrphans());SaveWithOptions(ctx, true)withSave(ctx).LoadConvertingStrings(...)withLoad(..., ConvertStringsToText()).NewPureGo/LoadPureGowithNew/Load.Object.Text()toObject.Text(ctx).ApplyChanges(ctx, [][]byte)toApplyChanges(ctx, []Change).pkg/probo/document_collaboration_service.goforwards[]Changedirectly in merge paths and wraps replayed bytes asChange.Tests
+1177-61Load/Saveoptions,Object.Text(ctx), and[]ChangeinApplyChanges.Written for commit 6542341. Summary will update on new commits.