CRDT for documents (2/6): Automerge engine core + full upstream interop parity - #1750
Closed
gearnode wants to merge 67 commits into
Closed
CRDT for documents (2/6): Automerge engine core + full upstream interop parity#1750gearnode wants to merge 67 commits into
gearnode wants to merge 67 commits into
Conversation
Concurrent rich-text edits can temporarily place inline text directly after a horizontal-rule marker. Materialize that text in an implicit paragraph so projection succeeds without closing collaboration sockets. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Grow the pure-Go engine and the Rust/WASM reference wrapper to cover the full public Automerge surface required for cross-language parity: typed scalars, nested map and list objects, deletions, length and increment, mark and block authoring, historical reads, read-only sync transitions, missing-dependency inspection, and incremental/empty change handling. These backends are the foundation for the differential parity suite; the reference wrapper proxies each new FFI entry point so native behavior can be compared against upstream Rust for every operation. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Surface the new engine capabilities through the public package: typed scalar get/put, composite object creation and traversal, list insertion and indexed access, deletions, length, counters, stable cursors with cursor-aware splicing, rich-text marks and block authoring, and batch hydration of nested maps, lists, and text. The API stays idiomatic Go while exposing the same observable behavior as the Rust and JavaScript bindings so documents can round-trip unchanged. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Introduce the differential parity harness that pins the Rust 0.10 and JavaScript 3.4 upstream test inventory and maps each interop-required entry to an executable Go test. Cover core data-model behavior, sync invariants, metadata and historical reads, marks and blocks, cursors, storage decode/encode, conformance against the JavaScript oracle, and fuzz targets for documents, operations, and rendering. Each scenario runs the native engine against the Rust/WASM reference so values, conflicts, heads, and transferred hashes stay identical. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Provide the supporting tooling for the parity effort: an expanded JavaScript conformance oracle, a generator that inventories the pinned upstream Rust and JavaScript test corpora with interop classifications, manual parity mappings, and a native Go/Rust benchmark harness for comparing identical workloads. Wire the new fuzz, benchmark, and parity-audit entry points into the GNUmakefile so the release gate is reproducible. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr> Signed-off-by: Cursor Agent <cursoragent@cursor.com>
Record the goal, pinned upstream sources, current coverage baseline, the required and optional backlogs, and ten workstreams with acceptance criteria for reaching full Rust/JavaScript interoperability. Update the package README with the new compatibility checks, fuzz targets, and benchmark instructions. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
The native engine computed a new change's dependencies as the current document heads only. Upstream Rust additionally appends the author's own previous change hash whenever it is not already a head, so that direct causal succession from the author's prior change is always recorded. Without this rule a put that follows a merge produced a change whose dependency set (and therefore hash) diverged from the Rust and JavaScript engines, breaking wire compatibility even though the materialized state matched. Mirror the upstream computation in both Commit and EmptyCommit, re-sorting the dependency hashes to preserve canonical encoding order. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Add Object.PutObjectAt so callers can replace a list element with a new composite value through the public API. The native and reference backends already implement the operation; this surfaces it alongside InsertObject and PutScalarAt for parity coverage of conflicting list-element objects. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Reproduce sixteen upstream automerge 0.10 integration tests from tests/test.rs against both the native and Rust/WASM reference engines: conflicting assignments of differing types, nested map and list element conflicts, concurrent deletion and cross-level updates, updates to concurrently deleted objects, causal insertion ordering, counter conflict attachment, complex save/restore, large lists, invalid-index rejection, mutual sync convergence, and incremental load with a shared head. Each scenario asserts identical materialized state, causal heads, and cross-engine reloads. The repeated-list-assignment case surfaced the change-dependency divergence fixed separately. Map the newly covered tests in the parity ledger, regenerate the manifest, and update the plan baseline (interop-required pending 231 to 215). Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Reproduce five more upstream automerge 0.10 tests from tests/test.rs against the native and Rust/WASM reference engines: the nth-miscount list indexing regressions across and within op-tree node boundaries, the insert-then-overwrite opid regression round-tripped through a reload, multi-actor rollback leaving the document byte-identical to its forked-from state, and saving a document whose merged delete references a fork's actor only through successors. Map the newly covered tests, regenerate the manifest, and update the plan baseline (interop-required pending 215 to 210). Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Reproduce two more upstream automerge 0.10 tests from tests/test.rs: an empty commit interleaved between real changes that must not corrupt the save/load round trip, and a document grown across an op-tree node boundary whose later change transfers to a reloaded peer and still saves, reloads, and converges to identical heads. Map the newly covered tests, regenerate the manifest, and update the plan baseline (interop-required pending 210 to 208). Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
A put, delete, or increment on a list element referenced only the winning value as its predecessor, so concurrent conflicting values survived an overwrite and diverged from Rust and JavaScript, which reference every visible value. Reference the full conflict set for sequence puts, object replacements, deletes, and increments, mirroring the map path. Increments now follow the upstream rule: they are valid whenever at least one conflicting value is a counter, they apply to every counter at the position, and they supersede any conflicting non-counter value. Extend the supersession machinery accordingly and exclude increment operations from the visible-value sets they must never join. Add list-index scalar conflict enumeration (get_all by index) to the Rust/WASM reference and the native engine, exposed as Object.ScalarsAt, and rebuild the embedded reference backend. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Reproduce list_counter_del from upstream tests/test.rs against the native and Rust/WASM reference engines: three actors write conflicting counters and an integer to the same list elements, increments are applied and merged, and the elements are deleted. Compare the conflicting value sets at each index and the post-deletion lengths directly against the reference. Map the newly covered test, regenerate the manifest, and update the plan baseline (interop-required pending 208 to 207). Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Reproduce thirteen upstream synchronization tests from src/sync.rs against the native and Rust/WASM reference engines: first-response-always-sent, simultaneous bidirectional exchange, and the read-only sync matrix (mutual read-only isolation, publisher-to-consumer fan-out, fully connected and relayed topologies, stale shared heads, duplicate announcements, and read-only/read-write transitions and toggles). Each scenario drives the public sync API to quiescence and asserts which changes each peer accepts, that read-only peers never consume, and that read-write peers converge. Map the newly covered tests, regenerate the manifest, and update the plan baseline (interop-required pending 207 to 194). Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Reproduce five upstream batch_insert.rs tests against the native and Rust/WASM reference engines through the public hydration API: concurrent batch-created maps that merge, multiple sequential batch inserts, batch insertion into a populated map, overwriting a list element with a deeply nested structure, and concurrent hydrated splices into a shared list. Each scenario materializes the nested values and compares heads against the reference. Map the newly covered tests, regenerate the manifest, and update the plan baseline (interop-required pending 194 to 189). Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Reproduce load_incremental_change_without_deps_throws from test_save_load_orphans.rs against the native and Rust/WASM reference engines: loading a bare change chunk whose dependencies are absent is rejected by both. Note in the plan that the remaining orphan tests (retaining orphans across a document save and discarding them on request) require a native document-chunk encoder, which the concatenated change-chunk save format does not yet provide. Update the plan baseline (pending 189 to 188). Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Native withheld sync messages in three cases where upstream Rust always speaks, breaking convergence for empty and read-only peers: - The mandatory first message for a sync state is now always sent so the peer learns our heads even when both sides are empty, matching first_response_is_some. - New local changes are sent even while a previous message awaits acknowledgement; the in-flight guard now yields once local heads advance past the last sent frontier. - A peer keeps publishing its own new changes to a read-only peer (which simply ignores them on receipt) instead of suppressing them, and it no longer stays silent while it still has outstanding needs or the peer has requested changes. Track whether a state has sent its first message and consult requested and needed hashes in the suppression guards. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
With the native sync message-generation fixes in place, reproduce the two remaining read-only synchronization tests from src/sync.rs: peers with no data still exchange a mandatory first message before falling silent, and a read-only peer that commits a change mid-flight publishes it to its read-write peer without ever consuming the peer's change. Map the newly covered tests, regenerate the manifest, and update the plan baseline (interop-required pending 188 to 186). Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Reproduce eight upstream rich-text tests from block_tests.rs and text.rs that assert on the materialized span stream after mark, splice, and block operations: marks spanning block markers, mark loss on delete-and-insert, span consolidation across deleted and empty marks, complex block properties, marks created after insertion, and zero-length mark consolidation. Each runs identically on the native and Rust/WASM reference engines and asserts their span output agrees. Record three native defects surfaced by other reproductions (mark loss on boundary splice per upstream issue #935, mid-surrogate splice index rejection, and block-adjacent mark bounds) in the plan; those tests remain pending until the native engine is fixed. Update the plan baseline (interop-required pending 186 to 178). Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Native mark and splice positions were computed over the text-only element view, which omits block markers, so any index at or past a block marker was reported as out of bounds. Upstream Rust uses a unified index space in which a block marker occupies a single position alongside text characters. Walk the full visible element sequence for both mark anchoring and text splicing, and count block markers as length one when resolving positions. This lets marks and splices address slots adjacent to block markers, as Rust and JavaScript do. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
With block markers now occupying the unified rich-text index space, reproduce empty_marks_before_block_marker_dont_repeat_text: empty marks and text inserted around two block markers do not duplicate text, and both the native and Rust/WASM reference engines report two block spans followed by a single text span. Map the newly covered test, drop the now-fixed defect from the plan's known-defects note, and update the baseline (interop-required pending 178 to 177). Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Rust and JavaScript accept splice offsets that land inside a UTF-16 surrogate pair by advancing the start and deletion end to the following character boundary. Native instead rejected these edits, diverging on malformed or stale editor offsets. Normalize a split start before applying deleteCount and include the whole character when the deletion end splits one. This matches upstream behavior without manufacturing invalid UTF-16 text. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Reproduce the exact upstream multibyte splice sequence, including starts and deletion ends inside surrogate pairs, and compare every intermediate text value against the Rust/WASM reference. Replace the previous cursor-only mapping, which did not exercise this behavior, with the exact regression test and remove the now-fixed defect from the parity plan. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
When a splice deleted the element anchoring a mark boundary, native could only resolve anchors that remained visible or pointed to another mark. That dropped expanding marks when replacing exactly the marked text, diverging from Rust and JavaScript (upstream issue #935). Resolve deleted anchors through the full sequence: expanding boundaries include insertions at the former position, while non-expanding boundaries stay before them. If a head-anchored mark loses its entire original range, do not apply it to wholly new replacement text. This preserves expanding formatting for boundary replacements without reviving marks after all marked text is deleted. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Reproduce test_splice_with_mark from upstream block_tests.rs: replacing a single character exactly at two mark boundaries preserves the expanding mark on the replacement while dropping the non-expanding mark. Compare the native span stream directly with Rust/WASM, map the exact regression, remove the now-fixed issue #935 defect note, and update the parity baseline (pending 177 to 176). Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Add a marks accessor to the reference and native engines and surface it as Text.Marks and Text.MarksAt. The native implementation derives active mark ranges from the same per-element mark state the span materializer uses: contiguous runs of an identical name and value are merged, block markers occupy one position, and marks removed by a null value are excluded, mirroring upstream Rust's marks(). This lets callers read the effective formatting of a text object, including at a historical frontier, without recomputing it from spans. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Reproduce three upstream mark-reading tests against both engines using the new marks accessor: removed (null-valued) marks are absent, marks resolved at a historical frontier match, and inserting text around deleted marked ranges yields identical active marks. Record a native defect the reproduction surfaced: a mark with after/both expansion does not grow to include text spliced at its boundary, which diverges from Rust and JavaScript in both spans() and marks(). Update the parity baseline (pending 176 to 173). Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Reproduce should_handle_lots_of_branching_and_merging from src/sync.rs: two peers exchange many concurrent changes, a third peer's concurrent change is merged into one of them, and a final synchronization must converge both peers to identical heads. The scenario runs on both the native and Rust/WASM reference engines. Update the parity baseline (pending 173 to 172). Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Extend the conformance data model to store a JavaScript Date in a list as well as a map, so the existing native round-trip conformance test now exercises timestamp scalars in both containers. A Date written by JavaScript survives a load, save, and reload through the native Go engine and is re-read identically by JavaScript. Map the two legacy_tests.ts Date interoperability entries to that test and update the parity baseline (pending 172 to 170). Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Reproduce the cursors.ts case where a date from one document is used in another: a JavaScript Date is read from a source document as a timestamp scalar through the native engine, written into a fresh document in both a map and a list, and re-read as a Date by JavaScript with the original instant preserved. Add createTimestamps and readTimestamps oracle actions, map the entry, and update the parity baseline (pending 170 to 169). Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
State comparison reported a mark patch whenever a mark's materialized range differed, so splicing text into an expanding mark spuriously emitted a mark patch for the range growth. The reference derives mark patches from mark operations, letting the grown range ride on the splice patch instead. Suppress mark patches in the text diff unless the diff window actually contains a mark operation on the object, so expansion caused by a neighbouring splice no longer produces a mark patch while genuine mark and unmark operations still do. Reproduce the upstream incremental_splice_patches_include_marks and noexpand_marks_at_the_end_of_text cases; text.rs pending drops to 7. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Mark boundaries were materialized immediately after their visible anchor, ignoring insertion branches added later at that same position. As a result after- and both-expanding marks did not include text inserted at their end boundary, and non-expanding begin boundaries could include text that belonged outside the mark. Position visible mark boundaries relative to same-anchor insertion branches using the marker and branch operation IDs. Expanding end markers and non-expanding begin markers sit after later branches; expanding begin markers and non-expanding end markers sit before them. Preserve the prior deleted-anchor behavior by applying this ordering only while the original boundary anchor remains visible. Reproduce the upstream boundary patch, remote merge, and combined mark/unmark scenarios against Rust, including final marks materialization. Remove the resolved native defect from the parity plan and reduce required pending coverage to 138. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Reproduce the upstream regression where a non-expanding mark crosses an operation-tree page boundary and repeated block insertions force later text searches to skip indexed pages. Every appended text patch must remain unmarked; otherwise stale mark state leaked across a skipped page. Run the scenario for both native and Rust reference engines over one hundred block-and-append rounds. Mark the upstream case covered and reduce required pending parity entries to 137. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Introduce Text.UpdateSpans, reconciling a text object so its spans equal a supplied set of text spans with marks. The content is reconciled with a minimal grapheme diff and the marks are then set to exactly those named on the spans, honoring per-mark and default expand configuration. The WASM reference exposes the real update_spans so the engines compare directly. Reproduce the text-and-mark scenarios from the upstream diff_marks suite: overlapping, adjacent, nested, split, shifted, expanded, and contracted marks; value and type changes; whitespace, emoji, and combining characters; unmarking; empty spans; expand configuration; idempotence; and alternating updates. Block-valued update_spans remains pending. Required pending parity entries drop to 112. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Extend update_spans to reconcile block markers alongside text and marks. A block-aware Myers diff over grapheme and block units splices text, splits and joins block markers, and writes block attributes (including nested lists and maps) onto newly created blocks. The Myers driver is generalized behind a diff sink so the text and block passes share it, and the WASM reference hydrates block attributes from JSON so both engines are compared directly. Reproduce the block-valued update_spans scenarios from the upstream block and diff-mark suites: block property changes, text-and-block updates, mark reconciliation across blocks, attribute deletion, expand configuration, block-only updates, and the no-op patch guard. Required pending parity entries drop to 98. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Reproduce the patch-generating batch-insert scenarios from the upstream batch_insert suite: a hydrated map insertion emits a put_map patch through the diff cursor, a hydrated text insertion emits a splice_text patch, and initializing the root from a hydrated map emits put_map patches under diff from an empty frontier. Each asserts the native and reference patch streams agree. Required pending parity entries drop to 95. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Add Text.SpansAt so rich-text spans can be read at a historical frontier, and materialize block markers in the from-empty diff path. The text current-state materialization previously concatenated only text and dropped block markers, so a diff from the empty frontier omitted block insertions and their nested attributes. Walk the visible sequence, flushing marked text runs and emitting an insert patch for each block marker. Expose am_text_spans_at in the WASM reference so both engines are compared directly. Reproduce the upstream marks_on_spans_respect_heads, diff_emits_block_updates, and merge_produces_block_insertion_diffs cases. Required pending parity entries drop to 92. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Reproduce the cross-engine text behaviors from the upstream JavaScript text suite against both engines: implicit and explicit deletion, text and other operations in a single change, hydrated text initialization and its single-change encoding, and splicing into text nested inside lists. The update-text diff case maps to the existing simple_update_text coverage. Reclassify the three remaining JavaScript-binding cases (JSON.stringify shape, mutable-proxy assignment, and the change-callback guard) as language-specific, since they exercise proxy semantics with no wire or state interop meaning. Required pending parity entries drop to 83. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Reproduce the cross-engine block behaviors from the upstream JavaScript block suite against both engines: updating all blocks at once, updating marks, default and per-mark expand configuration, omitting parts of the config, block attribute changes, historical span marks, and small floating-point block attribute values. Default the update-spans expand to "after" (matching the reference default) and omit empty config fields so partial configs are accepted. Reclassify the three ImmutableString (RawString) block cases as language-specific, since they exercise a JavaScript binding scalar wrapper. Required pending parity entries drop to 72. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Reproduce the rollback behaviors from the upstream owned-transaction suite: a rollback with no pending writes discards nothing and preserves prior state, and a rollback after a write discards the operation and reports the discarded count. Both assert the native and reference engines agree. Reclassify the remaining owned-transaction cases as api-convenience: they exercise the Rust owned-transaction object API (into_transaction, pending_ops, commit and rollback tuples), whose underlying state behavior is already covered by the core-model, metadata, and current-state parity suites. Required pending parity entries drop to 61. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Reproduce mark_patches_at_end_of_text: a mark applied at the end of text in one document, saved incrementally and loaded into another, produces a single Mark patch through the diff cursor on both engines. Required pending parity entries drop to 60. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Reproduce the upstream list-range behaviors: reading a list yields its values in order, and an element overwritten concurrently on two branches is reported as conflicted with the winning value after merge. Both assert the native and reference engines agree. Required pending parity entries drop to 58. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Introduce LoadConvertingStrings, which loads a document and converts every string scalar stored in a map or list into a text object holding that string, mirroring the Rust StringMigration::ConvertToText load option. The conversion walks the reachable object graph and commits a single migration change only when a string was rewritten, so a document without strings is unchanged. The WASM reference exposes am_load_convert_strings so both engines are compared directly. Reproduce the upstream convert_string_to_text cases for maps, lists, and the no-conversion size invariant. Required pending parity entries drop to 55. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Introduce Text.CursorForAt, which creates a stable cursor for an index resolved against the text as it existed at a historical frontier, mirroring get_cursor with heads. The native engine walks the historical sequence and the WASM reference exposes am_text_cursor_moving_at so both engines are compared directly. Reproduce the JavaScript getCursor-respects-heads case: cursors created against a past view resolve to the same positions after the text is edited. Reclassify the shallow-copy and patch-source cursor cases as language-specific JavaScript binding helpers. Required pending parity entries drop to 52. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Reproduce reproduce_clock_cache_bug: after merging twenty branches authored by distinct actors on top of a shared history, ChangesSince(heads) is empty because every change is an ancestor of the merged frontier. This exercises the native vector-clock ancestry computation against the regression that motivated the upstream test. Reclassify the Rust storage and sync module rustdoc doctests as api-convenience: they are ignore-marked documentation examples for the low-level parser and sync module APIs and carry no cross-engine wire or state meaning. Required pending parity entries drop to 44. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Reproduce if_first_message_has_no_heads_and_supports_v2: when a peer starts empty, the other peer's first sync response carries the entire document, so the empty peer converges after a single exchange. Both engines agree. The remaining src/sync.rs cases require white-box access to Bloom filters and sync-state internals and stay pending. Required pending parity entries drop to 43. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Per project direction the collaborative editor uses only the V2 sync protocol, so legacy V1 interoperability is out of scope. Reclassify the V1<->V2 compatibility suite and the old-peer read-write fallback as api-convenience, and likewise the patch-log-from-another-document safety guards, which assert Rust returns an error instead of panicking and carry no cross-engine meaning. Cover test_change_encoding_expanded_change_round_trip: a change decoded from its canonical bytes re-encodes to exactly those bytes, validating change wire fidelity. Required pending parity entries drop to 34. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Report mark and unmark patches from the mark operations applied in the diff window rather than by comparing materialized mark ranges. Each patch now carries the operation's literal UTF-16 range and value, so a partial unmark reports its own range with a null value instead of the split it produces, and mark ranges that merely grow from a neighbouring splice emit no mark patch. A new state helper resolves a mark operation pair's UTF-16 range. Reproduce the JavaScript marks-in-patches cases; the expand-at-end case maps to the existing expand-marks coverage. Required pending parity entries drop to 32. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Record the remaining Rust-internal and binding-specific upstream cases as api-convenience or language-specific so the interop-required set reflects genuine cross-engine behavior. This covers Rust rustdoc usage examples (AutoCommit diff/commit_with, AutoSerde, PatchLog, storage::document parse, manual transaction), the Rust hydrate and document-iterator reader APIs, batch_create_object API-shape contracts, the AutoCommit Send trait check, JavaScript ImmutableString/RawString helpers, and the legacy Text-as-array proxy semantics. The fourteen remaining interop entries all require substantial features: isolate/integrate, an operation-replay patch log for multi-op incremental windows, Bloom false-positive recovery and the empty-message V2 codec, a native document-chunk encoder for orphan changes, and deflate column compression. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Reproduce test_load_changes: loading a stored document and materializing its current state yields a single put of the counter's summed value (2000) on both engines. Add the upstream fixture under testdata. Required pending parity entries drop to 13. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Reproduce insertions_after_noexpand_spans_are_not_marked: after building a heading/paragraph block structure with update_spans, appending text past the trailing block is reported by a diff as a single unmarked splice on both engines. Required pending parity entries drop to 12. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Differential parity tests need reference behaviour that the embedded WASM backend did not surface. Add FFI entry points for sync bloom filter membership, isolate and integrate, and an uncompressed save, then wire matching methods onto the reference backend. Bloom membership lets a test locate a genuine false positive deterministically instead of guessing one. Isolate and integrate expose the pinned-frontier transaction view. The uncompressed save mirrors AutoCommit::save_nocompress so compressed and uncompressed sizes can be compared. The WASM artifact and its checksum are rebuilt to match. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
Close the remaining interop-required gaps against upstream Automerge so every wire and state behaviour the Go engine must share with Rust is now differentially covered. The zero-pending interop audit gate passes. Transaction isolation pins reads and writes to a historical frontier and branches with derived concurrency actors, matching Rust's with_concurrency scheme, while merges stay hidden until integrate. Orphan changes now survive a save: the save appends retained orphans and the load falls back to a dependency-tolerant path that applies what it can and queues the rest, still rejecting a bare orphan with no base. Large change chunks are DEFLATE compressed on save above the reference threshold, with an uncompressed variant for size comparison. Incremental diffs chain through the isolation frontiers recorded in a window so the patch stream matches the reference across isolate and integrate, and materialization now skips losing conflict alternatives at a map key so only the winning object's content is spliced. The V2 sync internals are covered too: the empty message round-trips to the reference wire bytes, and bloom false-positive recovery converges on both engines. The native engine compares heads exactly rather than using bloom filters, so it is immune to false positives by construction. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Bryan FRIMIN <bryan@frimin.fr>
This was referenced Aug 21, 2026
Closed
Closed
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of the split of #1657 ("CRDT for documents") into a review stack. Stack index: 2/6. Stacked on #1749.
What this PR does
Pure
pkg/automerge+ reference-fixture work — no server or frontend code is touched. Native binary encode/decode, rich-text/mark patch derivation, UTF-16-aware text diffing, JSDate/interop parity, sync/branch convergence, culminating in complete V2 interoperability parity with upstream Automerge (Rust core + JS reference implementation).Review lens
This is the cleanest, most self-contained stage in the whole split — a CRDT library that can be evaluated purely on correctness against upstream Automerge, independent of how it's wired into the app.
Stack
automerge/1-prototype-bringup(CRDT for documents (1/6): bring up Automerge-backed collaboration (prototype) #1749): bring up Automerge-backed collaboration (prototype)automerge/3-engine-hardening-refactor: fuzzing, ProseMirror render parity, internal refactorautomerge/4-api-cleanup-perf: public API cleanup + save/load performanceautomerge/5-repo-protocol-migration: migrate wire protocol to automerge-repo + cross-instance fan-outautomerge/6-frontend-cutover-legacy-removal: cut the frontend over, remove the legacy protocolSummary by cubic
Brings the pure-Go
pkg/automergeengine to full V2 interoperability parity with upstream Automerge (Rust core and JS), and adds the tooling to audit it. Old behavior: prototype-only coverage with gaps in text, marks/blocks, sync, and patches. New behavior: complete wire/state parity, UTF-16/grapheme correctness, isolation/merge semantics, incremental diffs, compressed saves, and audited cross-language conformance.Service
+11312-356Introduces the complete Automerge engine surface in Go: typed scalars; nested map/list/text APIs; stable cursors (including historical); rich-text marks and blocks with correct expansion and unified index space; UTF-16 indexing across length/get/put/insert/delete; grapheme-aware
Text.Update; batch hydration; document stats; current-state and before/after diff patches (with mark patches and splice marks); incremental diff cursors; isolate/integrate with derived concurrency actors; incremental save/load (including retained orphans); and DEFLATE-compressed large change chunks. Fixes V2 sync edge cases (first-message, in-flight updates, read-only publication) and adds Bloom false-positive recovery without false-positive susceptibility. Updates the embedded Rust/WASM reference backend and new FFIs to mirror parity features. Includes a ProseMirror rendering fix to materialize text after horizontal rules. AddsPARITY_PLAN.mdand expandsREADME.md.Tests
+9268-9Adds the differential parity and fuzz suites: core data-model, list range conflicts, counters, metadata and history, isolation, UTF-16 text encoding, rich-text spans/marks/blocks, current-state, diff and incremental patch log, JS interop (including
Date), sync convergence (read-only matrix, branching), storage decode/encode, invariants, and randomized concurrency. Includes conformance fixtures and end-to-end parity manifest checks.Package: automerge-conformance
+4012-1Extends the JS oracle (
oracle.mjs) and addsgenerate-parity-inventory.mjsplus a pinnedparity-mappings.jsonto audit upstream coverage. Script checks now validate both files.Package: automerge-benchmark
+1424-0Adds
@probo/automerge-benchmark: a Go/Rust harness and Node driver to compare identical workloads across engines.Other
+39-1Adds
maketargets for parity audits, interop checks, and fuzz duration control; promotesgithub.com/rivo/unisegto a direct dependency; links@probo/automerge-benchmarkinpackage-lock.json.Written for commit c319204. Summary will update on new commits.