[Incremental Reprocessing] [WIP] Proof-of-concept incremental reprocessing for MongoDB - #468

Draft

rkistner wants to merge 212 commits into

mainfrom

incremental-reprocessing

Commits on Nov 17, 2025

Refactor replication connections.
rkistner
committed
fda46b6
Start streaming concurrently - lets see what breaks.
rkistner
committed
7a812ee
Split out snapshot logic from streaming logic.
rkistner
committed
be3390a
Quick workaround for now.
rkistner
committed
fcd539d
Fix test issues.
rkistner
committed
7701a9a

Commits on Nov 18, 2025

WIP.
rkistner
committed
7e3b0d1
Split out "snapshot done" check.
rkistner
committed
98da657
Tweaks to tests.
rkistner
committed
db5d578
Fix for postgres storage.
rkistner
committed
b7a5d5f
Refactor for snapshotting.
rkistner
committed
df0d4cb

Commits on Nov 20, 2025

Refactor commit logic to better handle concurrent replication.
rkistner
committed
ce500a1
Cover case of no tables to snapshot.
rkistner
committed
f5ec031
Implement missing method for Postgres storage.
rkistner
committed
049b32a
Port changes to postgres storage.
rkistner
committed
531e887
Fix data storage tests - markSnapshotComplete is required.
rkistner
committed
64e2abc
Fix storage sync tests.
rkistner
committed
333bdad
Fix snapshot_lsn handling.
rkistner
committed
29f4302
Fix snapshot_lsn in postgres storage.
rkistner
committed
cd19ba2
Fix test promise handling.
rkistner
committed
8dc726f
Fix more tests.
rkistner
committed
ac790a7

Commits on Nov 24, 2025

Make schema test more stable.
rkistner
committed
12d60ed

Commits on Dec 1, 2025

Merge remote-tracking branch 'origin/main' into stream-during-snapshot
rkistner
committed
85aaccc
Refactor streaming promise management.
rkistner
committed
f09dea3
Fix more tests.
rkistner
committed
b3a23ef
More stable abort handling.
rkistner
committed
8b9ef4f
Skip empty checkpoints on Postgres again.
rkistner
committed
1d0088f
More test fixes.
rkistner
committed
1c85ca2
Add tests for empty checkpoints.
rkistner
committed
8a55aa1
Implement createEmptyCheckpoints filter for mongodb storage.
rkistner
committed
77cd69b

Commits on Dec 2, 2025

Implement soft deletes for current_data.
rkistner
committed
0041ff3
Cleaner abort handling.
rkistner
committed
b5a4428
Implement current_data soft deletes on postgres storage; add tests.
rkistner
committed
44afe81
Fix compacting tests.
rkistner
committed
df8b2dc
Add missing migration.
rkistner
committed
8f33247
Fix more tests.
rkistner
committed
1d17232
Update snapshot.
rkistner
committed
d3bfdd6
Improve handling of abort errors.
rkistner
committed
b68ffa6
Improve transaction isolation for postgres storage batches to avoid
rkistner
committed
8cc27bd
More abort error improvements.
rkistner
committed
ec82bb6

Commits on Dec 3, 2025

Fix truncate on MongoDB storage.
rkistner
committed
3129507
Fix postgres storage truncate; typed SourceTable id.
rkistner
committed
3c77f43
Fix build errors.
rkistner
committed
7b87214
Fix another build error.
rkistner
committed
4961662
Merge remote-tracking branch 'origin/main' into stream-during-snapshot
rkistner
committed
4a6a5ae
Workaround for tests.
rkistner
committed
d45c212
Same workaround in different test.
rkistner
committed
d968d39

Commits on Dec 4, 2025

Better fix for tests.
rkistner
committed
4958054
Better handling of deadlocks on current transactions.
rkistner
committed
effa749
Another error test tweak.
rkistner
committed
2a7a572
Restructure TEST_TABLE for tests.
rkistner
committed
bfc1259
Checksums are different for postgres now.
rkistner
committed
6a5590f
Merge remote-tracking branch 'origin/main' into stream-during-snapshot
rkistner
committed
eefddef
Fix merge conflict.
rkistner
committed
9dab9ef
Revert postgres changes.
rkistner
committed
8447d1d
Re-apply some postgres changes.
rkistner
committed
874ccb6
Re-apply more test changes.
rkistner
committed
ff32dab
Changeset.
rkistner
committed
d12daac
Postgres: stream while snapshotting.
rkistner
committed
510173c
Merge remote-tracking branch 'origin/main' into concurrent-storage-batches
rkistner
committed
e72c189
Fix some post-merge conflicts.
rkistner
committed
841273e
Tweaks to commit logs and logic.
rkistner
committed
5c18559

Commits on Dec 8, 2025

Fix results of markTableSnapshotDone.
rkistner
committed
0fb42f9

Commits on Jan 5, 2026

Better casting.
rkistner
committed
5935598
Null safety.
rkistner
committed
95c6ab5
Merge remote-tracking branch 'origin/main' into concurrent-storage-batches
rkistner
committed
7ade959

Commits on Jan 7, 2026

Merge remote-tracking branch 'origin/concurrent-storage-batches' into postgres-concurrent-streaming
rkistner
committed
9d50970
Fix error code check and improve test stability.
rkistner
committed
689dcee
Fix error code check.
rkistner
committed
0de45b7
Merge branch 'concurrent-storage-batches' into postgres-concurrent-streaming
rkistner
committed
5a1caed
Initial implementation for MongoDB.
rkistner
committed
f08bb44
Fix tests to use the new structure.
rkistner
committed
eae4aef
Fix race conditions with table snapshot state.
rkistner
committed
96cb151
Fix race condition on completing snapshots.
rkistner
committed
53efacb

Commits on Jan 8, 2026

Refactor ChangeStream implementation.
rkistner
committed
a845758
Fixes.
rkistner
committed
8e28411
Fix potential race condition leading to locks held for longer than
rkistner
committed
f2b0b6c

Commits on Jan 9, 2026

Implement actual multiplexing for mongodb change streams.
rkistner
committed
ffe752f
Fix tests.
rkistner
committed
60e5897

Commits on Jan 12, 2026

Consistent sort order for sync rules.
rkistner
committed
22f5904
Persist mappings.
rkistner
committed
55adf3a
Refactor deletes.
rkistner
committed
ed9e90c
Scoped bucket_data.
rkistner
committed
46ca53e
Fix sync rules tests.
rkistner
committed
24c5484
Tests build again.
rkistner
committed
3e45cc7
Minor restructure and test fixes.
rkistner
committed
9f50346

Commits on Jan 13, 2026

Fixes for checksums and some tests.
rkistner
committed
25344b8
Refactor current_data.
rkistner
committed
99bca49
Fix more tests.
rkistner
committed
017e6dd
More fixes for clearing data.
rkistner
committed
03b07dd
Fix test build issues.
rkistner
committed
df561c0
Initial fixes for parameter lookups.
rkistner
committed
5714d9f
Cleanup.
rkistner
committed
d26a01f
Hack: re-use existing mappings.
rkistner
committed
306beb6
Initial writer restructuring.
rkistner
committed
2d14290

Commits on Jan 14, 2026

WIP: merged processing.
rkistner
committed
d73fad6
Initial working through errors.
rkistner
committed
872d3a9
Restructure snapshotter.
rkistner
committed
d816b6b
Fix job wiring up.
rkistner
committed
f9dfbfc
Fixes.
rkistner
committed
44ac2b7
Fixes to source table filtering.
rkistner
committed
8b923a5
Fix source table filtering.
rkistner
committed
436e6f4
Fix initialization of sync rule rules where no new snapshots are
rkistner
committed
bb63762

Commits on Jan 15, 2026

Work around build issues.
rkistner
committed
596343d
Merge remote-tracking branch 'origin/main' into concurrent-storage-batches
rkistner
committed
8c67e14
Merge branch 'concurrent-storage-batches' into postgres-concurrent-streaming
rkistner
committed
40baa93
Merge branch 'postgres-concurrent-streaming' into mongo-concurrent-streaming
rkistner
committed
8075d95
Merge branch 'mongo-concurrent-streaming' into incremental-reprocessing
rkistner
committed
cbd97b0
Fix test build errors.
rkistner
committed
e71bc49
Support parameter lookups again; simplify a bit.
rkistner
committed
9cffa33
Fix parameter row filtering.
rkistner
committed
7acc0f7
Minor code simplification.
rkistner
committed
24ba4d0
Use TablePattern[] instead of Set<TablePattern>.
rkistner
committed
54e5f59

Commits on Jan 19, 2026

Refactor to pull through TablePattern.
rkistner
committed
bb9fb76
Improve lookup performance in some cases.
rkistner
committed
13d2a31

Commits on Jan 20, 2026

Re-add check for resumeToken order.
rkistner
committed
b9d517d
Fix batch logic in tests.
rkistner
committed
048675f
Fix metadata going missing.
rkistner
committed
08089fd
Fix detecting new changes.
rkistner
committed
370bbdd
Initial round of storage test fixes.
rkistner
committed
3d9bbe8
Round 2 of storage test fixes.
rkistner
committed
604461e
Test fix round 3.
rkistner
committed
4f802cc
Initial compacting fixes.
rkistner
committed
c58bdd6
Fix parameter compacting.
rkistner
committed
b5baf5c
Add missing test file.
rkistner
committed
e81181a

Commits on Jan 21, 2026

Test fix round 4.
rkistner
committed
c46ce88
Another test fix.
rkistner
committed
ebdae0e
Improve queue responsiveness.
rkistner
committed
bafd322
Postgres storage fixes part 1.
rkistner
committed
ab4a867
Some postgres storage test fixes.
rkistner
committed
a85771e
Implement getTable.
rkistner
committed
18f2d99
Fix updating table progress.
rkistner
committed
130041f
Remove some startBatch() usage from tests.
rkistner
committed
a93c031
Move WalStream to new BucketDataWriter.
rkistner
committed
b70f3be
Further compatibility fixes for postgres snapshotter.
rkistner
committed
8d9ca0d
Partial SQL Server refactor to new APIs.
rkistner
committed
7aa1754
Refactor BinLogStreams to use the new APIs (untested).
rkistner
committed
7d3a091
Remove startBatch.
rkistner
committed
a209643
Remove resolveTable from the public API.
rkistner
committed
a30c825

Commits on Jan 22, 2026

Remove some BucketStorageBatch usages.
rkistner
committed
93f6103
Simplify createWriter usage.
rkistner
committed
50b4b5b
Rename methods.
rkistner
committed
fdd7668
Rename BucketDataWriters.
rkistner
committed
99a7590
Map by table id instead of reference.
rkistner
committed
422d90f
Remove mapping for source tables - it is not reliable enough.
rkistner
committed
fb1bb7f
Fix dropping tables.
rkistner
committed
21517d7
Fix skipIf.
rkistner
committed
5b30f54
Fix sorting in test.
rkistner
committed
2489278
Restructure logic to drop tables.
rkistner
committed
b4e03b7
Change drop/snapshot order for test compatibility.
rkistner
committed
e32357d

Commits on Feb 2, 2026

Fix more tests.
rkistner
committed
c2cfe8d
Remove duplicate processing issue.
rkistner
committed
a1aebc2
Another test fix.
rkistner
committed
a759c74
Rewrite parameter tests to be independent of storage format.
rkistner
committed
52ae861
Merge remote-tracking branch 'origin/main' into concurrent-storage-batches
rkistner
committed
d53eeec
Merge branch 'concurrent-storage-batches' into postgres-concurrent-streaming
rkistner
committed
8681011
Merge branch 'postgres-concurrent-streaming' into mongo-concurrent-streaming
rkistner
committed
1ee9e94
Merge branch 'mongo-concurrent-streaming' into incremental-reprocessing
rkistner
committed
e10bc50
Fix initial compile issues post merge.
rkistner
committed
d169562
Fix circular import.
rkistner
committed
ec975c8
Fix some tests.
rkistner
committed
b236426

Commits on Feb 3, 2026

Fix test build issue.
rkistner
committed
127d416
Add missing file.
rkistner
committed
d885186
Restructure cast functions.
rkistner
committed
d45305f
Fix storage parameter tests.
rkistner
committed
997145b
Fix type issues.
rkistner
committed
cdbece1
Disable more unstable tests.
rkistner
committed
745d85f
Fix MS SQL replication.
rkistner
committed
dfa7e32
Restructure parameter lookups.
rkistner
committed
44edd30
Fix test.
rkistner
committed
6285c9f
Fix sync-rule tests to use the correct source.
rkistner
committed
6c06fd0
Fix truncating tables.
rkistner
committed
2c31123

Commits on Feb 17, 2026

Merge remote-tracking branch 'origin/main' into concurrent-storage-batches
rkistner
committed
e27e04e
Fix formatting issue.
rkistner
committed
e0d344e
Merge branch 'concurrent-storage-batches' into postgres-concurrent-streaming
rkistner
committed
674f05a
Merge branch 'postgres-concurrent-streaming' into mongo-concurrent-streaming
rkistner
committed
3b24010
Merge branch 'mongo-concurrent-streaming' into incremental-reprocessing
rkistner
committed
11de7bb
Post-merge type fixes.
rkistner
committed
8bc2aec
Fix sync-rule tests.
rkistner
committed
dfe726a
Fix more type issues.
rkistner
committed
6184ebf

Commits on Feb 18, 2026

Fix tsconfig.
rkistner
committed
ada05fd
Fix hydration issue.
rkistner
committed
d6807a2
Merge remote-tracking branch 'origin/main' into concurrent-storage-batches
rkistner
committed
26f5150
Post-merge type fixes.
rkistner
committed
25f7f3c
Some TEST_TABLE fixes.
rkistner
committed
98d63ff
Fix more tests.
rkistner
committed
bae9551
Re-order migrations.
rkistner
committed
3e0c2bc
Fix issue with delete + insert.
rkistner
committed
93a2615
Restructure checkpoint results.
rkistner
committed
290bc54
Fix CDC handling of checkpoints.
rkistner
committed
ade7be0
Fix tests after merge conflicts.
rkistner
committed
f431360

Commits on Feb 23, 2026

Merge branch 'concurrent-storage-batches' into incremental-reprocessing
rkistner
committed
b47dae5
Post-merge type fixes.
rkistner
committed
cfffd40
Fix core tests.
rkistner
committed
628214b
Fix bucket versioning.
rkistner
committed
81219b6
Update snapshots.
rkistner
committed
a597ded
Fix more tests.
rkistner
committed
d340625

Commits on Mar 5, 2026

Merge remote-tracking branch 'origin/main' into incremental-reprocessing
rkistner
committed
b3d2f70
Fix sync rule merge issues.
rkistner
committed
08d7566
Fix service-core.
rkistner
committed
3c9d1fe
Fix build issues.
rkistner
committed
0b5981a
Minor fixes.
rkistner
committed
b766cf6
Move BucketDefinitionMapping to service-core.
rkistner
committed
1e879ed
Fix some tests.
rkistner
committed
3b87e79
This test is storage v3+ only.
rkistner
committed
f17b531
Update snapshot.
rkistner
committed
23df303
Fix postgres merge issue.
rkistner
committed
c2358b3
Update snapshot.
rkistner
committed
e310fc8

Commits on Mar 11, 2026