feat: Use Compiler for Event Definitions - #762
Draft
stevensJourney wants to merge 10 commits into
Draft
Conversation
🦋 Changeset detectedLatest commit: 0ada267 The changes in this PR will be included in the next version bump. This PR includes changesets to release 19 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part 1 of 3
Summary
Replication events defined through
event_definitionsare currently parsed and evaluated by a separate legacy implementation. Sync streams and buckets already use the sync-plan compiler.This PR compiles replication events into the sync plan and evaluates them through the same row-projection machinery used by streams. It also introduces behavioral equality for the compiler's JavaScript event model, which the next PR will use to decide whether an event definition can be reused during incremental reprocessing.
This PR does not assign persisted event ids or change incremental-reprocessing behavior. Those storage concerns remain in part 2.
Motivation
Replication events map rows from a source table to payloads consumed by the service. Custom checkpoint requests are one use of these events:
Incremental reprocessing needs to distinguish a behavioral event change from a non-functional SQL edit. Comparing raw SQL would treat formatting and equivalent operand ordering as changes. For example, these definitions compile to equivalent event behavior:
The comparator works from the compiler's JavaScript model instead of serialized SQL or a content-derived UUID. This keeps compatibility decisions independent of the serialized plan format and allows the comparison rules to evolve with the compiler.
What changed
event_definitionsare now compiled for every supported sync-config edition and stored in an additiveeventsfield in the serialized sync plan. Compiled events use the shared row-projection implementation, allowing the separateSqlEventDescriptorandSqlEventSourceQueryevaluators to be removed.Behavioral equality includes the event name, source tables, filters, and projected payload. It ignores raw SQL formatting and payload-query order. Reordered and regrouped
ANDandORexpressions compare as equivalent, as do swapped=andISoperands. Reversed comparisons such asa > bandb < aare normalized to the same representation.Comparison remains conservative for constructs that are not explicitly normalized. Failing to recognize an equivalence may cause a later deployment to reprocess unnecessarily, while different behavior must never be treated as reusable.
The existing event-query restrictions remain. Each payload query must read from one physical source table and cannot depend on request parameters. An event can define at most one payload query for a given source table.
Persisted-plan compatibility
Compiled events are an additive sync-plan field and do not require a plan-version bump.
The service continues to dual-write the raw event SQL alongside the compiled plan so older service versions can ignore the new field and use their legacy event evaluator. When a new service loads an older persisted plan without compiled events, it compiles that sidecar SQL at the loading boundary and uses the compiled evaluator from then on.
Follow-up
Part 2 uses the behavioral equality added here to assign or reuse stream-local event ids, store those ids as source-table memberships, and resnapshot only the source tables associated with new or changed event definitions.
Validation
Tests cover event compilation, serialization, deserialization, hydration, and row evaluation. They verify behavioral equality across formatting, payload-query ordering, commutative operands, and reversed comparisons, while ensuring changed behavior is detected.
Compatibility tests cover both newly compiled plans and older plans containing only raw event descriptors. Validation tests cover unsupported joins, request parameters, and duplicate source tables.
AI usage: Guided Codex GPT 5.6 to implement this. Reviewed manually and with Claude Opus 4.8.