Skip to content

events: ask the index for a window, and match in two stages - #1088

Open
tamirms wants to merge 2 commits into
tamirms/events-slab-matcherfrom
tamirms/events-windowed-matches
Open

tamirms wants to merge 2 commits into
tamirms/events-slab-matcherfrom
tamirms/events-windowed-matches

Conversation

@tamirms

@tamirms tamirms commented Oct 5, 2026

Copy link
Copy Markdown
Contributor

What

  • Reader.LookupKeys now takes the range of event ids the query needs (the window) and returns, alongside the bitmaps, the range its answer covers.
  • Matches asks first for the leading four slabs of the window, in the walk's direction. It asks for the rest only if the page is still not full and the first answer didn't already cover it.
  • Both readers still return whole terms and report the whole id space as covered, so every query still runs in one stage.

Why

Groundwork for the next PR, which stores a popular term's bitmap in parts. With a window, the cold reader can read only the parts a page reaches and report exactly what it read. On its own, this PR changes neither results nor I/O.

Known limitations

N/A

🤖 Generated with Claude Code

@tamirms
tamirms added this pull request to stack #1089 October 5, 2026 16:24
@tamirms
tamirms removed this pull request from stack #1089 October 8, 2026 08:18
@tamirms
tamirms force-pushed the tamirms/events-windowed-matches branch from cb87b5b to 1661193 Compare October 8, 2026 08:18
@tamirms
tamirms added this pull request to stack #1102 October 8, 2026 08:19
@tamirms
tamirms force-pushed the tamirms/events-windowed-matches branch 3 times, most recently from 4fde3e9 to 551b86a Compare October 9, 2026 06:02
@tamirms
tamirms force-pushed the tamirms/events-windowed-matches branch from 551b86a to 39598c7 Compare October 15, 2026 18:13
@tamirms
tamirms force-pushed the tamirms/events-windowed-matches branch from 39598c7 to 5c9e548 Compare October 15, 2026 19:15
@tamirms
tamirms force-pushed the tamirms/events-windowed-matches branch 2 times, most recently from 3fa9a24 to d4a90ae Compare October 21, 2026 08:48
@tamirms
tamirms force-pushed the tamirms/events-windowed-matches branch from d4a90ae to 82ef797 Compare October 21, 2026 09:12
@tamirms
tamirms force-pushed the tamirms/events-windowed-matches branch 2 times, most recently from 18d433a to 050c9de Compare October 21, 2026 09:52
@tamirms
tamirms force-pushed the tamirms/events-windowed-matches branch from 050c9de to 6914f1c Compare October 21, 2026 11:16
@tamirms
tamirms force-pushed the tamirms/events-windowed-matches branch 2 times, most recently from 6b86dec to c402249 Compare October 21, 2026 14:07
@tamirms
tamirms force-pushed the tamirms/events-windowed-matches branch from c402249 to 9f50fcb Compare October 21, 2026 14:43
@tamirms
tamirms force-pushed the tamirms/events-windowed-matches branch from 9f50fcb to 6b2ae83 Compare October 21, 2026 17:05
@tamirms
tamirms force-pushed the tamirms/events-windowed-matches branch from 6b2ae83 to 009e40e Compare October 21, 2026 17:15
@tamirms
tamirms force-pushed the tamirms/events-windowed-matches branch 2 times, most recently from 9262ddb to 1a3daac Compare October 22, 2026 19:46
@tamirms
tamirms force-pushed the tamirms/events-windowed-matches branch 5 times, most recently from e9be794 to 133ce8b Compare October 24, 2026 02:49
@tamirms
tamirms removed this pull request from stack #1102 October 27, 2026 10:19
@tamirms
tamirms added this pull request to stack #1157 October 27, 2026 10:21
@tamirms tamirms moved this from To Do to In Progress in Platform Scrum Oct 27, 2026
LookupKeys now takes the range of event ids the query needs and reports
the range its answer covers. Matches first asks for the leading four slabs
of the window, in the walk's direction. It asks for the rest only if the
page is still not full and the first answer did not already cover it.

Both readers still return whole terms, so they report the whole id space
and every query runs in one stage, as before. The point is the next step:
a reader that stores a large term in pieces can then read only the pieces
a page reaches.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QHnF5BhsuoxpWxGmatQzmt
Comments only, apart from two test-file moves: everyID now sits above
wholeChunk's doc comment instead of inside it, and the candidate-free-slab
fixture, which has one caller, is back inline in its test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@tamirms
tamirms marked this pull request as ready for review October 29, 2026 16:02
@tamirms
tamirms requested a balanced review from Copilot October 29, 2026 16:02

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The regression test stays within the first stage, so its one-lookup assertion cannot detect whether Matches ignores whole-term coverage.

1 open issue
What changed in this PR

Prepares event queries for partially stored term bitmaps by adding lookup windows and coverage-aware, two-stage matching. Current hot and cold readers still return whole terms, preserving single-stage execution and existing results and I/O.

Changes:

  • Extend Reader.LookupKeys to accept a requested window and report its covered range.
  • Match the leading four slabs first, requesting the remainder only when needed.
  • Update reader implementations, test doubles, and tests for coverage boundaries and batch hints.
File Description
cmd/​stellar-rpc/​internal/​rpcv2/​stores/​hotchunk/​hotchunk_test.go Update lookups to pass whole-term windows.
cmd/​stellar-rpc/​internal/​rpcv2/​stores/​event/​slab_match.go Scope slab plans to stages and count emitted matches.
cmd/​stellar-rpc/​internal/​rpcv2/​stores/​event/​slab_match_test.go Add coverage-contract and stage-scheduling tests.
cmd/​stellar-rpc/​internal/​rpcv2/​stores/​event/​reader.go Define windowed lookup and coverage contracts.
cmd/​stellar-rpc/​internal/​rpcv2/​stores/​event/​matches_differential_test.go Adapt the differential reader to exercise staged matching.
cmd/​stellar-rpc/​internal/​rpcv2/​stores/​event/​match.go Implement coverage-aware stages and carry batch hints forward.
cmd/​stellar-rpc/​internal/​rpcv2/​stores/​event/​match_test.go Test batch hints across stage boundaries.
cmd/​stellar-rpc/​internal/​rpcv2/​stores/​event/​hot_store.go Adopt the lookup signature while retaining whole-term snapshots.
cmd/​stellar-rpc/​internal/​rpcv2/​stores/​event/​hot_store_test.go Update calls and assert whole-space coverage.
cmd/​stellar-rpc/​internal/​rpcv2/​stores/​event/​cold_reader.go Report whole-space coverage without changing index reads.
cmd/​stellar-rpc/​internal/​rpcv2/​stores/​event/​cold_reader_test.go Update calls and verify reported coverage.
cmd/​stellar-rpc/​internal/​rpcv2/​query/​event_page_test.go Adapt the paging test reader to windowed lookups.
cmd/​stellar-rpc/​internal/​rpcv2/​ingest/​ingest_test.go Update ingest readback tests for the lookup signature.

Review effort: Balanced

// TestMatches_WholeTermLookupIsOneStage pins that a lookup covering every id
// makes one stage, without moving the stream.
func TestMatches_WholeTermLookupIsOneStage(t *testing.T) {
f := newShapedFixture(t)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: In Progress

Development

Successfully merging this pull request may close these issues.

2 participants