Skip to content

fix(kafka): read past a tombstone instead of stopping at it - #309

Merged
andrevanzuydam merged 2 commits into
v3from
fix/kafka-tombstone-skip-not-stop
Nov 13, 2026
Merged

andrevanzuydam merged 2 commits into
v3from
fix/kafka-tombstone-skip-not-stop

Conversation

@M1gael

@M1gael M1gael commented Nov 10, 2026 •

Copy link
Copy Markdown
Collaborator

Follow-up to #304 for #303. #304 stops the crash, but answers a tombstone with None. A single-pass drain (consume(poll_interval=0), e.g. a cron consumer) reads that as an empty queue, so it never gets past the tombstone: run 1 gets the records before it, and every later run gets nothing. The raw-socket client (no confluent-kafka installed) never moved past a tombstone at all.

Fix: both clients skip tombstones and keep reading within the same dequeue, and still stop at its deadline (1 s steady state, the assign timeout on first subscribe). Once past a tombstone, the raw client caps the broker's long-poll wait to the time left. It stops rather than spins if the offset would not advance, and steps over an empty value like a null one. Nothing is committed on the tombstone path. On confluent, acknowledging the next record commits past the skipped tombstones, so a restart gets past them; the raw client keeps its offset in memory only, as before, so a restart reads from the start and skips them again. Acknowledge, retry and dead-letter are unchanged.

Limits: a run of tombstones longer than one dequeue can scan (300,000 in a row on confluent; not measured on the raw client) can still end a single-pass drain early, and a fresh process starts again at the same offset. A long-running worker gets through. The raw client still decodes only the first record of a fetched batch (a separate issue): with a producer that batches records it redelivers a batch's first record, or stops on it if that record is a tombstone, as on v3. Confluent's handling of an empty value is unchanged.

Tests (tests/test_queue_kafka_tombstone.py, live broker, 12 tests): 7 fail on v3; the crash tests (fixed by #304), the no-spin test and the stream-deadline tests pass there, because it stops at the first tombstone. Kafka and queue suites match v3.

Please squash-merge: the second commit's message says acknowledging the next record commits past the skipped tombstones without limiting it to confluent; the raw client commits nothing, as above.

Skipping a tombstone by returning None made a single-pass drain
(consume(poll_interval=0)) treat it as an empty queue: the records after
it were never reached, and a fresh process stopped at the same offset
every run. The raw-socket client (no confluent-kafka) never moved its
offset past a tombstone at all and returned None for ever.

Both clients now skip the tombstone and keep reading within the same
dequeue. The raw client stops, rather than spins, if the offset would not
advance.

Tests: single-pass drain past a tombstone, raw client past a tombstone,
and a capped no-spin case.

Signed-off-by: Michael <michael@codeinfinity.co.za>
@M1gael

M1gael commented Nov 10, 2026

Copy link
Copy Markdown
Collaborator Author

I have read the Tina4 Contributor Licence Agreement and I agree to it.

@M1gael

M1gael commented Nov 10, 2026

Copy link
Copy Markdown
Collaborator Author

Withdrawn for rework; a corrected PR will follow.

@M1gael M1gael closed this Nov 10, 2026
@M1gael
M1gael deleted the fix/kafka-tombstone-skip-not-stop branch November 10, 2026 13:38
@M1gael
M1gael restored the fix/kafka-tombstone-skip-not-stop branch November 10, 2026 13:39
Skipping a tombstone went straight back to poll and never reached the
deadline check, so a steady stream of tombstones held one dequeue() for
as long as the stream lasted (8 s for an 8 s stream). Both clients now
stop at the deadline while skipping.

The raw client also holds the broker's long-poll wait to the time left
once it is past a tombstone, so an empty topic behind tombstones no
longer costs the full 5 s, and it steps over an empty value as it does a
null one.

Nothing is committed on the tombstone path. Acknowledging the next record
commits past the skipped tombstones, so a restart gets past them.

Tests: the deadline under a tombstone stream (both clients); the raw
deadline at the end of the topic and through a backlog; a record arriving
during the capped wait is still delivered; the raw client steps over an
empty value; the record key after a skip; a restart gets past a tombstone
through the normal acknowledge.

Signed-off-by: Michael <michael@codeinfinity.co.za>
@M1gael M1gael reopened this Nov 11, 2026
@M1gael

M1gael commented Nov 11, 2026

Copy link
Copy Markdown
Collaborator Author

Reworked after review and reopened.

@M1gael
M1gael marked this pull request as ready for review November 11, 2026 09:54
@andrevanzuydam
andrevanzuydam merged commit 4b159b8 into v3 Nov 13, 2026
14 checks passed
@andrevanzuydam
andrevanzuydam deleted the fix/kafka-tombstone-skip-not-stop branch November 13, 2026 09:07
@andrevanzuydam andrevanzuydam mentioned this pull request Nov 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants