Repository navigation
fix(kafka): read past a tombstone instead of stopping at it - #309
Merged
Merged
Conversation
Skipping a tombstone by returning None made a single-pass drain (consume(poll_interval=0)) treat it as an empty queue: the records after it were never reached, and a fresh process stopped at the same offset every run. The raw-socket client (no confluent-kafka) never moved its offset past a tombstone at all and returned None for ever. Both clients now skip the tombstone and keep reading within the same dequeue. The raw client stops, rather than spins, if the offset would not advance. Tests: single-pass drain past a tombstone, raw client past a tombstone, and a capped no-spin case. Signed-off-by: Michael <michael@codeinfinity.co.za>
Collaborator
Author
|
I have read the Tina4 Contributor Licence Agreement and I agree to it. |
Collaborator
Author
|
Withdrawn for rework; a corrected PR will follow. |
Skipping a tombstone went straight back to poll and never reached the deadline check, so a steady stream of tombstones held one dequeue() for as long as the stream lasted (8 s for an 8 s stream). Both clients now stop at the deadline while skipping. The raw client also holds the broker's long-poll wait to the time left once it is past a tombstone, so an empty topic behind tombstones no longer costs the full 5 s, and it steps over an empty value as it does a null one. Nothing is committed on the tombstone path. Acknowledging the next record commits past the skipped tombstones, so a restart gets past them. Tests: the deadline under a tombstone stream (both clients); the raw deadline at the end of the topic and through a backlog; a record arriving during the capped wait is still delivered; the raw client steps over an empty value; the record key after a skip; a restart gets past a tombstone through the normal acknowledge. Signed-off-by: Michael <michael@codeinfinity.co.za>
Collaborator
Author
|
Reworked after review and reopened. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #304 for #303. #304 stops the crash, but answers a tombstone with
None. A single-pass drain (consume(poll_interval=0), e.g. a cron consumer) reads that as an empty queue, so it never gets past the tombstone: run 1 gets the records before it, and every later run gets nothing. The raw-socket client (no confluent-kafka installed) never moved past a tombstone at all.Fix: both clients skip tombstones and keep reading within the same dequeue, and still stop at its deadline (1 s steady state, the assign timeout on first subscribe). Once past a tombstone, the raw client caps the broker's long-poll wait to the time left. It stops rather than spins if the offset would not advance, and steps over an empty value like a null one. Nothing is committed on the tombstone path. On confluent, acknowledging the next record commits past the skipped tombstones, so a restart gets past them; the raw client keeps its offset in memory only, as before, so a restart reads from the start and skips them again. Acknowledge, retry and dead-letter are unchanged.
Limits: a run of tombstones longer than one dequeue can scan (300,000 in a row on confluent; not measured on the raw client) can still end a single-pass drain early, and a fresh process starts again at the same offset. A long-running worker gets through. The raw client still decodes only the first record of a fetched batch (a separate issue): with a producer that batches records it redelivers a batch's first record, or stops on it if that record is a tombstone, as on v3. Confluent's handling of an empty value is unchanged.
Tests (
tests/test_queue_kafka_tombstone.py, live broker, 12 tests): 7 fail on v3; the crash tests (fixed by #304), the no-spin test and the stream-deadline tests pass there, because it stops at the first tombstone. Kafka and queue suites match v3.Please squash-merge: the second commit's message says acknowledging the next record commits past the skipped tombstones without limiting it to confluent; the raw client commits nothing, as above.