Skip to content

Kafka queue consumer crashes on tombstone (null-value) records — uncaught AttributeError kills the consumer #303

Closed

Description

@SAB13711

Title: Kafka queue consumer crashes on tombstone (null-value) records — uncaught AttributeError kills the consumer

Version: tina4-python 3.13.160

Summary

A Kafka consumer permanently dies when it reaches a tombstone — a record whose
value is null. These are normal on Kafka: Debezium emits them for deletes, and
log-compacted topics use them as delete markers.

In tina4_python/queue_backends/kafka_backend.py, _dequeue() decodes the record like this:

# ~line 174 (confluent path)
try:
    message = json.loads(msg.value().decode("utf-8"))
except (json.JSONDecodeError, UnicodeDecodeError):
    return None, None

For a tombstone, msg.value() is None, so None.decode("utf-8") raises
AttributeError: 'NoneType' object has no attribute 'decode'. That exception is
not in the except, so it propagates out of the consume() generator and the
consumer thread exits.

Because the tombstone sits at the consumer group's committed offset, it's a
poison pill: every restart re-reads the same offset and crashes again, so the
topic silently stops consuming. (In our case a Debezium CDC topic went unconsumed
for days before we traced it.)

The same pattern exists in the protocol-fetch path

# ~line 686 (_send_fetch, non-confluent path)
value, msg_offset = decoded
try:
    message = json.loads(value.decode("utf-8"))
except (json.JSONDecodeError, UnicodeDecodeError):
    return None

If value can be None for a null-value record, it fails the same way.

Steps to reproduce

  1. Point a Tina4 Kafka Queue consumer at a topic that contains a null-value
    record (any Debezium CDC topic after a delete, or any compacted topic with a
    delete marker).
  2. Consume. When the consumer reaches the tombstone it raises
    AttributeError: 'NoneType' object has no attribute 'decode' and the
    consume() generator ends.

Expected

A null-value record should not crash the consumer. It should be skipped (and its
offset allowed to advance) — or surfaced to the caller as a tombstone/None-payload
job so application code can handle deletes explicitly.

Suggested fix

Guard the null value before decoding, at both decode sites, e.g.:

raw = msg.value()
if raw is None:
    # Tombstone / delete marker: nothing to decode. Skip it; the next record's
    # commit advances the offset past it.
    return None, None
try:
    message = json.loads(raw.decode("utf-8"))
except (json.JSONDecodeError, UnicodeDecodeError):
    return None, None

An explicit is None check is preferable to simply adding AttributeError to the
except, since it won't mask unrelated attribute errors. If you'd rather expose
deletes to consumers, yielding a job with a None/empty payload (and committing it)
would also work — then for job in queue.consume(topic) can act on tombstones.

Impact

High for anyone consuming Debezium CDC or compacted topics: a single tombstone at
the committed offset permanently stops that consumer, and it can't recover across
restarts without manually moving the offset.

Activity

  1. M1gael commented on Nov 10, 2026

    @M1gael
    Collaborator

    Landed on v3 @ 39074d4a (#304), merged: the confluent-kafka consumer now skips a tombstone instead of crashing.

  2. andrevanzuydam commented on Nov 13, 2026

    @andrevanzuydam
    Collaborator

    Fixed and released.

    Each fix is verified against a live broker producing real tombstones. Release notes: https://tina4.com/python/36-releases/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions