[zannis]

Problem

A Kafka consumer accepted topologies it could not honour and failed silently at runtime:

• External topologies with a DLQ. declare treated the DLQ as shove-owned. It created the DLQ, and expanded an existing one to 8 partitions, even when infra owns it (including a dlq_named pointing at an infra topic). An expansion cannot be undone, and it remaps keys to partitions.
• Only the consumer registry verified anything. The direct run, supervisor members, run_batch, run_fifo, broadcast subscriptions and run_dlq never declare.
  • A missing external topic mapped UnknownTopicOrPartition to a retryable Connection, and the reconnect wrapper looped on it forever.
  • A missing DLQ failed each dead-letter publish, and the record was committed as a lost discard.
• Producer auto-create. With with_producer_auto_create_topics(true), a dead-letter publish after an external DLQ disappeared would recreate the infra-owned topic with broker defaults.

Change

• declare on an external topology: the DLQ is verify-only. A missing one is ShoveError::Topology ("must be provisioned"); it is never created or expanded. Owned topologies are unchanged.

• Startup check on every non-registry consumer path: a metadata-only probe that runs once, before the reconnect loop, so no record is consumed first. It checks:

  • the main topic, when the topology is external;
  • the DLQ, wherever the path publishes dead letters (owned or external). A missing owned DLQ is skipped when the producer auto-creates topics, since the first dead-letter publish creates it;
  • on a dead-letter drain of an external topology, both its topics.

  A missing topic is ShoveError::Topology naming it. An unreachable broker or a metadata timeout is not: it stays Connection and is retried under the consumer's backoff and max_reconnect_attempts, and shutdown ends it. Owned main topics are never probed, because another process may still declare them. Registry members skip the probe, since register already declared the topology.

• Auto-create refusal: an external topology with a DLQ is refused at register and at every consumer start that publishes dead letters when the client's producer auto-creates topics. A publish-only declare is not refused.

• The probe builds no producer. It is a metadata request with allow.auto.create.topics=false, so it creates nothing and needs only Describe. An external topology without a DLQ still never requests a producer id.

Behaviour changes for existing users

Each of these turns a silent runtime failure into a startup error:

• An external topology's DLQ must already exist; shove no longer creates or expands it.
• An owned topology with a DLQ, consumed without declare and without producer auto-create, now fails at startup if the DLQ is missing. Before, its dead letters were lost.
• An external topology with a DLQ on a producer-auto-create client is refused at consumer startup.
• With max_reconnect_attempts set, an unreachable broker takes up to the metadata timeout (about 10 s) per attempt to fail at startup, instead of failing immediately.

No public API changes. The only new item is a test-support-gated, doc-hidden probe counter.

Tests

Every refusal test below was red on main and is green now.

• tests/kafka_startup_topic_checks_mock.rs, new, on librdkafka's mock cluster with no Docker. It covers each entry point refusing a missing external topic or DLQ, the auto-create refusal on direct run, batch and register, and the dead-letter drain checking both external topics. It also covers the cases that must keep starting:

  • an owned main topic;
  • a present DLQ;
  • auto-create on with an owned DLQ;
  • auto-create on with an external topology that has no DLQ;
  • a publish-only declare.

  It adds an unreachable-broker guard (not Topology), and checks that registry members make no probe.

• tests/kafka_integration.rs, real Kafka:

  • an external DLQ is never expanded at declare;
  • a missing external DLQ fails declare and is not created;
  • a direct run on a missing external topic fails and creates neither topic, on a broker with auto.create.topics.enable=true.

• tests/kafka_consume_only_mock.rs: the startup check builds no producer.

• Fifteen existing tests now provision their DLQ before consuming without declare.

Ran locally:

• cargo nextest run with the mock, Kafka integration, broadcast, rebalance, batch, auto-create, offset-reset and metrics suites;
• --lib and --no-default-features;
• cargo fmt -- --check;
• cargo clippy -D warnings on all features, no default features, and the Kafka --lib feature sets.

The ssl/gssapi/msk-iam feature set does not link locally; CI covers it.

[cloudflare-workers-and-pages]

Deploying with    Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status	Name	Latest Commit	Preview URL	Updated (UTC)
✅ Deployment successful! View logs	shove-docs	25c4974	Commit Preview URL Branch Preview URL	Oct 10 2026, 04:43 AM

[codecov]

Codecov Report

❌ Patch coverage is 96.13095% with 13 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
src/backends/kafka/client.rs	81.57%	7 Missing ⚠️
src/backends/kafka/startup.rs	97.19%	5 Missing ⚠️
src/backends/kafka/consumer.rs	99.02%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!