Repository navigation
Conversation
Switch KafkaProducer from arroyo's AsyncKafkaProducer to ReloadingKafkaProducer. Sends still wait for delivery, using the new produce_record_with_delivery. Config still comes from Taskbroker's own config; loading the shared configmap is tracked separately. Points sentry_arroyo at the getsentry/arroyo#591 branch until that is released. ref STREAM-1964
Keeps send() as it was; only the constructor changes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Switches Taskbroker's
KafkaProducerfrom arroyo'sAsyncKafkaProducertoReloadingKafkaProducer. Only the constructor changes:ReloadingKafkaProducerimplements the sameAsyncProducertrait (getsentry/arroyo#591), sosend()still waits for the broker to confirm delivery. Upkeep relies on that to only mark retries and deadletters as done once they are in Kafka.Config still comes from Taskbroker's own config through
from_config. Loading the shared configmap is STREAM-1967.ignore_key_ordering: true: Taskbroker sends without keys, so a reload swaps clients right away instead of holding back sends.KafkaProducer::newtakes a topic, which only selects the producer's entry in the shared config: the deadletter topic for the upkeep and batcher producers, the demoted topic for upkeep's forwarding producer. Producers still send to other topics on the same cluster (retry, demoted). Moving to one producer per topic is left for later; Taskbroker builds its own Kafka config for now, so it already knows every topic's cluster.Draft until arroyo#591 is released.
sentry_arroyopoints at that branch for now.ref STREAM-1964