feat: retry transient item failures in bulk ingestion - #4243
Open
falscherwiener1-svg wants to merge 3 commits into
Open
feat: retry transient item failures in bulk ingestion#4243falscherwiener1-svg wants to merge 3 commits into
falscherwiener1-svg wants to merge 3 commits into
Conversation
Add per-item retry logic for transient HTTP status codes (429, 503) in the bulk ingestion sink. Previously, individual items that failed with transient errors inside a successful bulk response were sent directly to the ErrorMonitor without retry, which could trigger a premature TooManyErrors abort on a temporarily overloaded cluster. Changes: - Add TRANSIENT_STATUS_CODES constant (429, 503) - Add _determine_action() helper to extract the bulk action key from a response item - Add _extract_transient_failed_operations() to collect retryable items from a bulk response - Add _retry_transient_item_failures() with exponential backoff that retries only the failed items - Rewire _batch_bulk to call _retry_transient_item_failures before _process_bulk_response, so the ErrorMonitor only tracks failures after all retry attempts are exhausted - Remove the now-unused retryable import and the dead _bulk_api_call function that was never invoked Closes elastic#4002
|
💚 CLA has been signed |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Add per-item retry logic for transient HTTP status codes (429, 503)
in the bulk ingestion sink. Previously, individual items that failed
with transient errors inside a successful bulk response were sent
directly to the ErrorMonitor without retry, which could trigger a
premature TooManyErrors abort on a temporarily overloaded cluster.
Changes
TRANSIENT_STATUS_CODESconstant (429, 503)_determine_action()helper to extract the bulk action key from a response item_extract_transient_failed_operations()to collect retryable items from a bulk response_retry_transient_item_failures()with exponential backoff that retries only the failed items_batch_bulkto call_retry_transient_item_failuresbefore_process_bulk_response, so the ErrorMonitor only tracks failures after all retry attempts are exhaustedretryableimport and the dead_bulk_api_callfunction that was never invokedTesting
test_extract_transient_failed_operations_returns_empty_when_all_successfultest_extract_transient_failed_operations_returns_429_itemstest_retry_transient_item_failures_retries_429_itemstest_retry_transient_item_failures_gives_up_after_exhaustiontest_batch_bulk_with_transient_item_retries_integrationCloses
Closes #4002