Repository navigation
Support 413 handling - #48548
Support 413 handling#48548Radhika Gupta (rads-1996) wants to merge 18 commits into
Conversation
|
Azure Pipelines: Successfully started running 1 pipeline(s). 9 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
d675163 to
f1ae240
Compare
There was a problem hiding this comment.
Pull request overview
Adds HTTP 413 handling to the Azure Monitor OpenTelemetry exporter.
Changes:
- Splits oversized batches and persists sub-batches for retry.
- Avoids logging oversized telemetry payloads.
- Adds tests, a reproduction sample, and release notes.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 5 comments.
| File | Description |
|---|---|
export/_base.py |
Implements payload sizing, splitting, persistence, and safer logging. |
tests/test_base_exporter.py |
Tests 413 handling and payload splitting. |
sample_test_413.py |
Adds a manual 413 reproduction script. |
CHANGELOG.md |
Documents the feature. |
💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.
8a54ccb to
3d17a68
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.
Suppressed comments (3)
sdk/monitor/azure-monitor-opentelemetry-exporter/azure/monitor/opentelemetry/exporter/export/_base.py:346
- The 413 contract requires splitting and resending down to individual items, but this branch drops the entire batch whenever offline storage is disabled. Users who opt out of persistence therefore receive none of the spec-aligned 413 handling described by this PR. Split/retry in memory regardless of storage availability, using persistence only as a fallback.
if not self.storage:
if not self._is_stats_exporter():
logger.debug(
"Payload too large but offline storage is disabled; dropping %d envelope(s): %s.",
len(envelopes),
response_error.message,
)
if self._should_collect_customer_sdkstats():
track_dropped_items(envelopes, DropCode.CLIENT_STORAGE_DISABLED)
return ExportResult.FAILED_NOT_RETRYABLE
sdk/monitor/azure-monitor-opentelemetry-exporter/CHANGELOG.md:22
- This feature is introduced in the unreleased
1.0.0b57section above, but these duplicate notes place it in the already released1.0.0b56history (2026-08-06), incorrectly implying that published version contains the new behavior. Remove this entry from the historical release.
- Handle `413 Payload Too Large` responses by splitting the batch and persisting each half to
local storage for retry at a smaller size, instead of dropping the entire batch. A single
envelope that is too large to split is dropped.
sdk/monitor/azure-monitor-opentelemetry-exporter/azure/monitor/opentelemetry/exporter/export/_base.py:359
- When a 413 occurs while draining storage, the original blob remains on disk until
_transmitreturns (_transmit_from_storagelines 294–302), while this line writes the split copies immediately. With the default 50 MiB cap, a large original can consume enough capacity that only the first split is persisted; the later split is dropped, and then the original is deleted, permanently losing telemetry. Replace the original without temporarily duplicating its contents (or resend the splits directly), and cover a 413 encountered during storage drain.
result_from_storage_put = self.storage.put([x.as_dict() for x in sub_batch])
f64b806 to
a679ce6
Compare
|
Azure Pipelines: Successfully started running 1 pipeline(s). 9 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.
Suppressed comments (1)
sdk/monitor/azure-monitor-opentelemetry-exporter/azure/monitor/opentelemetry/exporter/export/_base.py:333
- Disabling offline storage currently disables 413 recovery entirely: a batch of multiple individually valid envelopes is dropped here without ever being split. Offline persistence should only control whether failed retries are cached, not whether the request can be retried at a smaller size. Split and retransmit the sub-batches in memory when
self.storageis absent (while retaining the single-envelope termination case), and persist only sub-batches that still need a later retry.
if not self.storage:
if not self._is_stats_exporter():
logger.debug(
"Payload too large but offline storage is disabled; dropping %d envelope(s): %s.",
len(envelopes),
response_error.message,
)
return ExportResult.FAILED_NOT_RETRYABLE
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.
Suppressed comments (1)
sdk/monitor/azure-monitor-opentelemetry-exporter/azure/monitor/opentelemetry/exporter/export/_base.py:329
- 413 recovery is incorrectly gated on optional offline storage. With
disable_offline_storage=True, a multi-item batch is dropped without ever being split; when storage exists, the split blobs are only drained after a later live export succeeds, so workloads whose live batches keep receiving 413 will only fill storage and eventually lose telemetry. Split and transmit the smaller batches immediately, and use offline storage only for sub-batches that remain eligible for a later retry.
# Handle a 413 (Payload Too Large) response by splitting and persisting for retry.
if not self.storage:
if not self._is_stats_exporter():
logger.debug(
"Payload too large but offline storage is disabled; dropping %d envelope(s): %s.",
6fbd296 to
e84fa4b
Compare
There was a problem hiding this comment.
🔵 Needs a closer look
Splitting a stored near-capacity blob can discard both its children and original copy.
Review details
Suppressed comments (1)
sdk/monitor/azure-monitor-opentelemetry-exporter/azure/monitor/opentelemetry/exporter/export/_base.py:346
- When this runs while draining a stored blob, the original leased blob still counts toward
storage_max_sizeuntil_transmit_from_storagedeletes it. Persisting copies of its split contents can therefore hit capacity, drop every child, and then delete the only original copy. Replace the source blob atomically or retry the children directly so the original remains recoverable until all children are retained.
for sub_batch in _split_oversized_batch(envelopes):
result_from_storage_put = self.storage.put([x.as_dict() for x in sub_batch])
- Files reviewed: 7/7 changed files
- Comments generated: 0 new
- Review effort level: Balanced
e84fa4b to
08c5c92
Compare
There was a problem hiding this comment.
🟡 Changes recommended
Permanent persistence failures need visible operational logging, and the release note should match the implemented splitting behavior.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
Suppressed comments (1)
sdk/monitor/azure-monitor-opentelemetry-exporter/azure/monitor/opentelemetry/exporter/export/_base.py:352
- A failed
storage.putpermanently drops this sub-batch, but this message is emitted only at DEBUG and is therefore hidden under normal production logging. Emit it at WARNING, as the new test's contract describes and as the neighboring permanent-drop paths do, so operators can detect telemetry loss.
logger.debug(
- Files reviewed: 7/7 changed files
- Comments generated: 1
- Review effort level: Balanced
There was a problem hiding this comment.
🔵 Needs a closer look
Stored-batch replacement can lose telemetry near capacity, and persistence failures are hidden at DEBUG level.
Review details
Suppressed comments (2)
sdk/monitor/azure-monitor-opentelemetry-exporter/azure/monitor/opentelemetry/exporter/export/_base.py:346
- When a stored blob is retried and receives another 413, the original blob is still on disk while these replacement sub-batches are written; it is deleted only after
_transmitreturns. BecauseLocalFileStorage.put()rejects writes based on the current directory size, a near-capacity store can reject and drop the children even though replacing the parent would require no additional steady-state space. Replace the leased parent atomically (or otherwise exclude its size/reserve replacement capacity) before counting a child persistence failure as data loss.
for sub_batch in _split_oversized_batch(envelopes):
result_from_storage_put = self.storage.put([x.as_dict() for x in sub_batch])
sdk/monitor/azure-monitor-opentelemetry-exporter/azure/monitor/opentelemetry/exporter/export/_base.py:352
- This branch permanently drops the sub-batch, but the message is emitted at DEBUG even though the new test and docstring describe it as a warning. At normal logging levels this data loss is invisible; emit it at WARNING as intended.
logger.debug(
- Files reviewed: 7/7 changed files
- Comments generated: 0 new
- Review effort level: Balanced
Description
Add support to handle the 413 payload too large status code, in alignment with spec.
All SDK Contribution checklist:
General Guidelines and Best Practices
Testing Guidelines