Skip to content

Support 413 handling - #48548

Open
Radhika Gupta (rads-1996) wants to merge 18 commits into
Azure:mainfrom
rads-1996:support-413-handling
Open

Radhika Gupta (rads-1996) wants to merge 18 commits into
Azure:mainfrom
rads-1996:support-413-handling

Conversation

@rads-1996

@rads-1996 Radhika Gupta (rads-1996) commented Aug 12, 2026 •

Copy link
Copy Markdown
Member

Description

Add support to handle the 413 payload too large status code, in alignment with spec.

All SDK Contribution checklist:

  • The pull request does not introduce [breaking changes]
  • CHANGELOG is updated for new features, bug fixes or other significant changes.
  • I have read the contribution guidelines.

General Guidelines and Best Practices

  • Title of the pull request is clear and informative.
  • There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, see this page.

Testing Guidelines

  • Pull request includes test coverage for the included changes.

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).
9 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

@github-actions github-actions Bot added the Monitor - Exporter Monitor OpenTelemetry Exporter label Aug 12, 2026
@rads-1996 Radhika Gupta (rads-1996) changed the title Support 413 handling (WIP) Support 413 handling Aug 12, 2026
@rads-1996
Radhika Gupta (rads-1996) requested a balanced review from Copilot August 12, 2026 17:31

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds HTTP 413 handling to the Azure Monitor OpenTelemetry exporter.

Changes:

  • Splits oversized batches and persists sub-batches for retry.
  • Avoids logging oversized telemetry payloads.
  • Adds tests, a reproduction sample, and release notes.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 5 comments.

File Description
export/_base.py Implements payload sizing, splitting, persistence, and safer logging.
tests/test_base_exporter.py Tests 413 handling and payload splitting.
sample_test_413.py Adds a manual 413 reproduction script.
CHANGELOG.md Documents the feature.

💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.

Comment thread sdk/monitor/azure-monitor-opentelemetry-exporter/CHANGELOG.md Outdated
Comment thread sdk/monitor/azure-monitor-opentelemetry-exporter/sample_test_413.py Outdated

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

Suppressed comments (3)

sdk/monitor/azure-monitor-opentelemetry-exporter/azure/monitor/opentelemetry/exporter/export/_base.py:346

  • The 413 contract requires splitting and resending down to individual items, but this branch drops the entire batch whenever offline storage is disabled. Users who opt out of persistence therefore receive none of the spec-aligned 413 handling described by this PR. Split/retry in memory regardless of storage availability, using persistence only as a fallback.
        if not self.storage:
            if not self._is_stats_exporter():
                logger.debug(
                    "Payload too large but offline storage is disabled; dropping %d envelope(s): %s.",
                    len(envelopes),
                    response_error.message,
                )
                if self._should_collect_customer_sdkstats():
                    track_dropped_items(envelopes, DropCode.CLIENT_STORAGE_DISABLED)
            return ExportResult.FAILED_NOT_RETRYABLE

sdk/monitor/azure-monitor-opentelemetry-exporter/CHANGELOG.md:22

  • This feature is introduced in the unreleased 1.0.0b57 section above, but these duplicate notes place it in the already released 1.0.0b56 history (2026-08-06), incorrectly implying that published version contains the new behavior. Remove this entry from the historical release.
- Handle `413 Payload Too Large` responses by splitting the batch and persisting each half to
  local storage for retry at a smaller size, instead of dropping the entire batch. A single
  envelope that is too large to split is dropped.

sdk/monitor/azure-monitor-opentelemetry-exporter/azure/monitor/opentelemetry/exporter/export/_base.py:359

  • When a 413 occurs while draining storage, the original blob remains on disk until _transmit returns (_transmit_from_storage lines 294–302), while this line writes the split copies immediately. With the default 50 MiB cap, a large original can consume enough capacity that only the first split is persisted; the later split is dropped, and then the original is deleted, permanently losing telemetry. Replace the original without temporarily duplicating its contents (or resend the splits directly), and cover a 413 encountered during storage drain.
            result_from_storage_put = self.storage.put([x.as_dict() for x in sub_batch])

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).
9 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.

Suppressed comments (1)

sdk/monitor/azure-monitor-opentelemetry-exporter/azure/monitor/opentelemetry/exporter/export/_base.py:333

  • Disabling offline storage currently disables 413 recovery entirely: a batch of multiple individually valid envelopes is dropped here without ever being split. Offline persistence should only control whether failed retries are cached, not whether the request can be retried at a smaller size. Split and retransmit the sub-batches in memory when self.storage is absent (while retaining the single-envelope termination case), and persist only sub-batches that still need a later retry.
        if not self.storage:
            if not self._is_stats_exporter():
                logger.debug(
                    "Payload too large but offline storage is disabled; dropping %d envelope(s): %s.",
                    len(envelopes),
                    response_error.message,
                )
            return ExportResult.FAILED_NOT_RETRYABLE

@rads-1996 Radhika Gupta (rads-1996) changed the title (WIP) Support 413 handling Support 413 handling Aug 21, 2026
Copilot AI review requested due to automatic review settings August 21, 2026 19:14

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.

Suppressed comments (1)

sdk/monitor/azure-monitor-opentelemetry-exporter/azure/monitor/opentelemetry/exporter/export/_base.py:329

  • 413 recovery is incorrectly gated on optional offline storage. With disable_offline_storage=True, a multi-item batch is dropped without ever being split; when storage exists, the split blobs are only drained after a later live export succeeds, so workloads whose live batches keep receiving 413 will only fill storage and eventually lose telemetry. Split and transmit the smaller batches immediately, and use offline storage only for sub-batches that remain eligible for a later retry.
        # Handle a 413 (Payload Too Large) response by splitting and persisting for retry.
        if not self.storage:
            if not self._is_stats_exporter():
                logger.debug(
                    "Payload too large but offline storage is disabled; dropping %d envelope(s): %s.",

Copilot AI review requested due to automatic review settings August 21, 2026 19:34
Copilot AI review requested due to automatic review settings September 10, 2026 20:07

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The implementation handles storage outcomes and edge cases with comprehensive focused tests.

Review details
  • Files reviewed: 7/7 changed files
  • Comments generated: 0 new
  • Review effort level: Balanced

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Splitting a stored near-capacity blob can discard both its children and original copy.

Review details

Suppressed comments (1)

sdk/monitor/azure-monitor-opentelemetry-exporter/azure/monitor/opentelemetry/exporter/export/_base.py:346

  • When this runs while draining a stored blob, the original leased blob still counts toward storage_max_size until _transmit_from_storage deletes it. Persisting copies of its split contents can therefore hit capacity, drop every child, and then delete the only original copy. Replace the source blob atomically or retry the children directly so the original remains recoverable until all children are retained.
        for sub_batch in _split_oversized_batch(envelopes):
            result_from_storage_put = self.storage.put([x.as_dict() for x in sub_batch])
  • Files reviewed: 7/7 changed files
  • Comments generated: 0 new
  • Review effort level: Balanced

Copilot AI review requested due to automatic review settings September 11, 2026 16:39

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Permanent persistence failures need visible operational logging, and the release note should match the implemented splitting behavior.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (1)

sdk/monitor/azure-monitor-opentelemetry-exporter/azure/monitor/opentelemetry/exporter/export/_base.py:352

  • A failed storage.put permanently drops this sub-batch, but this message is emitted only at DEBUG and is therefore hidden under normal production logging. Emit it at WARNING, as the new test's contract describes and as the neighboring permanent-drop paths do, so operators can detect telemetry loss.
                    logger.debug(
  • Files reviewed: 7/7 changed files
  • Comments generated: 1
  • Review effort level: Balanced

Comment thread sdk/monitor/azure-monitor-opentelemetry-exporter/CHANGELOG.md Outdated

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Stored-batch replacement can lose telemetry near capacity, and persistence failures are hidden at DEBUG level.

Review details

Suppressed comments (2)

sdk/monitor/azure-monitor-opentelemetry-exporter/azure/monitor/opentelemetry/exporter/export/_base.py:346

  • When a stored blob is retried and receives another 413, the original blob is still on disk while these replacement sub-batches are written; it is deleted only after _transmit returns. Because LocalFileStorage.put() rejects writes based on the current directory size, a near-capacity store can reject and drop the children even though replacing the parent would require no additional steady-state space. Replace the leased parent atomically (or otherwise exclude its size/reserve replacement capacity) before counting a child persistence failure as data loss.
        for sub_batch in _split_oversized_batch(envelopes):
            result_from_storage_put = self.storage.put([x.as_dict() for x in sub_batch])

sdk/monitor/azure-monitor-opentelemetry-exporter/azure/monitor/opentelemetry/exporter/export/_base.py:352

  • This branch permanently drops the sub-batch, but the message is emitted at DEBUG even though the new test and docstring describe it as a warning. At normal logging levels this data loss is invisible; emit it at WARNING as intended.
                    logger.debug(
  • Files reviewed: 7/7 changed files
  • Comments generated: 0 new
  • Review effort level: Balanced

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Persistence, retry, and concurrent storage-draining behavior warrants final human validation.

Review details
  • Files reviewed: 7/7 changed files
  • Comments generated: 1
  • Review effort level: Balanced

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Monitor - Exporter Monitor OpenTelemetry Exporter

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants