Skip to content

[Prometheus Remote Write] Update request size limits #16962

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
gizas wants to merge 11 commits into from
Closed

[Prometheus Remote Write] Update request size limits #16962

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
2418429
update request size limits
gizas Jan 14, 2026
6faf09b
update changelog
gizas Jan 14, 2026
ce915d6
Merge branch 'main' into prometheus_body_bytes
gizas Jan 26, 2026
5fa0edd
change value to integer
gizas Jan 26, 2026
180fb75
Merge branch 'prometheus_body_bytes' of github.com:elastic/integratio…
gizas Jan 26, 2026
87d9441
fix versions
gizas Jan 30, 2026
a40fe07
Merge branch 'main' into prometheus_body_bytes
gizas Jan 30, 2026
7270dc6
update changelog
gizas Feb 18, 2026
977e90e
Merge branch 'main' into prometheus_body_bytes
gizas Feb 18, 2026
dd4ccac
update doc
gizas Feb 18, 2026
b0d6774
Merge branch 'prometheus_body_bytes' of github.com:elastic/integratio…
gizas Feb 18, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions packages/prometheus/_dev/build/docs/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -326,6 +326,14 @@ will be handled as a histogram, even if it has the suffix `_total` which is a de

The Prometheus integration's `remote_write` dataset provides a `Metrics Count` parameter, which is disabled by default. When enabled, it counts the total number of Prometheus metrics within each Elasticsearch document. This count is stored in a field called `metrics_count` and its value is calculated prior to any enrichments by Ingest Pipelines or Agent Processors, ensuring consistency. This field name is reserved for internal use and must not be altered using Agent Processors or Ingest Pipelines.

#### Request Size Limits

To protect against resource exhaustion from malicious or oversized payloads, the remote_write metricset enforces configurable size limits on incoming requests:

- `max_compressed_body_bytes`: Maximum size of the compressed (snappy-encoded) request body in bytes. Requests exceeding this limit are rejected with HTTP 413 before being read into memory. Default: 2 MB (2097152 bytes).
- `max_decoded_body_bytes`: Maximum size of the decompressed request body in bytes. The server checks the declared decoded size in the snappy header before allocating memory for decompression, preventing decompression bomb attacks. Default: 10 MB (10485760 bytes).


### Prometheus Queries (PromQL)

The Prometheus `query` dataset executes specific Prometheus queries against [Promethes Query API](https://prometheus.io/docs/prometheus/latest/querying/api/#expression-queries).
Expand Down
5 changes: 5 additions & 0 deletions packages/prometheus/changelog.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,9 @@
# newer versions go on top
- version: "1.25.0"
changes:
- description: Add request size limits to the remote_write data stream.
type: enhancement
link: https://github.com/elastic/integrations/pull/16962
- version: "1.24.2"
changes:
- description: Fix a typo in the Prometheus integration documentation.
Expand Down
2 changes: 2 additions & 0 deletions packages/prometheus/data_stream/remote_write/agent/stream/stream.yml.hbs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,8 @@ period: {{period}}
metrics_count: {{metrics_count}}
rate_counters: {{rate_counters}}
use_types: {{use_types}}
max_compressed_body_bytes: {{max_compressed_body_bytes}}
Copy link
Copy Markdown
Contributor Author

@gizas gizas Jan 14, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See beats relevant config options: https://github.com/elastic/beats/blob/main/metricbeat/module/prometheus/remote_write/config.go#L35C58-L35C80

max_decoded_body_bytes: {{max_decoded_body_bytes}}
types_patterns.exclude:
{{#each types_patterns.counter_patterns}}
- {{this}}
Expand Down
16 changes: 16 additions & 0 deletions packages/prometheus/data_stream/remote_write/manifest.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,22 @@ streams:
required: true
show_user: true
default: true
- name: max_compressed_body_bytes
type: integer
title: Max Compressed Body Bytes
description: Maximum size of the compressed (snappy-encoded) request body in bytes. Default is 2 MB (2097152 bytes).
multi: false
required: false
show_user: false
default: 2097152
- name: max_decoded_body_bytes
type: integer
title: Max Decoded Body Bytes
description: Maximum size of the decompressed request body in bytes. Default is 10 MB (10485760 bytes).
multi: false
required: false
show_user: false
default: 10485760
- name: types_patterns.counter_patterns
type: text
title: Counter Type Patterns
Expand Down
8 changes: 8 additions & 0 deletions packages/prometheus/docs/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -535,6 +535,14 @@ will be handled as a histogram, even if it has the suffix `_total` which is a de

The Prometheus integration's `remote_write` dataset provides a `Metrics Count` parameter, which is disabled by default. When enabled, it counts the total number of Prometheus metrics within each Elasticsearch document. This count is stored in a field called `metrics_count` and its value is calculated prior to any enrichments by Ingest Pipelines or Agent Processors, ensuring consistency. This field name is reserved for internal use and must not be altered using Agent Processors or Ingest Pipelines.

#### Request Size Limits

To protect against resource exhaustion from malicious or oversized payloads, the remote_write metricset enforces configurable size limits on incoming requests:

- `max_compressed_body_bytes`: Maximum size of the compressed (snappy-encoded) request body in bytes. Requests exceeding this limit are rejected with HTTP 413 before being read into memory. Default: 2 MB (2097152 bytes).
- `max_decoded_body_bytes`: Maximum size of the decompressed request body in bytes. The server checks the declared decoded size in the snappy header before allocating memory for decompression, preventing decompression bomb attacks. Default: 10 MB (10485760 bytes).


### Prometheus Queries (PromQL)

The Prometheus `query` dataset executes specific Prometheus queries against [Promethes Query API](https://prometheus.io/docs/prometheus/latest/querying/api/#expression-queries).
Expand Down
4 changes: 2 additions & 2 deletions packages/prometheus/manifest.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
format_version: "3.3.0"
name: prometheus
title: Prometheus
version: 1.24.2
version: 1.25.0
description: Collect metrics from Prometheus servers with Elastic Agent.
type: integration
categories:
Expand All @@ -10,7 +10,7 @@ categories:
- containers
conditions:
kibana:
version: "^8.16.0 || ^9.0.0"
version: "^9.2.4 || ^9.3.1 || ^9.4.0"
Copy link
Copy Markdown
Contributor Author

@gizas gizas Jan 14, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

According to https://github.com/elastic/beats/pull/48218/files#diff-3bca358ec7bb955db5fa88789cf4e1155ee015b30b28abd2b1892725b52190afR63

Copy link
Copy Markdown
Member

@ishleenk17 ishleenk17 Jan 28, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think changing kibana version to this is a good idea.
Any future bugfixes will be skipped for 18.9, 9.0, 9.1, 9.3.0.
Can we instead the agent version requirment for this configuration int he documentation of the conig parameters ?

Copy link
Copy Markdown
Contributor Author

@gizas gizas Feb 18, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See doc: update doc

Copy link
Copy Markdown
Member

@jsoriano jsoriano Jan 28, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

^9.2.4 already includes ^9.3.1 and ^9.4.0, were you intending to set something like ~9.2.4 || ~9.3.1 || ^9.4.0?

Copy link
Copy Markdown
Contributor Author

@gizas gizas Jan 29, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let me clarify with an example: If someone tries 9.2.3 should not match.
And also if someone tries 9.3.0 should also not match. So I would like to include 9.2.4 but not 9.3.0

Is not this the correct approach?

Copy link
Copy Markdown
Member

@jsoriano jsoriano Jan 29, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

^ matches all the versions in the same major over the indicated version. ^9.2.4 is equal to ">= 9.2.4 AND < 10.0.0", so ^9.2.4 || ^9.3.1 || ^9.4.0 includes 9.3.0.

You need to use ~ to match only versions of the same minor, so ~9.2.4 || ~9.3.1 || ^9.4.0.

Copy link
Copy Markdown
Contributor Author

@gizas gizas Jan 30, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you ! done in fix versions

Copy link
Copy Markdown
Member

@ishleenk17 ishleenk17 Jan 30, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@gizas : I still think this should not be done. As doing this we are actually removing any future support to versions like 8.19, 9.0, 9.1, 9.2 (lesser than 9.2.4), 9.3.0.
Can we instead update about this agent version dependency in the readme.

Copy link
Copy Markdown
Contributor

@tommyers-elastic tommyers-elastic Jan 30, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for core integrations like prometheus we really want to retain support for 8.x customers.

Copy link
Copy Markdown
Contributor Author

@gizas gizas Jan 30, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The reason for that was that the beats fix elastic/beats#48218 was not imported to 8.19 versions.

See above PR for relevant backports.
This relates to previous comment from Ishleen.

If you agree I can do:

^8.16.0 || ^9.2.4 || ^9.3.1 || ^9.4.0

and just document the limitations in README?

Copy link
Copy Markdown
Member

@jsoriano jsoriano Feb 4, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

^8.16.0 || ^9.2.4 || ^9.3.1 || ^9.4.0 is equivalent to ^8.16.0 || ^9.2.4, so the package will be available in 9.3.0.

You probably want ^8.16.0 || ~9.2.4 || ^9.3.1.

Copy link
Copy Markdown
Contributor Author

@gizas gizas Feb 4, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @jsoriano ! @tommyers-elastic , @ishleenk17 do u agree to proceed with above versions and also document the fact that the existing variables will not work in 8.x ?

Copy link
Copy Markdown
Contributor Author

@gizas gizas Feb 18, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tommyers-elastic , @ishleenk17 I did the backport to 8.19. See elastic/beats#48918

So now "^8.19.3 || ~9.2.5 || ^9.3.1" is the final one.
Can I have another review please?

elastic:
subscription: basic
screenshots:
Expand Down