PMM-15198 Indexstats fix. - #1321
Conversation
|
@copilot review |
Reviewed the current PR changes in the indexstats area; I don’t see additional actionable fixes to apply right now. I also re-ran targeted tests for the updated helper logic ( |
Co-authored-by: Cursor <cursoragent@cursor.com>
|
@copilot review |
ademidoff
left a comment
ademidoff
left a comment
There was a problem hiding this comment.
A few minor suggestions, nothing blocking.
|
@copilot review |
|
@copilot review |
|
@copilot review |
There was a problem hiding this comment.
Two things that don't map to a line:
Stale PR description. The second paragraph ("raises the go test timeout from 30s to 1m in the Makefile") no longer applies: main already carries -timeout 1m (Makefile:124, :127, :130) and no Makefile change appears in this diff. Commit bf207f6b is in the branch history but contributes nothing against the current base. Worth trimming so reviewers aren't hunting for it.
Cardinality, for the release notes. The duplicate indexstats series were previously deduplicated silently by Gather; now they're all exposed. Scraping mongos in discovering mode goes from collections x indexes to collections x indexes x shards series. That's the correct data, but it's a real increase users should be told about.
…no shards." This reverts commit 118859d.
|
@copilot review |
PMM-15198
FB: Percona-Lab/pmm-submodules#4507
Summary
mongodb_indexstats_* metrics are silently dropped when the exporter scrapes a sharded cluster through mongos.
Steps to reproduce
Run a sharded MongoDB cluster (2+ shards) with a sharded collection that has at least one index.
Point mongodb_exporter at mongos with --collector.indexstats enabled.
Scrape /metrics.
Actual result
Only one shard's index statistics are exposed. The exporter logs duplicate-metric errors from the Prometheus registry, and the remaining shards' data is discarded.
Expected result
One series per index per shard, each distinguished by a shard label.
Root cause
When scraping via mongos, the $indexStats aggregation returns one document per shard for every index. The indexstats collector built its label set per document but never read the shard field, so every shard produced an identical metric descriptor. Prometheus rejects duplicate descriptors, so all but one shard was thrown away.
A related defect exists in the collstats collector: the labels map is allocated once before the document loop, so a shard value from one document leaks into subsequent documents that report no shard.
Fix
Add a shared setShardLabel() helper in exporter/common.go that sets the shard label unconditionally (empty string when absent), so all series of a metric family carry identical label dimensions and no registration conflict occurs.
Call it from both the indexstats and collstats collectors.
Move the collstats label map allocation inside the document loop to stop cross-document leakage.
Impact on existing series
Non-sharded deployments see no change — Prometheus drops empty label values at ingestion. Sharded deployments gain one series per shard per index instead of losing them.
Testing
Unit tests for setShardLabel() (shard present / empty / missing / non-string). Integration tests TestIndexStatsCollectorSharded and TestCollStatsCollectorSharded assert metrics from ≥2 shards with no duplicates. Note: these integration tests currently skip because the compose test cluster registers no shards — a separate test-infrastructure issue, tracked independently of this fix.