feat(otel-collector): expose rollouts-pod-template-hash on spanmetrics

[guimoreirar]

Proposal (for o11y review)

Expose the Argo Rollouts pod-template hash on the spanmetrics so that
progressive-delivery analysis can isolate a canary's RED metrics from the
stable version. This is a prerequisite for automatic, metric-gated canary
promotion/rollback (Argo Rollouts AnalysisTemplate) across our services.

Opening this for the o11y team to analyze the cardinality/dimension policy
before we adopt it. Nothing else depends on it merging immediately.

Problem

Our apps don't expose /metrics; RED signals come from the OTel spanmetrics
connector (calls_total, duration_milliseconds_bucket) shipped to Mimir. A
canary gate must compare only the new version's pods against thresholds —
otherwise, at low traffic weight, canary errors are diluted by the stable
version and the gate is blind.

Argo Rollouts distinguishes canary vs stable pods with the pod label
rollouts-pod-template-hash. Today the spanmetrics carry k8s.pod.name,
k8s.namespace.name, k8s.deployment.name — but no pod labels, so there is
no way to slice a series by canary vs stable. (Verified on benedita: the label
rollouts_pod_template_hash returns an empty value set in Mimir.)

Change (2 additive edits, otel-collector-lerian)

1. k8sattributes.extract.labels — extract the pod label into a resource
   attribute:

   labels:
     - tag_name: rollouts_pod_template_hash
       key: rollouts-pod-template-hash
       from: pod

2. spanmetrics.dimensions — add it as a dimension so it becomes a metric
   label:

   - name: rollouts_pod_template_hash
     default: ""

Chart version: 4.2.0-beta.3 → 4.2.0-beta.4.

Cardinality (please sanity-check)

• New label is empty ("") for every pod not managed by a Rollout → one extra
  value cluster-wide for the vast majority of series.
• For an app under rollout it adds the stable + canary hashes, and only
  while a rollout is in progress; it collapses back to the stable hash after.
• The connector already carries the far-higher-cardinality k8s.pod.name, and
  aggregation_cardinality_limit: 50000 is unchanged. Net impact is expected to
  be negligible, but flagging for your call on the dimension policy.

Backward compatibility

Purely additive. Existing queries/dashboards are unaffected (the new label just
appears with value "" on existing series). No pipeline/processor reordering.

Validation

• helm lint → 0 failures.
• helm template renders the label in both places (k8sattributes tag_name
rollouts_pod_template_hash + spanmetrics dimension rollouts_pod_template_hash).

Questions for o11y

• OK to add this as a default dimension for all envs, or should it be gated
  behind a values flag (e.g. spanmetrics.rolloutDimension: true) so only
  clusters doing progressive delivery pay the (tiny) cardinality?
• Naming: rollouts_pod_template_hash vs a namespaced k8s.pod.label.* form?

[coderabbitai]

📝 Walkthrough

Walkthrough

The Helm chart extracts the Argo Rollouts pod template hash as a Kubernetes resource attribute and adds it as a spanmetrics dimension, defaulting to an empty string when absent.

Changes

Rollout-aware span metrics

Layer / File(s)	Summary
Rollout hash metric dimension charts/otel-collector-lerian/values.yaml	The k8sattributes processor extracts the Rollouts pod template hash, and spanmetrics uses it as a dimension with an empty default for non-Rollout pods.

✨ Finishing Touches

✨ Simplify code

• Create PR with simplified code
• Commit simplified code in branch feat/otel-spanmetrics-rollouts-hash

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@charts/otel-collector-lerian/values.yaml`:
- Around line 396-401: The spanmetrics configuration around
rollouts_pod_template_hash currently reads a resource attribute that remains
unavailable in pinned version 0.142.0. Copy the k8sattributes resource attribute
onto each span before the spanmetrics connector, or upgrade to a version
supporting resource-attribute dimensions; preserve the existing empty default
for non-Rollout pods.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: 67c2263d-e429-45e8-8e8d-9ce57e3e1193

📥 Commits

Reviewing files that changed from the base of the PR and between b3594cd and fb5f8a4.

📒 Files selected for processing (1)

• charts/otel-collector-lerian/values.yaml

[coderabbitai]

🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/usr/bin/env bash
set -euo pipefail

file="charts/otel-collector-lerian/values.yaml"
rg -n 'tag: 0\.142\.0|rollouts_pod_template_hash|spanmetrics:' "$file"

curl -fsSL \
  https://raw.githubusercontent.com/open-telemetry/opentelemetry-collector-contrib/v0.142.0/connector/spanmetricsconnector/config.go \
  | grep -F "fetched from the span's attributes"

Repository: LerianStudio/helm

Length of output: 447

---

🏁 Script executed:

#!/usr/bin/env bash
set -euo pipefail

file="charts/otel-collector-lerian/values.yaml"

echo "== relevant sections =="
sed -n '340,480p' "$file"

echo
echo "== any attribute-copying processor or transform around spanmetrics =="
rg -n 'transform|attributes|spanmetrics|rollouts_pod_template_hash|resource' "$file"

Repository: LerianStudio/helm

Length of output: 10578

---

🏁 Script executed:

#!/usr/bin/env bash
set -euo pipefail

echo "== upstream spanmetrics config for v0.142.0 =="
curl -fsSL \
  https://raw.githubusercontent.com/open-telemetry/opentelemetry-collector-contrib/v0.142.0/connector/spanmetricsconnector/config.go \
  | sed -n '1,220p'

Repository: LerianStudio/helm

Length of output: 9245

---

rollouts_pod_template_hash stays empty on spans in 0.142.0
k8sattributes writes this as a resource attribute, but spanmetrics in this pinned version reads dimensions from span attributes only. Without a resource→span copy before the connector, canary and stable series collapse to "". Either upgrade to a version that supports resource-attribute dimensions or copy it onto the span first.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@charts/otel-collector-lerian/values.yaml` around lines 396 - 401, The
spanmetrics configuration around rollouts_pod_template_hash currently reads a
resource attribute that remains unavailable in pinned version 0.142.0. Copy the
k8sattributes resource attribute onto each span before the spanmetrics
connector, or upgrade to a version supporting resource-attribute dimensions;
preserve the existing empty default for non-Rollout pods.