Repository navigation
fix(telemetry): view http.client.request.duration onto 8 buckets - #347
Merged
Merged
Conversation
The meter provider registered no Views, so the histogram otelhttp records on all four outbound paths (teranode, merkleservice, bump/datahub, webhook) kept whatever bucket set the instrumentation advises: 14 boundaries, 15 buckets per attribute set, on a metric already multiplied by target host, method and status code. Nothing downstream can re-bucket a histogram, so the lever has to exist here. Register one View narrowing it to 8 boundaries. The unit is seconds — the instrument is declared with unit "s" and the transport records durationToSeconds — so the boundaries are seconds too, and each is an edge of the shared latency buckets in metrics/metrics.go so quantiles off this histogram and off arcade_teranode_request_duration_seconds interpolate over the same edges. The top boundary is 30s, the client timeout on the teranode and merkleservice paths, so a timeout is still distinguishable from a slow success instead of both falling in the overflow bucket. The arcade_* families are untouched: they reach the reader through the Prometheus bridge producer, whose ScopeMetrics are appended at collection time and never pass through view resolution, and the View matches one wildcard-free instrument name anyway. Tests drive a real otelhttp transport and assert both the narrowed boundaries and a bridged arcade_* histogram keeping the buckets its promauto definition declares, with the pre-View advisory pinned as the control. Closes #333
galt-tr
approved these changes
Sep 22, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #333.
What
The meter provider in
telemetry/telemetry.goregistered no Views, sohttp.client.request.duration— the histogramotelhttp.NewTransportrecords on all four outbound paths (teranode/client.go,merkleservice/client.go,bump/datahub.go,services/webhook/service.go) — kept whatever bucket set the instrumentation advises. That is 14 boundaries / 15 buckets per attribute set, on a metric already multiplied by target host, method and status code.Nothing downstream can re-bucket an explicit histogram, so the lever has to exist here.
This registers one View narrowing it to 8 boundaries:
Nine buckets instead of fifteen.
Correction to the issue
#333 claims a possible unit mismatch — "semconv puts
http.client.request.durationin seconds, but the SDK default boundary set is millisecond-shaped, so every bucket above the first may be empty". That is wrong, and there is no correctness bug here. The instrumentation ships its own seconds-shaped advisory, which the SDK honours:otel/semconv/v1.43.0/httpconv/metric.godeclares the instrumentmetric.WithUnit("s")with a default advisory of0.005 … 10otelhttp/internal/semconv/client.go:41passes that same seconds-shaped list when constructing the instrumentotelhttp/internal/semconv/client.go:336recordsdurationToSeconds(md.RequestDuration)So the metric is already in the right unit against the right buckets. The issue's series-count argument stands; its correctness argument does not. The count is also 14 boundaries, not 15.
Why these boundaries
Every value is also an edge of
latencyBucketsinmetrics/metrics.go, so a quantile taken off this histogram and one taken offarcade_teranode_request_duration_secondsfor the same calls interpolate over shared edges rather than two unrelated grids.The top boundary is 30s rather than the 10s the issue suggested, because
defaultTimeoutis30 * time.Secondon bothteranode/client.goandmerkleservice/client.go. With a 10s ceiling a timed-out request and a merely slow one are the same datapoint in the overflow bucket — which is exactly the case worth alerting on.arcade_*metrics are structurally unaffectedThey reach the reader through
prometheusbridge.NewMetricProducer.PeriodicReader.Collectruns the view-resolved pipeline and then appends the producer'sScopeMetricsdirectly, so external producer output bypasses view resolution entirely. Theirpromautobucket definitions stay authoritative regardless of how a View is scoped. The View criterion is also the exact, wildcard-free instrument name, which could not match anarcade_*name in any case.Both facts are asserted by tests rather than just reasoned about.
Deliberately not included
No
AttributeFilter.url.scheme,server.portandnetwork.protocol.versionlook like an 8x multiplier but are functionally determined byserver.addressfor a fixed set of targets — removing them saves bytes per datapoint and zero series.server.addressdoes multiply, but it is the attribute that separates the four outbound paths and is worth its cost.http.response.status_codeis the remaining independent dimension. Unlike bucket boundaries it can be collapsed downstream, so it is left alone here.Tests
telemetry/telemetry_test.godrives a realotelhttp.NewTransportagainst anhttptestserver through aManualReaderconfigured with the same bridge producerinitMetricsuses, then reads the collectedmetricdata:view_narrows_the_boundary_set— assertsUnit == "s"(unit-drift guard) and that the collected bounds equalhttpClientDurationBoundaries. Fails if the View does not reach the instrument otelhttp actually creates.baseline_without_the_view— the control: same setup, no Views, yields exactly the 14-value advisory. This is what makes the assertion above meaningful, and doubles as a drift alarm if a dependency bump ever reverts to the SDK default.TestMetricViews_LeaveBridgedArcadeMetricsAlone— observes on the realmetrics.TeranodeRequestDurationand compares the bridged family's bounds against whatprometheus.DefaultGathererreports, read from the registry rather than hardcoded.All 9 pre-existing tests in the package still pass.
gofmt -l telemetry/is empty.golangci-lintcould not run locally — the installed binary is built with go1.26 against this module's go 1.27.0 target — so CI will need to cover it.Separate bug worth its own issue
http.client.request.body.sizeandhttp.client.response.body.sizehave no advisory (httpconv/metric.gosets only description and unitBy). They therefore do fall back to the SDK's default0, 5, 10, 25 … 10000— which for a byte-valued histogram means everything above ~10KB lands in the overflow bucket. That is the genuinely-mismatched histogram #333 suspected, just on a different metric, and it is 15 more buckets each. Out of scope here; happy to follow up.