OTEL improvements

[EvanDietzMorris]

OTEL improvements: trace noise reduction and richer callback attributes

Summary

Four related changes to the OpenTelemetry instrumentation: drop noisy low-value ASGI spans, add payload/result-size attributes to ARA callbacks, standardize every custom attribute onto dotted namespaces, and split callback delivery timing into HTTP time vs. retry backoff.

Changes

1. Drop ASGI send/receive spans (shepherd_server/server.py)

FastAPIInstrumentor.instrument_app now passes exclude_spans=["receive", "send"]. These per-ASGI-message spans represent individual events within a larger message and flood the backend with traces that carry no useful signal.

2. Add attributes to the ARA callback span (shepherd_server/base_routes.py)

The callback span now records the shape of what arrived, not just its ID: callback.results, callback.kg_nodes, callback.kg_edges, and callback.payload_bytes. This makes it possible to see how much each ARA actually returned.

3. Standardize attribute names onto namespaces (all workers + server)

Flat keys became dotted, so attributes group consistently in the backend:

Before	After
callback_id	callback.id
query_id	query.id
response_id	response.id
drained_callbacks	merge.drained_callbacks
callback_attempt_failed (event)	callback.attempt_failed
attempt, duration_ms (event attrs)	callback.attempt, callback.attempt_duration_ms

Touches aragorn_lookup, aragorn_pathfinder, bte_lookup, merge_message, finish_query, and base_routes.

4. Separate wait time from backoff time (workers/finish_query/worker.py)

callback.duration_ms conflated time spent in HTTP requests with time spent sleeping between retries. Two new attributes split them: callback.wait_ms (summed elapsed across all attempts, successful and failed) and callback.backoff_ms. A slow callback receiver and a flaky one now look different in the trace.

[codecov]

Codecov Report

❌ Patch coverage is 74.07407% with 7 lines in your changes missing coverage. Please review.
✅ Project coverage is 54.88%. Comparing base (806a000) to head (8940967).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
workers/merge_message/worker.py	0.00%	5 Missing ⚠️
shepherd_server/server.py	0.00%	1 Missing ⚠️
workers/aragorn_pathfinder/worker.py	50.00%	1 Missing ⚠️

Files with missing lines	Coverage Δ
shepherd_server/base_routes.py	53.47% <100.00%> (+1.27%)	⬆️
workers/aragorn_lookup/worker.py	76.44% <100.00%> (ø)
workers/bte_lookup/worker.py	85.45% <100.00%> (ø)
workers/finish_query/worker.py	85.80% <100.00%> (+0.77%)	⬆️
shepherd_server/server.py	0.00% <0.00%> (ø)
workers/aragorn_pathfinder/worker.py	63.71% <50.00%> (ø)
workers/merge_message/worker.py	68.61% <0.00%> (ø)

---

Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 731f987...8940967. Read the comment docs.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.