diff --git a/docs/ADRs/0071-forge-pointer-correlation-for-agent-traces.md b/docs/ADRs/0071-forge-pointer-correlation-for-agent-traces.md new file mode 100644 index 000000000..451b48dba --- /dev/null +++ b/docs/ADRs/0071-forge-pointer-correlation-for-agent-traces.md @@ -0,0 +1,93 @@ +--- +title: "71. Harness snapshot and forge pointer correlation for agent traces" +status: Accepted +relates_to: + - operational-observability +topics: + - observability + - telemetry + - opentelemetry + - forge +--- + +# 71. Harness snapshot and forge pointer correlation for agent traces + +Date: 2026-07-22 + +## Status + +Accepted + +## Context + +[ADR 0050](0050-distributed-tracing-instrumentation.md) defines how fullsend +generates traces (OTel Go SDK; Level 1 = `run-telemetry.jsonl`; Level 2 = +OTLP). [Operational observability](../problems/operational-observability.md) +requires forge **pointers** (repo + SHA, change id, pipeline run) rather than +duplicating diffs, and a stable answer to: “what harness/config produced this +run?” Forge-neutral hosting concepts remain those in +[ADR 0005](0005-forge-abstraction-layer.md); this ADR only decides how join +pointers are recorded on a run. + +After the OTel SDK migration, `run-summary.json` was removed; the sole Level 1 +telemetry file today is `run-telemetry.jsonl`. Eval wrappers that `export` +ambient CI variables are the wrong layer: harness identity is a **run-time** +concern of `fullsend run`, and downstream tools must not scrape CI env as the +source of truth. Cross-project join field names live in a shared join contract; +this ADR decides only what **fullsend** writes. + +## Options + +**A. Root-span attributes only (no snapshot file).** Join keys exist in OTLP +backends, but offline/forensic consumers must parse spans, and keys disappear +when span export is disabled. + +**B. Embed a snapshot event in `run-telemetry.jsonl`.** One Level 1 file, but +couples the harness/config contract to telemetry format and exporter semantics. + +**C. Eval-script / ambient CI env exports.** Easy to prototype; unstable across +local runs, child dispatches, and forges; encourages scraping rather than an +explicit artifact. + +**D. Dedicated `harness-snapshot.json` plus mirrored root-span attributes +(chosen).** Stable local run-start contract for offline consumers, plus +backend-friendly correlation for OTLP. + +## Decision + +Every `fullsend run` records run-start join keys by writing +**`harness-snapshot.json`** next to `run-telemetry.jsonl` **and**, when tracing +is enabled, setting the same keys on the root `run` span (`vcs.*`, `cicd.*`, +harness content SHA, forge platform). The file holds pointers and a config +fingerprint only (harness identity, content hash, forge/CI pointers when known) +— no diffs, prompts, or skill bodies. Forge/CI fields are filled at write time +(`FULLSEND_*` overrides first, then standard CI env). Consumers read the JSON +(or stores that ingest it), not ambient CI env. + +When tracing is enabled, write the snapshot after the root span exists and +include `trace_id` / `traceparent` so the file joins to `run-telemetry.jsonl` +and OTLP. When `OTEL_SDK_DISABLED=true`, still write the snapshot (config/forge +contract, not span export) but **omit** `trace_id` / `traceparent` and skip +root-span attribute mirroring — there is no usable trace context. + +## Consequences + +- Operators and downstream loggers can join a run to forge/CI via a single + decided artifact; when tracing is on, `trace_id` joins to `run-telemetry.jsonl`. +- OTLP backends receive join keys without parsing the JSON when tracing is on; + local JSON remains the forensic / handoff contract. +- Dispatched child runs must inherit forge context via env so child snapshots + are complete. +- GitLab/Bitbucket coverage depends on CI vars or `FULLSEND_*` overrides. +- Implementation is tracked separately ([#5449](https://github.com/fullsend-ai/fullsend/issues/5449)); + detailed field lists for cross-project consumers stay in the shared join + contract, not duplicated here. + +## Related + +- [ADR 0050](0050-distributed-tracing-instrumentation.md) +- [ADR 0005](0005-forge-abstraction-layer.md) +- Shared join contract: [provenance_forge_pointers.md](https://github.com/RHEcosystemAppEng/ABEvalFlow/blob/main/Docs/provenance_forge_pointers.md) +- Implementation: [#5449](https://github.com/fullsend-ai/fullsend/issues/5449) +- [#2368](https://github.com/fullsend-ai/fullsend/issues/2368) +- [#294](https://github.com/fullsend-ai/fullsend/issues/294) diff --git a/docs/architecture.md b/docs/architecture.md index 43f934c43..a61c81b95 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -262,6 +262,7 @@ Observability is a cross-cutting concern that touches every other component. Eac - JSONL reasoning trace exposure: raw JSONL conversation transcripts are extracted from sandboxes and stored with owner-scoped access. Credential scanning acts as an invariant check on [ADR 0017](ADRs/0017-credential-isolation-for-sandboxed-agents.md)'s isolation model. Agents handling data from protected sources beyond the target repo can opt in to JSONL suppression via configuration ([ADR 0021](ADRs/0021-jsonl-reasoning-trace-exposure.md)). - Event-driven stage dispatch remains traceable end-to-end in the GitHub Actions UI by using synchronous `workflow_call` dispatch (see [ADR 0041](ADRs/0041-synchronous-workflow-call-event-dispatch.md)). - Distributed tracing: framework-native OpenTelemetry instrumentation with zero-configuration baseline. Every run produces `run-telemetry.jsonl` locally; optional live OTLP export to any compatible backend. W3C trace context propagation links multi-agent pipelines into unified traces. OTEL GenAI semantic conventions enable LLM-aware backends ([ADR 0050](ADRs/0050-distributed-tracing-instrumentation.md)). +- Decided (pending [#5449](https://github.com/fullsend-ai/fullsend/issues/5449)): harness snapshot join keys — every run will write `harness-snapshot.json` (harness fingerprint + forge/CI pointers) and, when tracing is enabled, mirror those keys on the root span so offline and OTLP consumers can correlate without scraping CI env ([ADR 0071](ADRs/0071-forge-pointer-correlation-for-agent-traces.md)). **Open questions:** diff --git a/docs/guides/infrastructure/distributed-tracing.md b/docs/guides/infrastructure/distributed-tracing.md index 037055051..dd7266e61 100644 --- a/docs/guides/infrastructure/distributed-tracing.md +++ b/docs/guides/infrastructure/distributed-tracing.md @@ -7,16 +7,21 @@ Decided in [ADR 0050](../../ADRs/0050-distributed-tracing-instrumentation.md). ## Zero-configuration baseline (Level 1) -Every `fullsend run` produces one file in the run output directory with no -configuration required: - -- **`run-telemetry.jsonl`** — OTLP JSON spans covering the run lifecycle - (sandbox creation, agent iterations, validation) with timestamps, durations, - trace IDs, and token/cost attributes. +Every `fullsend run` produces **`run-telemetry.jsonl`** in the run output +directory with no configuration required — OTLP JSON spans covering the run +lifecycle (sandbox creation, agent iterations, validation) with timestamps, +durations, trace IDs, and token/cost attributes. This file is written on every run unless `OTEL_SDK_DISABLED=true`, which -suppresses all telemetry output including the local file. It contains -metadata only — no prompts, completions, or source code content. +suppresses span export (local file and OTLP). It contains metadata only — no +prompts, completions, or source code content. + +> **Planned:** `harness-snapshot.json` (run-start harness fingerprint and +> forge/CI pointers, mirrored on the root span) is decided in +> [ADR 0071](../../ADRs/0071-forge-pointer-correlation-for-agent-traces.md). +> Implementation is tracked in +> [#5449](https://github.com/fullsend-ai/fullsend/issues/5449). Until that +> ships, only `run-telemetry.jsonl` is guaranteed on disk. ## Prerequisites