Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
93 changes: 93 additions & 0 deletions docs/ADRs/0071-forge-pointer-correlation-for-agent-traces.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,93 @@
---
title: "71. Harness snapshot and forge pointer correlation for agent traces"
status: Accepted
relates_to:
- operational-observability
topics:
- observability
- telemetry
- opentelemetry
- forge
---

# 71. Harness snapshot and forge pointer correlation for agent traces

Date: 2026-07-22

## Status

Accepted

## Context

[ADR 0050](0050-distributed-tracing-instrumentation.md) defines how fullsend
generates traces (OTel Go SDK; Level 1 = `run-telemetry.jsonl`; Level 2 =
OTLP). [Operational observability](../problems/operational-observability.md)
requires forge **pointers** (repo + SHA, change id, pipeline run) rather than
duplicating diffs, and a stable answer to: “what harness/config produced this
run?” Forge-neutral hosting concepts remain those in
[ADR 0005](0005-forge-abstraction-layer.md); this ADR only decides how join
pointers are recorded on a run.

After the OTel SDK migration, `run-summary.json` was removed; the sole Level 1
telemetry file today is `run-telemetry.jsonl`. Eval wrappers that `export`
ambient CI variables are the wrong layer: harness identity is a **run-time**
concern of `fullsend run`, and downstream tools must not scrape CI env as the
source of truth. Cross-project join field names live in a shared join contract;
this ADR decides only what **fullsend** writes.

## Options

**A. Root-span attributes only (no snapshot file).** Join keys exist in OTLP
backends, but offline/forensic consumers must parse spans, and keys disappear
when span export is disabled.

**B. Embed a snapshot event in `run-telemetry.jsonl`.** One Level 1 file, but
couples the harness/config contract to telemetry format and exporter semantics.

**C. Eval-script / ambient CI env exports.** Easy to prototype; unstable across
local runs, child dispatches, and forges; encourages scraping rather than an
explicit artifact.

**D. Dedicated `harness-snapshot.json` plus mirrored root-span attributes
(chosen).** Stable local run-start contract for offline consumers, plus
backend-friendly correlation for OTLP.

## Decision

Every `fullsend run` records run-start join keys by writing
**`harness-snapshot.json`** next to `run-telemetry.jsonl` **and**, when tracing
is enabled, setting the same keys on the root `run` span (`vcs.*`, `cicd.*`,
harness content SHA, forge platform). The file holds pointers and a config
fingerprint only (harness identity, content hash, forge/CI pointers when known)
— no diffs, prompts, or skill bodies. Forge/CI fields are filled at write time
(`FULLSEND_*` overrides first, then standard CI env). Consumers read the JSON
(or stores that ingest it), not ambient CI env.

When tracing is enabled, write the snapshot after the root span exists and
include `trace_id` / `traceparent` so the file joins to `run-telemetry.jsonl`
and OTLP. When `OTEL_SDK_DISABLED=true`, still write the snapshot (config/forge
contract, not span export) but **omit** `trace_id` / `traceparent` and skip
root-span attribute mirroring — there is no usable trace context.

## Consequences

- Operators and downstream loggers can join a run to forge/CI via a single
decided artifact; when tracing is on, `trace_id` joins to `run-telemetry.jsonl`.
- OTLP backends receive join keys without parsing the JSON when tracing is on;
local JSON remains the forensic / handoff contract.
- Dispatched child runs must inherit forge context via env so child snapshots
are complete.
- GitLab/Bitbucket coverage depends on CI vars or `FULLSEND_*` overrides.
- Implementation is tracked separately ([#5449](https://github.com/fullsend-ai/fullsend/issues/5449));
detailed field lists for cross-project consumers stay in the shared join
contract, not duplicated here.

## Related

- [ADR 0050](0050-distributed-tracing-instrumentation.md)
- [ADR 0005](0005-forge-abstraction-layer.md)
- Shared join contract: [provenance_forge_pointers.md](https://github.com/RHEcosystemAppEng/ABEvalFlow/blob/main/Docs/provenance_forge_pointers.md)
- Implementation: [#5449](https://github.com/fullsend-ai/fullsend/issues/5449)
- [#2368](https://github.com/fullsend-ai/fullsend/issues/2368)
- [#294](https://github.com/fullsend-ai/fullsend/issues/294)
1 change: 1 addition & 0 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -262,6 +262,7 @@ Observability is a cross-cutting concern that touches every other component. Eac
- JSONL reasoning trace exposure: raw JSONL conversation transcripts are extracted from sandboxes and stored with owner-scoped access. Credential scanning acts as an invariant check on [ADR 0017](ADRs/0017-credential-isolation-for-sandboxed-agents.md)'s isolation model. Agents handling data from protected sources beyond the target repo can opt in to JSONL suppression via configuration ([ADR 0021](ADRs/0021-jsonl-reasoning-trace-exposure.md)).
- Event-driven stage dispatch remains traceable end-to-end in the GitHub Actions UI by using synchronous `workflow_call` dispatch (see [ADR 0041](ADRs/0041-synchronous-workflow-call-event-dispatch.md)).
- Distributed tracing: framework-native OpenTelemetry instrumentation with zero-configuration baseline. Every run produces `run-telemetry.jsonl` locally; optional live OTLP export to any compatible backend. W3C trace context propagation links multi-agent pipelines into unified traces. OTEL GenAI semantic conventions enable LLM-aware backends ([ADR 0050](ADRs/0050-distributed-tracing-instrumentation.md)).
- Decided (pending [#5449](https://github.com/fullsend-ai/fullsend/issues/5449)): harness snapshot join keys — every run will write `harness-snapshot.json` (harness fingerprint + forge/CI pointers) and, when tracing is enabled, mirror those keys on the root span so offline and OTLP consumers can correlate without scraping CI env ([ADR 0071](ADRs/0071-forge-pointer-correlation-for-agent-traces.md)).

**Open questions:**

Expand Down
21 changes: 13 additions & 8 deletions docs/guides/infrastructure/distributed-tracing.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,16 +7,21 @@ Decided in [ADR 0050](../../ADRs/0050-distributed-tracing-instrumentation.md).

## Zero-configuration baseline (Level 1)

Every `fullsend run` produces one file in the run output directory with no
configuration required:

- **`run-telemetry.jsonl`** — OTLP JSON spans covering the run lifecycle
(sandbox creation, agent iterations, validation) with timestamps, durations,
trace IDs, and token/cost attributes.
Every `fullsend run` produces **`run-telemetry.jsonl`** in the run output
directory with no configuration required — OTLP JSON spans covering the run
lifecycle (sandbox creation, agent iterations, validation) with timestamps,
durations, trace IDs, and token/cost attributes.

This file is written on every run unless `OTEL_SDK_DISABLED=true`, which
suppresses all telemetry output including the local file. It contains
metadata only — no prompts, completions, or source code content.
suppresses span export (local file and OTLP). It contains metadata only — no
prompts, completions, or source code content.

> **Planned:** `harness-snapshot.json` (run-start harness fingerprint and
> forge/CI pointers, mirrored on the root span) is decided in
> [ADR 0071](../../ADRs/0071-forge-pointer-correlation-for-agent-traces.md).
> Implementation is tracked in
> [#5449](https://github.com/fullsend-ai/fullsend/issues/5449). Until that
> ships, only `run-telemetry.jsonl` is guaranteed on disk.

## Prerequisites

Expand Down