feat(telemetry): generated metrics schema and CI compatibility gate

[sirahd]

Stacked on #919 — base is sira/client-transfer-telemetry, so the diff here is just this change. Merge #919 first; the base will retarget to main automatically.

Publishes the telemetry metric vocabulary as a generated, committed JSON Schema, and enforces the compatibility rules consumers depend on. This is PR 2 of 2.

telemetry/metrics.schema.json is the source of truth for what we collect

All 65 properties carry a description of what they measure and their unit, so a consumer needs nothing from this repo's source to interpret a document:

"defrag_prevented_dedup_bytes": {
  "description": "Bytes that could have been deduplicated but were re-uploaded anyway,
                  because doing otherwise would have fragmented the xorb past the defrag
                  threshold. The cost of the fragmentation-vs-dedup tradeoff.",
  "type": "integer", "format": "uint64", "minimum": 0
}

A test fails the build if any property reaches the schema without a description. A metric documented only by its name is a metric nobody outside this repo can read.

xet_data/src/telemetry/payload.rs remains the source of truth for the code; the schema is generated from it and must never be hand-edited:

UPDATE_TELEMETRY_SCHEMA=1 cargo test -p xet-data --lib telemetry::schema

Without that variable, the same test asserts the committed file is current — so a payload change that forgets to regenerate fails cargo test.

schemars is a dev-dependency and the derives are cfg(test); cargo tree -p xet-data -e normal shows zero occurrences of it or its transitive deps, so nothing ships.

Scope: what this repo publishes

What the client emits — property names, JSON types, meanings.

It deliberately does not describe how any consumer stores, indexes, or aggregates these documents. That's the consumer's concern, this repo is public, and it has no way to keep a description of someone else's storage layer correct. A consumer types its own storage from each property's type:

JSON Schema	Meaning
"integer"	fits an unsigned 64-bit integer
"number"	finite double; never NaN or infinity
"boolean"	—
"string"	short, low-cardinality except transfer_id

Every value is a scalar: never null, never nested, never an array.

Two things the api_changes note flags for whoever builds the ingestion side: the numeric properties must be stored as numbers, or throughput and duration can't be averaged or range-queried — which is the entire point of collecting them. And the mapping should tolerate an unknown property, since consumers pin a tag and a client can ship ahead of a consumer rebuild.

CI gate

scripts/check_telemetry_schema_compat.py diffs the branch schema against the merge target in a new telemetry-schema-compat job.

Change	Verdict
Property added	pass
Property removed	fail
Property type changed	fail

A missing baseline is treated as the introducing commit. Checkout needs fetch-depth: 0 — the shallow default can't read the base ref.

Testing

6 schema tests: drift, every property documented, scalars only, both directions present, cross-direction type agreement, alerting metrics are numeric. I verified the drift test actually fires by tampering with the committed artifact, and exercised the CI script against all six cases (identical / missing baseline / added / removed / retyped / malformed).

• cargo test --features "strict simulation internal-tools" — 38 test binaries, all green
• cargo clippy -r --features "strict simulation internal-tools" — clean
• cargo +nightly fmt — clean

Changed since first review

Dropped telemetry/es-index-template.json. It described the receiving service's document shape and storage layout, which doesn't belong in a public repo. The generated JSON Schema now carries the type information a consumer needs to derive its own mapping, and the storage-specific guidance moved to prose in the api_changes note. Related wording elsewhere in these files was made storage-agnostic for the same reason.

Note: api_changes/update_260728_client_transfer_telemetry.md from #919 still names the storage technology in several places. I left it alone rather than editing a file from an open PR — say the word and I'll clean it up in either PR.

---

Note

Low Risk
Mostly contract, tests, and CI; runtime telemetry behavior is unchanged and schemars stays dev-only.

Overview
Introduces a published, generated contract for client transfer telemetry: committed telemetry/metrics.schema.json is built from xet_data payload types (via test-only schemars + schema.rs), not hand-edited. Payload fields gain /// doc comments that become schema descriptions; tests require every property to be documented and keep the committed file in sync (UPDATE_TELEMETRY_SCHEMA=1 to regenerate).

Adds scripts/check_telemetry_schema_compat.py and a telemetry-schema-compat CI job that diffs the branch schema against the merge target: new metrics allowed, removals or JSON type changes fail. .gitattributes pins the schema to LF for stable byte comparisons on Windows.

Documents the contract in api_changes/update_260729_telemetry_schema_contract.md (pin-by-tag URL, regeneration, consumer typing rules). Cargo.lock updates reflect schemars as a dev-only dependency path.

^{Reviewed by Cursor Bugbot for commit 7819742. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 45a0729. Configure here.}

[seanses]

Trying to understand this PR, what's the purpose of generating a JSON schema? We don't expect users to consume or use the telemetry details right?