feat(telemetry): client transfer performance telemetry

[sirahd]

Adds client-side transfer performance telemetry so we can detect and alert on throughput regressions shipped in an hf-xet release. Each upload or download reports one summary document to POST /v1/telemetry, the endpoint added by a companion server-side change.

This is PR 1 of 2. A follow-up (#922) adds the generated schema (telemetry/metrics.schema.json) and a CI compatibility gate. See Rollout — there is a decision needed before this is enabled against a real index.

Verified end-to-end, not just in tests: a real hf CLI upload and download against a local Hub + a local CAS server carrying the new endpoint. Details in Verification — that exercise is what surfaced the download gap fixed below.

Why

Server-side metrics cover CAS request latency but not end-to-end client throughput, dedup effectiveness, or where a transfer's wall time goes. Without that we can't tell a slow client from a slow server, and a regression in a released client is invisible.

Design

One hook point covers every surface. All four client entry points bottom out in FileUploadSession / FileDownloadSession, so the measurement lives entirely in xet_data and xet_client:

Surface	Reaches
XetUploadCommit::commit, XetFileDownloadGroup::finish	the two sessions
Legacy data_client (what shipped huggingface_hub calls today)	same
git_xet / xtool	same
XetDownloadStreamGroup	new finish(), plus Drop as the fallback

Notable choices:

• Client::transfer_telemetry() has a default None body, so 6 of the 7 implementors are untouched and local/memory/simulation clients are excluded for free.
• The sink clones RemoteClient's authenticated HTTP client rather than building one. A second build_auth_http_client would create a second AuthMiddleware with its own TokenProvider, giving telemetry an independent token-refresh cycle against the Hub.
• TransferTelemetry is direction-agnostic. RemoteClient has no notion of direction, and threading one through would have touched all 18 of its construction sites to benefit 2. xet_data knows which kind of session it holds and supplies it when building the payload.
• The emit path is compiled out on wasm — XetRuntime has no spawn there, so there is no way to report without blocking a transfer. The outcome vocabulary (telemetry::outcome) does compile everywhere, because it appears in FileDownloadSession's public signatures and gating it would push #[cfg] onto every caller that merely names an outcome.

Making the production paths actually emit

The measurement side above was not sufficient on its own. Wiring it up needed changes in xet_pkg and hf_xet:

• XetFileDownloadGroup::finish/finish_blocking and legacy download_async now finalize the download session, on the error path as well as the success path. Nothing previously called FileDownloadSession::finalize(), which is the only route to a download document — so downloads reported nothing at all through the Python bindings while every test passed. huggingface_hub was already doing the right thing on its side; the Rust finish_blocking simply read the progress report and returned.
• FileDownloadSession::finalize_with(outcome, error_class) — finalize() hardcoded Ok(()) as the result, so a download could only ever report outcome: ok. The plumbing for error outcomes existed but was unreachable, which would have left download failure-rate alerting reading zero forever. XetError::telemetry_class() maps xet_pkg's error categories onto the same coarse class vocabulary xet_data uses, so both paths aggregate together.
• XetDownloadStreamGroup::finish (plus finish() and context-manager support on the Python class). Streams are consumed independently, so the group cannot detect completion itself. Purely additive — a group that is never finished behaves exactly as before and still reports, just as dropped; no existing caller has to change.
• The Drop fallback no longer requires an ambient tokio runtime. The send is spawned on the XetRuntime's own stored handle, so Handle::try_current() was never needed — and requiring it silently disabled the fallback for embedders that release the last Arc from a foreign thread, which is exactly what the Python bindings do.

Fire-and-forget

Never retried (a 429 is the server shedding load), 5s per-request timeout, an in-flight cap that drops rather than queues, and every failure swallowed at DEBUG.

One deliberate exception: the terminal document is awaited, bounded by final_flush_timeout (default 2s). A fully detached final send is usually lost, because host processes routinely exit within milliseconds of a transfer returning. It runs after all transfer work is done — teardown, not the data path — and HF_XET_TELEMETRY_FINAL_FLUSH_TIMEOUT=0 makes it fully detached. There's a wiremock test asserting a hanging endpoint can't hold a transfer past the budget.

The removed hf_xet/src/telemetry.rs (#441) used a tracing layer, a process-global OnceLock, its own runtime, and a Mutex held across sends — which hung around os.fork(). This design has none of those; all state hangs off XetContext/RemoteClient and dies with them.

Privacy

The payload carries no file names, paths, hashes, repository ids, or user ids. Enabled by default, and disabled by HF_XET_TELEMETRY_ENABLED=0, HF_HUB_DISABLE_TELEMETRY, or HF_HUB_OFFLINE — the opt-outs win over an explicit enable.

Worth stating plainly: the resulting documents are user-identifiable server-side, because cas_server attaches the JWT claim set (userId, repoId) and clientIp. That's a property of the ingest endpoint, not of this change, but it shouldn't be implied away by the client-side "no PII" claim.

Metrics

~26 common keys (byte counts, throughput, duration, outcome, error class, concurrency, client version/os/arch, endpoint host only), plus 16 upload-only from DeduplicationMetrics and shard progress, and expansion_ratio for downloads. xet_data/src/telemetry/payload.rs is the source of truth.

The key set is a contract. Consumers assign each property a type on first sight and cannot change it in place, so adding a key is safe but changing an existing key's JSON type breaks ingestion for every document carrying it, and recovering means rebuilding the stored data. Two tests enforce this. Relatedly, every f64 goes through a finite guard — serde_json renders NaN and infinity as null, and one such document poisons that property's type for a consumer.

Testing

~70 tests: 9 config gating, 24 in xet_client (envelope, sink, in-flight cap, heartbeat weak-reference, four against wiremock), 18 payload/outcome/emit, and 19 end-to-end against the simulation server (13 in xet_data, 6 in xet_pkg).

Two failure modes these are shaped around, both of which actually occurred:

1. transfer_telemetry has a default body, so an override with a mistyped signature would compile cleanly and silently never be called. xet_data/tests/test_transfer_telemetry.rs drives a real transfer through a real HTTP server to catch that.
2. The emit machinery can work perfectly while no production caller reaches it — which is exactly what happened to downloads. Tests that drive FileDownloadSession directly cannot see this, because they call finalize() themselves. xet_pkg/tests/test_download_telemetry.rs goes through XetFileDownloadGroup::finish_blocking() and asserts on what the server received. Please keep new coverage at that altitude.

Both new regression tests were confirmed to fail when their fix is reverted, not merely to pass with it.

Run as CI runs it:

• cargo test --features "strict simulation internal-tools" — all green
• cargo clippy -r -- -D warnings, for the workspace and for hf_xet — clean
• cargo +nightly fmt — clean
• hf_xet_wasm + hf_xet_thin_wasm (wasm32) build

Verification

Beyond the test suite, the full path was exercised against real infrastructure: hf CLI → huggingface_hub 1.26 → this branch's hf-xet → a local Hub → a local CAS server with the new endpoint → the receiving datastore.

Confirmed there:

• Upload and download documents both index, with the server correctly stamping env, casVersion, the JWT claim set, and clientIp from the rightmost X-Forwarded-For hop.
• Both documents validate against feat(telemetry): generated metrics schema and CI compatibility gate #922's telemetry/metrics.schema.json.
• Granularity differs by direction, which matters for dashboards. Upload emits one document per commit batch (n_files > 1); download emits one per file, because hf_hub_download opens a fresh download group per file — so n_files is always 1 there and a 200-shard model produces 200 documents.
• max_in_flight is per-TransferTelemetry, i.e. per transfer, not a global cap. It bounds heartbeats within one long transfer but does not throttle a concurrent multi-file snapshot download.

Rollout

Point the ingestion side at a scratch destination for now. Property types are pinned by the first documents that carry them and cannot be changed in place, and this PR starts writing before any storage mapping is designed — so whatever those early documents imply is what we are stuck with.

What automatic type inference actually produced when measured, rather than assumed:

• Numeric typing came out correct — every ratio and throughput property as a float, every counter as an integer. The specific hazard flagged earlier (throughput_bps serializing as 0.0 pinning the property to an integer type) does not fire, because serde_json always renders an f64 with a decimal point. It would fire if a metric were ever changed from f64 to an integer type: a bare 0 pins the property, and every later float is then silently truncated with no error surfaced.
• Every string was stored as analyzed full text with an exact-match variant alongside it, including pure identifiers like transfer_id and sessionId and low-cardinality enums like outcome. Those want exact-match only.
• Floats landed in single precision, so the 4-decimal values the client computes do not round-trip (71910674.2857 stores as 71910672.0). Relative error is ~1e-7, so aggregations are unaffected, but exact values are not preserved. Double precision is a choice worth making deliberately.

None of that is fixable in place, hence the scratch destination. Deriving the storage mapping is the ingestion side's call — this repo publishes JSON types via metrics.schema.json and deliberately does not carry the receiving service's storage layout.

This PR alone cannot deliver the regression alerting that motivates the work — until the ingestion side types these fields as numbers, they are not aggregatable. That work and this shouldn't be left to drift apart.

Follow-ups

• feat(telemetry): generated metrics schema and CI compatibility gate #922: generated JSON Schema, drift test, CI breaking-change gate.
• Companion server-side work: an explicit storage mapping derived from the published schema, replacing automatic type inference.
• Consider whether download telemetry should be aggregated per snapshot rather than per file — that is a huggingface_hub change (one group for many files), not a xet-core one.
• HTTP-level counters (retries, per-API latency, status classes, chunk-cache hit rate) — these are what distinguish "slow client" from "slow server". RetryWrapper is the chokepoint; scoped in the plan.

🤖 Generated with Claude Code

---

Note

Medium Risk
Touches core upload/download finalize and Python stream-group lifecycle with default-on reporting, but sends are bounded, non-blocking for data movement, and carry no client-side PII; mis-finalization or dropped telemetry would affect observability more than transfer correctness.

Overview
Adds best-effort transfer performance telemetry that POSTs one terminal summary (and optional heartbeats) per upload/download to POST /v1/telemetry, with a new telemetry config group, HF_HUB_* opt-outs, and README/OpenAPI/docs.

xet-client gains TransferTelemetry (non-wasm): snake_case envelope, fire-and-forget sink with process-wide in-flight cap, bounded terminal flush, and a default Client::transfer_telemetry() hook implemented on RemoteClient (shared auth HTTP client, concurrency sampling).

xet-data builds flat metric payloads from progress/dedup state, starts heartbeats from sessions, emits on finalize (including finalize_with / upload reported_failure), and Drop when sessions are abandoned; GroupProgress::all_items_complete() drives download outcomes when finish() was never called.

Production wiring fixes: download groups and legacy download paths now finalize the download session (so Python-bound downloads actually report); XetDownloadStreamGroup exposes finish/abort and Python __exit__ no longer reports ok on exceptions.

Tests/simulation: local server records telemetry docs; integration tests assert end-to-end delivery through finish_blocking.

^{Reviewed by Cursor Bugbot for commit 2c45835. Bugbot is set up for automated code reviews on this repo. Configure here.}

[rajatarya]

Design overview — a reviewer's map

Posting this as a shared reference so anyone picking up the review can orient quickly. It's a big diff (~3.6k lines across 39 files), but the shape is smaller than the line count suggests: a self-contained telemetry subsystem plus the wiring needed to actually reach it. All diagrams below are just a redraw of what's in the code — no new claims.

1. Where the pieces live

The measurement is split across two crates by what each can see: xet_data owns the metric definitions (it's the only crate that can read DeduplicationMetrics and the progress report), while xet_client owns identity, timing, and delivery.

flowchart TD
    subgraph entry["Client entry points (all bottom out in the two sessions)"]
        A["XetUploadCommit.commit"]
        C["Legacy data_client"]
        B["XetFileDownloadGroup.finish"]
        D["XetDownloadStreamGroup (finish or Drop)"]
    end

    subgraph xetdata["xet_data — owns the metric definitions"]
        US["FileUploadSession"]
        DS["FileDownloadSession"]
        EMIT["emit.rs — bridges a finished session to the sink"]
        PAY["payload.rs — the metric contract (u64 / f64 / bool / String)"]
        OUT["outcome.rs — outcome and coarse error_class"]
    end

    subgraph xetclient["xet_client — owns identity, timing, delivery"]
        TT["TransferTelemetry — one per transfer per direction"]
        SINK["TelemetrySink — in-flight cap, never retries"]
        ENV["TelemetryEnvelope — the wire body"]
    end

    CONF["xet_runtime — telemetry config group plus HF_HUB opt-outs"]
    SERVER["POST /v1/telemetry — corresponding server-side endpoint"]

    A --> US
    C --> US
    B --> DS
    D --> DS
    US --> EMIT
    DS --> EMIT
    EMIT --> PAY
    EMIT --> OUT
    EMIT --> TT
    TT --> SINK
    SINK --> ENV
    ENV --> SERVER
    CONF -. gates .-> TT

Loading

• One hook point covers every surface. Because all four entry points bottom out in FileUploadSession / FileDownloadSession, the measurement lives entirely on those two sessions — nothing had to be threaded through each caller.
• Client::transfer_telemetry() defaults to None, so only RemoteClient implements it; local / in-memory / simulation clients are excluded for free, and every telemetry call site is a cheap no-op in tests.
• The whole subsystem is compiled out on wasm (no XetRuntime::spawn there); only the outcome vocabulary compiles everywhere, because it appears in FileDownloadSession's public signatures.

2. The lifecycle of one transfer's telemetry

sequenceDiagram
    participant Caller
    participant Session as Session in xet_data
    participant TT as TransferTelemetry
    participant Sink as TelemetrySink
    participant Server as Server-side endpoint

    Note over Session,TT: one TransferTelemetry per transfer and direction
    Session->>TT: start_heartbeat (weak ref so it never keeps the session alive)
    Session->>TT: record_concurrency on each permit acquired
    loop only after heartbeat_after elapses
        TT->>Sink: heartbeat (detached and dropped if the in-flight cap is full)
        Sink-->>Server: POST heartbeat
    end
    alt finalize path (success or error)
        Session->>TT: emit_terminal (awaited up to final_flush_timeout)
        TT->>Sink: submit_awaited
        Sink-->>Server: POST terminal summary
    else dropped without finalize
        Session->>TT: emit_terminal_detached
        TT->>Sink: submit_detached
        Sink-->>Server: POST terminal summary (best effort)
    end
    Note over TT: a terminal_sent guard means exactly one terminal document per transfer

Loading

Fire-and-forget is the governing principle: never retried, a per-request timeout, an in-flight cap that drops rather than queues, and every failure swallowed at DEBUG. The one deliberate exception is the terminal document, which is awaited up to final_flush_timeout (default 2s) — because a fully detached final send is usually lost when the host process exits milliseconds after the transfer returns. That wait happens in teardown, after all data movement.

3. How the outcome field is chosen

flowchart TD
    T{"How did the transfer end?"}
    T -->|"finalize returned Ok"| OK["outcome = ok"]
    T -->|"finalize returned Err"| CL["classify_error"]
    T -->|"upload dropped before finalize"| AB["outcome = aborted"]
    T -->|"download dropped before finalize"| DR["outcome = dropped"]
    T -->|"still running (heartbeat)"| IP["outcome = in_progress"]
    CL -->|"cancelled class"| CA["outcome = cancelled — kept out of failure alerts"]
    CL -->|"any other class"| ER["outcome = error"]

Loading

error_class is a deliberately coarse closed set (auth / network / timeout / rate_limited / server_error / not_found / io / format / cancelled / internal / other), so failure-rate dashboards group cleanly and no error text — which could contain paths — ever reaches the wire.

Scope of changes, by bucket

Bucket	Files	What to look at
New telemetry subsystem	xet_client/.../telemetry/{mod,sink,envelope}.rs, xet_data/src/telemetry/{payload,emit,outcome,mod}.rs	The core; payload.rs is the metric contract
Session hooks	file_upload_session.rs, file_download_session.rs	terminal on finalize, Drop fallback, heartbeat start
Making prod paths emit	xet_pkg/{error.rs, legacy/data_client.rs, xet_session/*}, hf_xet/src/py_download_stream_group.rs	the behavioral changes (below)
Config	xet_runtime/src/config/groups/telemetry.rs, utils/configuration_utils.rs	HF_XET_TELEMETRY_* + the HF_HUB_* opt-outs
Test infra	simulation server records POSTs; test_transfer_telemetry.rs, test_download_telemetry.rs	end-to-end coverage
Docs / generated	README.md, openapi/cas.openapi.yaml, api_changes/*, Cargo.locks	—

Where behavior actually changed (worth close eyes)

The measurement side alone wasn't sufficient — a few production paths had to change to reach it, and these are the non-additive parts:

1. Downloads previously reported nothing through the Python bindings. Nothing called FileDownloadSession::finalize(), so XetFileDownloadGroup::finish/finish_blocking and legacy download_async now finalize the session — on the error path as well as success.
2. finalize_with(outcome, error_class) is new: the old finalize() hardcoded Ok(()), so a download could only ever report outcome: ok and failure-rate would have read zero forever.
3. XetDownloadStreamGroup::finish + Python context-manager support — purely additive; a group that is never finished still reports, as dropped.
4. The Drop fallback no longer requires an ambient tokio runtime — it spawns on XetRuntime's stored handle, which is what the Python bindings need (they release the last Arc from a foreign thread).

Privacy, in one line

The client payload carries no file names, paths, hashes, repo ids, or user ids — only scalar performance metrics. Enabled by default; HF_XET_TELEMETRY_ENABLED=0, HF_HUB_DISABLE_TELEMETRY, or HF_HUB_OFFLINE turn it off, and the opt-outs win over an explicit enable. (Documents do become user-identifiable server-side, since the endpoint attaches request identity from the JWT — a property of the ingest side, not of this payload.)

I'll follow up with review comments separately.

[rajatarya]

Thanks, Sam — this is really carefully built. Splitting the subsystem by what each crate can see (metric definitions in xet_data, identity/timing/delivery in xet_client) is clean, and the fire-and-forget discipline is consistent all the way down — no retries, drop-not-queue backpressure, everything swallowed at DEBUG, and a ()-return send so a telemetry error physically cannot propagate into a transfer. The terminal_sent dedup across the finalize and Drop paths, the weak-reference heartbeat, the finite-guard on every f64 so a NaN can't poison a field type, and the key-set/type contract tests are all the right instincts. I also appreciate how directly the description owns the fact that downloads were reporting nothing before this — that's the kind of thing that's easy to leave unsaid.

I've posted a separate design-overview comment with a few diagrams so other reviewers can orient without reading all 39 files first.

Everything below is a question rather than a blocker — mostly about the interaction between the per-file download granularity and a few per-transfer scoped mechanisms:

1. The awaited terminal flush is per-transfer, but downloads are per-file (inline on file_download_session.rs). When the endpoint is degraded, each file's finalize can wait up to final_flush_timeout; a large snapshot with limited download concurrency could accumulate that. Wondering whether downloads want the detached path.
2. outcome: dropped will be the common case for successful stream-group downloads (inline on emit.rs), since existing callers won't adopt finish(). Flagging so dashboards don't read dropped as failure — and floating whether Drop could infer ok from a fully-completed progress report.
3. The in-flight cap is per-transfer, so aggregate telemetry request volume is unbounded (inline on sink.rs) — relevant precisely because a snapshot download fans out into many per-file transfers at once.

None of these change the correctness of what's here; they're about how the feature behaves under the per-file download shape once it's pointed at a real endpoint. Really nice work.

[rajatarya]

The awaited flush is a great call for the terminal document — a fully detached final send really does get lost when the process exits milliseconds later. But it's worth thinking about how it composes with download granularity: an upload amortizes this wait over one commit, whereas a download emits one document per file (a fresh group per hf_hub_download), so this awaited path runs once per file.

Under normal conditions that's just the round-trip and invisible. When the endpoint is degraded, though, each file's finalize() waits up to final_flush_timeout (2s default). Concurrent file downloads overlap that wait, but if download concurrency is well below the file count — a large model fetched a handful at a time — the 2s stacks across teardown batches.

Would it make sense for the download terminal to use the detached path (or a much shorter budget), and reserve the awaited flush for cases where the process is actually about to exit? The upload commit is the clean fit for awaiting; per-file downloads seem like the case where the multiplied cost outweighs the delivery gain. Curious whether you'd already weighed this.

[rajatarya]

Since XetDownloadStreamGroup::finish() is purely additive and existing callers won't adopt it right away, the Drop-based path here will be the common outcome for stream-group downloads — including ones that consumed every stream and fully succeeded. So outcome: dropped won't mean "abandoned"; for a while it'll mostly mean "succeeded but the caller didn't call finish()."

That's fine as long as everyone downstream knows to treat dropped as neutral rather than as a failure or an incomplete transfer — but it's an easy trap for a failure-rate or completion dashboard, and it's the metric the whole feature is meant to feed.

Since Drop already has the progress report in hand (self.report()), could it infer Outcome::Ok when the report shows everything completed, and reserve dropped for a genuinely partial transfer? That would keep dropped meaning "actually abandoned" and let success be read straight off the outcome. If the intent is instead that finish() becomes the norm and dropped stays a real signal, a note to that effect would help the query side.

[rajatarya]

The in-flight cap lives on the sink, which is per-TransferTelemetry, i.e. per transfer. That bounds heartbeats within one long transfer nicely, but it doesn't bound the aggregate: a snapshot download fans out into many concurrent per-file transfers, each with its own sink and its own cap, so the process-wide number of in-flight telemetry POSTs is max_in_flight × (concurrent transfers) with no ceiling.

For best-effort telemetry that's not dangerous — worst case a burst of small POSTs that the endpoint sheds — but it's the opposite of where you'd want the backpressure: the heaviest telemetry moment (a wide snapshot) is exactly when there's no aggregate limit. Would a process-wide cap (a shared counter, or a shared sink) fit the model better than a per-transfer one? Not urgent, but it interacts with the per-file granularity in the other comments, so worth deciding deliberately rather than by default.

[jgodlew]

Overall looks good! Just a couple of nits and a comment on the behavior of the heartbeats if we experience contention.

[jgodlew]

Should failing a snapshot cause all future heartbeats to be skipped?

From my understanding of the snapshot behavior, the snapshot can fail if this task is unable to try_lock() on the xorb_upload task. We could just skip this heartbeat and wait for the next one or sleep for sleep for a smaller time interval to retry?

[sirahd]

Good catch. I updated the task to skip the heartbeat instead of failing if snapshot fails.

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

Want higher recall? High effort reviews run extra passes and find more bugs. A team admin can switch effort levels in the Cursor dashboard.

^{Reviewed by Cursor Bugbot for commit fea0f48. Configure here.}

[seanses]

Let's remove mentioning of HF_HUB env vars as disabling telemetry is done programmatically.

[seanses]

Let's remove mentioning of HF_HUB env vars as disabling telemetry is done programmatically.

[seanses]

LGTM! Let's fix the above two doc issues and merge!

[sirahd]

Closing this PR in favor of the stacked PRs at #932 - #937