Skip to content

docs: list every value with_high_performance() sets and its memory cost - #954

Closed
7487 wants to merge 1 commit into
huggingface:mainfrom
7487:doc-high-performance
Closed

7487 wants to merge 1 commit into
huggingface:mainfrom
7487:doc-high-performance

Conversation

@7487

@7487 7487 commented Aug 31, 2026

Copy link
Copy Markdown

Fixes #926.

Answering the scoping question in the issue thread: this PR takes the xet-core code comment half. The hub docs describe each HF_XET_* variable individually, but the doc comment on with_high_performance() is the one place that defines what the preset itself does, and it summarized twelve assignments in one sentence with no memory figure.

Since #943 the buffer values are memory-derived, so the comment now lists:

  • all twelve fields with default -> high performance values, the buffer entries as their derivation formulas (usable/16 -> usable/8 etc.) with floors/ceilings, plus the static fallbacks used when memory cannot be probed;
  • the worst-case buffer allocation, download_buffer_size + n_active * download_buffer_perfile_size capped at download_buffer_limit (verified against FileReconstructor's target formula) — up to 32GB at the ceilings with the default 8 concurrent downloads;
  • that data.max_concurrent_file_downloads is not raised by the preset, only its ingestion twin;
  • that every field has its own HF_XET_* variable, so parts of the preset can be adopted individually on memory-constrained machines.

All values checked against the field defaults in config/groups/ and the constants in utils/system_memory.rs on current main.

Comment-only diff; no Rust toolchain on this machine so I did not run cargo locally, but no code is touched and the [floor, ceiling] notation matches the existing field docs in reconstruction.rs.

Written with Claude Code, values verified against the source as described above.


Note

Low Risk
Comment-only change with no code or configuration logic modified.

Overview
Expands the with_high_performance() doc comment so the high-performance preset is documented in one place instead of a single vague sentence.

The comment now lists twelve fields with default → preset values (ingestion concurrency, adaptive concurrency bounds, reconstruction fetch sizes, and memory-derived download buffer formulas with clamps). It explains how usable memory is computed, static 16GB / 2GB / 64GB fallbacks when probing is disabled or fails, and the worst-case download buffer formula (up to ~32GB at ceilings with default concurrent downloads). It also notes that max_concurrent_file_downloads is not raised by the preset, and that each field can be tuned via its own HF_XET_* env var without enabling the full preset.

No runtime behavior changes — documentation only on xet_config.rs.

Reviewed by Cursor Bugbot for commit 49fb4e8. Bugbot is set up for automated code reviews on this repo. Configure here.

The doc comment summarized the preset in one prose sentence. List all
twelve fields with their default and high performance values, the
memory-derived buffer formulas, the worst-case buffer allocation, and
the fact that each field can also be set individually via its own
HF_XET_* variable.

Fixes huggingface#926

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
@seanses

seanses commented Sep 11, 2026

Copy link
Copy Markdown
Collaborator

There's no need to repeat the code in the comment.

@seanses seanses closed this Sep 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

with_high_performance() documents one of the twelve values it sets, and none of the memory cost

2 participants