diff --git a/docs/hub/xet/using-xet-storage.md b/docs/hub/xet/using-xet-storage.md index bbe757df2e..252c1d630b 100644 --- a/docs/hub/xet/using-xet-storage.md +++ b/docs/hub/xet/using-xet-storage.md @@ -155,8 +155,8 @@ By default, `xet-core` uses adaptive concurrency — dynamically adjusting paral | Environment Variable | Default | Description | |---|---|---| | `HF_XET_CLIENT_ENABLE_ADAPTIVE_CONCURRENCY` | `true` | Enable or disable adaptive concurrency control. When disabled, concurrency stays at the initial value. | -| `HF_XET_CLIENT_AC_INITIAL_UPLOAD_CONCURRENCY` | `1` | Starting number of concurrent upload streams. HP mode: `16`. | -| `HF_XET_CLIENT_AC_INITIAL_DOWNLOAD_CONCURRENCY` | `1` | Starting number of concurrent download streams. HP mode: `16`. | +| `HF_XET_CLIENT_AC_INITIAL_UPLOAD_CONCURRENCY` | `2` | Starting number of concurrent upload streams. HP mode: `16`. | +| `HF_XET_CLIENT_AC_INITIAL_DOWNLOAD_CONCURRENCY` | `4` | Starting number of concurrent download streams. HP mode: `16`. | | `HF_XET_CLIENT_AC_MIN_UPLOAD_CONCURRENCY` | `1` | Lower bound for upload concurrency. HP mode: `4`. | | `HF_XET_CLIENT_AC_MIN_DOWNLOAD_CONCURRENCY` | `1` | Lower bound for download concurrency. HP mode: `4`. | | `HF_XET_CLIENT_AC_MAX_UPLOAD_CONCURRENCY` | `64` | Upper bound for upload concurrency. HP mode: `124`. | @@ -178,7 +178,7 @@ By default, `xet-core` uses adaptive concurrency — dynamically adjusting paral | `HF_XET_CLIENT_RETRY_BASE_DELAY` | `3000ms` | Base delay between retries (with exponential backoff). | | `HF_XET_CLIENT_RETRY_MAX_DURATION` | `360s` | Maximum total time to spend retrying a request. | | `HF_XET_CLIENT_CONNECT_TIMEOUT` | `60s` | TCP connection timeout. | -| `HF_XET_CLIENT_READ_TIMEOUT` | `120s` | Read timeout for HTTP responses. | +| `HF_XET_CLIENT_READ_TIMEOUT` | `300s` | Read timeout for HTTP responses. | | `HF_XET_CLIENT_IDLE_CONNECTION_TIMEOUT` | `60s` | Timeout before idle connections are closed. | | `HF_XET_CLIENT_MAX_IDLE_CONNECTIONS` | `16` | Maximum number of idle connections in the pool. | diff --git a/docs/xet/api.md b/docs/xet/api.md index 2147ca992c..a8e3b86d34 100644 --- a/docs/xet/api.md +++ b/docs/xet/api.md @@ -229,7 +229,7 @@ A typical successful stream: If validation stalls, the server periodically re-emits the last progress event as a heartbeat so the stream never looks dead, which lets clients keep a short read timeout. > [!NOTE] -> Additive change: `/v1/shards` is unchanged. Clients SHOULD try `/v2/shards` and fall back to `/v1/shards` on `404 Not Found`. +> Additive change: `/v1/shards` is unchanged. Clients SHOULD try `/v2/shards` and fall back to `/v1/shards` on a `404` or `501`. The reference client caches the detected shard API version for the session, so this fallback probe happens once rather than per shard. - **Error Responses**: A malformed or unauthorized request can still fail before streaming starts; see [Error Cases](./api#error-cases). Once the stream has started (`200 OK`), validation and registration failures are reported through the terminal `error` event instead. - `401 Unauthorized`: Refresh the token to continue making requests, or provide a token in the `Authorization` header. diff --git a/docs/xet/shard.md b/docs/xet/shard.md index d58aaad096..0fd93046d7 100644 --- a/docs/xet/shard.md +++ b/docs/xet/shard.md @@ -47,6 +47,8 @@ A shard file consists of the following sections in order: ├─────────────────────┤ │ CAS Info Section │ ├─────────────────────┤ +│ Lookup Tables │ (optional) +├─────────────────────┤ │ Footer │ └─────────────────────┘ ``` @@ -68,7 +70,7 @@ Offset footer.file_info_offset: │ │ └───────────────────────────────────────────────────────┘ -Offset footer.cas_info_offset: +Offset footer.xorb_info_offset: ┌───────────────────────────────────────────────────────┐ │ │ │ CAS Info Section │ ← Variable size @@ -77,6 +79,14 @@ Offset footer.cas_info_offset: │ │ └───────────────────────────────────────────────────────┘ +Offset footer.file_lookup_offset: +┌───────────────────────────────────────────────────────┐ +│ │ +│ Lookup Tables │ ← Variable size, +│ (file, then xorb, then chunk lookup) │ may be absent +│ │ +└───────────────────────────────────────────────────────┘ + Offset footer.footer_offset: ┌───────────────────────────────────────────────────────┐ │ Footer (200 bytes, sometimes omitted) │ ← Fixed size @@ -137,7 +147,7 @@ struct MDBShardFileHeader { ## 2. File Info Section -**Location**: `footer.file_info_offset` to `footer.cas_info_offset` or directly after the header +**Location**: `footer.file_info_offset` to `footer.xorb_info_offset` or directly after the header This section contains a sequence of 0 or more file information (File Info) blocks, each consisting at least a header and at least 1 data sequence entry, and OPTIONAL verification entries and metadata extension section. The file info section ends when reaching the bookend entry. @@ -334,7 +344,7 @@ The file info section begins right after the header and ends when the bookend is ## 3. CAS Info Section -**Location**: `footer.cas_info_offset` to `footer.footer_offset` or directly after the file info section bookend +**Location**: `footer.xorb_info_offset` up to the section's bookend, or directly after the file info section bookend This section contains CAS (Content Addressable Storage) block information. Each CAS Info block represents a xorb by first having a `CASChunkSequenceHeader` which contains the number of `CASChunkSequenceEntries` to follow that make up this block. The CAS Info section ends when reaching the bookend entry. @@ -362,7 +372,7 @@ This section contains CAS (Content Addressable Storage) block information. Each **Deserialization steps**: -1. Seek to `footer.cas_info_offset` +1. Seek to `footer.xorb_info_offset` 2. Read `CASChunkSequenceHeader` 3. Check if `cas_hash` is all 0xFF (bookend marker) - if so, stop 4. Read `cas_chunk_sequence_header.num_entries` × `CASChunkSequenceEntry` structures @@ -401,23 +411,37 @@ struct CASChunkSequenceEntry { chunk_hash: Hash, // 32-byte chunk hash chunk_byte_range_start: u32, // Start position in CAS block unpacked_segment_bytes: u32, // Size when unpacked - _unused: [u8; 8], // Reserved space 8 bytes + flags: u32, // Chunk flags, see below + _unused: u32, // Reserved space 4 bytes } ``` **Memory Layout**: ```txt -┌────────────────────────────────────────────────────────────────┬─────────┬─────────┬─────────────────┐ -│ chunk_hash (32 bytes) │chunk_ │unpacked │ _unused │ -│ Chunk Hash │byte_ │segment_ │ (8 bytes) │ -│ │range_ │bytes │ │ -│ │start │(4 bytes)│ │ -│ │(4 bytes)│ │ │ -└────────────────────────────────────────────────────────────────┴─────────┴─────────┴─────────────────┘ -0 32 36 40 48 +┌────────────────────────────────────────────────────────────────┬───────────┬───────────┬───────────┬───────────┐ +│ chunk_hash (32 bytes) │ chunk_ │ unpacked │ flags │ _unused │ +│ Chunk Hash │ byte_ │ segment_ │ (4 bytes) │ (4 bytes) │ +│ │ range_ │ bytes │ │ │ +│ │ start │ (4 bytes) │ │ │ +│ │ (4 bytes) │ │ │ │ +└────────────────────────────────────────────────────────────────┴───────────┴───────────┴───────────┴───────────┘ +0 32 36 40 44 48 ``` +**Chunk flags** + +`flags` is a bitfield. One bit is currently defined: + +| Bit | Name | Meaning | +|---|---|---| +| `1 << 31` | `MDB_CHUNK_WITH_GLOBAL_DEDUP_FLAG` | This chunk is eligible for global deduplication — its hash may be used as a key into the global dedup index. | + +All other bits are reserved and MUST be zero. Readers MUST mask out undefined bits rather than comparing `flags` for equality, so that later additions stay backward compatible. + +> [!NOTE] +> Earlier revisions of this document described bytes 40-48 as a single 8-byte `_unused` field. Only bytes 44-48 are unused; bytes 40-44 are `flags`. The entry size is unchanged at 48 bytes, so byte offsets of surrounding fields are unaffected. + ### CAS Info Bookend The end of the cas info sections is marked by a bookend entry. @@ -440,12 +464,20 @@ Since the cas info section immediately follows the file info section bookend, a struct MDBShardFileFooter { version: u64, // Footer version (must be 1) file_info_offset: u64, // Offset to file info section - cas_info_offset: u64, // Offset to CAS info section - _buffer: [u8; 48], // Reserved space (48 bytes) + xorb_info_offset: u64, // Offset to CAS (xorb) info section + file_lookup_offset: u64, // Offset to the file lookup table + file_lookup_num_entry: u64, // Number of file lookup entries + xorb_lookup_offset: u64, // Offset to the xorb lookup table + xorb_lookup_num_entry: u64, // Number of xorb lookup entries + chunk_lookup_offset: u64, // Offset to the chunk lookup table + chunk_lookup_num_entry: u64, // Number of chunk lookup entries chunk_hash_hmac_key: Hash, // HMAC key for chunk hashes (32 bytes) shard_creation_timestamp: u64, // Creation time (seconds since epoch) shard_key_expiry: u64, // Expiry time (seconds since epoch) - _buffer2: [u8; 72], // Reserved space (72 bytes) + _buffer: [u64; 6], // Reserved space (48 bytes) + stored_bytes_on_disk: u64, // Accounting: bytes as stored on disk + materialized_bytes: u64, // Accounting: materialized bytes + stored_bytes: u64, // Accounting: deduplicated stored bytes footer_offset: u64, // Offset where footer starts } ``` @@ -456,33 +488,73 @@ struct MDBShardFileFooter { > Fields are not exactly to scale ```txt -┌─────────┬─────────┬─────────┬─────────────────────────────────────────────────────────────┬─────────────────────────────────────┐ -│ version │file_info│cas_info │ _buffer (reserved) │ chunk_hash_hmac_key │ -│(8 bytes)│offset │offset │ (48 bytes) │ (32 bytes) │ -│ │(8 bytes)│(8 bytes)│ │ │ -└─────────┴─────────┴─────────┴─────────────────────────────────────────────────────────────┴─────────────────────────────────────┘ -0 8 16 24 72 104 - -┌─────────┬──────────┬─────────────────────────────────────────────────────────────────────────────┬─────────┐ -│creation │shard_ │ _buffer (reserved) │footer_ │ -│timestamp│key_expiry│ (72 bytes) │offset │ -│(8 bytes)│ (8 bytes)│ │(8 bytes)│ -└─────────┴──────────┴─────────────────────────────────────────────────────────────────────────────┴─────────┘ -104 112 120 192 200 +┌───────────┬───────────┬───────────┐ +│ version │ file_info │ xorb_info │ +│ (8 bytes) │ offset │ offset │ +│ │ (8 bytes) │ (8 bytes) │ +└───────────┴───────────┴───────────┘ +0 8 16 24 + +┌───────────┬───────────┬───────────┬───────────┬───────────┬───────────┐ +│ file_ │ file_ │ xorb_ │ xorb_ │ chunk_ │ chunk_ │ +│ lookup_ │ lookup_ │ lookup_ │ lookup_ │ lookup_ │ lookup_ │ +│ offset │ num_entry │ offset │ num_entry │ offset │ num_entry │ +│ (8 bytes) │ (8 bytes) │ (8 bytes) │ (8 bytes) │ (8 bytes) │ (8 bytes) │ +└───────────┴───────────┴───────────┴───────────┴───────────┴───────────┘ +24 32 40 48 56 64 72 + +┌─────────────────────────────────────┬───────────┬────────────┐ +│ chunk_hash_hmac_key │ shard_ │ shard_ │ +│ (32 bytes) │ creation_ │ key_expiry │ +│ │ timestamp │ (8 bytes) │ +│ │ (8 bytes) │ │ +└─────────────────────────────────────┴───────────┴────────────┘ +72 104 112 120 + +┌───────────────────────────────────────────────────────┬───────────┬────────────┬───────────┬───────────┐ +│ _buffer (reserved) │ stored_ │ material- │ stored_ │ footer_ │ +│ (48 bytes) │ bytes_ │ ized_bytes │ bytes │ offset │ +│ │ on_disk │ (8 bytes) │ (8 bytes) │ (8 bytes) │ +│ │ (8 bytes) │ │ │ │ +└───────────────────────────────────────────────────────┴───────────┴────────────┴───────────┴───────────┘ +120 168 176 184 192 200 ``` **Deserialization steps**: 1. Seek to `file_size - footer_size` -2. Read all fields sequentially as u64 values +2. Read the fields sequentially in declaration order. All fields are `u64` except `chunk_hash_hmac_key`, which is a 32-byte hash, and `_buffer`, which is 48 reserved bytes. 3. Verify version equals 1 ### Use of Footer Fields -#### file_info_offset and cas_info_offset +#### file_info_offset and xorb_info_offset These offsets allow you to seek into the shard data buffer to reach these sections without deserializing linearly. +> [!NOTE] +> `xorb_info_offset` was previously called `cas_info_offset`. The name changed when xorb terminology replaced the older "CAS object" terminology; the field's position and meaning are unchanged. The section it points at is still referred to as the CAS Info Section in this document. + +#### Lookup Tables + +The six `*_lookup_*` fields describe three lookup tables that a shard MAY carry after the CAS info section, each given as a byte offset and an entry count. They let a reader resolve a hash without scanning the info sections linearly, and they are placed after the info sections so a shard can still be read incrementally without seeking to the footer first. + +Every key is a **truncated hash**: the first 8 bytes of the full 32-byte hash, read as a little-endian `u64`. Values are **indices** into the corresponding info section, not byte offsets. + +| Offset field | Count field | Entry size | Key | Value | +|---|---|---|---|---| +| `file_lookup_offset` | `file_lookup_num_entry` | 12 bytes | truncated file hash (`u64`) | index into the file info section (`u32`) | +| `xorb_lookup_offset` | `xorb_lookup_num_entry` | 12 bytes | truncated xorb hash (`u64`) | index into the CAS info section (`u32`) | +| `chunk_lookup_offset` | `chunk_lookup_num_entry` | 16 bytes | truncated chunk hash (`u64`) | xorb index (`u32`) followed by chunk index (`u32`) | + +Because keys are truncated, a match is not conclusive — two distinct hashes can share their first 8 bytes. A reader MUST confirm a hit against the full hash in the info section before relying on it. + +A count of `0` means that table is absent. A reader MUST NOT assume the tables are present and MUST be able to fall back to scanning the info sections; in particular, the tables sit between the CAS info section and the footer, so a shard serialized without its footer (as with the shard upload API body) gives a reader no way to locate them. + +#### Accounting Fields + +`stored_bytes_on_disk`, `materialized_bytes`, and `stored_bytes` are bookkeeping totals describing the data this shard references. They do not affect parsing and MAY be zero. Readers that only reconstruct files can ignore them. + #### HMAC Key Protection If `footer.chunk_hash_hmac_key` is non-zero (as a response shard from the global dedupe API), chunk hashes in the CAS Info section are protected with [HMAC](https://en.wikipedia.org/wiki/HMAC): @@ -530,11 +602,11 @@ footer = read_footer(shard) // 3. Read file info section seek(footer.file_info_offset) -file_info = read_file_info_section(shard) // until footer.cas_info_offset +file_info = read_file_info_section(shard) // until footer.xorb_info_offset // 4. Read CAS info section -seek(footer.cas_info_offset) -cas_info = read_cas_info_section(shard) // until footer.footer_offset +seek(footer.xorb_info_offset) +cas_info = read_cas_info_section(shard) // until the cas info bookend ``` ## Version Compatibility diff --git a/docs/xet/xorb.md b/docs/xet/xorb.md index 57ebd7f015..127d13c17b 100644 --- a/docs/xet/xorb.md +++ b/docs/xet/xorb.md @@ -10,7 +10,11 @@ It is advantageous to collect series of chunks in Xorbs such that they can be re Suppose a file is chunked into chunks A, B, C, D in the order ABCD. Then create a Xorb X1 with chunks A, B, C, D in this order (starting at chunk index 0), let's say this Xorb's hash is X1. Then to reconstruct the file we ask for Xorb X1 chunk range `[0, 4)`. -While there's no explicit limit on the number of chunks in a Xorb, there is a limit of 64MiB on the total size of the Xorb as serialized. +There is a limit of 64 MiB on the total size of the Xorb as serialized. The reference client additionally caps a Xorb at **8192 chunks** (`MAX_XORB_CHUNKS`), cutting the Xorb as soon as either limit is reached. + +> [!NOTE] +> The 8192-chunk cap is a producer-side limit, not part of the wire format: the serialized Xorb encodes no chunk-count limit, and the CAS server does not reject a Xorb for chunk count (only for exceeding the 64 MiB serialized size). A reader MUST NOT assume a Xorb has at most 8192 chunks; a writer SHOULD apply the cap so that Xorbs stay comparable to those produced by the reference client. + Since some chunks will get compressed, it is generally advised to collect chunks until their total uncompressed length is near 64 MiB then serialize the struct. Namely, Xorbs point to roughly 64 MiB worth of data. (Recall that the target chunk size is 64 KiB so expect roughly ~1024 chunks per Xorb). @@ -22,6 +26,24 @@ It is RECOMMENDED to pack chunks from multiple files into a Xorb if the size req ## Xorb Format A Xorb is a series of "Chunks" that is serialized according to a specific format that enables accessing chunks of ranges and builds in chunk level compression. +The chunk sequence is followed by a metadata footer (`XorbObjectInfo`) and a trailing 4-byte length, which together are how a reader locates the metadata. + +```txt +Offset 0: +┌───────────────────────────────────────────────────────┐ +│ │ +│ Chunk sequence │ ← variable size +│ (Chunk 0 .. Chunk N) │ +│ │ +├───────────────────────────────────────────────────────┤ +│ XorbObjectInfo (metadata footer) │ ← variable size +├───────────────────────────────────────────────────────┤ +│ XorbObjectInfo length: u32 (final 4 bytes) │ ← fixed size +└───────────────────────────────────────────────────────┘ +[END OF XORB] +``` + +The chunk sequence itself: ```txt ┌─────────┬─────────────────────────────────┬─────────┬─────────────────────────────────┬─────────┬─────────────────────────────────┬────────── @@ -38,6 +60,42 @@ Each chunk has an index within the Xorb it is in, starting at 0. Chunks can be addressed individually by their index but are usually addressed or fetched in range. Chunk ranges are always specified start inclusive and end exclusive i.e. `[start, end)`. +### Xorb Footer (XorbObjectInfo) + +After the last chunk, a Xorb carries a variable-length `XorbObjectInfo` metadata footer, followed by a 4-byte length. +The footer holds the Xorb hash, the hash of every chunk, and the chunk boundary offsets that make range reads possible. + +The final 4 bytes of a Xorb are the reader's entry point. They hold the length of the `XorbObjectInfo` block as a little-endian `u32`, which does **not** count itself. A reader therefore: + +1. Seeks to `end - 4` and reads `info_length`. +2. Seeks to `end - 4 - info_length` and deserializes `XorbObjectInfo` from there. + +For this reason `info_length` MUST remain the final 4 bytes of the serialized Xorb. + +The tail of the footer is fixed-size and at a known position, so a reader that only needs the chunk boundaries can seek straight to it: + +```txt +┌─────────┬──────────┬───────────┬─────────┬──────────┬─────────┐ +│ num_ │ hashes_ │ boundary_ │ nonce │ reserved │ info_ │ +│ chunks │ section_ │ section_ │ (4 B) │ (12 B) │ length │ +│ (4 B) │ offset_ │ offset_ │ │ │ (4 B) │ +│ │ from_end │ from_end │ │ │ │ +│ │ (4 B) │ (4 B) │ │ │ │ +└─────────┴──────────┴───────────┴─────────┴──────────┴─────────┘ +-32 -28 -24 -20 -16 -4 0 + (offsets relative to the end of the Xorb) +``` + +The trailing 16-byte buffer is an extensibility buffer: + +- The leading 4 bytes are a per-upload **uniqueness nonce**. The remaining 12 bytes are reserved for future use and stay zero. +- The nonce is **excluded from the Xorb hash**, which is a Merkle tree over chunk contents only. Writing a nonce therefore does not change the Xorb's content address or its storage key — only its serialized bytes. Two Xorbs with identical content can be made to serialize to distinct byte streams. +- Xorbs written before the nonce existed carry an all-zero buffer and remain valid. The buffer's size is unchanged, so the format is wire-compatible in both directions. +- Readers MUST ignore the contents of this buffer. + +> [!NOTE] +> Because the nonce does not participate in the hash, byte-identical content does not imply a byte-identical serialized Xorb. Do not treat serialized Xorb bytes as a stable identity for content; use the Xorb hash. + ## Chunk Format A chunk consists of a header followed by compressed data. The header contains metadata about the chunk, particularly the compression scheme required to know how to deserialize the chunk.