Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
43 commits
Select commit Hold shift + click to select a range
2e54806
feat: add CAS file composition APIs for range writes
XciD Mar 16, 2026
7c602c6
fix: prevent data corruption when two dirty regions produce identical…
XciD Mar 16, 2026
53d8742
refactor: improve code quality in CAS client
XciD Mar 16, 2026
44774c2
fix: correct append, truncation, and hash dedup in upload_ranges
XciD Mar 16, 2026
8da6741
fix: update WASM cleaner for new finalize() signature
XciD Mar 16, 2026
32183f7
refactor: improve code quality from review feedback
XciD Mar 16, 2026
269cb35
refactor: stream boundary data directly to cleaner without buffering
XciD Mar 16, 2026
b4984bf
fix: validate dirty_ranges preconditions at runtime, not just debug
XciD Mar 16, 2026
e0db8b9
refactor: improve readability and error handling in range_upload
XciD Mar 17, 2026
9f9ce99
fix: read truncation boundary bytes from CAS instead of staging
XciD Mar 19, 2026
cf4275d
refactor: replace ReadSeek with per-range AsyncRead in upload_ranges
XciD Mar 19, 2026
fc3e41a
fix: clamp CAS prefix/suffix reads to original_size
XciD Mar 19, 2026
dee0643
fix: correct already_covered check in assert_range_edit test helper
XciD Mar 20, 2026
840b6e9
chore: merge main, resolve conflicts
XciD Mar 20, 2026
61a3003
fix: adapt to main API changes in tests
XciD Mar 20, 2026
79a1ef3
fix: reject append with uncovered gap beyond original_size
XciD Mar 20, 2026
e055a58
fix: use associated function in map_err closure
XciD Mar 20, 2026
e9a3232
chore: merge main, resolve conflicts
XciD May 1, 2026
78de688
feat: adapt Client::get_file_chunk_hashes to xetcas multi-range API
XciD May 1, 2026
a9d9cfe
refactor: drop ChunkHashList composition, use MerkleHashSubtree::merge
XciD May 1, 2026
e4c2b36
refactor(range_upload): cleanup pass
XciD May 1, 2026
29c38aa
fix(range_upload): handle mid-edit+append, empty original, empty result
XciD May 1, 2026
2b572dc
refactor(range_upload): address PR feedback from seanses
XciD May 1, 2026
5657765
feat(range_upload): support resize edits (insert / delete / arbitrary…
XciD May 2, 2026
d6cb7a9
refactor(file_cleaner): split chunks-returning finish into finish_wit…
XciD May 2, 2026
87b9061
fix(range_upload): promote response-shape invariants to runtime errors
XciD May 2, 2026
a8c8d01
fix(range_upload): error if edits not all assigned to a window
XciD May 2, 2026
45f5937
fix(range_upload): consume gap_verification from FileChunkHashesResponse
XciD May 6, 2026
adb25b1
fix(range_upload): always emit verification section in composed shard
XciD May 6, 2026
46e1be3
fix(range_upload): address PR review comments
XciD May 6, 2026
c41c195
fix(range_upload): avoid orphan window MDB entries in session shard
XciD May 6, 2026
c180ca6
refactor(cas_types): remove unnecessary serde(default) on gap_verific…
XciD May 6, 2026
678609a
refactor(range_upload): extract compose_mdb, remove dead conditional,…
XciD May 7, 2026
16022bd
fix(ci): pin WASM nightly to 2026-05-05 to fix wasm-bindgen __heap_ba…
XciD May 7, 2026
e3cf4de
Revert "fix(ci): pin WASM nightly to 2026-05-05 to fix wasm-bindgen _…
XciD May 7, 2026
a1c8b19
Merge remote-tracking branch 'origin/main' into feat/file-chunk-hashe…
XciD May 7, 2026
954ba6f
refactor(range_upload): address review nits from #717
XciD May 18, 2026
2ebcfcc
docs(range_upload): link boundary-case comment to exercising tests
XciD May 18, 2026
624ba7c
refactor(file_cleaner): use mut self instead of this in finish_inner
XciD May 21, 2026
3fd592e
refactor(file_upload_session): drop unnecessary destructure+rewrap of…
XciD May 21, 2026
54e0498
Added stress testing; use of next_stable_chunk_boundary logic (#845)
May 21, 2026
cbee472
Merge remote-tracking branch 'origin/main' into feat/file-chunk-hashe…
XciD May 21, 2026
2f4cee4
Merge remote-tracking branch 'origin/feat/file-chunk-hashes-and-compo…
XciD May 21, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions api_changes/update_260424_next_stable_chunk_boundary.md
Original file line number Diff line number Diff line change
@@ -1,15 +1,17 @@
This update adds a new public deduplication helper for computing restart-safe chunk boundaries from existing chunk boundary metadata.

What changed
- Added `xet_data::deduplication::next_stable_chunk_boundary(starting_position, chunk_boundaries) -> Option<usize>`.
- Re-exported it from `xet_data::deduplication` so downstream crates can use it directly.
- Added `next_stable_chunk_boundary(starting_position, chunk_boundaries) -> Option<usize>`.
- Canonical implementation lives in `xet_core_structures::xorb_object::constants` (alongside the chunk-size constants it uses).
- Re-exported from `xet_data::deduplication` for convenience.
- The function scans forward from `starting_position` and returns the next chunk boundary that satisfies the stable-boundary condition:
- two consecutive chunk sizes in `[2 * min_chunk, max_chunk - min_chunk)`,
- where `min_chunk` and `max_chunk` are derived from chunking constants.

Why this matters
- Callers that already have chunk-boundary metadata can locate a stable resume boundary without re-reading file bytes.
- This enables deterministic alignment behavior for resumed/partial workflows that need chunk boundaries robust to prefix changes.
- The server-side `build_file_chunk_hashes_response` now extends dirty ranges to stable chunk boundaries before building windows, so that the client's rechunking around each dirty window is guaranteed to converge to the same chunk boundaries as the original file.

Usage notes
- `chunk_boundaries` should be monotonically increasing chunk-end offsets produced by the same chunking configuration.
Expand Down
3 changes: 2 additions & 1 deletion wasm/hf_xet_wasm/src/wasm_file_cleaner.rs
Original file line number Diff line number Diff line change
Expand Up @@ -184,7 +184,8 @@ impl SingleFileCleaner {
let metadata_ext = FileMetadataExt::new(sha256);

// Now finish the deduplication process.
let (file_hash, remaining_file_data, deduplication_metrics) = self.dedup_manager.finalize(Some(metadata_ext));
let (file_hash, _chunk_hashes, remaining_file_data, deduplication_metrics) =
self.dedup_manager.finalize(Some(metadata_ext));

// Let's check some things that should be invariants
{
Expand Down
Loading
Loading