Skip to content

feat(rs-client): Add automatic resumable uploads - #700

Open
lcian wants to merge 27 commits into
mainfrom
lcian/feat/rust-automatic-resumable-uploads
Open

lcian wants to merge 27 commits into
mainfrom
lcian/feat/rust-automatic-resumable-uploads

Conversation

@lcian

@lcian lcian commented Oct 21, 2026 •

Copy link
Copy Markdown
Member

Changelog Entry

  • Session::put now uses resumable uploads automatically for buffers and paths to regular files of 256 MiB or more, and resends only what the server hasn't stored after a failed write
    • Change the threshold with Usecase::with_resumable_threshold_bytes or per request with .resumable_threshold_bytes(...); None disables automatic resumable uploads.
    • Set ClientBuilder::spool_dir to choose where payloads compressed on the fly are spooled (defaults to the system temp dir).
    • The resumable-upload-api feature only gates the manual resumable upload API. This new behavior is enabled regardless of that feature flag.

Details

Session::put now uses resumable uploads automatically for eligible uploads, matching the Python client (#667). An upload is eligible if:

  • It's a buffer or a path to a regular file (put, put_path)
  • Its input size, before client compression, is at least the resumable threshold (default 256 MiB)
  • It's sent standalone. Uploads queued through many() keep their existing batch or direct routing and ignore the threshold.

Open file handles (put_file), streams, generic readers, non-regular paths and inputs of unknown size use a direct upload.
The API and behavior mirrors the Python client:

let usecase = Usecase::from_static("debug_files")
    .with_resumable_threshold_bytes(Some(256 * 1024 * 1024)); // None disables; Some(0) includes every supported payload

session.put(payload).resumable_threshold_bytes(None).send().await?; // per-request override

ClientBuilder::new(url).spool_dir("/var/tmp/objectstore"); // optional

Path and reader lifetime. Each attempt opens an independent reader. Source paths must remain readable, unchanged, and must not be renamed, removed, or replaced until the upload completes. Compressed payloads spool to named temporary files and retries reopen them. The HTTP layer may retain old readers after the upload returns or is dropped, delaying handle closure and temporary-storage reclamation.

For reviewers

  • The Error variants ResumableUploadUnavailable, UploadOffsetMismatch and the chunk-granularity variant are no longer feature-gated. Since Error is #[non_exhaustive] (feat(rs-client)!: Mark Error as non-exhaustive #706), this is additive.
  • Most of the logic is in put.rs (routing), put/payload.rs (preparation and readers) and put/automatic.rs (recovery loop).

Closes FS-561

Resume eligible buffer and file uploads from authoritative server progress, with configurable thresholds and temporary-file compression. Keep automatic uploads available without the manual protocol feature.
@codecov

This comment has been minimized.

@lcian lcian changed the title feat(rust): Add automatic resumable uploads feat(rust-client): Add automatic resumable uploads Oct 21, 2026
lcian and others added 3 commits October 21, 2026 14:48
Separate payload preparation and reader lifetime from protocol recovery. Stop and join each request reader before returning its result, report local IO errors directly, and keep progress probes in a dedicated retry loop.
Unify direct fallbacks, make sizing optional, and limit resumables to standalone uploads.
# Conflicts:
#	Cargo.lock
#	clients/rust/src/error.rs

Co-Authored-By: Lorenzo Cian <17258265+lcian@users.noreply.github.com>
@sentry-junior sentry-junior Bot changed the title feat(rust-client): Add automatic resumable uploads feat(rs-client): Add automatic resumable uploads Oct 22, 2026
@github-actions

github-actions Bot commented Oct 22, 2026 •

Copy link
Copy Markdown

Semver Impact of This PR

This PR lands in these workspaces:

  • clients/rust: 🟡 Minor (new features)
📋 Changelog Preview

This is how your changes will appear in the changelog.
Entries from this PR are highlighted with a left border (blockquote style).


New Features

  • Session::put now uses resumable uploads automatically for buffers and paths to regular files of 256 MiB or more, and resends only what the server hasn't stored after a failed write by lcian with sentry-junior in #700
    • Change the threshold with Usecase::with_resumable_threshold_bytes or per request with .resumable_threshold_bytes(...); None disables automatic resumable uploads.
    • Set ClientBuilder::spool_dir to choose where payloads compressed on the fly are spooled (defaults to the system temp dir).
    • The resumable-upload-api feature only gates the manual resumable upload API. This new behavior is enabled regardless of that feature flag.

🤖 This preview updates automatically when you update the PR.

sentry-junior Bot and others added 7 commits October 23, 2026 11:30
…ated docs

Co-Authored-By: Lorenzo Cian <17258265+lcian@users.noreply.github.com>
Keep both test dependencies and support gzip when spooling resumable payloads.
Limit automatic resumables to buffers and stable paths.

Reopen source paths and named spool files per attempt to remove reader coordination.
@lcian

This comment has been minimized.

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Comment thread clients/rust/src/put.rs Outdated
@lcian

This comment has been minimized.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 3f64519. Configure here.

Comment thread clients/rust/src/put/payload.rs Outdated
@lcian
lcian marked this pull request as ready for review October 28, 2026 16:03
@lcian
lcian requested a review from a team as a code owner October 28, 2026 16:03
@linear-code

This comment has been minimized.

Comment thread clients/rust/src/resumable/mod.rs Outdated
Comment thread clients/rust/src/resumable/api.rs Outdated
Comment thread clients/rust/src/resumable/mod.rs Outdated
Comment thread clients/rust/src/client.rs Outdated
name: Arc::new(UsecaseName::new(name)?),
compression: Some(Compression::Zstd),
expiration_policy: ExpirationPolicy::default(),
resumable_threshold_bytes: Some(256 * 1024 * 1024),

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This means automatic resumable uploads are enabled by default. This is something we should call out explicitly in the changelog.

Let's check with Ingest team what their thoughts on the behavior are.

Comment thread clients/rust/src/put/payload.rs Outdated
Comment thread clients/rust/src/put/payload.rs Outdated
Comment thread clients/rust/src/put/payload.rs Outdated
Comment thread clients/rust/src/put.rs
Comment thread clients/rust/src/put/automatic.rs Outdated
@lcian
lcian marked this pull request as draft October 29, 2026 12:25
@lcian
lcian marked this pull request as ready for review October 29, 2026 12:26
lcian added a commit that referenced this pull request Oct 29, 2026
Fixes the changelog preview added in #737, which fails at startup on
every PR ([run on
#700](https://github.com/getsentry/objectstore/actions/runs/40873702019):
"likely failed because of a workflow file issue").

Craft's reusable preview job requests `contents: read`, `pull-requests:
write` and `statuses: write`. A called workflow can't get more
permissions than its caller, and our caller only granted the first two.
This adds `statuses: write`, matching the old per-client preview
workflows.

The preview runs from the base branch (`pull_request_target`), so it
starts working on other PRs once this merges.

#skip-changelog

---------

Co-authored-by: sentry-junior[bot] <264270552+sentry-junior[bot]@users.noreply.github.com>
Co-authored-by: Lorenzo Cian <17258265+lcian@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants