Document the streaming attachment transport (JS SDK) - #563
Conversation
…t SDK - Clarified the format of `localUri` for local storage references. - Added details about the `Attachment Transport` and its role in managing remote operations. - Introduced the concept of a streaming transport for large files, including configuration examples. - Updated notes on the React Native local storage adapter requirements. - Provided additional context on the `Attachment Queue` and its lifecycle management. This update aims to improve clarity and usability for developers working with attachments.
|
Preview deployment for your docs. Learn more about Mintlify Previews.
💡 Tip: Enable Workflows to automatically generate PRs for you. |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
- Updated the description of the remote storage adapter to emphasize the use of streaming transports for large file transfers. - Removed outdated sections on the Attachment Transport, consolidating information for clarity. - Enhanced the explanation of the Attachment Queue and its role in managing attachment lifecycles. - Clarified the handling of `localUri` in the web SDK and its implications for image loading. These changes aim to improve the documentation's clarity and usability for developers working with attachments.
| - `downloadFile(attachment)` - Download file from cloud storage | ||
| - `deleteFile(attachment)` - Delete file from cloud storage | ||
|
|
||
| In the JavaScript/TypeScript SDK, apps that transfer large files can replace this adapter with a streaming transport; see [Transferring Large Files Without Buffering](#transferring-large-files-without-buffering). |
There was a problem hiding this comment.
Are we calling it TypeScript SDK anywhere else?
| In the JavaScript/TypeScript SDK, apps that transfer large files can replace this adapter with a streaming transport; see [Transferring Large Files Without Buffering](#transferring-large-files-without-buffering). | |
| In the JavaScript SDK, apps that transfer large files can replace this adapter with a streaming transport; see [Transferring Large Files Without Buffering](#transferring-large-files-without-buffering). |
This also still makes it sound like all other SDKs necessarily buffer the file, when that is not true for Kotlin and Dart.
There was a problem hiding this comment.
I have fixed this by changing every reference on the "JavaScript/TypeScript SDK" to "JavaScript SDKs" as it is consistent with the rest of the docs.
I have also made this specific part more explicit about it being JS only.
| - **Performs cleanup** - Removes archived files that are no longer needed | ||
| - **Verifies integrity** - Checks local files exist and repairs inconsistencies | ||
|
|
||
| In the JavaScript/TypeScript SDK, remote transfers buffer each file through JS memory by default. Apps that handle large files can swap in a streaming transport adapter; see [Transferring Large Files Without Buffering](#transferring-large-files-without-buffering). |
There was a problem hiding this comment.
Let's also only mention this once (it's also mentioned under attachment table, I don't think it belongs there).
There was a problem hiding this comment.
I have kept both references but reworded them to their actual context so they don't appear like repetition.
| - **Resumable transfers** - The queue retries a failed operation by calling the transport again on the next sync interval. A transport built on a resumable protocol such as [tus](https://tus.io) or S3 multipart upload can continue from the last confirmed offset instead of restarting from zero. Downloads can resume a partial file with HTTP `Range` requests | ||
| - **Encryption** - Encrypt files before upload and decrypt them after download for end-to-end encrypted attachments, without holding the whole file in memory | ||
|
|
||
| ```typescript |
There was a problem hiding this comment.
Instead of this empty snippet, linking to the relevant interface in our tsdoc reference might be better.
There was a problem hiding this comment.
This was addressed as per the comment.
|
|
||
| Throwing from any method marks the operation as failed; the queue retries it on the next sync interval, subject to your [error handler](#error-handling). |
There was a problem hiding this comment.
IMO this is obvious enough that it doesn't need to be mentioned.
| Throwing from any method marks the operation as failed; the queue retries it on the next sync interval, subject to your [error handler](#error-handling). |
There was a problem hiding this comment.
I have reworded this around retries and improved the wording. Let me know if you still think it should be removed
| // saveFileFromUri queues the upload without reading the file into memory. | ||
| // Requires a streaming-capable local storage adapter (StreamingLocalStorageAdapter: | ||
| // Node.js, Expo, or React Native FS; not available on web). | ||
| async function attachRecording(localUri: string, recordingId: string) { |
There was a problem hiding this comment.
Maybe we should add a second snippet for this (to have one with a blob and one streaming example) instead of combining it into one?
There was a problem hiding this comment.
Not sure I understood you well, but I have created a new section just below, dedicated to streaming and scoped to JS. This, however, takes it out of the <CodeGroup> and into the main content.
|
@khawarizmus in 7503b48 I added some general polish based on our docs writing standards - hopefully nothing too contentious, mainly:
I think all of these were existing things, nothing new from this PR, just noticed it now. |
…t SDK - Clarified the format of `localUri` for local storage references. - Added details about the `Attachment Transport` and its role in managing remote operations. - Introduced the concept of a streaming transport for large files, including configuration examples. - Updated notes on the React Native local storage adapter requirements. - Provided additional context on the `Attachment Queue` and its lifecycle management. This update aims to improve clarity and usability for developers working with attachments.
- Updated the description of the remote storage adapter to emphasize the use of streaming transports for large file transfers. - Removed outdated sections on the Attachment Transport, consolidating information for clarity. - Enhanced the explanation of the Attachment Queue and its role in managing attachment lifecycles. - Clarified the handling of `localUri` in the web SDK and its implications for image loading. These changes aim to improve the documentation's clarity and usability for developers working with attachments.
These changes aim to improve clarity and provide developers with better guidance on managing attachments.
2514df6 to
459d1e4
Compare
…ync-docs into attachment-transport
|
@khawarizmus In commit 2539369, I further updated the "flow" of this page, especially in regard to this new streaming feature for large files. Overall, the problem still was, that this feature was mentioned multiple times in early sections and it was unnecessarily noisy IMO. It's of course still up for discussion, so here's my reasoning (I asked Claude for some specifics, so please tell me if any of my current understanding is incorrect):
My main changes (mostly summarized by Claude):
You implemented feedback from Simon as I pushed this so just some notes on how I merged those with the above updates:
A remaining question I have is where we say "JavaScript SDKs" shouldn't we technically say just React Native and Node.js? I don't fully grok what this new feature adds for web users. |
|
@benitav the reason I initially added the section that has now been removed and why I still think the two short references at the beginning of the document are worth keeping is mainly about helping the reader build a mental model as they go. My thinking was:
From my personal experience reading docs, my brain naturally starts asking questions as I go. Knowing that another concept exists, or that the thing I'm reading is related to something else, helps me keep track of those questions even if I don't follow the reference immediately. It essentially creates a shelf and a pointer that I can come back to when I need it. I think this is useful because it lets the reader understand what they don't know yet without forcing them to learn it upfront. They can stop at the level they need, follow a reference when it becomes relevant, or jump directly to an advanced use case if that's what they're looking for. |
|
@khawarizmus I agree with that sentiment when it comes to common/core concepts that most/all readers should know/care about. But my reasoning is that with this feature those concepts is not useful to what 80/90%? (please correct me if you think I'm wrong!) of readers, hence unnecessary mental overhead and a distraction for most readers in this case. Please let me know if you think this feature has a much wider audience and then more practically, why you think it's useful to introduce those readers to the different transport setup for these large files early on - since the use case itself also feels advanced and like I tried to explain, it seems safe to just plug that in, and learn about it later, when users run into this use case? I migth just be misunderstanding the feature, or be missing something so I'm also happy to jump on a quick call to discuss if easier. |
- Added a tip for React Native users to utilize streaming transport instead of a remote storage adapter to prevent memory issues on low-end devices. - Clarified the implications of buffering large files and emphasized the need for a transport adapter for efficient file transfers. These updates aim to improve guidance for developers managing attachments in the SDK.
| ### Transferring Large Files Without Buffering | ||
|
|
||
| <Note> | ||
| This section applies to the JavaScript SDKs only. In the Dart and Kotlin SDKs, the remote storage interface is already stream-based (`Stream`/`Flow`), so transfers can avoid buffering. The Swift SDK currently receives files as `Data` and has no streaming equivalent yet. |
There was a problem hiding this comment.
| This section applies to the JavaScript SDKs only. In the Dart and Kotlin SDKs, the remote storage interface is already stream-based (`Stream`/`Flow`), so transfers can avoid buffering. The Swift SDK currently receives files as `Data` and has no streaming equivalent yet. | |
| This section applies to React Native, Expo and Node.js platforms only. In the Dart and Kotlin SDKs, the remote storage interface is already stream-based (`Stream`/`Flow`), so transfers can avoid buffering. The Swift SDK currently receives files as `Data` and has no streaming equivalent yet. |
There was a problem hiding this comment.
Can we say something about .Net and Kotlin here as well since they are also referenced on this page.
| - `ReactNativeFileSystemStorageAdapter` (`@powersync/attachments-storage-react-native`) - The transport streams with `uploadFiles`/`downloadFile` from `@dr.pogodin/react-native-fs`, uploading as a raw binary `PUT` by default | ||
| - `NodeFileSystemAdapter` (`@powersync/node`) - The transport streams with `fetch` and Node.js filesystem streams | ||
|
|
||
| The web IndexedDB adapter is not streaming-capable and cannot create a transport. |
There was a problem hiding this comment.
| The web IndexedDB adapter is not streaming-capable and cannot create a transport. |
| All three take the same options. `resolveUpload` and `resolveDownload` map an attachment to the HTTP request that transfers its bytes, typically a signed URL from your backend. `deleteFile` performs the remote delete, which is a plain remote call rather than a byte transfer. | ||
|
|
||
| <Note> | ||
| The transport API requires `@powersync/web` v2.2.0, `@powersync/react-native` v2.0.3, or `@powersync/node` v0.21.0 or later. React Native also requires `@powersync/attachments-storage-react-native` v0.1.0 or later. |
There was a problem hiding this comment.
In what cases would @powersync/web be relevant, since I thought this is not usable/available on web.
|
|
||
| ### Custom Transport Adapters | ||
|
|
||
| In the JavaScript SDKs, you can also write your own [transport adapter](#transferring-large-files-without-buffering). A custom remote storage adapter always receives the file as one full in-memory buffer. A custom transport receives the file's path instead. This makes the following possible: |
There was a problem hiding this comment.
Does this also just apply to RN and Node.js or web as well?
| ``` | ||
|
|
||
| #### Saving Files Already on Disk | ||
|
|
There was a problem hiding this comment.
| <Note> | |
| This section applies to React Native, Expo and Node.js platforms only. | |
| </Note> | |
3 participants
Documents the streaming attachment transport shipped in powersync-js powersync-ja/powersync-js#1039 (
@powersync/common@2.1.0), and fixes a few pre-existing issues in the JS examples found along the way.New content (all JavaScript/TypeScript only; other SDK tabs untouched):
AttachmentTransportAdapterinterface, and the either-remoteStorage-or-transportAdapter configuration rulecreateTransportAdapter, with a full Expo examplesaveFileFromUriexample for registering on-disk files without buffering, plusStreamingLocalStorageAdaptercoverage in the Local Storage Adapter sectionFixes:
state === 'SYNCED', butAttachmentStateis a numeric enum, so the checks never matchedProfilePhotoweb example passedlocal_uri(anindexeddb://reference) straight to<img src>, which fails; it now converts through the storage adapter to an object URL, with a note explaining the platform differenceThe content in this PR was AI-assisted using Claude Code.