-
Notifications
You must be signed in to change notification settings - Fork 19
Document the streaming attachment transport (JS SDK) #563
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
11a13a6
993cf4e
f83fb4f
7503b48
87eec6b
baf075d
110444f
459d1e4
2539369
4f915df
8dd346d
853ff63
0a0bbc8
9c8b7e6
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,6 +1,6 @@ | ||
| --- | ||
| title: "Attachments / Files" | ||
| description: "Sync file attachments like images and PDFs without storing them in the database, using an offline-first queue for uploads and downloads." | ||
| description: "Sync file attachments like images and PDFs without storing them in the database, using a local queue that uploads and downloads files in the background." | ||
| --- | ||
|
|
||
| ## Introduction | ||
|
|
@@ -15,7 +15,7 @@ Instead, PowerSync uses a **metadata + storage provider pattern**: sync small me | |
|
|
||
| - **Optimal performance** - Database stays small and fast | ||
| - **Automatic queue management** - Background uploads/downloads with retry logic | ||
| - **Offline-first support** - Local files available immediately, sync happens in background | ||
| - **Works offline** - Local files available immediately, sync happens in background | ||
| - **Cache management** - Automatic cleanup of unused files | ||
| - **Platform flexibility** - Works across web, mobile, and desktop | ||
|
|
||
|
|
@@ -69,12 +69,12 @@ Most demo applications use Supabase Storage as the storage provider, but the pat | |
|
|
||
| ### Attachment Table | ||
|
|
||
| The **Attachment Table** is a local-only table that stores metadata about each file. It's not synced through PowerSync's Sync Streams/Rules - instead, it's managed entirely by the attachment queue on each device. | ||
| The **Attachment Table** is a local-only table that stores metadata about each file. It's not synced through PowerSync's Sync Streams - Instead, it's managed entirely by the attachment queue on each device. | ||
|
|
||
| **Metadata stored:** | ||
| - `id` - Unique attachment identifier (UUID) | ||
| - `filename` - File name with extension (e.g., `photo-123.jpg`) | ||
| - `localUri` - Path to file in local storage | ||
| - `localUri` - Reference to the file in local storage. The format is platform-specific: a file path on native platforms and Node.js, or an internal `indexeddb://` reference on web | ||
| - `size` - File size in bytes | ||
| - `mediaType` - MIME type (e.g., `image/jpeg`) | ||
| - `state` - Current sync state (see states above) | ||
|
|
@@ -89,15 +89,21 @@ The **Attachment Table** is a local-only table that stores metadata about each f | |
|
|
||
| ### Remote Storage Adapter | ||
|
|
||
| The **Remote Storage Adapter** is an interface you implement to connect PowerSync with your cloud storage provider. It's completely platform-agnostic - Implementations can use S3, Supabase Storage, Cloudflare R2, Azure Blob, or even IPFS. | ||
| The **Remote Storage Adapter** is an interface you implement to connect PowerSync with your cloud storage provider. It's completely platform-agnostic - implementations can use S3, Supabase Storage, Cloudflare R2, Azure Blob, or even IPFS. | ||
|
|
||
| **Interface methods:** | ||
| - `uploadFile(fileData, attachment)` - Upload file to cloud storage | ||
| - `downloadFile(attachment)` - Download file from cloud storage | ||
| - `deleteFile(attachment)` - Delete file from cloud storage | ||
|
|
||
| **Common pattern:** | ||
| For security reasons, client-side implementations should use **signed URLs** | ||
| In the JavaScript SDKs, this adapter receives the full file as one in-memory buffer. | ||
|
|
||
| <Tip> | ||
| On React Native, use a streaming transport instead of a remote storage adapter. You cannot control your users' devices or file sizes, and buffering large files can exhaust memory on low-end devices. Node.js supports the same setup. See [Transferring Large Files Without Buffering](#transferring-large-files-without-buffering). | ||
| </Tip> | ||
|
|
||
| For security reasons, client-side implementations should use signed URLs: | ||
|
|
||
| 1. Request a signed upload/download URL from your backend | ||
| 2. Your backend validates permissions and generates a temporary URL | ||
| 3. Client uploads/downloads directly to storage using the signed URL | ||
|
|
@@ -145,51 +151,11 @@ The `watchAttachments` queries are reactive and execute whenever the watched tab | |
|
|
||
| There are a few scenarios you might encounter: | ||
|
|
||
| **Single Attachment Type** | ||
|
|
||
| For a single attachment type, you watch one table. For example, if users have profile photos: | ||
|
|
||
| ```sql | ||
| SELECT photo_id FROM users WHERE photo_id IS NOT NULL | ||
| ``` | ||
|
|
||
| **Multiple Attachment Types - Single Queue** | ||
|
|
||
| You can watch multiple attachment types using a single queue by combining queries with SQL `UNION` or `UNION ALL`. This allows you to monitor attachments across different tables (e.g., `users.photo_id`, `documents.document_id`, `videos.video_id`) in one queue. Each attachment type may have different file extensions, which can be handled in the query by selecting the extension from your data model or using type-specific defaults. | ||
|
|
||
| For example: | ||
|
|
||
| ```sql | ||
| SELECT photo_id as id, photo_file_extension as file_extension | ||
| FROM users | ||
| WHERE photo_id IS NOT NULL | ||
|
|
||
| UNION ALL | ||
| - **Single attachment type** - Watch one table. For example, if users have profile photos: `SELECT photo_id FROM users WHERE photo_id IS NOT NULL` | ||
| - **Multiple attachment types, single queue** - Combine queries with SQL `UNION ALL` to watch attachments across different tables (e.g., `users.photo_id`, `documents.document_id`) in one queue | ||
| - **Multiple attachment types, multiple queues** - Create a separate queue per attachment type. Each queue watches its own table(s) with a simpler query, allowing independent configuration, at the cost of some extra memory | ||
|
|
||
| SELECT document_id as id, document_file_extension as file_extension | ||
| FROM documents | ||
| WHERE document_id IS NOT NULL | ||
|
|
||
| UNION ALL | ||
|
|
||
| SELECT video_id as id, video_file_extension as file_extension | ||
| FROM videos | ||
| WHERE video_id IS NOT NULL | ||
| ``` | ||
|
|
||
| Use `UNION ALL` when you want to include all rows (including duplicates), or `UNION` when you want to automatically deduplicate results. For attachment watching, `UNION ALL` is typically preferred since attachment IDs should already be unique. | ||
|
|
||
| <Note> | ||
| The UNION query executes whenever any of the watched tables change, which may have higher database overhead compared to watching a single table. Implementation examples are shown in the [Initialize Attachment Queue](#initialize-attachment-queue) section below. | ||
| </Note> | ||
|
|
||
| **Multiple Attachment Types - Multiple Queues** | ||
|
|
||
| Alternatively, you can create separate queues for different attachment types. Each queue watches its own specific table(s) with simpler queries, allowing for independent configuration and management. | ||
|
|
||
| <Note> | ||
| Multiple queues may use more memory, but each queue watches simpler queries. Implementation examples are shown in the [Initialize Attachment Queue](#initialize-attachment-queue) section below. | ||
| </Note> | ||
| Implementation examples for all three are shown in the [Initialize Attachment Queue](#initialize-attachment-queue) section below. | ||
|
|
||
| ## Implementation Guide | ||
|
|
||
|
|
@@ -845,9 +811,11 @@ The `watchAttachments` callback is crucial - it tells the queue which files your | |
|
|
||
| #### Watching Multiple Attachment Types | ||
|
|
||
| When watching multiple attachment types, you need to provide the `fileExtension` for each attachment. You can store this in your data model tables or derive it from other fields. Here are examples for both patterns: | ||
| When watching multiple attachment types, you need to provide the `fileExtension` for each attachment. You can store this in your data model tables or derive it from other fields. | ||
|
|
||
| **Single Queue with UNION ALL** | ||
|
|
||
| **Pattern 2: Single Queue with UNION** | ||
| Combining queries with `UNION ALL` lets one queue watch attachments across different tables. Use `UNION ALL` rather than `UNION`: attachment IDs should already be unique, so deduplication is unnecessary. The combined query executes whenever any of the watched tables change, which may have higher database overhead than watching a single table. | ||
|
|
||
| <CodeGroup> | ||
|
|
||
|
|
@@ -1073,7 +1041,9 @@ internal sealed class AttachmentRef | |
|
|
||
| </CodeGroup> | ||
|
|
||
| **Pattern 3: Multiple Queues** | ||
| **Multiple Queues** | ||
|
|
||
| Alternatively, create separate queues for different attachment types. Each queue watches its own table(s) with a simpler query, allowing independent configuration and management, at the cost of some extra memory. | ||
|
|
||
| <CodeGroup> | ||
|
|
||
|
|
@@ -1483,11 +1453,15 @@ async Task UploadProfilePhotoAsync(Stream imageStream, string currentUserId) | |
| The `updateHook` parameter is the recommended way to link attachments to your data model. It runs in the same database transaction, ensuring data consistency. | ||
| </Info> | ||
|
|
||
| On React Native and Node.js, files already on disk (such as camera captures or recordings) can be queued without reading them into memory; see [Saving Files Already on Disk](#saving-files-already-on-disk). | ||
|
|
||
| ### Download/Access an Attachment | ||
|
|
||
| <CodeGroup> | ||
|
|
||
| ```typescript JavaScript/TypeScript | ||
| import { AttachmentState } from '@powersync/web'; | ||
|
|
||
| // Downloads happen automatically when watchAttachments references a file | ||
|
|
||
| async function getProfilePhotoUri(userId: string): Promise<string | null> { | ||
|
|
@@ -1509,42 +1483,63 @@ async function getProfilePhotoUri(userId: string): Promise<string | null> { | |
| return null; | ||
| } | ||
|
|
||
| if (attachment.state === 'SYNCED' && attachment.local_uri) { | ||
| if (attachment.state === AttachmentState.SYNCED && attachment.local_uri) { | ||
| return attachment.local_uri; | ||
| } | ||
|
|
||
| return null; | ||
| } | ||
|
|
||
| // Example: Display image in React with watch query | ||
| // Example: display the image in React on web. On web, local_uri is an | ||
| // internal indexeddb:// reference, not a URL the browser can load, so | ||
| // passing it straight to <img src> fails with net::ERR_UNKNOWN_URL_SCHEME. | ||
| // Read the file through the local storage adapter and convert it to an | ||
| // object URL instead, as below. On React Native and Node.js, local_uri is | ||
| // a real file path that can be used directly | ||
| // (e.g. <Image source={{ uri: localUri }} /> in React Native). | ||
| function ProfilePhoto({ userId }: { userId: string }) { | ||
| const [photoUri, setPhotoUri] = useState<string | null>(null); | ||
| const [photoUrl, setPhotoUrl] = useState<string | null>(null); | ||
|
|
||
| useEffect(() => { | ||
| let objectUrl: string | null = null; | ||
|
|
||
| const watch = db.watch( | ||
| `SELECT a.local_uri, a.state | ||
| `SELECT a.local_uri, a.media_type, a.state | ||
| FROM users u | ||
| LEFT JOIN attachments a ON a.id = u.photo_id | ||
| WHERE u.id = ?`, | ||
| [userId], | ||
| { | ||
| onResult: (result) => { | ||
| onResult: async (result) => { | ||
| const row = result.rows?._array[0]; | ||
| if (row?.state === 'SYNCED' && row?.local_uri) { | ||
| setPhotoUri(row.local_uri); | ||
| if (row?.state === AttachmentState.SYNCED && row?.local_uri) { | ||
| const buffer = await localStorage.readFile(row.local_uri); | ||
| const nextUrl = URL.createObjectURL( | ||
| new Blob([buffer], { type: row.media_type ?? 'image/jpeg' }) | ||
| ); | ||
| if (objectUrl) { | ||
| URL.revokeObjectURL(objectUrl); | ||
| } | ||
| objectUrl = nextUrl; | ||
| setPhotoUrl(nextUrl); | ||
| } | ||
| } | ||
| } | ||
| ); | ||
|
|
||
| return () => watch.close(); | ||
| return () => { | ||
| watch.close(); | ||
| if (objectUrl) { | ||
| URL.revokeObjectURL(objectUrl); | ||
| } | ||
| }; | ||
| }, [userId]); | ||
|
|
||
| if (!photoUri) { | ||
| if (!photoUrl) { | ||
| return <div>Loading photo...</div>; | ||
| } | ||
|
|
||
| return <img src={photoUri} alt="Profile" />; | ||
| return <img src={photoUrl} alt="Profile" />; | ||
| } | ||
| ``` | ||
|
|
||
|
|
@@ -2151,6 +2146,101 @@ var queue = new AttachmentQueue(new AttachmentQueueOptions | |
| ``` | ||
| </CodeGroup> | ||
|
|
||
| ### Transferring Large Files Without Buffering | ||
|
|
||
| <Note> | ||
| This section applies to React Native, Expo and Node.js platforms only. In the Dart and Kotlin SDKs, the remote storage interface is already stream-based (`Stream`/`Flow`), so transfers can avoid buffering. The Swift SDK currently receives files as `Data` and has no streaming equivalent yet. | ||
| </Note> | ||
|
|
||
| By default, the queue transfers files by buffering them through JS memory: the entire file is read into an `ArrayBuffer` before it is handed to the remote storage adapter, and the reverse for downloads. This works well for small files but limits the practical attachment size, particularly in React Native, where a large video can exhaust the JS heap on lower-end devices. | ||
|
|
||
| To stream instead, configure the queue with a transport adapter in place of the remote storage adapter (you provide one or the other, not both; TypeScript enforces this). A transport owns all remote operations through the three methods of the [`AttachmentTransportAdapter`](https://powersync-ja.github.io/powersync-js/common/interfaces/AttachmentTransportAdapter.html) interface: | ||
|
|
||
| - `upload(attachment)` - Transfer the file at `attachment.localUri` to remote storage | ||
| - `download(attachment)` - Fetch the remote file into `attachment.localUri` (the queue assigns the destination path before the call) | ||
| - `delete(attachment)` - Remove the file from remote storage | ||
|
|
||
| You usually don't implement these methods yourself. The streaming-capable local storage adapters each create a ready-made transport through their `createTransportAdapter` method: | ||
|
|
||
| - `ExpoFileSystemStorageAdapter` (`@powersync/attachments-storage-react-native`) - The transport streams with Expo's native `File.upload`/`File.downloadFileAsync`. Using the transport requires Expo 56 or later; using only the storage adapter requires Expo 54 | ||
| - `ReactNativeFileSystemStorageAdapter` (`@powersync/attachments-storage-react-native`) - The transport streams with `uploadFiles`/`downloadFile` from `@dr.pogodin/react-native-fs`, uploading as a raw binary `PUT` by default | ||
| - `NodeFileSystemAdapter` (`@powersync/node`) - The transport streams with `fetch` and Node.js filesystem streams | ||
|
|
||
|
|
||
| All three take the same options. `resolveUpload` and `resolveDownload` map an attachment to the HTTP request that transfers its bytes, typically a signed URL from your backend. `deleteFile` performs the remote delete, which is a plain remote call rather than a byte transfer. | ||
|
|
||
| <Note> | ||
| The transport API requires `@powersync/web` v2.2.0, `@powersync/react-native` v2.0.3, or `@powersync/node` v0.21.0 or later. React Native also requires `@powersync/attachments-storage-react-native` v0.1.0 or later. | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. In what cases would
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. A custom transport adapter can be used for other use cases, for example, resumable downloads/uploads. The ones we offer out of the box (built-in) are for the sole purpose of streaming files, and that does not apply to web |
||
| </Note> | ||
|
|
||
| ```typescript | ||
| import { AttachmentQueue } from '@powersync/react-native'; | ||
| import { ExpoFileSystemStorageAdapter } from '@powersync/attachments-storage-react-native'; | ||
|
|
||
| const localStorage = new ExpoFileSystemStorageAdapter(); | ||
|
|
||
| // Streams bytes natively and owns upload/download/delete. No remoteStorage needed. | ||
| const transportAdapter = localStorage.createTransportAdapter({ | ||
| resolveUpload: async (attachment) => ({ | ||
| url: await getSignedUploadUrl(attachment.filename), // from your backend | ||
| mimeType: attachment.mediaType ?? 'application/octet-stream' | ||
| }), | ||
| resolveDownload: async (attachment) => ({ | ||
| url: await getSignedDownloadUrl(attachment.filename) | ||
| }), | ||
| deleteFile: async (attachment) => { | ||
| await deleteFromStorage(attachment.filename); // your backend or storage SDK call | ||
| } | ||
| }); | ||
|
|
||
| const attachmentQueue = new AttachmentQueue({ | ||
| db, | ||
| localStorage, | ||
| transportAdapter, // owns all remote operations; used in place of remoteStorage | ||
| watchAttachments: (onUpdate) => { | ||
| // Same as in Initialize Attachment Queue | ||
| } | ||
| }); | ||
|
|
||
| await attachmentQueue.startSync(); | ||
| ``` | ||
|
|
||
| #### Saving Files Already on Disk | ||
|
|
||
|
khawarizmus marked this conversation as resolved.
|
||
| <Note> | ||
| This section applies to React Native, Expo and Node.js platforms only. | ||
| </Note> | ||
|
|
||
| For files your app produces on disk (camera captures, recordings, exports), `saveFileFromUri` queues the upload without reading the file into memory. `saveFile` would read the file into an `ArrayBuffer` just to write it back to disk; `saveFileFromUri` moves it into managed storage instead. This requires a streaming-capable local storage adapter: those adapters implement the `StreamingLocalStorageAdapter` subinterface, which adds `moveFile(sourceUri, targetUri)`. Combined with a transport adapter, the file is saved and uploaded without ever passing through JS memory: | ||
|
|
||
| ```typescript | ||
| async function attachRecording(localUri: string, recordingId: string) { | ||
| return attachmentQueue.saveFileFromUri({ | ||
| localUri, // path to the existing file | ||
| fileExtension: 'm4a', | ||
| mediaType: 'audio/mp4', | ||
| updateHook: async (tx, attachment) => { | ||
| await tx.execute( | ||
| 'UPDATE recordings SET audio_id = ? WHERE id = ?', | ||
| [attachment.id, recordingId] | ||
| ); | ||
| } | ||
| }); | ||
| } | ||
| ``` | ||
|
|
||
| ### Custom Transport Adapters | ||
|
|
||
| In the JavaScript SDKs, you can also write your own [transport adapter](#transferring-large-files-without-buffering). A custom remote storage adapter always receives the file as one full in-memory buffer. A custom transport receives the file's path instead. This makes the following possible: | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Does this also just apply to RN and Node.js or web as well?
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I believe it does apply to web as well. |
||
|
|
||
| - **Buffer-free transfers** - Let a native package transfer directly between the file system and the network, bypassing JS entirely, as the built-in transports do | ||
| - **Resumable transfers** - The queue retries a failed operation by calling the transport again on the next sync interval. A transport built on a resumable protocol such as [tus](https://tus.io) or S3 multipart upload can continue from the last confirmed offset instead of restarting from zero. Downloads can resume a partial file with HTTP `Range` requests | ||
| - **Encryption** - Encrypt files before upload and decrypt them after download for end-to-end encrypted attachments, without holding the whole file in memory | ||
|
|
||
| To build your own transport, implement the [`AttachmentTransportAdapter`](https://powersync-ja.github.io/powersync-js/common/interfaces/AttachmentTransportAdapter.html) interface. It has three methods: `upload(attachment)`, `download(attachment)`, and `delete(attachment)`. For a working reference, see the built-in [`NodeFileSystemTransportAdapter`](https://github.com/powersync-ja/powersync-js/blob/main/packages/node/src/attachments/NodeFileSystemTransportAdapter.ts), which streams with `fetch` and Node.js filesystem streams. | ||
|
|
||
| The queue retries failed operations on the next sync interval, subject to your [error handler](#error-handling). | ||
|
|
||
| ### Custom Storage Adapters | ||
|
|
||
| The following is an example of how to implement a custom storage adapter for IPFS: | ||
|
|
@@ -2370,24 +2460,20 @@ This method does the following: | |
| await attachmentQueue.verifyAttachments(); | ||
| ``` | ||
|
|
||
| ```dart Flutter | ||
| Coming soon, need to expose the function publicly | ||
| ``` | ||
|
|
||
| ```swift Swift | ||
| try await attachmentQueue.waitForInit() | ||
| ``` | ||
|
|
||
| ```kotlin Kotlin | ||
| Coming soon, need to expose the function publicly | ||
| ``` | ||
|
|
||
| ```csharp .NET | ||
| await queue.VerifyAttachmentsAsync(); | ||
| ``` | ||
|
|
||
| </CodeGroup> | ||
|
|
||
| <Note> | ||
| In the Flutter and Kotlin SDKs, this method is not yet exposed publicly. It still runs automatically during `startSync()`. | ||
| </Note> | ||
|
|
||
| ### Cache Management | ||
|
|
||
| Control archived file retention: | ||
|
|
@@ -2452,11 +2538,11 @@ await queue.ExpireCacheAsync(); | |
|
|
||
| </CodeGroup> | ||
|
|
||
| ### Offline-First Considerations | ||
| ### Offline Behavior | ||
|
|
||
| The attachment queue is designed for offline-first apps: | ||
| The attachment queue keeps working in poor or no network conditions: | ||
|
|
||
| - **Local-first operations** - Files are saved locally immediately, synced later | ||
| - **Local saves** - Files are saved locally immediately, synced later | ||
| - **Automatic retry** - Failed uploads/downloads retry when connection returns | ||
| - **Queue persistence** - Queue state survives app restarts | ||
| - **Conflict-free** - Files are immutable, identified by UUID | ||
|
|
||
Uh oh!
There was an error while loading. Please reload this page.