Chore/presign urls

CHANGELOG.md

@@ -3,7 +3,38 @@
 All notable changes to buckt are documented here. This project follows
 [Semantic Versioning](https://semver.org/).
 
-## [1.10.0] — unreleased
+## [1.9.2] — unreleased
+
+A backward-compatible **minor** release adding presigned (direct-download) URLs.
+Additive; no API removals.
+
+### ✨ Added
+
+- **Presigned URLs** (`Client.PresignedURL` / `PresignedURLContext`,
+  `PresignedDerivativeURL` / `…Context`, and the `domain.PresignBackend`
+  capability). Mint a time-limited URL that downloads a file (or an image
+  derivative) **directly from the storage backend**, so reads bypass the
+  application process entirely — hand the URL to a browser or CDN instead of
+  streaming bytes through your server. Implemented for the S3/R2 backend
+  (`cloud/aws`), designed as an optional capability so Azure/GCS can follow;
+  backends that can't presign (the local filesystem, or during a migration)
+  return the new `ErrUnsupported`. For an S3/R2 object the correct key form is
+  resolved first, so a URL for a migrated object doesn't 404. The web client
+  adds `GET /presign/:file_id?ttl=15m` (501 when unsupported). Presigned URLs
+  bypass buckt's auth for their lifetime — keep the TTL short.
+- **Direct (presigned) uploads — register/confirm** (`Client.PresignUpload` /
+  `FinalizeUpload`, `PresignBackend.PresignPutURL`). `PresignUpload` reserves a
+  file (returning a **stable file ID** immediately, for your app's tracking) plus
+  a presigned PUT URL the client uploads to **directly**, bypassing your process
+  on the write path; `FinalizeUpload` confirms the object landed and makes the
+  file live, firing `file.uploaded` so handlers (e.g. derivative generation) run
+  lazily. The reserved file is `pending` (new column, schema migration v7) and
+  hidden from listings until finalized. Because buckt never sees the bytes on
+  this path, these uploads are **not** deduplicated, scanned, or content-hashed.
+  Web endpoints `POST /upload/presign` and `POST /upload/finalize` (form or
+  JSON). `example/migration/s3` is a copy-paste local→S3 migration template.
+
+## [1.9.1] — unreleased
 
 A backward-compatible **minor** release adding file expiry / temp files.
 Additive; no API removals.

README.md

@@ -78,6 +78,7 @@ In fact, Buckt can use MinIO as its storage backend, allowing you to combine Min
   - [Quick Start](#quick-start)
   - [Configuration](#configuration)
     - [All Options](#all-options)
+  - [Presigned URLs](#presigned-urls)
   - [Storage Backends](#storage-backends)
     - [Local Filesystem](#local-filesystem)
     - [AWS S3](#aws-s3)
@@ -241,6 +242,51 @@ Or with options:
 
 ---
 
+## Presigned URLs
+
+Hand clients a time-limited URL that downloads a file **directly from the storage backend**, so reads don't stream through your process — the standard, CDN-friendly way to serve media (images, video, downloads).
+
+> **Integrating an app?** See the step-by-step [S3 migration + presigned URLs guide](docs/s3-migration-and-presigned-urls.md) — local→S3 cut-over, presigned downloads, and direct uploads, with Go backend and Astro/Svelte frontend snippets.
+
+```go
+  // 15-minute direct-download link for the file's bytes:
+  url, err := client.PresignedURL(fileID, 15*time.Minute)
+
+  // ...or for a generated image derivative (thumbnail, etc.):
+  thumbURL, err := client.PresignedDerivativeURL(fileID, "thumbnail", 15*time.Minute)
+```
+
+Presigning is a **cloud-backend capability**: it works with S3/R2 (`cloud/aws`) today (Azure/GCS can follow). The local filesystem backend — and a Client mid-migration — can't presign, and return `ErrUnsupported`:
+
+```go
+  url, err := client.PresignedURL(fileID, ttl)
+  if errors.Is(err, buckt.ErrUnsupported) {
+    // fall back to streaming via GetFileStream / the /serve handler
+  }
+```
+
+> A presigned URL grants access to whoever holds it — bypassing buckt's own auth — for the whole TTL, so keep the TTL short and don't log the URLs. The web client exposes `GET /presign/:file_id?ttl=15m` (returns 501 when the backend can't presign).
+
+### Direct uploads (register / confirm)
+
+The reverse direction: let the client upload **straight to the bucket** so the bytes never pass through your server. It's a two-step handshake, and the **file ID is stable from step 1** — so your app can track it immediately.
+
+```go
+  // 1. Reserve — returns a stable file ID + a presigned PUT URL:
+  fileID, uploadURL, _ := client.PresignUpload(userID, parentID, "video.mp4", "video/mp4", 15*time.Minute)
+
+  // 2. The client PUTs the bytes directly to uploadURL (browser → bucket).
+
+  // 3. Confirm — makes the file live and fires file.uploaded (derivatives, etc.):
+  client.FinalizeUpload(fileID, sizeInBytes)
+```
+
+Until you finalize, the file is **pending** and hidden from listings. Trade-off: because buckt never sees the bytes on this path, these uploads are **not deduplicated, scanned, or content-hashed**, and image derivatives are generated lazily at finalize (via the upload event). Keep bytes-through-the-backend uploads (`UploadFile`) for anything that needs those; use direct uploads for large/opaque files.
+
+Web endpoints for the frontend: `POST /upload/presign` → `{file_id, upload_url}`, then `POST /upload/finalize` (both accept form or JSON). Cloud backends only — `ErrUnsupported`/501 otherwise.
+
+---
+
 ## Storage Backends
 
 Buckt's `FileBackend` interface lets you swap storage providers without changing your application code. The cloud backends are separate Go modules, so you only pull in the SDKs you actually use.
@@ -804,6 +850,10 @@ UploadFile(userID, parentID, name, contentType string, data []byte) (string, err
 UploadFileFromReader(userID, parentID, name, contentType string, r io.Reader) (string, error)
 GetFile(fileID string) (*FileModel, error)
 GetFileStream(fileID string) (*FileModel, io.ReadCloser, error)
+PresignedURL(fileID string, ttl time.Duration) (string, error)                 // direct-download URL (cloud backends)
+PresignedDerivativeURL(fileID, name string, ttl time.Duration) (string, error) // direct URL for a derivative
+PresignUpload(userID, parentID, name, contentType string, ttl time.Duration) (fileID, uploadURL string, err error) // reserve a direct upload
+FinalizeUpload(fileID string, size int64) (string, error)                      // confirm a direct upload landed
 ListFiles(folderID string) ([]FileModel, error)
 ListFilesMetadata(folderID string) ([]FileModel, error)
 MoveFile(fileID, newParentID string) error
@@ -848,6 +898,7 @@ Branch on failures with `errors.Is` using the re-exported sentinels (also availa
 | `buckt.ErrUploadRejected` | Rejected by an upload scanner | 422 |
 | `buckt.ErrTrashBatchExceeded` | Folder delete exceeds the trash batch cap | 409 |
 | `buckt.ErrBackendUnavailable` | Backend unreachable / feature not enabled | 503 |
+| `buckt.ErrUnsupported` | Operation not supported by the active backend (e.g. presign on local) | 501 |
 
 ---
 
@@ -865,6 +916,7 @@ Branch on failures with `errors.Is` using the re-exported sentinels (also availa
 | [Azure Blob Storage](example/cloud/azure/main.go) | Azure setup |
 | [Full-featured UI](example/client/web/ui/main.go) | Metrics, dedup, derivatives, and upload events across `local`/`migrate`/`r2` modes |
 | [Headless migration](example/migration/headless/main.go) | Bulk `MigrateAll` + progress polling without the UI |
+| [Local → S3 migration](example/migration/s3/main.go) | Copy-paste template for the local→S3 swap + cut-over |
 | [Expiring / temp files](example/expiry/headless/main.go) | `SetFileTTL` → `PurgeExpired` lifecycle |
 
 ---

buckt.go

@@ -1156,6 +1156,97 @@ func (b *Client) startExpirySweeper(interval time.Duration) {
 	}()
 }
 
+/* Presigned URLs */
+
+// PresignedURL returns a time-limited URL that downloads the file's bytes
+// directly from the storage backend, so reads bypass this process entirely —
+// ideal for serving media (hand the URL to a browser/CDN instead of streaming
+// through your server). Valid for ttl.
+//
+// Only cloud backends can presign. With the local filesystem backend, or while
+// a migration is in progress, this returns ErrUnsupported. The URL grants access
+// for its whole lifetime regardless of buckt's own auth, so keep ttl short.
+func (b *Client) PresignedURL(file_id string, ttl time.Duration) (string, error) {
+	return b.PresignedURLContext(context.Background(), file_id, ttl)
+}
+
+// PresignedURLContext is PresignedURL with an explicit context.
+func (b *Client) PresignedURLContext(ctx context.Context, file_id string, ttl time.Duration) (string, error) {
+	p, ok := b.backend.(domain.PresignBackend)
+	if !ok {
+		return "", fmt.Errorf("backend %q does not support presigned URLs: %w", b.backend.Name(), ErrUnsupported)
+	}
+	file, err := b.fileService.GetFile(ctx, file_id)
+	if err != nil {
+		return "", err
+	}
+	return p.PresignGetURL(ctx, file.Path, ttl)
+}
+
+// PresignedDerivativeURL returns a time-limited direct-download URL for a
+// generated image derivative (e.g. "thumbnail"), valid for ttl. Like
+// PresignedURL it needs a cloud backend (else ErrUnsupported); the URL 404s if
+// the derivative hasn't been generated. See GenerateDerivatives.
+func (b *Client) PresignedDerivativeURL(file_id, name string, ttl time.Duration) (string, error) {
+	return b.PresignedDerivativeURLContext(context.Background(), file_id, name, ttl)
+}
+
+// PresignedDerivativeURLContext is PresignedDerivativeURL with an explicit context.
+func (b *Client) PresignedDerivativeURLContext(ctx context.Context, file_id, name string, ttl time.Duration) (string, error) {
+	p, ok := b.backend.(domain.PresignBackend)
+	if !ok {
+		return "", fmt.Errorf("backend %q does not support presigned URLs: %w", b.backend.Name(), ErrUnsupported)
+	}
+	return p.PresignGetURL(ctx, derivativeKey(file_id, name), ttl)
+}
+
+/* Direct (presigned) uploads — register / confirm */
+
+// PresignUpload reserves a file and returns its stable ID plus a presigned PUT
+// URL the client uploads the bytes to **directly** (bypassing this process on
+// the write path). The file ID is valid immediately for your app's tracking,
+// but the file stays hidden from listings until you call FinalizeUpload once the
+// client's upload finishes.
+//
+// Because buckt never sees the bytes on this path, the upload is NOT
+// deduplicated or scanned, and no content hash is computed; image derivatives
+// are generated lazily at FinalizeUpload (via the file.uploaded event). Needs a
+// cloud backend — returns ErrUnsupported on local or during a migration.
+func (b *Client) PresignUpload(user_id, parent_id, file_name, content_type string, ttl time.Duration) (fileID, uploadURL string, err error) {
+	return b.PresignUploadContext(context.Background(), user_id, parent_id, file_name, content_type, ttl)
+}
+
+// PresignUploadContext is PresignUpload with an explicit context.
+func (b *Client) PresignUploadContext(ctx context.Context, user_id, parent_id, file_name, content_type string, ttl time.Duration) (fileID, uploadURL string, err error) {
+	p, ok := b.backend.(domain.PresignBackend)
+	if !ok {
+		return "", "", fmt.Errorf("backend %q does not support presigned uploads: %w", b.backend.Name(), ErrUnsupported)
+	}
+	fileID, key, err := b.fileService.CreatePendingUpload(ctx, user_id, parent_id, file_name, content_type)
+	if err != nil {
+		return "", "", err
+	}
+	uploadURL, err = p.PresignPutURL(ctx, key, ttl)
+	if err != nil {
+		return "", "", err
+	}
+	return fileID, uploadURL, nil
+}
+
+// FinalizeUpload confirms a presigned upload completed (the object is present in
+// the backend), makes the file live (visible in listings, readable), records its
+// size, and fires the file.uploaded event so handlers such as derivative
+// generation run. Call it after the client's PUT to the URL from PresignUpload
+// succeeds. Returns the file ID.
+func (b *Client) FinalizeUpload(file_id string, size int64) (string, error) {
+	return b.FinalizeUploadContext(context.Background(), file_id, size)
+}
+
+// FinalizeUploadContext is FinalizeUpload with an explicit context.
+func (b *Client) FinalizeUploadContext(ctx context.Context, file_id string, size int64) (string, error) {
+	return b.fileService.FinalizeUpload(ctx, file_id, size)
+}
+
 /* Helper Methods */
 
 func initializeCache(conf CacheConfig, bucktLog domain.BucktLogger) (domain.CacheManager, domain.LRUCache) {

buckt_errors.go

@@ -21,4 +21,5 @@ var (
 	ErrTrashBatchExceeded = buckterr.ErrTrashBatchExceeded
 	ErrBackendUnavailable = buckterr.ErrBackendUnavailable
 	ErrUploadRejected     = buckterr.ErrUploadRejected
+	ErrUnsupported        = buckterr.ErrUnsupported
 )

buckt_migration_test.go

@@ -121,6 +121,17 @@ func (m *memBackend) count() int {
 	return len(m.objs)
 }
 
+// PresignGetURL / PresignPutURL make memBackend satisfy domain.PresignBackend so
+// the presigned-upload flow can be tested. The "URL" just encodes the key, so a
+// test can extract it and simulate the client's direct upload with a plain Put.
+func (m *memBackend) PresignGetURL(_ context.Context, key string, _ time.Duration) (string, error) {
+	return "memget://" + key, nil
+}
+
+func (m *memBackend) PresignPutURL(_ context.Context, key string, _ time.Duration) (string, error) {
+	return "memput://" + key, nil
+}
+
 func TestMigration_BulkMigrateExistingFiles(t *testing.T) {
 	dir := t.TempDir()
 	mediaDir := filepath.Join(dir, "media")

buckt_presign_test.go

@@ -0,0 +1,113 @@
+package buckt
+
+import (
+	"context"
+	"database/sql"
+	"path/filepath"
+	"strings"
+	"testing"
+	"time"
+
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+// TestPresignedURL_UnsupportedOnLocal verifies that presigning is reported as
+// unsupported (not a crash) on the local filesystem backend, which can't mint
+// direct URLs. Cloud presigning is covered in cloud/aws (no network needed).
+func TestPresignedURL_UnsupportedOnLocal(t *testing.T) {
+	c := newTestClient(t) // local backend
+	fileID, err := c.UploadFile("u1", "", "a.txt", "text/plain", []byte("x"))
+	require.NoError(t, err)
+
+	_, err = c.PresignedURL(fileID, 15*time.Minute)
+	assert.ErrorIs(t, err, ErrUnsupported, "local backend cannot presign")
+
+	_, err = c.PresignedDerivativeURL(fileID, "thumbnail", 15*time.Minute)
+	assert.ErrorIs(t, err, ErrUnsupported)
+
+	_, _, err = c.PresignUpload("u1", "", "b.txt", "text/plain", 15*time.Minute)
+	assert.ErrorIs(t, err, ErrUnsupported, "local backend cannot presign uploads")
+}
+
+// presignClient builds a Client whose backend is a presign-capable in-memory
+// backend (memBackend), so the register/confirm flow can run without a real
+// cloud store.
+func presignClient(t *testing.T) (*Client, *memBackend) {
+	t.Helper()
+	dir := t.TempDir()
+	sqlDB, err := sql.Open("sqlite3", filepath.Join(dir, "b.db"))
+	require.NoError(t, err)
+	t.Cleanup(func() { _ = sqlDB.Close() })
+
+	target := newMemBackend()
+	c, err := New(Config{
+		DB:       DBConfig{Driver: SQLite, Database: sqlDB},
+		MediaDir: filepath.Join(dir, "media"),
+		Log:      LogConfig{Silence: true},
+		Backend:  BackendConfig{Source: target},
+	})
+	require.NoError(t, err)
+	t.Cleanup(func() { _ = c.Close() })
+	return c, target
+}
+
+// TestPresignUpload_RegisterConfirmFlow drives the full direct-upload lifecycle:
+// reserve (file ID valid, but hidden from listings) → client uploads straight to
+// storage → finalize (file becomes live and readable).
+func TestPresignUpload_RegisterConfirmFlow(t *testing.T) {
+	c, target := presignClient(t)
+	const user = "u1"
+	ctx := context.Background()
+
+	folderID, err := c.NewFolder(user, "", "Docs", "")
+	require.NoError(t, err)
+
+	// A normal file so the folder has one visible entry to compare against.
+	normalID, err := c.UploadFile(user, folderID, "a.txt", "text/plain", []byte("a"))
+	require.NoError(t, err)
+
+	// Reserve a presigned upload.
+	pendingID, url, err := c.PresignUpload(user, folderID, "b.pdf", "application/pdf", 15*time.Minute)
+	require.NoError(t, err)
+	require.NotEmpty(t, pendingID)
+	require.True(t, strings.HasPrefix(url, "memput://"), "got a presigned PUT URL")
+
+	// Before finalize the reserved file is hidden from listings.
+	files, err := c.ListFiles(folderID)
+	require.NoError(t, err)
+	require.Len(t, files, 1)
+	assert.Equal(t, normalID, files[0].ID.String(), "pending upload is not listed")
+
+	// Simulate the client PUTting the bytes directly to storage.
+	key := strings.TrimPrefix(url, "memput://")
+	body := []byte("PDF-BYTES")
+	require.NoError(t, target.Put(ctx, key, body))
+
+	// Finalize — the file becomes live.
+	gotID, err := c.FinalizeUpload(pendingID, int64(len(body)))
+	require.NoError(t, err)
+	assert.Equal(t, pendingID, gotID)
+
+	// Now it's listed and readable, with the right size.
+	files, err = c.ListFiles(folderID)
+	require.NoError(t, err)
+	assert.Len(t, files, 2, "finalized file now appears in listings")
+
+	f, err := c.GetFile(pendingID)
+	require.NoError(t, err)
+	assert.Equal(t, body, f.Data)
+	assert.False(t, f.Pending)
+	assert.Equal(t, int64(len(body)), f.Size)
+}
+
+// TestFinalizeUpload_RejectsMissingObject verifies finalize fails if the client
+// never actually uploaded the object (nothing at the reserved key).
+func TestFinalizeUpload_RejectsMissingObject(t *testing.T) {
+	c, _ := presignClient(t)
+	pendingID, _, err := c.PresignUpload("u1", "", "never.pdf", "application/pdf", 15*time.Minute)
+	require.NoError(t, err)
+
+	_, err = c.FinalizeUpload(pendingID, 10)
+	assert.ErrorIs(t, err, ErrNotFound, "finalize refuses a file whose object never landed")
+}