feat(s3): Add S3-compatible storage backend for Tigris/AWS#3
Open
perigrin wants to merge 8 commits intomagnet-mainfrom
Open
feat(s3): Add S3-compatible storage backend for Tigris/AWS#3perigrin wants to merge 8 commits intomagnet-mainfrom
perigrin wants to merge 8 commits intomagnet-mainfrom
Conversation
Implements Convos::Util::S3 module with sign_request function for cryptographic signing of S3-compatible API requests (AWS, Tigris, etc.). Features: - AWS4-HMAC-SHA256 signing algorithm - Canonical request building with sorted headers - SHA256 payload hashing - HMAC-SHA256 signing key derivation chain - Support for GET/PUT/POST methods - Query parameter handling - Flexible region support (including 'auto' for Tigris) Tests cover: - GET request signing - PUT request with JSON body - Query parameter signing
- Auto-inject Host header from URL if not provided by caller - Sort query parameter values (not just keys) per AWS spec - Add POD documentation with SYNOPSIS and parameter docs
- Extends Backend::File to inherit local log handling
- save_object_p: PUT JSON to S3
- load_object_p: GET from S3, returns {} on 404
- delete_object_p: DELETE from S3
- Uses Convos::Util::S3 for AWS4 signing
- Configurable via CONVOS_S3_* env vars
- Includes mocked UA tests
Implement S3 LIST operations to support user and connection discovery:
- Add _s3_list_p helper method to query S3 with prefix/delimiter
- Parses XML response using Mojo::DOM
- Returns both object keys and common prefixes
- Add users_p method to list all users
- Lists directories under users/ prefix
- Loads each user.json file
- Sorts by registration date and email (matches File backend)
- Add connections_p method to list user connections
- Lists directories under users/{email}/ prefix
- Loads each connection.json file
- Removes state field (matching File backend behavior)
- Add comprehensive tests with XML mocking for all operations
This completes Component 3 of the S3 backend implementation.
Implements files_p method that lists files from the user's upload directory in S3, matching the File backend's pagination behavior. Files are identified by .json metadata files with corresponding .data content files. Note: File content (.data) remains on local filesystem for now, consistent with the hybrid approach for message logs.
Adds integration test suite that runs against a real S3-compatible endpoint. Tests are skipped unless CONVOS_S3_TEST_BUCKET, CONVOS_S3_KEY, and CONVOS_S3_SECRET environment variables are set. Tests cover: - Save and load user objects - List users - Save and load connections - List connections - Delete operations - Empty file listing
The is_success method was calling shift twice, which broke the mock. Also fixed URL query param regex to accept URL-encoded slashes.
In async sub functions, returning Mojo::Promise->reject() wraps the promise incorrectly. Use die() to properly reject the promise.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Convos::Util::S3- AWS Signature Version 4 signing utility for S3-compatible APIsConvos::Core::Backend::S3- Storage backend extending File backend to store objects in S3/TigrisArchitecture
Backend::S3extendsBackend::Filevia inheritance:save_object_p,load_object_p,delete_object_p→ S3users_p,connections_p,files_p→ S3 LIST operationsmessages_p,notifications_p→ local files (append-heavy, different solution needed)Configuration
CONVOS_BACKEND=Convos::Core::Backend::S3 CONVOS_S3_ENDPOINT=https://fly.storage.tigris.dev CONVOS_S3_BUCKET=my-bucket CONVOS_S3_KEY=tid_xxxx CONVOS_S3_SECRET=tsec_xxxx CONVOS_S3_REGION=auto CONVOS_HOME=/data/convos # still needed for logsStats
Test Plan
t/util-s3.t)t/backend-s3.t)t/backend-s3-integration.t- requires credentials)