Skip to content

Test/esti lakectl large upload - #10591

Closed
itaigilo wants to merge 4 commits into
masterfrom
test/esti-lakectl-large-upload
Closed

itaigilo wants to merge 4 commits into
masterfrom
test/esti-lakectl-large-upload

Conversation

@itaigilo

@itaigilo itaigilo commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

TBD

Covers the multipart path fixed in #9283 for both pre-signed and
regular uploads. Extends the output sanitizers to handle multipart
etags (md5-N) and relative physical addresses with part suffixes.

Closes #9284
Drop the golden file and the sanitizer changes it needed. The test now
checks the uploaded size through lakectl output and the stat API, and
deletes each large object from the backing store once the subtest ends,
since deleting the repository leaves its data in place.
Look up the object's physical address in the cleanup itself, so the
large object is removed from storage even if the upload or a later
check fails.
@itaigilo itaigilo added exclude-changelog PR description should not be included in next release changelog minor-change Used for PRs that don't require issue attached labels Nov 3, 2026
@github-actions github-actions Bot added the area/testing Improvements or additions to tests label Nov 3, 2026

itaigilo commented Nov 3, 2026

Copy link
Copy Markdown
Contributor Author

The compatibility job fails in the new test's cleanup, not in the upload. Both subtests of TestLakectlFsUploadLarge hit this at esti/lakectl_test.go:204 (the adapter.Delete call):

operation error S3: DeleteObject, resolve auth scheme: resolve endpoint: endpoint rule error, Invalid region: region was not a valid DNS name.

That is the AWS SDK v2 error for an empty region. The ESTI S3 adapter builds its client with config.LoadDefaultConfig, which only picks the region up from the environment. The other S3 ESTI jobs go through bootstrap-test-lakefs, which runs aws-actions/configure-aws-credentials with aws-region: us-east-1 and so exports AWS_REGION into the job. The compatibility-checks job in .github/workflows/lakectl-compatibility-tests.yaml never runs that action and sets no region, and no TestLakectl* test used the adapter before this PR.

Suggested fix, in the "Run lakectl tests" step:

         env:
           AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
           AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
+          AWS_REGION: us-east-1
           ESTI_AWS_ACCESS_KEY_ID: ${{ secrets.ESTI_AWS_ACCESS_KEY_ID }}

An alternative that covers any future caller is to default the region in newS3Adapter (config.LoadDefaultConfig(ctx, config.WithDefaultRegion("us-east-1"))), but the workflow change matches how the other S3 jobs are configured.

Side note: since the delete failed, the two 5 GiB objects from run 1460 were left behind under s3://esti-system-testing/lakectl-compat/1460/.


Generated by Claude Code

The ESTI S3 adapter resolves its region from the environment, and the
compatibility job exported none, so deleting the large upload failed
with an invalid-region endpoint error.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016H9g6KrGzdcBdLerkWzG8S
@itaigilo itaigilo closed this Nov 7, 2026
@itaigilo
itaigilo deleted the test/esti-lakectl-large-upload branch November 7, 2026 09:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/ci area/testing Improvements or additions to tests exclude-changelog PR description should not be included in next release changelog minor-change Used for PRs that don't require issue attached

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants