Skip to content

CLDSRV-901: Add functional test coverage for ListObjectsV2 optional object attributes#6174

Merged
bert-e merged 5 commits into
development/9.2from
improvement/CLDSRV-901-listobjectsv2-optional-attributes-tests
May 22, 2026
Merged

CLDSRV-901: Add functional test coverage for ListObjectsV2 optional object attributes#6174
bert-e merged 5 commits into
development/9.2from
improvement/CLDSRV-901-listobjectsv2-optional-attributes-tests

Conversation

@dvasilas
Copy link
Copy Markdown
Contributor

This PR adds functional test coverage for x-amz-optional-object-attributes header on ListObjectsV2 and ListObjectVersions (these tests will also run in scality/Integration).

This feature has been implemented for Artesca. For context see requirements and design.

It also includes two fixes:

  • If the request included RestoreStatus multiple times, then the response included multiple <RestoreStatus> blocks. The fix is to deduplicate <RestoreStatus> in the response (requests with multiple RestoreStatus are still allowed).
  • Delete markers included optional attributes. The fix is to omit optional attributes from delete markers (this matches AWS behavior).

Adds scality:ListBucketOptionalObjectAttributes

NODE_AUTH_TOKEN is set to a dummy value as a workaround.
Vault 7.89.0 image ships a .npmrc with ${NODE_AUTH_TOKEN}.
Yarn parses the .npmrc on `yarn start` and refuses to launch with
"Failed to replace env in config: ${NODE_AUTH_TOKEN}" when the
variable is unset.
This is temporary; a future version of Vault will remove this.
@bert-e
Copy link
Copy Markdown
Contributor

bert-e commented May 21, 2026

Hello dvasilas,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Available options
name description privileged authored
/after_pull_request Wait for the given pull request id to be merged before continuing with the current one.
/bypass_author_approval Bypass the pull request author's approval
/bypass_build_status Bypass the build and test status
/bypass_commit_size Bypass the check on the size of the changeset TBA
/bypass_incompatible_branch Bypass the check on the source branch prefix
/bypass_jira_check Bypass the Jira issue check
/bypass_peer_approval Bypass the pull request peers' approval
/bypass_leader_approval Bypass the pull request leaders' approval
/approve Instruct Bert-E that the author has approved the pull request. ✍️
/create_pull_requests Allow the creation of integration pull requests.
/create_integration_branches Allow the creation of integration branches.
/no_octopus Prevent Wall-E from doing any octopus merge and use multiple consecutive merge instead
/unanimity Change review acceptance criteria from one reviewer at least to all reviewers
/wait Instruct Bert-E not to run until further notice.
Available commands
name description privileged
/help Print Bert-E's manual in the pull request.
/status Print Bert-E's current status in the pull request TBA
/clear Remove all comments from Bert-E from the history TBA
/retry Re-start a fresh build TBA
/build Re-start a fresh build TBA
/force_reset Delete integration branches & pull requests, and restart merge process from the beginning.
/reset Try to remove integration branches unless there are commits on them which do not appear on the source branch.

Status report is not available.

@bert-e
Copy link
Copy Markdown
Contributor

bert-e commented May 21, 2026

Incorrect fix version

The Fix Version/s in issue CLDSRV-901 contains:

  • None

Considering where you are trying to merge, I ignored possible hotfix versions and I expected to find:

  • 9.2.41

  • 9.3.8

  • 9.4.0

Please check the Fix Version/s of CLDSRV-901, or the target
branch of this pull request.

@bert-e
Copy link
Copy Markdown
Contributor

bert-e commented May 21, 2026

Request integration branches

Waiting for integration branch creation to be requested by the user.

To request integration branches, please comment on this pull request with the following command:

/create_integration_branches

Alternatively, the /approve and /create_pull_requests commands will automatically
create the integration branches.

@claude
Copy link
Copy Markdown

claude Bot commented May 21, 2026

  • The outer listObjectsV2WithOptionalAttributes helper (line 499) and the pre-existing inner one (line 1022 inside describeBypass) have different signatures and return types (outer returns { data, parsedXml, rawXml }, inner returns response.data). The inner definition shadows the outer for all tests in the describeBypass block. Anyone adding tests there will silently get the old, limited helper. Consider removing the inner definition and updating the few tests that use it to use the outer helper instead.

    Review by Claude Code

@DarkIsDude DarkIsDude requested review from a team, DarkIsDude and maeldonn May 21, 2026 07:59
Comment thread lib/api/bucketGet.js Outdated
@codecov
Copy link
Copy Markdown

codecov Bot commented May 21, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 84.50%. Comparing base (d535e47) to head (ea65b8e).
⚠️ Report is 5 commits behind head on development/9.2.
✅ All tests successful. No failed tests found.

Additional details and impacted files

Impacted file tree graph

Files with missing lines Coverage Δ
lib/api/bucketGet.js 96.12% <100.00%> (+3.27%) ⬆️

... and 2 files with indirect coverage changes

@@                 Coverage Diff                 @@
##           development/9.2    #6174      +/-   ##
===================================================
+ Coverage            84.44%   84.50%   +0.06%     
===================================================
  Files                  204      204              
  Lines                13216    13217       +1     
===================================================
+ Hits                 11160    11169       +9     
+ Misses                2056     2048       -8     
Flag Coverage Δ
file-ft-tests 68.17% <100.00%> (+0.13%) ⬆️
kmip-ft-tests 28.41% <0.00%> (-0.01%) ⬇️
mongo-v0-ft-tests 69.41% <100.00%> (+0.21%) ⬆️
mongo-v1-ft-tests 69.40% <100.00%> (+0.19%) ⬆️
multiple-backend 35.68% <50.00%> (-0.01%) ⬇️
sur-tests 35.99% <50.00%> (+0.02%) ⬆️
sur-tests-inflights 37.80% <50.00%> (-0.01%) ⬇️
unit 70.34% <100.00%> (+0.04%) ⬆️
utapi-v2-tests 34.69% <50.00%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@claude
Copy link
Copy Markdown

claude Bot commented May 21, 2026

LGTM

The two fixes are correct:
- Using Set for optionalAttributes properly deduplicates RestoreStatus (the inner userMetadata Set already handled user-metadata dedup, but RestoreStatus XML was pushed directly in the loop, so duplicate tokens caused duplicate <RestoreStatus> blocks).
- The delete-marker guard in processVersions brings it in line with processMasterVersions which already skips delete markers entirely (line 230).

Downstream consumers of listParams.optionalAttributes (metadata backend) don't inspect the field, so the Array-to-Set change is safe. The shadowed helper function inside describeBypass preserves compatibility with the existing vault-dependent tests.

Review by Claude Code

@maeldonn maeldonn self-requested a review May 21, 2026 09:10
Copy link
Copy Markdown
Contributor

@maeldonn maeldonn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

- ENABLE_LOCAL_CACHE=true
- REDIS_HOST=0.0.0.0
- REDIS_PORT=6379
# TODO: drop NODE_AUTH_TOKEN when VAULT-721 is done
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should mention that in the JIRA also to make sure the develop who will resolve the VAULT-721, will remove it from here 🙏. Why waiting to do it ?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment thread lib/api/bucketGet.js
);
?? []
);
for (const attr of optionalAttributes) {
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are we sure this match the AWS behaviour ? Of course no duplicate blocks, but should we reject the request ?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Checked with multiple RestoreStatus. AWS does not reject the request

aws s3api list-objects-v2 --bucket  cldsrv-probe-optattr-manual --optional-object-attributes RestoreStatus RestoreStatus --no-cli-pager
{
    "Contents": [
        {
            "Key": "obj1",
            "LastModified": "2026-05-21T10:15:25+00:00",
            "ETag": "\"74e11c9bddafedf3ed7cd8bea153e7c8\"",
            "ChecksumAlgorithm": [
                "CRC64NVME"
            ],
            "ChecksumType": "FULL_OBJECT",
            "Size": 19,
            "StorageClass": "STANDARD"
        }
    ],
    "RequestCharged": null,
    "Prefix": ""
}

Side note: We have a divergence with AWS. AWS does not include RestoreStatus in the response for STANDARD objects (but I didn't want to change this in this PR because it will break existing behavior in Artesca).

dvasilas added 4 commits May 22, 2026 10:26
If the request included RestoresStatus multiple times,
then the response included multiple <RestoreStatus> blocks.
Delete markers in a listing included <RestoreStatus> and <x-amz-meta-*>
elements.
@dvasilas dvasilas force-pushed the improvement/CLDSRV-901-listobjectsv2-optional-attributes-tests branch from eebfbd9 to ea65b8e Compare May 22, 2026 07:26
@dvasilas
Copy link
Copy Markdown
Contributor Author

/approve

@bert-e
Copy link
Copy Markdown
Contributor

bert-e commented May 22, 2026

Conflict

A conflict has been raised during the creation of
integration branch w/9.3/improvement/CLDSRV-901-listobjectsv2-optional-attributes-tests with contents from improvement/CLDSRV-901-listobjectsv2-optional-attributes-tests
and development/9.3.

I have not created the integration branch.

Here are the steps to resolve this conflict:

 git fetch
 git checkout -B w/9.3/improvement/CLDSRV-901-listobjectsv2-optional-attributes-tests origin/development/9.3
 git merge origin/improvement/CLDSRV-901-listobjectsv2-optional-attributes-tests
 # <intense conflict resolution>
 git commit
 git push -u origin w/9.3/improvement/CLDSRV-901-listobjectsv2-optional-attributes-tests

The following options are set: approve

@claude
Copy link
Copy Markdown

claude Bot commented May 22, 2026

LGTM

Review by Claude Code

@bert-e
Copy link
Copy Markdown
Contributor

bert-e commented May 22, 2026

Integration data created

I have created the integration data for the additional destination branches.

The following branches will NOT be impacted:

  • development/7.10
  • development/7.4
  • development/7.70
  • development/8.8
  • development/9.0
  • development/9.1

You can set option create_pull_requests if you need me to create
integration pull requests in addition to integration branches, with:

@bert-e create_pull_requests

The following options are set: approve

@bert-e
Copy link
Copy Markdown
Contributor

bert-e commented May 22, 2026

I have successfully merged the changeset of this pull request
into targetted development branches:

  • ✔️ development/9.2

  • ✔️ development/9.3

  • ✔️ development/9.4

The following branches have NOT changed:

  • development/7.10
  • development/7.4
  • development/7.70
  • development/8.8
  • development/9.0
  • development/9.1

Please check the status of the associated issue CLDSRV-901.

Goodbye dvasilas.

The following options are set: approve

@bert-e bert-e merged commit ea65b8e into development/9.2 May 22, 2026
54 of 55 checks passed
@bert-e bert-e deleted the improvement/CLDSRV-901-listobjectsv2-optional-attributes-tests branch May 22, 2026 08:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants