Support per-backend role and destination in replication engine#2743
Support per-backend role and destination in replication engine#2743maeldonn wants to merge 3 commits into
Conversation
Hello maeldonn,My role is to assist you with the merge of this Available options
Available commands
Status report is not available. |
Codecov Report❌ Patch coverage is
Additional details and impacted files
... and 8 files with indirect coverage changes
@@ Coverage Diff @@
## development/9.4 #2743 +/- ##
===================================================
- Coverage 74.73% 74.50% -0.23%
===================================================
Files 199 199
Lines 13650 13630 -20
===================================================
- Hits 10201 10155 -46
- Misses 3439 3465 +26
Partials 10 10
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| rule => rule.Status === 'Enabled' && | ||
| entry.getObjectKey().startsWith(rule.Prefix) && | ||
| rule.Destination && | ||
| rule.Destination.StorageClass === this.site); |
There was a problem hiding this comment.
Rules.find() picks the first matching rule in the API response, but MongoQueueProcessor._updateReplicationInfo selects backends from the highest-priority rule (via .sort((a, b) => (b.priority || 0) - (a.priority || 0))). If multiple enabled rules match the same object prefix and target the same StorageClass with different Destination.Account values, and the API returns them in non-priority order, the expectedDestRole derived here may differ from the role stored in the entry, causing a spurious BadRole rejection.
— Claude Code
|
Read destination bucket and role from each backends[] entry instead of the shared top-level fields, so a single source object can be replicated to multiple CRR destinations with their own role. Legacy entries without per-backend fields keep working via top-level fallback in ObjectQueueEntry's site-aware getters; MongoQueueProcessor's oplog path now matches every applicable rule and dedups backends per the design's (site, destination, role) rule. Issue: BB-762
maeldonn
force-pushed
the
improvement/BB-762/crr-multi
branch
from
36b459a to
98c878f
Compare
|
| if (roles.length !== 2) { | ||
| log.error('expecting two roles separated by a ' + | ||
| 'comma in bucket replication configuration', | ||
| if (roles.length < 1 || roles.length > 2) { |
There was a problem hiding this comment.
Minor: roles.length < 1 is unreachable — String.split(',') always returns at least one element (''.split(',') → ['']). Could simplify to roles.length > 2.
| if (roles.length < 1 || roles.length > 2) { | |
| if (roles.length > 2) { |
— Claude Code
| "@smithy/node-http-handler": "^3.3.3", | ||
| "JSONStream": "^1.3.5", | ||
| "arsenal": "git+https://github.com/scality/arsenal#8.3.9", | ||
| "arsenal": "git+https://github.com/scality/arsenal#21b9bb33ad77d21609a690a5709a645eab1a95d7", |
There was a problem hiding this comment.
Arsenal is pinned to a raw commit SHA instead of a tag. The yarn.lock resolves this to version 8.4.2 — if the tag exists, pin to 8.4.2 for readability and consistency with other git-based deps in this file (e.g. breakbeat#v1.0.3, werelogs#8.2.2).
— Claude Code
|
maeldonn
force-pushed
the
improvement/BB-762/crr-multi
branch
from
d5a834d to
83c75ef
Compare
| "@smithy/node-http-handler": "^3.3.3", | ||
| "JSONStream": "^1.3.5", | ||
| "arsenal": "git+https://github.com/scality/arsenal#8.3.9", | ||
| "arsenal": "git+https://github.com/scality/arsenal#39c1a642d77436d871305435dcf244a64423a4b8", |
There was a problem hiding this comment.
Arsenal is pinned to a commit hash instead of a tag. The yarn.lock shows this resolves to version 8.4.3. Per project conventions, git-based deps should be pinned to a tag for readability and consistency with the other deps (breakbeat#v1.0.3, bucketclient#8.2.4, etc.).
— Claude Code
|
| // the literal two-comma role equality check. | ||
| const matchingRule = data.ReplicationConfiguration.Rules.find( | ||
| rule => rule.Status === 'Enabled' && | ||
| entry.getObjectKey().startsWith(rule.Prefix) && |
There was a problem hiding this comment.
The matchingRule lookup here uses rule.Prefix, but the replicationEnabled check on line 258 uses rule.Filter?.Prefix ?? rule.Prefix ?? ''. For V2 replication rules that use Filter.Prefix instead of top-level Prefix, rule.Prefix will be undefined, causing startsWith(undefined) to always return false. The matching rule will never be found, and the code will silently fall back to roles[1] (the legacy destination role) instead of computing the per-account substituted role.
| entry.getObjectKey().startsWith(rule.Prefix) && | |
| rule => rule.Status === 'Enabled' && | |
| entry.getObjectKey().startsWith( | |
| rule.Filter?.Prefix ?? rule.Prefix ?? '') && |
— Claude Code
| "@smithy/node-http-handler": "^3.3.3", | ||
| "JSONStream": "^1.3.5", | ||
| "arsenal": "git+https://github.com/scality/arsenal#8.3.9", | ||
| "arsenal": "git+https://github.com/scality/arsenal#39c1a642d77436d871305435dcf244a64423a4b8", |
There was a problem hiding this comment.
Arsenal is pinned to a commit hash (39c1a642d77436d871305435dcf244a64423a4b8) instead of a tag. The yarn.lock resolves this to version 8.4.3 — if that tag exists, it should be used here to follow the project's dependency pinning convention.
— Claude Code
|
2 participants
Read destination bucket and role from each backends[] entry instead of the shared top-level fields, so a single source object can be replicated to multiple CRR destinations with their own role. Legacy entries without per-backend fields keep working via top-level fallback in ObjectQueueEntry's site-aware getters; MongoQueueProcessor's oplog path now matches every applicable rule and dedups backends per the design's (site, destination, role) rule.
Issue: BB-762