Repository navigation
Conversation
Contributor
Author
Baseline reproduction run —
|
| # | Test | JIRA | Result | Oracle / Failure message |
|---|---|---|---|---|
| 1 | test_19405 |
19405 | FAILED |
ConnectionRefusedError — RGW-A mid-restart from stale process; needs clean re-run (bug confirmed on prior run: HEAD=200 GET=404 bytes=0) |
| 2 | test_19406 |
19406 | FAILED 🐛 | IBMCEPH-19406 BUG CONFIRMED: HEAD=200 GET=404 after GC |
| 3 | test_19406v |
19406v | FAILED 🐛 | IBMCEPH-19406v BUG CONFIRMED: HEAD=200 GET=404 after GC (versioning-SUSPENDED) |
| 4 | test_19407 |
19407 | SKIPPED ⏭️ | retry=500 — lock still held at crash time (window missed) |
| 5 | test_19410 |
19410 | FAILED 🐛 | IBMCEPH-19410 BUG CONFIRMED: HEAD=200 GET=404 bytes=0/41943040 (gc_tails 0→17) |
| 6 | test_19417 |
19417 | FAILED 🐛 | IBMCEPH-19417 BUG CONFIRMED: HEAD=404 GET=404 — new object wiped by racing DELETE |
| 7 | test_19412 |
19412 | FAILED 🐛 | IBMCEPH-19412 BUG CONFIRMED: HEAD=404 GET=404 — tail leaked + object wiped |
| 8 | test_19414 |
19414 | FAILED 🐛 | IBMCEPH-19414 BUG CONFIRMED: HEAD=200 GET=404 bytes=0/20971520 — CopyObject-self data loss |
| 9 | test_19416 |
19416 | FAILED 🐛 | IBMCEPH-19416 BUG CONFIRMED: HeadObject ETag≠ListObjectsV2 ETag — index stale |
| 10 | test_19407bl |
19407bl | FAILED 🐛 | IBMCEPH-19407bl BUG CONFIRMED: HEAD=200 GET=404 bytes=0/41943040 (gc_tails 0→16) |
| 11 | test_19424 |
19424 | PASSED ✅ | 200s=1 412s=7 — only 1/8 If-Match writes succeeded (correct S3 CAS behaviour) |
Summary: 9 FAILED (bugs confirmed) · 1 SKIPPED (race window missed) · 1 PASSED
Notes
- 19424 PASSED on this build — only 1 of 8 concurrent If-Match conditional writes succeeded, which is the correct behaviour. May already be fixed or the window was not hit.
- 19405 failed due to RGW-A being mid-restart from earlier stale processes — not a genuine pass. The underlying bug was confirmed in a prior run (
oracle: HEAD=200 GET=404 bytes=0). - 19407 SIGKILL variant missed the narrow race window this run (retry returned 500 = lock still held at crash). The same underlying bug was confirmed via the blocklist variant (test_19407bl).
- Fixes for these bugs are on
private-tserlin-ceph-9.1-rhel-9-test-IBMCEPH19403-patchesandprivate-tserlin-ceph-9.2-rhel-10-test-IBMCEPH19403-patches. Will re-run suite against those builds once RPMs are available.
Logs saved on cluster: /root/ibmceph-pytest/run.log · /root/ibmceph-pytest/results.xml · /root/ibmceph-pytest/RESULTS_SUMMARY.txt
viduship
changed the title
Adds a pytest-based test suite reproducing all 11 race-condition bugs
tracked under the IBMCEPH-19403 epic:
19405 Lock renewal failure + concurrent CompleteMultipartUpload → data loss
19406 RGW crash between head-write and .meta-delete → ghost object
19406v Same as 19406 on versioning-SUSPENDED bucket
19407 Failed .meta-delete + immediate retry from second RGW → data loss (SIGKILL)
19407bl Same via OSD blocklist instead of SIGKILL
19410 Lifecycle AbortMultipartUpload ignores completion lock → premature GC
19412 DeleteObject racing overwrite leaks new object tail
19414 CopyObject-to-itself racing overwrite corrupts data
19416 Stalled write lost from bucket index
19417 DeleteObject missing RADOS ID-tag guard
19424 Concurrent conditional PUTs (If-Match) not atomic
Test semantics (standard pytest):
PASSED — correct behaviour or race window not caught
FAILED — bug confirmed (pytest.fail with BUG CONFIRMED message)
SKIPPED — race precondition not met (re-run to retry)
For test_19407 (SIGKILL variant), parts are uploaded via RGW-B so
credentials remain valid across the RGW-A kill/restart cycle, avoiding
InvalidAccessKeyId errors seen on grim017.
Files added:
rgw/v2/tests/s3_swift/test_ibmceph_19403_pytest.py — test module
rgw/v2/tests/s3_swift/run_ibmceph_19403_pytest.py — cephci runner
rgw/v2/tests/s3_swift/test_ibmceph_19403.yaml — config/YAML
Resolves: IBMCEPH-19403 IBMCEPH-19405 IBMCEPH-19406 IBMCEPH-19407
IBMCEPH-19410 IBMCEPH-19412 IBMCEPH-19414 IBMCEPH-19416
IBMCEPH-19417 IBMCEPH-19424
viduship
force-pushed
the
ibmceph-19403-race-repro
branch
from
October 11, 2026 22:01
f7e4f05 to
00413b7
Compare
The SIGKILL must land after the MPU lock is released but before .meta is deleted (sub-window 3b). Killing immediately after wait_get_ok() lands in sub-window 3a (lock still held), causing retry to return 500 → SKIPPED. A 4s sleep after GET 200 is confirmed gives RGW-A time to release the lock, ensuring the immediate retry on RGW-B returns 200 and triggers the data-loss path. Ref: QE_UPDATE.md — Case 4 observed GET readable at 19.1s then SIGKILL
Replace SIGKILL-based approach with deterministic GDB injection per JIRA reproduction steps: 1. Fire CompleteMultipartUpload on RGW-A with ms_inject_delay to widen window 2. Wait for GET 200 on RGW-B (head written, lock released) 3. GDB attach to radosgw host PID, breakpoint on delete_object, return -5 4. .meta deletion fails deterministically — CMU still returns 200 5. Verify: object readable, .meta present, lock cleared, gc_tails=0 6. Retry CompleteMultipartUpload on RGW-B immediately 7. Run GC — oracle: HEAD 200 GET 404 = BUG CONFIRMED This eliminates the narrow SIGKILL timing window that caused SKIPPED on previous runs (lock still held at crash time). Resolves: IBMCEPH-19407
…isarm after GDB must attach and set the breakpoint WHILE ms_inject_delay is still armed (RGW-A stalled mid-completion). A background thread disarms the delay 3s after GDB starts, so RGW-A resumes toward delete_object with the breakpoint already set. Also increase GDB timeout to 120s. Previous attempt timed out because delay was disarmed before GDB attached, so delete_object was reached and passed before the breakpoint was in place.
Replace GDB approach (fails on stripped binary without debuginfo) with
rgw_mp_lock_inject_delay=30 set persistently via ceph config at RGW-A
startup. This injects a 30s delay between lock release and .meta delete,
giving a guaranteed sub-window 3b SIGKILL target without debuginfo.
The test now:
- Waits for GET 200 (head written, lock released)
- Verifies lock cleared (0 lockers) — skips if still in sub-window 3a
- SIGKILLs RGW-A while .meta is still present
- Retries immediately on RGW-B
- Runs GC and checks for data loss
Prereq: ceph config set client.rgw rgw_mp_lock_inject_delay 30
+ RGW-A restarted (already done on grim017)
… delete rgw_mp_lock_inject_delay delays AFTER acquiring the lock (not after releasing it), so it doesn't help open sub-window 3b. Fix: - Poll until GET 200 AND lockers=0 (lock released, head written) - Then arm ms_inject_delay_max=30/prob=1 to stall the upcoming delete_object(.meta) RADOS call - SIGKILL RGW-A while stalled = guaranteed sub-window 3b hit This is the correct sequence matching the JIRA reproduction steps.
test_19423 (new): If-Match/If-None-Match prefix comparison bug - Tests GET/PUT with exact ETag (must pass) and ETag+suffix (must 412) - Tests If-None-Match exact (must 304) and +suffix (must 200) - Bug: RGW does prefix match so etag+EXTRA passes If-Match incorrectly - Fix: ceph PR ac6c204b3a3 'rgw: compare If-Match entity tags whole' test_19424 (fixed): use versioning-ENABLED bucket per JIRA description - Bug is in the versioned write path (check → write → make-current, no guard) - Now enables versioning before concurrent writes - Adds Part 2: If-None-Match:* concurrent creates (create-once semantics) - Proper cleanup: deletes all versions before bucket delete Refs: IBMCEPH-19423, IBMCEPH-19424
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a pytest-based test suite that reproduces all 10 data-loss / integrity bugs tracked under epic IBMCEPH-19403.
Tests have been on:
Files added:
test_ibmceph_19403_pytest.py — 11 pytest test functions
run_ibmceph_19403_pytest.py — cephci-compatible runner
test_ibmceph_19403.yaml — cluster config (rio86c defaults)
Bugs covered (all confirmed UNFIXED on 20.2.2-349.el10cp):
IBMCEPH-19405 Lock renewal failure + concurrent complete -> data loss
IBMCEPH-19406 RGW crash between head-write and meta-delete -> ghost object
IBMCEPH-19406v Same as 19406 on versioning-SUSPENDED bucket
IBMCEPH-19407 Failed meta-delete + retry from second RGW -> data loss (SIGKILL)
IBMCEPH-19407bl Same as 19407 via OSD blocklist (deterministic reproduction)
IBMCEPH-19410 Lifecycle AbortMPU ignores completion lock -> premature GC
IBMCEPH-19412 DeleteObject racing overwrite deletes new data
IBMCEPH-19414 CopyObject-self racing overwrite deletes data
IBMCEPH-19416 Stalled write lost from bucket index (pending-op expiry)
IBMCEPH-19417 DeleteObject lost RADOS-level ID-tag guard
IBMCEPH-19424 8 concurrent conditional PUTs (If-Match) all succeed
pytest markers:
ibmceph_mpu — multipart completion race tests
ibmceph_race — overwrite / bucket-index race tests
ibmceph_cond — conditional write atomicity tests
ibmceph_all — all of the above
Usage:
pytest test_ibmceph_19403_pytest.py -C test_ibmceph_19403.yaml -v
pytest test_ibmceph_19403_pytest.py -C test_ibmceph_19403.yaml -m ibmceph_mpu
Fixes: https://issues.redhat.com/browse/IBMCEPH-19403