EBS snapshots: IllegalMonitorStateException in EbsStatusManagerService.pollAndWait (fixed silently in 1.32.0) + non-idempotent createSnapshot cause rate-limit storms and orphan AWS snapshots

[mateo-moon]

Summary

Two related defects make EBS snapshots effectively unusable:

1. (≤1.31.x, incl. 1.30.4) Every EBS snapshot flow crashes the controller-side cleanup with IllegalMonitorStateException, aborting the "Clean up in-progress snapshots" step. The snapshot is then retried, and because of (2) each retry mints new AWS snapshots until AWS rate-limits the volume. This was silently fixed on the way to v1.32.0 by commit 2bfea15af ("Add @nullable where needed") — the fix is buried in an annotation sweep and never appears in any changelog as a bug fix.
2. (all versions incl. current master) EbsTargetProvider.createSnapshot() is not idempotent and the abort path never cleans up AWS snapshots it already created — so any retry storm (whatever its trigger) leaks orphan EBS snapshots and deterministically hits SnapshotCreationPerVolumeRateExceeded.

Defect 1: pollAndWait waits on a monitor it doesn't hold (≤1.31.x)

controller/src/main/java/com/linbit/linstor/core/ebs/EbsStatusManagerService.java at v1.30.4:

synchronized (queue)
{
    pollStatus = offer(new PollConfig());
    ...
    while (keepRunning && !pollStatus.answerReceived && remainingTimeout > 0)
    {
        pollStatus.wait(remainingTimeout);   // <-- monitor of pollStatus NOT held -> IllegalMonitorStateException
        ...

CtrlSnapshotCrtApiCallHandler.removeInProgressSnapshotsInTransaction() → pollEbs() → pollAndWait() throws on every EBS snapshot:

Category: RuntimeException
Class name: IllegalMonitorStateException
Error message: current thread is not owner
Error context: ... *__checkpoint ? Clean up in-progress snapshots

Consequence: the cleanup transaction aborts, the snapshot definition is never marked SUCCESSFUL, the flow retries, and each retry re-issues ec2:CreateSnapshot. Within seconds AWS answers:

The maximum per volume CreateSnapshot request rate has been exceeded.
(Service: AmazonEC2; Status Code: 400; Error Code: SnapshotCreationPerVolumeRateExceeded)

(AWS enforces a minimum of ~15s between CreateSnapshot calls per volume — see the AWS EBS team's guidance relayed in aws/aws-sdk-go#1284.)

In a single afternoon of testing we accumulated ~20 orphaned AWS snapshots from this loop (each retry created snapshots on both replica volumes; the aborts never deleted them).

The fix (synchronized (pollStatus) { pollStatus.wait(...); }) is present since v1.32.0 via commit 2bfea15af — could you note it in release-notes/errata? Users are effectively pinned to ≤1.30.4 for EBS anyway (see #506), which is exactly the version range where this crash lives.

Defect 2: createSnapshot non-idempotent + no abort cleanup (still on master)

satellite/src/main/java/com/linbit/linstor/layer/storage/ebs/EbsTargetProvider.java#createSnapshot() unconditionally calls client.createSnapshot(...); the snapshot ID is only recorded (setEbsSnapId) after waitUntilSnapshotCreated returns. If anything fails between the AWS call and that persist (rate limit on the other replica volume, the defect-1 crash, controller restart), the next device-manager cycle re-issues CreateSnapshot for the same volume — duplicating snapshots and triggering the per-volume rate limit — and the already-created AWS snapshot is never deleted on abort.

Suggested fix: before creating, look up an existing/in-flight AWS snapshot by the LinstorID tag the provider already writes (TAG_KEY_LINSTOR_ID = the snap LV identifier) and adopt it instead of re-creating; on abort, delete AWS snapshots that were created but not adopted. I'm happy to open a PR if that direction is acceptable.

Environment

Observed live on linstor-server 1.30.4 (Kubernetes / piraeus-operator v2.10.7, CSI snapshots via linstor-csi, arm64); defect 2 confirmed by inspection on current master.

Related

Part of a set of four native-EBS issues: #505 (DUPLICATE_UNAME), #506 (no resource dispatch to EBS targets), #507 (snapshot rate-limit storm), #508 (/dev/sdz attach collision).