Skip to content

fix(s3): notifications cleanup fails when bucket is already deleted - #39552

Open
abhiramshaji-cyber wants to merge 3 commits into
aws:mainfrom
abhiramshaji-cyber:fix/s3-notifications-no-such-bucket
Open

abhiramshaji-cyber wants to merge 3 commits into
aws:mainfrom
abhiramshaji-cyber:fix/s3-notifications-no-such-bucket

Conversation

@abhiramshaji-cyber

@abhiramshaji-cyber abhiramshaji-cyber commented Oct 29, 2026 •

Copy link
Copy Markdown

Issue # (if applicable)

Closes #35352.

Deleting a stack whose bucket has event notifications could get stuck forever once the bucket was gone; now the cleanup step sees the bucket is gone and lets the deletion finish.

What this code is for

bucket.addEventNotification() adds a Custom::S3BucketNotifications resource backed by a small Python Lambda, index.py in notifications-resource-handler. CloudFormation calls its handler() on Create, Update and Delete. On Delete it writes an empty (or, for an imported bucket, an external only) configuration with put_bucket_notification_configuration, and for an imported bucket it first reads the current one with get_bucket_notification_configuration.

Before this PR

A developer runs cdk destroy on a stack with a bucket and bucket.addEventNotification(...).

  1. The bucket is already gone when the Delete runs, for example after an earlier failed delete attempt (as @rittneje describes) or after it was removed outside the stack.
  2. CloudFormation sends a Delete request to handler().
  3. handler() calls put_bucket_notification_configuration (or get_bucket_notification_configuration first, for an imported bucket).
  4. S3 answers with a ClientError whose code is NoSuchBucket.
  5. The except Exception block reports FAILED for every error, including this one.
  6. The stack goes to DELETE_FAILED, and every retry repeats steps 2 to 5, so the stack can never be deleted.

After this PR

Same developer, same cdk destroy. Same as above through step 4.

  1. The except block now checks: is this a Delete request, and is the error a ClientError with code NoSuchBucket?
  2. Yes, so it prints Bucket no longer exists, nothing to clean up and reports SUCCESS.
  3. The stack finishes deleting.
  4. If the error is anything else, for example AccessDenied, or if NoSuchBucket comes back on a Create or Update, the handler reports FAILED exactly as before.

The diagram

flowchart LR

classDef box fill:#eef1ff,stroke:#4c5bd4,color:#111827
classDef bad fill:#ffd6d6,stroke:#cc0000,color:#111827,stroke-width:3px
classDef good fill:#c8f0c8,stroke:#008800,color:#111827,stroke-width:3px

subgraph BEFORE["BEFORE: every delete and every retry fails"]
direction TB
  B1["cdk destroy, bucket already gone"] --> B2["handler() gets RequestType Delete"]
  B2 --> B3["S3 returns ClientError NoSuchBucket"]
  B3 --> B4{"except Exception: any error?"}
  B4 -->|"NoSuchBucket counts as an error"| B5["FAILED, stack stuck in DELETE_FAILED"]
  B4 -->|"no error"| B6["SUCCESS"]
end

subgraph AFTER["AFTER: the delete finishes in one try"]
direction TB
  A1["cdk destroy, bucket already gone"] --> A2["handler() gets RequestType Delete"]
  A2 --> A3["S3 returns ClientError NoSuchBucket"]
  A3 --> A4{"THE CHANGE: Delete and code is NoSuchBucket?"}
  A4 -->|"yes, nothing left to clean"| A5["SUCCESS, stack deletes"]
  A4 -->|"no, e.g. AccessDenied"| A6["FAILED, same as before"]
end

B1 ~~~ A1

class B1,B2,B3,B6 box
class A1,A2,A3,A6 box
class B4,B5 bad
class A4,A5 good

linkStyle 3 stroke:#cc0000,stroke-width:5px
linkStyle 8 stroke:#008800,stroke-width:5px

style BEFORE fill:#fff5f5,stroke:#cc0000,color:#111827
style AFTER fill:#f3fff3,stroke:#008800,color:#111827
Loading

The change

The fix is in the except block of handler():

if event.get("RequestType") == "Delete" and is_missing_bucket(e):
  print("Bucket no longer exists, nothing to clean up")
else:
  # unchanged: log, FAILED, error message

is_missing_bucket checks the structured error, isinstance(error, ClientError) and error.response["Error"]["Code"] == "NoSuchBucket", not the message text. Because the check sits in handler(), it covers both managed buckets and imported buckets (Managed: false), whether the error comes from the read or the write. All destination types (SNS, SQS and Lambda) go through this one handler, so all of them are covered.

What changes for a user

  • cdk destroy on a stack with bucket notifications now completes when the bucket is already gone, for SNS, SQS and Lambda destinations alike.
  • A stack that is already stuck in DELETE_FAILED cannot receive the new handler through an update, so it still needs aws cloudformation delete-stack --retain-resources for the notifications resource, or the FORCE_DELETE_STACK deletion mode.
  • Real errors still fail the delete: AccessDenied, throttling, anything that is not NoSuchBucket.
  • NoSuchBucket on Create or Update still fails, because there it means a real problem.
  • Create and Update behaviour is otherwise unchanged.

Reason for this change

Thank you @aramfe for the clear report, and @pahud for confirming it and for describing the fix: let the Delete in NotificationsResourceHandler succeed on NoSuchBucket. @rittneje's point shaped this PR too: fixing the order between the two resources would not be enough, since CloudFormation still deletes the bucket when the custom resource fails, and the retry then fails forever. So the handler has to accept that the bucket is gone.

I also read the two earlier attempts, #35388 and #35481, which went in the same direction. In this PR:

  • The handler change is 2 files.
  • The decision uses the ClientError code only, and only on Delete, so every other failure is still reported.

On #35481, @TheRealAmazonKendra suggested fixing the dependency between the bucket and the notifications resource instead, and asked which destination types are affected. That resource already depends on the bucket, so CloudFormation normally deletes it first; the trouble starts when the bucket is gone anyway, and @rittneje's comment explains why ordering alone cannot help then. Every destination type (SNS, SQS, Lambda) shares this one handler, so they are all affected and all covered here. I would really value your view on whether this handler change is enough on its own, or whether you would like a dependency change as well.

Description of changes

See "The change" above. One new helper, is_missing_bucket, and one if in the except block of handler(). No change to the TypeScript constructs.

Describe any new or updated permissions being added

None.

Description of how you validated changes

  • New MissingBucketTest in test_index.py. It covers managed and imported buckets, NoSuchBucket raised from the read or the write, AccessDenied on Delete, and NoSuchBucket on Create and Update. The three Delete cases fail on main and pass with this change. All 25 handler tests pass on Python 3.9 and 3.13.
  • I could not deploy the integration tests myself, so I did not run them against an AWS account. Could a maintainer add pr/needs-integration-tests-deployment when you have a moment?
  • This PR has no new integ test file, because a bucket that is missing during Delete cannot be set up in an integ test, and the existing integ tests already deploy this handler. If that sounds reasonable to you, could a maintainer also add pr-linter/exempt-integ-test? I am glad to add a test instead if you see a way to set it up.

Checklist


By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

During stack deletion CloudFormation can remove the bucket before the
Custom::S3BucketNotifications resource runs its Delete. The handler then
calls S3 on a bucket that no longer exists, gets NoSuchBucket, reports
FAILED, and every retry fails the same way, so the stack can never be
deleted.

On Delete, a ClientError whose code is NoSuchBucket now reports SUCCESS,
because there is nothing left to clean up. This covers managed and
imported buckets, whether the error comes from the read or the write.
Every other error, and NoSuchBucket on Create or Update, still fails.

Closes aws#35352
The notifications handler is inlined into each template, so changing it
changes the inline code and the template hash in every snapshot that
uses it. Only those values change here, nothing else in the templates.
@github-actions github-actions Bot added beginning-contributor [Pilot] contributed between 0-2 PRs to the CDK bug This issue is a bug. effort/medium Medium work item – several days of effort p1 labels Oct 29, 2026

@aws-cdk-automation aws-cdk-automation left a comment •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The pull request linter fails with the following errors:

❌ Fixes must contain a change to an integration test file and the resulting snapshot.

If you believe this pull request should receive an exemption, please comment and provide a justification. A comment requesting an exemption should contain the text Exemption Request. Additionally, if clarification is needed, add Clarification Request to a comment.

✅ A exemption request has been requested. Please wait for a maintainer's review.

@abhiramshaji-cyber

Copy link
Copy Markdown
Author

Exemption Request: a bucket that is already gone during Delete cannot be set up in an integ test, and I cannot deploy to regenerate snapshots. The fix is covered by MissingBucketTest in test_index.py (test_delete_succeeds_when_bucket_is_gone, test_delete_fails_on_other_errors, test_create_and_update_fail_when_bucket_is_gone). Would you like an integ test, or is an exemption fine? I am happy to add one.

@aws-cdk-automation aws-cdk-automation added the pr-linter/exemption-requested The contributor has requested an exemption to the PR Linter feedback. label Oct 29, 2026
CONTRIBUTING.md requires integ snapshots to come from a real deployment, not from cdk-integ --dry-run or a manual update. These snapshots were built locally, so they are removed and left for a maintainer to regenerate with a deployed run.

This branch was previously deployed

1 inactive deployment
automation — acd01aa4 Deployed Oct 29, 2026 by abhiramshaji-cyber via validate-pr #375514
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

beginning-contributor [Pilot] contributed between 0-2 PRs to the CDK bug This issue is a bug. effort/medium Medium work item – several days of effort p1 pr-linter/exemption-requested The contributor has requested an exemption to the PR Linter feedback.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

addEventNotification causes "NoSuchBucket" errors during stack deletion regardless of bucket RemovalPolicy

2 participants