(toolkit-lib): stale DescribeStacks read fails fresh stack creates, and the error is masked as NoStack

[svozza]

Describe the bug

Under CloudFormation read pressure, a fresh stack create via Toolkit.deploy() intermittently fails with an error that says nothing useful:

NoStack: ❌  <StackName> failed: NoStack: CloudFormationStack object does not hold a stack

Two bugs compound.

1. A stale DescribeStacks read is treated as terminal. For a fresh CREATE via changeset, ExecuteChangeSet moves the stack REVIEW_IN_PROGRESS → CREATE_IN_PROGRESS. Because DescribeStacks is eventually consistent, a later poll can still see REVIEW_IN_PROGRESS. StackStatus.isInProgress excludes review, so that read falls into the isReviewInProgress carve-out in stabilizeStack (cfn-api.ts:426), which returns the stack as stable. waitForStackDeploy then throws StackDeployFailed because REVIEW_IN_PROGRESS isn't a success status. The carve-out is right at the start of a wait (abandoned changeset), but wrong mid-wait — a real stack can't go CREATE_IN_PROGRESS → REVIEW_IN_PROGRESS, so it can only be a stale read.

2. The diagnosis path then masks that error. In monitorDeployment (deploy-stack.ts:718), the catch block passes finalState.wrapped to diagnoseFromErrorCollection. finalState is the pre-deploy lookup, which for a new stack holds no Stack, so the wrapped getter throws NoStack. That happens while evaluating an argument inside the catch, so it replaces the original DeploymentError entirely.

Bug 2 isn't specific to bug 1 — it masks any deployment error on a fresh create, including a plain ROLLBACK_COMPLETE from a resource failure.

Observed on @aws-cdk/toolkit-lib 1.35.0; both paths unchanged on main at 65517408.

Expected Behavior

A stale read shouldn't fail a deployment that CloudFormation goes on to complete. And a fresh-create failure should surface its real cause — NoStack is an internal invariant violation that shouldn't be reachable from a normal failed deploy.

Current Behavior

The deployment succeeds in CloudFormation, but deploy throws NoStack roughly one stabilization interval after ExecuteChangeSet. CloudTrail across three failing stacks shows every DescribeStacks returning 200 with no errorCode (a genuinely absent stack returns ValidationError), under heavy ThrottlingException on CFN reads — consistent with a stale replica read rather than a missing stack.

This hit the Powertools for AWS Lambda (TypeScript) e2e CI: ~40 parallel jobs deploying small stacks into one account/region, ~15% of matrix cells failing per run (baseline run, 6/40 failed). Tracked downstream as aws-powertools/powertools-lambda-typescript#5537.

Reproduction Steps

Both bugs reproduce as unit tests against main, using the existing FakeCloudFormation / mockCloudFormationClient / advanceTime helpers:

• Bug 1: DescribeStacks returning CREATE_IN_PROGRESS → REVIEW_IN_PROGRESS → CREATE_COMPLETE should deploy successfully. Currently throws DeploymentError: ... failed to deploy: REVIEW_IN_PROGRESS at cfn-api.ts:399.
• Bug 2: a fresh create driven to ROLLBACK_COMPLETE (via the fake's alwaysFailResources) should surface StackCreationFailed. Currently throws NoStack from stack-helpers.ts:75, via deploy-stack.ts:718.

We also confirmed the masking directly: making wrapped non-throwing reveals the real error underneath — DeploymentError / StackCreationFailed / "The stack named X failed creation ...: ROLLBACK_COMPLETE".

Possible Solution

We're working on a fix and will open a PR.

Sketch: for bug 2, don't evaluate finalState.wrapped when the pre-deploy stack doesn't exist — guard on finalState.exists or re-look-up the stack for diagnosis, preserving the existing rethrow-when-noProblem behaviour. For bug 1, treat REVIEW_IN_PROGRESS seen after a real in-progress state as a stale read and keep polling, bounded so the abandoned-changeset escape hatch survives (stabilizeStack is stateless across polls today).

We validated both as a load-time monkey-patch in the Powertools repo — retrying waitForStackDeploy on the spurious error, and making wrapped non-throwing. The 40-cell matrix went 40/40 green vs 6/40 failing on baseline, and in a later run 13 genuine fresh-create failures all surfaced their real DeploymentErrors through the path that previously produced NoStack.

Additional Information/Context

Related: #1709 and #1723 (closed) made the polling intervals configurable — same read-pressure environment. #1777 (open) is the closest sibling: also monitorDeployment, also read pressure turning a successful deploy into a reported failure, via the unguarded monitor.stop().

CDK CLI Version

N/A — used programmatically via @aws-cdk/toolkit-lib ^1.35.0

Framework Version

aws-cdk-lib ^2.261.0

Node.js Version

24

OS

Linux (GitHub Actions ubuntu-latest)

Language

TypeScript

Language Version

TypeScript ^6.0.3