Repository navigation
fix(cloud-security): retry transient AssumeRole failures (first-scan "could not assume role") - #3055
Merged
Merged
Conversation
…"could not assume role") Customer-reported: the first AWS scan fails with a "Could not assume AWS role" finding; switching the scan engine and back, then re-running, works — on every account, with no change in AWS. Customer correctly concluded it's not the role (it assumes fine the second time). Root cause (verified): assuming the customer's cross-account role is a two-hop STS flow with NO application-level retry, and the AWS SDK's default retry only covers throttling/5xx/network — NOT the classes that actually occur here: - IAM/STS eventual consistency on a new or just-edited role/trust-policy returns AccessDenied on the first assume and succeeds seconds later (no AWS change). - The base/roleAssumer session can briefly return ExpiredToken. So a single transient failure surfaced as a sticky finding. The "switch switch" is INCIDENTAL — updateMode only writes metadata (idempotent, no cache/cred refresh); what actually fixes it is the elapsed time before the retry. The visible finding is written by the integration-platform checks path (resolveAwsSessionOrFail in manifests/aws/checks/shared.ts), not the Cloud Tests scan (which returns success:false with no finding) — so both assume implementations are fixed. - new shared helper retryAssume / isRetryableAssumeError (manifests/aws/checks/assume-retry.ts): bounded exponential backoff + jitter, retries only transient / eventual-consistency classes (AccessDenied, ExpiredToken, IDPCommunicationError, throttling, 5xx, network). Hard config errors (bad ARN, ValidationError) and a persistently-denied role still propagate; backoff capped so a genuinely broken role fails within seconds. - wrap both AssumeRole hops in shared.ts (checks path) and aws-security.service.ts assumeRole (Cloud Tests scan path). - export retryAssume from the package barrel for apps/api. - tests: retry-then-succeed, no-retry on config errors, give-up after N, classifier. Note: not yet live-verified; the exact STS class (AccessDenied vs ExpiredToken) is already captured in evidence.error on the failing finding — the fix covers both. cloud-security suite green (311); helper tests green; typecheck clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
Contributor
|
🎉 This PR is included in version 3.73.2 🎉 The release is available on GitHub release Your semantic-release bot 📦🚀 |
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Customer (with multiple AWS accounts, cross-account role auth) reported: the first AWS scan fails with a "Could not assume AWS role" finding; switching the scan engine and back, then re-running, works — on every account, with nothing changed in AWS. They correctly concluded it isn't the role (it assumes fine the second time) — it's our bug.
Root cause (investigated + adversarially verified)
Assuming the customer's cross-account role is a two-hop STS flow (task role → Comp roleAssumer → customer role) with no application-level retry, and the AWS SDK's default retry strategy only covers throttling / 5xx / network — not the classes that actually occur here:
AccessDeniedon the first assume and succeeds seconds later, with no AWS change.ExpiredToken.A single transient failure surfaced as a sticky finding, written by the integration-platform checks path (
resolveAwsSessionOrFail→ctx.fail({ title: 'Could not assume AWS role' })inmanifests/aws/checks/shared.ts) — not the Cloud Tests scan (which returnssuccess:falsewith no finding). So both assume implementations are fixed.The "switch switch" is incidental, not causal.
updateModeonly writesmetadata.awsScanMode(idempotent, no cache or credential refresh) and both modes share the same assume path. What actually makes the retry succeed is the elapsed time before re-running.Fix
retryAssume/isRetryableAssumeError(manifests/aws/checks/assume-retry.ts): bounded exponential backoff + jitter, retrying only transient / eventual-consistency classes (AccessDenied,ExpiredToken,IDPCommunicationError, throttling, 5xx, network). Hard config errors (bad ARN,ValidationError) and a persistently-denied role still propagate; backoff is capped so a genuinely broken role fails within a few seconds.AssumeRolehops inshared.ts(checks path — writes the finding) andaws-security.service.tsassumeRole(Cloud Tests scan path).retryAssumefrom the package barrel forapps/api.Tests
assume-retry.test.ts: retry-then-succeed, no-retry on config errors, give-up after N attempts, and the classifier (AccessDenied/ExpiredToken/throttling/5xx/network = retryable; ValidationError/bad-ARN = not). 7 passing.Honest notes
AccessDeniedvsExpiredToken) is already captured inevidence.erroron the failing finding — the fix covers both. Pulling that error string would confirm which hop/class it was.🤖 Generated with Claude Code
Summary by cubic
Retries transient AWS STS
AssumeRolefailures with bounded backoff to stop first‑scan “Could not assume AWS role” findings. Applies to both the checks path and the Cloud Tests scanner.retryAssumeandisRetryableAssumeErrorwith capped exponential backoff + jitter; retriesAccessDenied,ExpiredToken, throttling, 5xx, and network errors; skips config errors (e.g.,ValidationError).AssumeRolehops in the checks (shared.ts) and scanner (aws-security.service.ts) paths to ensure transient failures don’t surface as findings.@trycompai/integration-platformfor reuse inapps/api; added unit tests for retry behavior; no change to the success path.Written for commit 28ca06e. Summary will update on new commits.