Skip to content

ecs-patterns: When health check failed, ECS deployment stucks #24614

Closed

Description

@yoshizawa56

Describe the feature

When deploying an ECS service using AWS CDK, the deployment process gets stuck until it times out if the health checks fail. This is causing issues in our deployment process and we would like to find a solution.

Use Case

Same as above

Proposed Solution

I propose creating a custom resource that monitors the deployment status of the ECS service and detects if the health checks fail. If the health checks fail, the custom resource will trigger a deployment error to prevent the deployment from stucking. By implementing this custom resource, we can ensure that failed health checks do not cause our deployment process to stall indefinitely.

Other Information

No response

Acknowledgements

  • I may be able to implement this feature request
  • This feature might incur a breaking change

CDK version used

2.68.0

Environment details (OS name and version, etc.)

m1 mac

Activity

  1. pahud commented on Mar 14, 2023

    @pahud
    Contributor

    This is an interesting idea. I believe this is the design from cloudformation that only successful health check makes a successful ecs service deployment, and the health check usually comes from the load balancer.

    I think this will need a lot of consideration about the design:

    1. how do you implement the health check logic from a custom service for a common ecs service with ALB?
    2. what happens if your health checker returns service "unhealthy" while ALB is still checking?
    3. what happens if your health checker returns service "healthy" while ALB is still checking?

    I am converting this to discussion and any more feedback with details are appreciated.

  2. locked and limited conversation to collaborators on Mar 14, 2023
  3. converted this issue into a discussion #24618 on Mar 14, 2023
  4. removed
    needs-triageThis issue or PR still needs to be triaged.
    on Mar 14, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions