Skip to content

fix(registration): apply mitigations to Cryostat registration flooding#851

Merged
andrewazores merged 7 commits intocryostatio:mainfrom
andrewazores:discovery-retry-storm
Apr 27, 2026
Merged

fix(registration): apply mitigations to Cryostat registration flooding#851
andrewazores merged 7 commits intocryostatio:mainfrom
andrewazores:discovery-retry-storm

Conversation

@andrewazores
Copy link
Copy Markdown
Member

@andrewazores andrewazores commented Apr 22, 2026

Related to cryostatio/cryostat#1483

This should improve the robustness and self-healing ability of the Agent's registration as a Discovery Plugin.

  1. Applies a circuit-breaker and cooldown to registration attempts. Previously the Agent had a simple fixed timer interval that it would re-attempt registrations after a failure. Now, it will slow down attempts on failure until hitting the circuit breaker, at which point it will take a long pause before retrying.
  2. The Agent will now refuse requests on its own WebServer if it is in a state where it isn't expecting them - ie if it believes its own registration process has not yet started, or it's in circuit-breaker cooldown. This ensures that if Cryostat tries to ping plugins based on records it has in its database, but the Agent doesn't think it is actually registered, the Agent should not respond to Cryostat. This way Cryostat can mark the plugin as failed and prune it, so that the Agent can hopefully retry again later and succeed.
  3. Adds a CredentialTracker and some cleaner shutdown logic to ensure the Agent does a better best-effort job of cleaning up after itself and removing its credentials from Cryostat's keyring.

@mergify mergify Bot added the safe-to-test label Apr 22, 2026
@andrewazores andrewazores marked this pull request as ready for review April 22, 2026 20:23
@andrewazores andrewazores marked this pull request as draft April 22, 2026 20:24
@andrewazores andrewazores force-pushed the discovery-retry-storm branch from 959f912 to b2ef1c8 Compare April 24, 2026 18:29
@andrewazores andrewazores marked this pull request as ready for review April 24, 2026 18:29
@andrewazores andrewazores requested a review from a team April 24, 2026 21:03
@andrewazores andrewazores force-pushed the discovery-retry-storm branch from b2ef1c8 to 0432c1a Compare April 27, 2026 16:25
@andrewazores andrewazores merged commit ebe5fa2 into cryostatio:main Apr 27, 2026
9 checks passed
@andrewazores andrewazores deleted the discovery-retry-storm branch April 27, 2026 18:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants