Skip to content

fix(http-client): use thread-safe connection pool#858

Merged
andrewazores merged 3 commits intocryostatio:mainfrom
andrewazores:cooldown-pooling
May 1, 2026
Merged

fix(http-client): use thread-safe connection pool#858
andrewazores merged 3 commits intocryostatio:mainfrom
andrewazores:cooldown-pooling

Conversation

@andrewazores
Copy link
Copy Markdown
Member

@andrewazores andrewazores commented Apr 30, 2026

See #851

BasicHttpClientConnectionManager is NOT thread-safe and maintains only a single connection. It's designed for single-threaded use only, but the Agent uses a ScheduledExecutorService with 3 worker threads. Misuse of the connection manager by different threads leads to an IllegalStateException in the HTTP client the Agent uses to send requests to Cryostat.

This HTTP client failure results in the Agent entering a degraded cooldown state where it believes it's unable to register with the Cryostat server due to a server-side failure, which also results in it rejecting connection requests to its own WebServer. The cooldown period is too long and Cryostat repeatedly fails to ping the Agent because it's in cooldown for too long, resulting in Cryostat eventually pruning the plugin.

This patch does two things:

  1. reduces the cooldown period. The Agent should prevent itself from flooding Cryostat with re-registration requests, but it should also become ready to accept pings from Cryostat more quickly so that Cryostat can recognize that it is still alive and reachable.
  2. uses a pooling HTTP connection manager which is thread safe, preventing the Agent-side failure which triggers the degraded state to begin with.
2026-04-30 15:33:35:297 +0000 [cryostat-agent-worker-2] ERROR io.cryostat.agent.Registration - Registration failure (attempt 1, circuit state: CLOSED, cooldown: PT5M)
java.util.concurrent.CompletionException: java.lang.IllegalStateException: Connection 10.217.0.205:38574<->10.217.5.209:8282 is still allocated
	at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:315)
	at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:320)
	at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1770)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: java.lang.IllegalStateException: Connection 10.217.0.205:38574<->10.217.5.209:8282 is still allocated
	at io.cryostat.agent.shaded.org.apache.shaded.hc.core5.util.Asserts.check(Asserts.java:50)
	at io.cryostat.agent.shaded.org.apache.shaded.hc.client5.http.impl.io.BasicHttpClientConnectionManager.getConnection(BasicHttpClientConnectionManager.java:389)
	at io.cryostat.agent.shaded.org.apache.shaded.hc.client5.http.impl.io.BasicHttpClientConnectionManager$1.get(BasicHttpClientConnectionManager.java:311)
	at io.cryostat.agent.shaded.org.apache.shaded.hc.client5.http.impl.classic.InternalExecRuntime.acquireEndpoint(InternalExecRuntime.java:111)
	at io.cryostat.agent.shaded.org.apache.shaded.hc.client5.http.impl.classic.ConnectExec.execute(ConnectExec.java:127)
	at io.cryostat.agent.shaded.org.apache.shaded.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
	at io.cryostat.agent.shaded.org.apache.shaded.hc.client5.http.impl.classic.ProtocolExec.execute(ProtocolExec.java:192)
	at io.cryostat.agent.shaded.org.apache.shaded.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
	at io.cryostat.agent.shaded.org.apache.shaded.hc.client5.http.impl.classic.ContentCompressionExec.execute(ContentCompressionExec.java:150)
	at io.cryostat.agent.shaded.org.apache.shaded.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
	at io.cryostat.agent.shaded.org.apache.shaded.hc.client5.http.impl.classic.HttpRequestRetryExec.execute(HttpRequestRetryExec.java:113)
	at io.cryostat.agent.shaded.org.apache.shaded.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
	at io.cryostat.agent.shaded.org.apache.shaded.hc.client5.http.impl.classic.RedirectExec.execute(RedirectExec.java:110)
	at io.cryostat.agent.shaded.org.apache.shaded.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
	at io.cryostat.agent.shaded.org.apache.shaded.hc.client5.http.impl.classic.InternalHttpClient.doExecute(InternalHttpClient.java:183)
	at io.cryostat.agent.shaded.org.apache.shaded.hc.client5.http.impl.classic.CloseableHttpClient.execute(CloseableHttpClient.java:141)
	at io.cryostat.agent.shaded.org.apache.shaded.hc.client5.http.impl.classic.CloseableHttpClient.execute(CloseableHttpClient.java:55)
	at io.cryostat.agent.CryostatClient.executeQuiet(CryostatClient.java:553)
	at io.cryostat.agent.CryostatClient.lambda$supply$43(CryostatClient.java:548)
	at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1768)
	... 6 more
2026-04-30 15:33:35:297 +0000 [cryostat-agent-worker-1] DEBUG io.cryostat.agent.shaded.org.apache.shaded.hc.client5.http.impl.classic.InternalHttpClient - ep-0000000000 start execution ex-0000000014
2026-04-30 15:33:35:297 +0000 [cryostat-agent-worker-2] DEBUG io.cryostat.agent.Registration - Entering cooldown for PT5M after 1 consecutive failures
2026-04-30 15:33:35:297 +0000 [cryostat-agent-worker-1] DEBUG io.cryostat.agent.shaded.org.apache.shaded.hc.client5.http.impl.io.BasicHttpClientConnectionManager - ep-0000000000 Executing exchange ex-0000000014
2026-04-30 15:33:35:297 +0000 [cryostat-agent-worker-2] DEBUG io.cryostat.agent.WebServer - Performing cleanup before cooldown

@andrewazores andrewazores requested a review from a team April 30, 2026 16:13
@andrewazores andrewazores marked this pull request as draft April 30, 2026 16:41
@andrewazores andrewazores marked this pull request as ready for review April 30, 2026 18:52
@andrewazores andrewazores changed the title fix(http-client): use thread-safe connection pool and ensure clean connection release fix(http-client): use thread-safe connection pool Apr 30, 2026
andrewazores added a commit to andrewazores/cryostat-agent that referenced this pull request May 1, 2026
* fix(http-client): use thread-safe connection pooling

* fix(registration): reduce cooldown period
@andrewazores andrewazores merged commit 43e1742 into cryostatio:main May 1, 2026
16 checks passed
@andrewazores andrewazores deleted the cooldown-pooling branch May 1, 2026 18:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants