fix(memory): wrap boto3 clients in context-isolating proxy to silence OTel detach errors#555
Open
citizen204 wants to merge 1 commit into
Open
fix(memory): wrap boto3 clients in context-isolating proxy to silence OTel detach errors#555citizen204 wants to merge 1 commit into
citizen204 wants to merge 1 commit into
Conversation
… OTel detach errors When MemoryClient is used from an asyncio task that already has an OTel context attached (e.g. via aws-opentelemetry-distro), the boto3 auto-instrumentation calls opentelemetry.context.attach() then detach() around every API call. If the call crosses an asyncio task or thread-pool boundary, Python raises: ValueError: <Token> was created in a different Context because ContextVar.reset() only accepts tokens created in the current execution context. The error is logged at ERROR level by the OTel runtime on every boto3 call, polluting CloudWatch and obscuring real failures. _ContextIsolatingProxy wraps each method call in contextvars.copy_context().run() so the OTel attach/detach pair is fully contained within the copied context snapshot -- reset() always matches the token created in the same context. Applies to both gmdp_client and gmcp_client at construction time, covering every call site in MemoryClient and AgentCoreMemorySessionManager without any per-call changes. Fixes aws#456
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
When
MemoryClientis used inside an asyncio application withaws-opentelemetry-distro(or any OTel boto3 auto-instrumentation), every data-plane or control-plane call emits an ERROR-level log:This happens because the OTel boto3 instrumentation calls
context.attach()before the API call andcontext.detach()after. Python'sContextVar.reset()raisesValueErrorwhen the token was created in a different execution context -- which occurs when the SDK is used from an asyncio task that inherited OTel context from a parent task, or when aThreadPoolExecutorworker crossesawaitboundaries.At scale (many concurrent agents,
AgentCoreMemorySessionManagerwithasyncio.to_thread()), this fills CloudWatch with spurious ERROR entries that obscure real failures.Fix
Added
_ContextIsolatingProxy-- a thin wrapper around a boto3 client that runs each method call insidecontextvars.copy_context().run(). This creates an isolated context snapshot so the OTel attach/detach pair is always contained within the same context, makingContextVar.reset()succeed.Both
gmdp_client(data plane) andgmcp_client(control plane) are wrapped at construction time inMemoryClient.__init__. This fixes every call site -- including directself.gmdp_client.create_event(...)calls insession_manager.py-- without per-call changes.The proxy exposes
metadirectly (formeta.region_namelogging) and wraps all callable attributes; non-callable attributes are returned as-is.Changes
src/bedrock_agentcore/memory/client.py:import contextvars_ContextIsolatingProxyclass (27 lines, beforeMemoryClient)gmdp_clientandgmcp_clientwith the proxy in__init__Fixes #456