fix(langgraph/ts): preserve multimodal metadata + media type through round-trip#4
Draft
Nadine Nguyen (nadine-nguyen) wants to merge 489 commits into
Draft
Conversation
@ag-ui/langchain pinned `@langchain/core` as a hard `dependency` at
`^0.3.80`. For an adapter meant to run against the consumer's own
LangChain (0.3 → 1.x), a hard dep is wrong: pnpm installs the lib's
own core 0.3.80 alongside the consumer's core, producing a dual-core
install.
In the dojo, `@langchain/openai@1.0.0` (peer `@langchain/core@^1.0.0`)
resolves core 1.1.40, while @ag-ui/langchain dragged in 0.3.80. The two
`AIMessageChunk` / `MessageType` definitions then diverge, breaking
`demo-viewer:build` at agents.ts:257/263:
Type 'string & {}' is not assignable to type 'MessageType'
This cascaded to every dojo e2e job, the build, the typescript unit
job, and Vercel. main was green only because pnpm had previously
mis-resolved openai down to 0.3.80, which happened to match.
Move `@langchain/core` to a wide-range `peerDependency`
(`>=0.3.0 <2.0.0`) so the consumer provides a single core instance:
- <1.x consumers (0.5 / 0.6) stay satisfied — backwards compat kept
- 1.x consumers / dojo dedupe to one core 1.1.40 — types align, green
Keep a `^0.3.80` devDependency so the package still builds/tests
standalone.
Validated: `nx run demo-viewer:build` and `@ag-ui/langchain:build`
both pass locally after the change.
…62-google-add-a2ui-recoveryerror-handling-loop feat(a2ui): error-recovery loop — toolkit + middleware gate + LangGraph reference (OSS-162)
…id-publish-readiness chore(langroid): bring Python package to publish-readiness parity
…spec-publish-readiness fix(agent-spec): publish-readiness — correct license, real tests, fix langgraph KeyError
…ests Add verify-config-manifest-names.sh, which cross-checks every package `name` in release.config.json against the actual name in its manifest (package.json for TypeScript, pyproject.toml [project]/[tool.poetry] for Python). The config `name` feeds PR-body labels, AI release notes and human summaries; nothing previously enforced it matched the real published name. Fix the one existing drift this surfaced: integration-langroid's config name was the underscore form `ag_ui_langroid` while its pyproject (and PyPI distribution) is the hyphenated `ag-ui-langroid`.
Wire verify-config-manifest-names.sh into the Lint Release Workflows pipeline so config/manifest name drift fails CI on any PR touching scripts/release/**.
… with streaming path
Harden handle_tool_use_block / handle_tool_result_block so the
non-streaming fallback path is data-fidelity-consistent with the
streaming path in adapter.py:
- STATE_SNAPSHOT divergence: the non-streaming handler emitted a
STATE_SNAPSHOT unconditionally on successful parse. Now it computes
whether the merge actually changed state and suppresses no-op
snapshots, matching the streaming change check.
- state_updates extraction divergence: when the "state_updates" key is
absent, fall back to the whole parsed object (was: empty {}); a value
that is itself a JSON string is re-parsed. Mirrors streaming.
- tool-result content encoding: a bare-string result was json.dumps
-quoted while a list-of-text-blocks result was emitted unquoted, so
identical logical payloads reached the frontend differently. Both
shapes now route through one canonical text normaliser (try-JSON else
raw passthrough).
- parent_tool_use_id was accepted but inert: nested/sub-agent results
now surface the parent linkage via the protocol-standard raw_event
escape hatch (only when present), and only for nested results.
Adds red-green tests for each, plus coverage for the previously
untested non-list / scalar / unserializable result fallbacks.
…es, options, and worker lifecycle - State merge with None prior: make dict updates merge onto an empty dict explicitly instead of falling into the replace branch, keeping merge/replace semantics unambiguous and consistent with the handler. - Reasoning signature clobber: a message can contain multiple thinking blocks each with its own signature_delta. The encrypted value is now emitted tied to the reasoning block id (entity_id) rather than the enclosing message id, so a later block's signature can no longer attach to the wrong entity. - Invalid ClaudeAgentOptions kwargs: some whitelisted forwarded_props (e.g. temperature, max_tokens) are not ClaudeAgentOptions fields and would raise TypeError at construction, crashing the run. build_options now drops unknown kwargs with a warning so a forwarded prop can never wedge a run. - Worker lifecycle: (a) replace the per-entry "active" bool with an active_runs refcount so a finished run cannot mark a worker idle (and evictable) while a peer concurrent run on the same thread is still streaming; (b) retain strong references to fire-and-forget eviction stop() tasks in a set (discarded on completion) so they cannot be garbage-collected mid-flight. Adds red-green tests for each.
release: integration-langgraph-py + integration-langgraph-ts + middleware-a2ui + sdk-py-a2ui-toolkit + sdk-ts-a2ui-toolkit
…ase-pipeline-hardening chore(release): guard config↔manifest package names; flag deferred hardening
The refcount-aware dead-worker branch in run() — which must NOT pop+stop a cached worker that reports is_alive()==False while a concurrent peer still holds it (active_runs > 0) — had zero coverage. Add a test that pre-seeds a dead worker with active_runs=1 and asserts the shared entry survives and stop() is never called, so the peer is not torn down mid-stream.
Bump the error-path peer-skip log from debug to warning so it matches the analogous dead-worker peer-skip (leaving a shared worker cached for a live peer is an operationally notable event). Also add the active_runs field to the worker-cache-entry dict-shape comment so it matches the actual shape.
…n tests Drive two concurrent run() invocations on one thread_id through the REAL adapter + real SessionWorker (only ClaudeSDKClient is substituted), covering the per-thread active_runs refcount hardening that existing tests prove only against _Fake*Worker stand-ins: overlapping runs share one worker (refcount 2 -> 0, never duplicated/torn down), an erroring run does not evict a live peer's shared worker, and the worker is cleanly evictable afterward.
The example server defaults to port 8019 (examples/server.py, the uv `dev` script, and the dojo wiring all use 8019), but both READMEs documented the old 8888. Correct them so the documented port matches what the server binds.
…-sdk-adapter-hardening fix(claude-agent-sdk): harden adapter against deferred deep bugs (0.1.3)
When a cached worker's run-loop has exited (is_alive()==False) but a concurrent peer run still holds it (active_runs > 0), the arriving run can no longer be served: reusing the dead worker hangs forever (the peer's exited loop never drains the output queue) and evicting it would tear the worker out from under the live peer. Emit a descriptive RunErrorEvent and stop instead, leaving the peer's entry and refcount untouched (the run is never counted in, so the finally block does not decrement the peer's count). The single-run dead-worker case (active_runs == 0) still evicts and replaces as before.
Inject generate_a2ui only for adk-middleware's subagent demos via an explicit
whitelist (ADK_A2UI_INJECT_AGENTS): A2UIMiddleware({injectA2UITool}) is applied
per-agent in agents.ts rather than via the integration-wide route.ts flag.
ADK is the first integration shipping BOTH a2ui_fixed_schema (direct tools, must
not get generate_a2ui injected) and a subagent a2ui_dynamic_schema (relies on
injection). The integration-wide injectA2UITool flag can't distinguish them, so
flipping it on pollutes fixed_schema and leaving it off starves dynamic_schema
(the e2e failure). The whitelist gives per-agent granularity.
Whitelisted agents are excluded from the runtime-level a2ui config (route.ts)
to avoid double-applying the middleware -- the per-request clone copies
construction-time .use(). The demo drops its backend inject_a2ui_tool opt-in and
relies on the forwarded flag again (strands parity).
… drop custom-event relay
The render_a2ui sub-agent runs model.astream inside the graph, so its tool-call
arg deltas already surface as OnChatModelStream events, which the generic
agent.py / agent.ts translator turns into inner TOOL_CALL_START/ARGS/END and the
a2ui middleware paints progressively. The prior approach ALSO re-emitted those
deltas as explicit a2ui_render_{start,args,end} custom events, which on both the
FastAPI and langgraph platform paths duplicated the native stream: two
TOOL_CALL_START for one render id, tripping "tool call already in progress".
Remove the A2UI custom-event relay entirely: the dispatch in a2ui_tool.py /
a2ui-tool.ts, the handlers in agent.py / agent.ts, and the CustomEventNames
entries. OnChatModelStream is the single source and the translators carry no
A2UI knowledge. The toolkit recovery loop (error handling) is unchanged. Strands
keeps its own explicit push (its SDK does not surface a nested model stream).
Verified: FastAPI render emits 1 START, 183 progressive ARGS, 1 END, 0 custom
events, 0 errors; langgraph-python suite 211 passing.
…g-ui-protocol#1946) uv.lock hadn't been re-resolved since Python CI was added and still pinned google-adk 1.26.0, even though pyproject.toml declares google-adk>=1.16.0,<3.0.0. As a result CI never exercised any ADK >=1.30 behavior — including the Runner._resolve_invocation_id override path the middleware has dedicated workarounds and regression tests for — and 7 version-gated tests silently skipped (1 in test_adk_130_invocation_id_override.py, 3 in test_lro_tool_response_persistence.py, 3 in test_adk_2_0_compat.py). Pin google-adk to 1.35.2 (latest 1.x). The previously-gated tests now run (14 passed in those files, with only the genuine 2.x-only Workflow cases still skipping) and the full suite is green: 859 passed, 6 skipped, 0 failed. The 2.x story is tracked separately in ag-ui-protocol#1947 (suite is red under 2.2.0). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…i-protocol#1947) pyproject advertises google-adk>=1.16.0,<3.0.0 ("compatible with 1.x and 2.x"), but the lockfile resolves 1.x (ag-ui-protocol#1946), so CI never exercises 2.x — and the suite is currently red under google-adk 2.2.0 (35 failures), so the advertised 2.x compatibility is unverified and partially broken. Add an adk-middleware-python-adk-2x job that installs the locked env, force- installs google-adk>=2,<3 over it, and runs the suite. The job is marked continue-on-error so it surfaces the 2.x breakage in CI without blocking merges — the failures can then be burned down, or the advertised range narrowed, as a follow-up. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… a red check Job-level continue-on-error kept the workflow from failing, but each failing run still showed the adk-middleware-python-adk-2x job as a red X in the PR's check list — noisy on every PR. Move continue-on-error to the test step so the job stays green, and add a reporting step that emits a warning annotation and a job summary when the 2.x suite is red (and a "passed — consider making this required" note when it's green). Once the 2.x failures are burned down (ag-ui-protocol#1947), drop the step's continue-on-error to make this a blocking check. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…bump-uv-lock-google-adk-1x chore(adk-middleware): re-resolve uv.lock to latest google-adk 1.x (ag-ui-protocol#1946)
…dleware-adk-2x-leg ci(adk-middleware): add allowed-to-fail google-adk 2.x test leg (ag-ui-protocol#1947)
Resolve the adk-middleware uv.lock conflict from main's google-adk 1.x re-resolution (ag-ui-protocol#1946): keep the PR's google-adk>=1.28.1,<3.0.0 specifier (the OSS-158 floor bump) and the PR's resolved pin at 1.35.0. uv lock --check confirms 1.35.0 satisfies the merged pyproject, so no re-resolve is needed (and a full --upgrade would overshoot into adk 2.x, which the lock deliberately keeps on 1.x for CI). All other files auto-merged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…lver feat(adk): Add ADK endpoint agent resolver
- examples/agents/a2ui_dynamic_schema/agent.py: migrate from manual StateGraph + ToolNode to create_agent with directly-bound get_a2ui_tools - apps/dojo route.ts: enable injectA2UITool for langgraph integrations - regenerate files.json; sync examples uv.lock
…trands-lg-parity
release: integration-aws-strands-py + integration-aws-strands-ts + integration-langgraph-py + integration-langgraph-ts + middleware-a2ui
…58-port-new-a2ui-implementation-to-google-adk feat(adk-middleware): A2UI subagent generation + bounded recovery (OSS-158)
…g it
On a continuation run after a frontend/HITL tool, the legacy/session-manager
path overwrote the real tool result with the literal
'{tool_name} executed successfully with no return value.' before the model saw
it. An approval resolving to {"approved": false} was therefore reported to the
model as a no-value success, silently breaking HITL.
Now the actual result is forwarded ('{tool_name} returned: <result>'); the
synthetic acknowledgement is used only when the result is genuinely empty.
All 150 existing strands tests pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…-hitl-context-fix fix(strands): forward HITL tool result on resume instead of discarding it
Python Preview Packages — Publish FailedPreview publish failed for commit b01b70a. |
Nadine Nguyen (nadine-nguyen)
force-pushed
the
fix/langgraph-ts-multimodal-metadata-roundtrip
branch
2 times, most recently
from
10e795f to
b51fa93
Compare
…round-trip The two multimodal converters in utils.ts were lossy: forward collapsed every image/audio/video/document part to image_url and dropped InputContent.metadata, and reverse re-emitted every image_url as type "image" with no metadata. An attachment could not survive an AG-UI -> LangGraph -> AG-UI round-trip. Forward now carries metadata (parity with merged Python ag-ui-protocol#1832) plus the original media type stashed as __agui_type inside the metadata object, and reverse restores both with a "image" fallback for untagged legacy blocks. The extra key is inert for LangChain/model providers and survives the LangGraph checkpoint JSON round-trip. Closes ag-ui-protocol#2011.
Nadine Nguyen (nadine-nguyen)
force-pushed
the
fix/langgraph-ts-multimodal-metadata-roundtrip
branch
from
b51fa93 to
2c61bdc
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
17 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
InputContent.metadataand the original AG-UI media type when converting to LangChain multimodal blocksFixes ag-ui-protocol#2011. Extends the Python fix from ag-ui-protocol#1832 to TypeScript and closes the reverse-direction gap.
What changed
integrations/langgraph/typescript/src/utils.tsMEDIA_CONTENT_TYPESis nowas const;MediaContentTypeandAGUI_TYPE_KEYare derived from it so there is a single source of truth for the key name and the valid type valuesLangchainBlockMetadatatype (Record<string, unknown> & { [AGUI_TYPE_KEY]?: MediaContentType }) replaces the bareunknownonLangchainMultimodalBlock.metadataconvertAguiMultimodalToLangchain— spreads user metadata and adds[AGUI_TYPE_KEY]: mediaItem.typeon every media block; falls back to no metadata field only when neither metadata nor type needs to be carriedconvertLangchainMultimodalToAgui— reads[AGUI_TYPE_KEY]from metadata, strips it before returning soInputContent.metadatais clean, falls back toDEFAULT_MEDIA_CONTENT_TYPE("image") for legacy untagged blocksintegrations/langgraph/typescript/src/utils.test.tsdescribe("Metadata + media type round-trip")block covering:imagefor untagged blocksit.each(["image","audio","video","document"])round-trip assertingtype,source, andmetadataall surviveTo verify