[Feature]: use native multimodal function_call_output in OpenAI Responses formatter

[fancyboi999]

Background

The current OpenAI Responses API accepts either a string or an array of text, image, and file content items in function_call_output.output. See the official function-calling guide.

AgentScope's OpenAIResponseFormatter currently takes a multimodal ToolResultBlock through convert_tool_result_to_string() instead:

1. it emits a text-only function_call_output containing a system reminder;
2. it promotes supported DataBlock values into a separate synthetic user message.

The behavior is pinned in tests/formatter_openai_response_test.py and was introduced as part of #1621, when multimodal promotion was used as a provider-compatible fallback.

Why consider the native form

For the Responses API specifically, using the native output array would:

• preserve the direct association between each image/file and its call_id;
• avoid an extra synthetic user turn and identifier reminder text;
• reduce context overhead;
• align with current peer implementations: LangChain and Pi both pass multimodal tool output directly in function_call_output.output.

A live AgentScope probe against an OpenAI-compatible Responses endpoint completed a full tool-call → Base64 PNG tool result → second model call flow with the current promotion path, so the existing implementation is functional. This proposal is about preserving native protocol semantics, not fixing a broken tool loop.

Proposed behavior

For OpenAIResponseFormatter only:

• keep text-only tool results as the current string output;
• encode supported image/file DataBlock values directly as input_image / input_file items in the function_call_output.output array;
• retain an explicit fallback for unsupported media types;
• decide whether lagging OpenAI-compatible endpoints require an opt-out compatibility flag or a narrowly handled fallback.

Suggested verification:

• exact formatter tests for text + Base64 image, URL image, and PDF output;
• mixed text/data ordering and multiple parallel tool results;
• live OpenAI Responses round trip;
• at least one OpenAI-compatible endpoint probe before changing the default.

Related, not duplicate

• [Bug]: Random tool-result media identifiers make message formatting nondeterministic #2161 / fix(formatter): derive multimodal tool-result identifier from the block's own stable id #2165 make promotion identifiers deterministic across providers. They do not remove the synthetic user promotion for Responses.
• [Bug]: ToolResultDataDeltaEvent accepts missing or conflicting data sources #2363 validates that ToolResultDataDeltaEvent contains exactly one of data or url; it is independent of formatter serialization.

I would like maintainer feedback on the compatibility policy before implementing this, especially whether native arrays should be the default for OpenAIResponseFormatter or gated for OpenAI-compatible endpoints.

[DavdGao]

@fancyboi999 Thanks for pointing out this issue. @qbc2016 Plz take a look at this?

[qbc2016]

@fancyboi999 Thanks for the detailed investigation. I agree that OpenAIResponseFormatter should use the native multimodal tool-result representation defined by the official OpenAI Responses API.

The formatter should represent the official protocol accurately. Provider-specific differences in OpenAI-compatible endpoints should be handled separately and should not determine the behavior of the core OpenAI formatter.

Please proceed with the following behavior:

• Keep text-only tool results as strings.
• Serialize mixed text, image, and file results directly in the function_call_output.output content array.
• Preserve the original content-block ordering.
• Retain a textual fallback for unsupported media types.
• Remove the synthetic user-message promotion from OpenAIResponseFormatter.
• Add exact tests for Base64 images, URL images, files, mixed ordering, and parallel tool results.

I do not think a compatibility flag is necessary in OpenAIResponseFormatter for this change. We can document that support from OpenAI-compatible endpoints depends on the provider and address provider-specific differences separately when needed.