Make NATIVE thinking-content available in both blocking and streaming calls through ThinkingResponse/StreamingEvent

[jorander]

Problem

Today Embabel contains ThinkingResponse and StreamingEvent abstractions offering to return the result of a call to an AI-model together with any thinking-content the model might have produced. However, the current solution has two major drawbacks:

1. Methods offering ThinkingResponse (in LlmOperations) and StreamingEvent in (StreamingLlmOperations) is only available for methods producing objects, not for methods producing pure text (blocking or streamed).
2. Implementation for extracting thinking-content is only focused on content embedded in thinking/reasoning tags/blocks in the returned content, but many models/providers/api-implementations have stopped sending thinking-content together with the final output and instead started to use separate, provider dependent, fields in the response objects. See below how Spring AI provides thinking-content in three different locations depending on the underlying provider. After studying the Spring AI source code I think it is safe to say that although the object graph is the same, the way thinking-content is conveyed is different for each underlying provider (see Spring AI open issue).

Spring AI - OpenAI: Thinking-content provided as "reasoningContent" in metadata in the AssistantMessage.

Spring AI - Ollama: Thinking-content provided as "thinking" in the ChatGenerationMetadata.

Spring AI - Anthropic: Thinking-content provided as content in AssistantMessage with a metadata "thinking" set to true or "reducted" (encrypted) thinking-content as "data" in metadata in the AssistantMessage for streaming calls. Or in almost the same way for blocking calls, but then the metadata flag is called "signature" instead of "thinking".

This issue should include

• Complete the Emabel API with *withThinking versions of greneration-methods where those are missing.
• Extend the LlmMessageSender and LlmMessageStreamer API:s so that thinking-content is forwarded from the underlying implementations.
• Adapt the implementations of LlmOperations and StreamingLlmOperations to create ThinkingResponse/StreamingEvent from thinking-content provided by LlmMessageSender and LlmMessageStreamer and not only from the string content as per today.
• Adapt the Spring AI-implementations of LlmMessageSender and LlmMessageStreamer to pick up thinking/reasoning content from metadata entries for the different underlying providers.

Timing

There seem to be some changes/additions to thinking-content handling for some underlying providers in Spring AI 2.x.x, so it might be a good thing to hold Spring AI specific parts of the implementation for this issue until #1765 has been merged.

Thanks to @kenkc for doing the debugging in #1691 leading up to this issue being created.

[igordayen]

@jorander - thank you for the input.
Apparently, moving from the prompt-based contrrol to contract-based control has become a trend.
And since standardization takes time, we have to pay special attention to supporting provider-specific solutions.
I made this for release 2.0.0, as the area seems coupled with the new Spring AI 2.0.0 features.

[jorander]

@igordayen - since release 1.5.0 (#1765) now includes upgrade to Spring AI 2.0.0, this fix doesn't have to wait for release 2.0.0 of Embabel.

[igordayen]

This issue consists of 2 separate issues:

• native thinking support: consider an LLMOptions-dedicated map for native extensions (already there) and option converters, provider-specific;
• generate a stream with NL buffering

Perhaps it makes sense to split.
Thanks

[igordayen]

Added one more sub-issue.

[igordayen]

Native thinking gets tricky due to a lack of standardization. Instructed Codex to come up with a design.
Getting complex. But still doable.

embabel-native-thinking-extraction-design.md
embabel-native-thinking-request-design.md

Its initial draft; treat it accordingly:)

Thank you.

@jorander @kenkc @azanux @arnabnandy7 @simeshev @alexheifetz @arimu1 - FYI.

If I miss anyone involved in this sizable cross-provider effort, please distribute.