Add validation mechanism that does not require JSR-380 annotations

[johnsonr]

We presently support validation only via JSR-380 annotations. While this is valuable, such annotations might not exist on classes we use, and we might not control those classes. Also, validation may be too complex for JSR-280 annotations.

We need to provide a way of defining a validator that works programmatically and produces a compatible result, via InvalidLlmReturnType exception.

This is distinct from the problem of validating UserContent.

[igordayen]

Could using ValidatorUtils / SmartValidator from spring framework be considered. Custom annotations could also be developed with delegation to existing spring utils or special utils can be developed by implementing spring validator interface.

[igordayen]

Design notes:

validation-options-comparison.md

[johnsonr]

I'd rather not increase API dependence on Spring. Although this is certainly a potential path.

[igordayen]

I'd rather not increase API dependence on Spring. Although this is certainly a potential path.

noted, understood

[m-kostira]

Hi guys, pitching in:

I'm using the programatic API, and in some situations it would be nice to have a custom validation function applied to the generated object/text, so that an error message can be returned back to the llm as a hint how to fix the generation. E.g something like:

ai
   .withOutputValidator(result -> {
        // custom validation logic
        if (!validate(result)) {
            throw new Exception("Error..."); // framework propagates the error message to the LLM and it retries while avoiding making the same error 
        }
    })
    .createObject(promptString, Result.class);

Again, this is nice to have, not a deal breaker. I sometimes get around this by exposing a tool method to accept the result, and doing validation inside the method.

[igordayen]

Hi guys, pitching in:

I'm using the programatic API, and in some situations it would be nice to have a custom validation function applied to the generated object/text, so that an error message can be returned back to the llm as a hint how to fix the generation. E.g something like:

hello @m-kostira - thanks for suggestions. may i inquire: you would like to validate "raw" LLM response prior to object creation?

[m-kostira]

I'd like to validate the deserialized object rather than the raw text response.

…

On Thu, 4 Dec 2025, 01:43 Igor Dayen, ***@***.***> wrote: *igordayen* left a comment (embabel/embabel-agent#978) <#978 (comment)> Hi guys, pitching in: I'm using the programatic API, and in some situations it would be nice to have a custom validation function applied to the generated object/text, so that an error message can be returned back to the llm as a hint how to fix the generation. E.g something like: hello @m-kostira <https://github.com/m-kostira> - thanks for suggestions. may i inquire: you would like to validate "raw" LLM response prior to object creation? — Reply to this email directly, view it on GitHub <#978 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADRFXMXRFMAR3TJY4R7A2KT375YQTAVCNFSM6AAAAACKEZECLKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTMMBZGI4TOMBVHE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[igordayen]

I'd like to validate the deserialized object rather than the raw text
response.
@m-kostira

would be helpful if you re-confirm proper understanding of the ask:

1. LLM returns - sample output:
   July is the hottest month in New York, average tempreture is 78F
   {name: July, ranking:hottest}
2. so, you would like to validate: {name: July, ranking:hottest} adheres to schema prior to transforming to
   record MonthItem(name:String, ranking:String) ?

Would be also helpful if you can provide feedback on proposals:

• implicit validation by matching validation condition signature with action signature
• explicit validation by demarcating action with validation annotation
• hybrid, both

Thank you.

[m-kostira]

So my use case is something similar to the following: the user wants to plan a vacation and tells the LLM their preferred destinations are Greece and Italy and they can only take vacation in August, June, or September.

Then the user asks the LLM to come up with some destinations with cheap plane tickets for a one-week stay. Let's say the output is an object, basically a round trip flight. Even if the output adheres to the schema, I want to be able to verify whether the flight dates are in the requested months, whether the destinations are in Greece/Italy vs somewhere else like Spain or Turkey.

So basically we're not validating the schema, rather we're validating the reasoning according to some custom business logic. IMO that can be achieved by a custom validator method supplied by the client code.

Regarding the proposals, I'm using Embabel via the programatic API, so I don't really have any preference how it's implemented with regards to actions/goals/annotations, as long as there is a low level way to supply the validator method, like in my snippet in the previous comment.

Thanks!

Edit: come to think of it, maybe there is already functionality that supports this in Spring AI? E.g. if we can add a custom advisor in the advisor chain, maybe..