Interop idea: optimize/eval_* ↔ EvalPort (open interchange format for eval results)

[adhabnr-ux]

Context

I'm the maintainer of EvalPort — an Apache-2.0 open interchange spec (TestCase / Grader / ResultSet) for portable LLM evaluation data, with a Python/TS SDK and adapters merged for DeepEval, Ragas, MLflow, LangSmith, Braintrust, Phoenix, Weave, DSPy, and others (full list in the README). I'm not affiliated with UnicoLab.

@piotrlaczkowski pointed me here from FlowyML#5 — I'd originally filed an EvalPort interop idea against FlowyML's evals/ module, and the redirect was right: agentomatic/optimize/ has a more complete, more decoupled eval subsystem, so I read through it before filing this rather than assuming.

Where it maps

Looking at src/agentomatic/optimize/eval_contract.py, dataset.py, metrics.py, and eval_api.py:

• EvalContract (name, input_fields, output_format, required_output_fields, constraints) is a declarative input/output schema + quality gate — this is close in spirit to EvalPort's TestCase schema plus a Grader of type schema/format. EvalContract.validate_details() already returns a {score, passed, failed, checks} shape that's a short hop from EvalPort's grader_results[] entries.
• DataPoint (query, expected_answer, context, metadata) and Dataset (JSONL/CSV/list loaders, split()) line up directly with EvalPort's TestCase.input / expected_output / context — this is essentially the same shape as flowyml.evals.EvalDataset.create_genai(), which is what made me look here.
• BaseMetric.evaluate() → EvalResult(metric_name, score, reason, metadata) maps field-for-field onto EvalPort's grader_results[] entries (grader_id→metric_name, score→score, reason→reason, metadata→metadata). CompositeMetric (weighted sub-metrics with dimensions) would serialize as multiple grader_results entries under one test_case_id, or as one custom grader carrying dimensions in metadata.
• DeterministicMetric (contains/regex/json_valid/length checks) and LLMJudgeMetric/GEvalMetric map onto EvalPort's type: "custom" and type: "llm_judge" grader types respectively — DeepEvalMetric (wrapping any DeepEval metric) is already effectively bridging to a framework EvalPort has an adapter for.
• run_eval() / EvaluateResult (scores, example_results, dataset_sizes, model, split) is the run-level container that would become an EvalPort ResultSet — example_results[].scores → results[].grader_results, EvaluateResult.scores (aggregate) → ResultSet.summary.by_grader.

The part I'd flag as most valuable both ways: once an AgentoMatic EvaluateResult is serialized as a ResultSet, optimize/loop.py's optimization runs could be diffed against baselines produced by any other EvalPort adapter (DSPy, Ragas, DeepEval, whatever produced the reference), not just another AgentoMatic run — and conversely, results from AgentoMatic's judge/contract-based scoring become readable by any EvalPort-consuming tool.

Comparable adapters already merged

For the shape and rigor I'd aim for — real to_openeval()/from_openeval(), tests against EvalPort's actual validator, a README:

• adapters/dspy-openeval-adapter — closest analog, since EvalContract is explicitly modeled as "Agentomatic's deployment-first equivalent of DSPy signatures" and optimize/ is a DSPy-style optimization loop.
• adapters/deepeval-openeval-adapter — relevant since metrics.py already wraps DeepEval metrics directly (DeepEvalMetric, _make_deepeval_metric).

The actual ask

Is this something you'd want either as:

1. An agentomatic-openeval-adapter package living in EvalPort's adapters/, maintained by me with AgentoMatic as the upstream reference, or
2. A small to_openeval()/from_openeval() example living in agentomatic/optimize/ itself, if you'd rather it be first-party — you mentioned a plugin/module might make sense, so this could sit under plugins/ if that's a better fit than optimize/.

Filing as an idea, not a PR — happy to build it if there's interest, and happy to be told the optimize/ internals aren't meant to be a public interchange surface if that's the case.

— Sahi, independent contributor (not affiliated with this project)