(7/n) Complete declarative task runtime and shared pairwise orchestration by kargibora · Pull Request #86 · OpenEuroLLM/JudgeArena

kargibora · 2026-07-21T12:07:58Z

Summary

This PR connects packaged task YAML definitions to dataset loading, baseline selection, benchmark execution, and scoring, and splits the task schema into focused modules.

Previously, task definitions described benchmark settings, but parts of the runtime still selected behavior through hard-coded branches, repeated registry lookups, or direct function calls. Adding another task could therefore require changes across several unrelated files.

After this change, a task is resolved once and passed through the pairwise evaluation pipeline. Existing components can be reused by referencing their registered names from task YAML.

Before and after

 Before                                                       After
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
 Task information was partly defined in YAML and partly       Stable task behavior is defined in task YAML and
 selected in Python.                                          validated by typed schemas.
───────────────────────────────────────────────────────────  ───────────────────────────────────────────────────────────
 Runners and dataset loaders could resolve the same task      The dispatcher resolves the task once and passes the
 repeatedly.                                                  resulting specification downstream.
───────────────────────────────────────────────────────────  ───────────────────────────────────────────────────────────
 Pairwise tasks relied on benchmark-specific routing and      Compatible tasks share the same pairwise runner and
 duplicated setup logic.                                      dataset orchestration.
───────────────────────────────────────────────────────────  ───────────────────────────────────────────────────────────
 Scoring configuration was validated but not always used      The runner resolves and executes the scoring adapter
 by the runtime.                                              declared by the task.
───────────────────────────────────────────────────────────  ───────────────────────────────────────────────────────────
 Adding fields to one large schema made task definitions      Schemas are separated by responsibility and protocol-
 harder to maintain.                                          specific fields remain isolated.

Architecture

The runtime now follows this path:

Task YAML
    ↓
Schema validation and inheritance
    ↓
Resolved task specification
    ↓
Benchmark and dataset registries
    ↓
Shared or protocol-specific runner
    ↓
Generation, judging, scoring, and artifacts

The new schema package separates datasets, sources, baselines, pairwise protocols, MT-Bench protocols, and resolved task models. This prevents a single schema file from growing whenever a new benchmark family introduces specialized fields.

Most new tasks do not require a new schema or runner. A task using existing components only needs a YAML definition selecting its dataset adapter, runner, baseline policy, judge protocol, and scorer.

A new Python component is only necessary when a benchmark introduces genuinely different behavior:

A dataset adapter handles benchmark-specific downloading and normalization.
A benchmark runner handles a different evaluation workflow.
A protocol schema validates new task-owned algorithm settings.
A scoring adapter handles a different result interpretation.

Executable behavior remains in Python rather than YAML; YAML only selects registered and validated components.

Changes

Split the task schema into focused modules.
Added shared pairwise dataset and runner orchestration.
Passed resolved task specifications through the runtime instead of repeating lookups.
Made task-defined scoring adapters operational: the runner now resolves the scorer named by the task, and primary_metric / higher_is_better / parser were removed from task YAML since the scorer and prompt preset already own them.
Added documentation for defining tasks and extending the available components.
Preserved runtime overrides while keeping stable benchmark identity in task YAML.

Notes

Most of the changes are coming from splitting schema.py to multiple files, thus this is just a refactoring step as next PR's will introduce two more task which can inflate the same file if they require some special fields.

kargibora · 2026-08-04T12:10:31Z

@@ -0,0 +1,79 @@
+"""Public schema API for declarative task definitions.


Note:
All these related changes simply splits functions. As mt_bench can require some specific fields that others ignore, we only need a shared components + special components on their own file.

kargibora · 2026-08-04T12:12:45Z

@@ -0,0 +1,43 @@
+"""Baseline-selection policies shared by pairwise protocols."""


This was a design problem we have talked before. Baseline is used to determine which model should be used as default - and which tasks requires a baseline provided by the config file we send. We probalby dont need many fields other than the ones that requires us to introduce functionality like using multiple different baselines sucnh asCategoryDefaultBaseline. However if you think this is over-engineering happy to remove it

It is not clear why we need it.
Why cant we just have a baseline: str = "gpt-4o" config key with a default value in the task configurations?

Move dataset, baseline, pairwise, and MT-Bench contracts into focused modules while preserving the public schema imports and YAML behavior.

Explain task identity, adding YAML definitions, and when new dataset adapters or protocol schemas are required.

Pass resolved tasks through dispatch, normalize registered dataset inputs behind one contract, and move baseline planning out of the runner.

Let prompt presets own parsing, resolve scoring adapters at runtime, and remove repeated metric semantics from task YAML.

geoalgo · 2026-08-05T16:12:24Z



-def run_mt_bench_benchmark(cfg: RunConfig):
+def run_mt_bench_benchmark(


This feels slightly weird to me. I would imagine that tasks resolve a run function which gets executed.
Here it feels weird to have run_mt_bench_benchmark takes as input cfg: RunConfig and _resolved_task: ResolvedTaskSpec.

geoalgo · 2026-08-05T16:13:17Z

+        return sorted(self.baseline_by_index.dropna().unique().tolist())
+
+    @property
+    def is_flat(self) -> bool:


why is_flat rather than is_unique_model?

geoalgo · 2026-08-05T16:15:44Z

@@ -0,0 +1,43 @@
+"""Baseline-selection policies shared by pairwise protocols."""


It is not clear why we need it.
Why cant we just have a baseline: str = "gpt-4o" config key with a default value in the task configurations?

geoalgo · 2026-08-05T16:17:47Z

+
+import pandas as pd
+
+from judgearena.utils.eval import PrefSummary, compute_pref_summary


are we using compute_pref_summary anywhere else? if not lets put the code in the object bellow to simplify.

geoalgo · 2026-08-05T16:18:42Z

+
+_PAIRWISE_SCORERS = {
+    "pairwise_win_rate": PairwiseScorer(
+        name="pairwise_win_rate",


DRY, name appears twice.
Either we keep the dictionary and rename name field (best approach I think) or we have a list of PairwiseScorer.

geoalgo · 2026-08-05T16:23:08Z

+        required = {"instruction_index", "model", "output"}
+        missing = sorted(required - set(self.model_outputs.columns))
+        if missing:
+            raise ValueError(


This should not happen when querying the object but when creating it.

geoalgo · 2026-08-05T16:28:12Z

+            if preloaded is not None:
+                return preloaded.loc[instructions.index]
+        else:
+            preloaded = _try_load_legacy_dataset_completions(


cant we put the logic above in _try_load_legacy_dataset_completions?
(I mean this
if task_data is not None:
preloaded = task_data.completions_for(model_spec)
if preloaded is not None:
return preloaded.loc[instructions.index]
else:
)
otherwise the code is weird since the block is also trying to load dataset completions.

geoalgo · 2026-08-05T16:31:44Z

+from judgearena.tasks.schema.base import StrictFrozenModel
+
+
+class HuggingFaceDatasetSource(StrictFrozenModel):


Do we really need those sources?
Dataset are always loaded locally, we have code to download them so things are reproducible given a git hash.
Can we remove this as a simplification given that the PR is quite complex?

geoalgo · 2026-08-05T16:32:33Z

@@ -0,0 +1,113 @@
+# Task definitions


I wish I would have started the PR by this 😂.

geoalgo · 2026-08-05T16:34:33Z

+## Layout
+
+```text
+tasks/


We do have one folder per task right? (also we should support custom files from the folder so for instance mt_bench.py should probably be in mt_bench/)

kargibora changed the title ~~Refactor/task yaml pairwise~~ (7/n) Complete declarative task runtime and shared pairwise orchestration Jul 21, 2026

kargibora closed this Jul 22, 2026

kargibora reopened this Jul 23, 2026

kargibora mentioned this pull request Aug 4, 2026

(3/n) Introduce declarative YAML task definitions #81

Open

kargibora force-pushed the refactor/task-yaml-pairwise branch from cbc94ff to 06a7211 Compare August 4, 2026 12:04

kargibora commented Aug 4, 2026

View reviewed changes

kargibora marked this pull request as ready for review August 4, 2026 12:47

kargibora added 4 commits August 5, 2026 15:32

Split task schemas by responsibility

d096667

Move dataset, baseline, pairwise, and MT-Bench contracts into focused modules while preserving the public schema imports and YAML behavior.

Document declarative task extensions

e5c9ce0

Explain task identity, adding YAML definitions, and when new dataset adapters or protocol schemas are required.

Refactor pairwise task orchestration

b3f005a

Pass resolved tasks through dispatch, normalize registered dataset inputs behind one contract, and move baseline planning out of the runner.

Make task scoring operational

d16cdfb

Let prompt presets own parsing, resolve scoring adapters at runtime, and remove repeated metric semantics from task YAML.

geoalgo force-pushed the refactor/task-yaml-pairwise branch from 06a7211 to d16cdfb Compare August 5, 2026 13:32

geoalgo reviewed Aug 5, 2026

View reviewed changes

Navigation Menu

Uh oh!

(7/n) Complete declarative task runtime and shared pairwise orchestration - #86

(7/n) Complete declarative task runtime and shared pairwise orchestration#86
kargibora wants to merge 4 commits into
refactor/task-yaml-mt-benchfrom
refactor/task-yaml-pairwise

kargibora commented Jul 21, 2026 •

edited

Loading

Uh oh!

kargibora Aug 4, 2026

Uh oh!

kargibora Aug 4, 2026

Uh oh!

geoalgo Aug 5, 2026

Uh oh!

geoalgo Aug 5, 2026

Uh oh!

geoalgo Aug 5, 2026

Uh oh!

geoalgo Aug 5, 2026

Uh oh!

geoalgo Aug 5, 2026

Uh oh!

geoalgo Aug 5, 2026

Uh oh!

geoalgo Aug 5, 2026

Uh oh!

geoalgo Aug 5, 2026

Uh oh!

geoalgo Aug 5, 2026

Uh oh!

geoalgo Aug 5, 2026

Uh oh!

geoalgo Aug 5, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		@@ -0,0 +1,79 @@
		"""Public schema API for declarative task definitions.

		@@ -0,0 +1,43 @@
		"""Baseline-selection policies shared by pairwise protocols."""



		def run_mt_bench_benchmark(cfg: RunConfig):
		def run_mt_bench_benchmark(


		import pandas as pd

		from judgearena.utils.eval import PrefSummary, compute_pref_summary

		from judgearena.tasks.schema.base import StrictFrozenModel


		class HuggingFaceDatasetSource(StrictFrozenModel):

Uh oh!

Conversation

kargibora commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Before and after

Architecture

Changes

Notes

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

kargibora commented Jul 21, 2026 •

edited

Loading