(5/n) Package m-ArenaHard task families with language variants

[kargibora]

Summary

This PR migrates m-ArenaHard v0.1 and v2.0 into the declarative task registry.

m-ArenaHard supports multiple benchmark versions and many language-specific views. Defining every combination as a
separate YAML file would duplicate the same dataset, protocol, judge, and scoring configuration dozens of times.

This PR introduces task-family variants so each benchmark version is defined once while language and language-group
views are resolved from validated task suffixes.

Task-family resolution

The task definitions declare their supported variants:

variants:
  selector: language
  values: [ar, cs, de, en, es, fr, ...]
  groups:
    EU: [cs, de, el, en, es, fr, it, nl, pl, pt, ro, uk]

This allows the registry to resolve invocations such as:

  m-arena-hard-v2.0
  m-arena-hard-v2.0-de
  m-arena-hard-v2.0-fr
  m-arena-hard-v2.0-EU

The base ID selects all declared languages. A language suffix selects one language, while a group suffix expands to its
declared members.

The resolved task keeps both pieces of information:

• The stable definition ID, such as m-arena-hard-v2.0
• The runtime invocation and selected language view, such as m-arena-hard-v2.0-de

This selection is also included in CLI inspection and run metadata.

Basically ResolvedTaskSpec contains all the important detail about tasks.

Task organization

The definitions are organized as:

  judgearena/tasks/definitions/m_arena_hard/
  ├── _base.yaml
  ├── m-arena-hard-v0.1.yaml
  └── m-arena-hard-v2.0.yaml

The shared base defines:

• Supported language values and the EU group
• Pairwise runner and generation mode
• Judge prompt, parser, and swap policy
• Scoring configuration
• Canonical dataset fields
• Shared model-output source

Each version then defines its own:

• Dataset repository and pinned revision
• Download patterns
• Default baseline
• Reference implementation

m-ArenaHard v0.1 uses Aya Expanse 8B as its task default, while v2.0 uses Gemini 2.5 Flash. Both allow runtime baseline
overrides.

Dataset handling

The m_arena_hard dataset adapter reads the resolved language selection and:

Resolved task invocation
-> download the pinned dataset sources
-> select the requested languages
-> normalize prompts and question IDs
-> load matching reference outputs
-> return canonical rows to the pairwise runner

The generic runner does not need to understand language suffixes or m-ArenaHard dataset layouts.

Validation

Variant definitions are validated to prevent:

• Duplicate language values
• Empty language groups
• Groups referencing unknown languages
• Variant IDs conflicting with real task IDs
• Unsupported runtime suffixes

Legacy m-ArenaHard version, revision, baseline, and prompt maps are removed because this information is now owned by the task definitions.

Main architectural difference

Before:
m_arenahard.py = configuration + task parsing + downloading + normalization

Now:
Task YAML = stable benchmark configuration
Task registry = task and language resolution
m_arenahard.py = downloading and dataset normalization
Pairwise runner = generation and evaluation

Notes

• New added tests is ~200 line, that is why the PR looks a bit larger
• As can be seen, we have defined new resolution - otherwise we would need to define lots of new tasks. It simply looks at suffixes and compare this with the known values, also defined in the task specification. If matched then it looks at language column and filter it. Similarly for category, we can use it to filter instruction categories.

[kargibora]

As we dont want to populate every language task with its own file, this allow us to basically select which sub-tasks we want to execute.

[kargibora]

Now task can be m-arena-hard-en for example, which is basically m-arena-hard but only en samples. Thus we need an extra step for resolving this.

[geoalgo]

not for now, but perhaps having another character to split task configuration would be easier, for instance m-arena-hard:en (parsing seems too complicated)

[kargibora]

An example for adding an functionality in the schema.

In TaskSpec, we can specify whether to selefct a specific subset (task or language based) easily.

[geoalgo]

LGTM

[geoalgo]

not for now, but perhaps having another character to split task configuration would be easier, for instance m-arena-hard:en (parsing seems too complicated)