Add meta evaluation task - #76
Conversation
ErlisLushtaku
changed the title
|
|
||
| def parse_pairscore_pref(judge_completion: str, *, temperature: float) -> float: | ||
| score = pair_score_parser(temperature).parse_model_raw(judge_completion) | ||
| if score is None or np.isnan(score): |
There was a problem hiding this comment.
Why we return tie=0.5 for parsing errors? We have to make sure this does not change the Cohen's Kappa or any accuracy
There was a problem hiding this comment.
Yes, malformed outputs become ties and therefore affect accuracy and the Cohen kappa score. This matches the reference implementation here. I kept this behavior to match the reference results, since dropping them or adding an invalid label would change the reported metrics. Let me know if you think we should intentionally diverge.
| replace=False, | ||
| random_state=int(rng.integers(0, 2**32 - 1)), | ||
| ) | ||
| if exclude_ties: |
There was a problem hiding this comment.
Sampling num_battles row then excluding tie means that our battles will have n <= num_battles
| args.prompt_mode, | ||
| provide_explanation=args.provide_explanation, | ||
| ) | ||
| judge_chat_model = make_model( |
There was a problem hiding this comment.
Should we call build_default_judge_model_kwargs before? Or this is already enough
| annotation_cache: AnnotationCache | None = None, | ||
| ) -> pd.DataFrame: | ||
| n_total = len(df_sample) | ||
| n_batches = (n_total + args.batch_size - 1) // args.batch_size |
There was a problem hiding this comment.
Not so problematic but we never check args.batch_size > 0. It can raise division by zero by mistake.
|
Apart from these I feel like there are some duplications: (1) The existing function already supports continuous preferences: fit_bradley_terry(df, pref_col="pref_llm")(2) The PR introduces: PromptModeSpec
resolve_prompt_mode()
_read_prompt()JudgeArena already has: JudgePromptPreset
ResolvedJudgePrompt
resolve_judge_prompt()Are these exchangable or we are required to implement this specifically? |
|
@kargibora Regarding each point:
Good point, I removed it and now use
|
Introduce the core judge meta-evaluation building blocks and package prompt resources so later CLI wiring can reuse them unchanged.
Add the annotation orchestration path and expose meta-eval as a flat --task route alongside generate+judge and ELO workflows.
Add unit/integration coverage for meta-eval behavior and ensure prompt resources remain loadable from wheel and sdist installs.
Describe the new meta-eval CLI path, prompt modes, swap-mode semantics, and temporary SQLite cache limitations for operators.
Store rendered prompt text rather than LangChain prompt objects so SQLite-backed meta-eval runs complete after annotation. Includes-AI-Code: true
Reuse shared Bradley-Terry and judge defaults, and reject invalid annotation batch sizes. Includes-AI-Code: true
Keep the meta-eval tests aligned with the hierarchical RunConfig interface retained from main. Includes-AI-Code: true
56949c0 to
340fffd
Compare
2 participants
Summary
--task meta-evalfor evaluating judge models against human-labeled arena battles.RunConfigCLI.Usage
judgearena \ --task meta-eval \ --judge.model OpenRouter/deepseek/deepseek-v3.2 \ --meta_eval.reference_arena LMArena-140k \ --meta_eval.prompt_mode standard \ --meta_eval.languages '["en", "es"]' \ --meta_eval.top_models 20 \ --meta_eval.battles_per_model 50 \ --meta_eval.n_bootstraps 1000 \ --run.seed 0Outputs
Runs produce:
annotations.parquetresults.jsonsummary.csvargs.jsonrun-metadata.v1.jsonWith
--judge.swap_mode both, annotations contain one row per judge pass with normalized model ordering and anorientationcolumn.Cache note
Meta-eval temporarily retains the existing per-battle SQLite WAL cache. It should be used from a single host and not concurrently across NFS-mounted compute nodes. Cache unification remains deferred to the dedicated caching work.