Proposal: an EvalPort adapter for PawBench's tasks + submissions (interchange format for portable eval data)

[adhabnr-ux]

Proposal: an evalport adapter for PawBench's tasks + submissions

Hi — I maintain EvalPort, an Apache-2.0 interchange format (JSON Schema + Python/TS SDK, evalport-sdk) for portable LLM eval test cases, graders, and results. I read through the actual task/grading/submission code in this repo before writing this (not just the README), and I think PawBench's data model is a genuinely good fit — enough that I'd like to build and contribute the adapter myself if that's welcome.

What I actually looked at

• data/pawbench-v1.0/tasks/T001_claweval_M005_score_canon.md (and neighboring task files) — the YAML front-matter + Prompt/Expected Behavior/Grading Criteria/Automated Checks/LLM Judge Rubric structure
• pawbench/grader.py — grade_task(), the three grading_types (automated/llm_judge/hybrid), GradeResult (task_id, score, max_score, grading_type, breakdown, notes, score_simple), and _combine_grades()'s weighted combination
• submissions/pawbench-4models-opusjudge-20260529__qwen3.6-plus__qwenpaw.json — a real committed submission (run, model, harness, overall, automated, judge, tasks, tasks_errored, tasks_missing, plus the by_complexity/by_environment/by_scenario/by_scenario_top/by_modality/by_channel/by_capability/by_source/by_category/by_subcategory/by_grading slice breakdowns)

The mapping

Task → TestCase

PawBench (real fields, T001_claweval_M005_score_canon.md)	EvalPort TestCase
front-matter id: M005_score_canon	id
## Prompt body	input
## Expected Behavior body	expected_output
labels.capabilities, labels.modality.type/.channels, labels.scenario, labels.complexity, labels.environment, plus the submission's by_source/by_category keys	tags (e.g. ["capability:Tool_Use", "scenario:Content_Creation/Design", "complexity:L3", "environment:closed", "modality:multimodal", "source:claweval"])
timeout_seconds: 1200	timeout_ms (×1000)

Grading → graders[], using the real grading_weights

T001 has grading_type: hybrid with grading_weights: {automated: 0.3, llm_judge: 0.7}. That maps onto two Grader entries on the same TestCase, weighted the same way EvalPort's own weight field already supports:

graders = [
    {
        "id": "T001_automated",
        "type": "code",
        "weight": 0.3,
        "params": {"language": "python", "source": automated_checks_source},  # the grade() body, verbatim
    },
    {
        "id": "T001_llm_judge",
        "type": "llm_judge",
        "weight": 0.7,
        "params": {"model": "claude-opus-4-5-20251101", "prompt": llm_judge_rubric_text},  # real default from grader.py
    },
]

One honest wrinkle I want to flag rather than paper over: PawBench's grade() function signature is grade(transcript: list, workspace_path: str) -> dict — it scores the whole agent transcript plus final workspace filesystem state, not a single actual_output string. EvalPort's code grader type doesn't currently pin down a calling convention for params.source beyond "language + source," so this isn't a hard blocker, but it's worth resolving explicitly (e.g. does the adapter serialize workspace file contents into TestCase.context, or is this a case for a pawbench.transcript/pawbench.workspace_snapshot metadata extension?) before I write code that guesses.

Submission JSON → ResultSet

The committed submissions/*.json files are already aggregated (one JSON per model×harness run, not one row per task) — there's no raw per-task GradeResult dump in the repo, only the rolled-up overall/by_* breakdowns. So a faithful adapter has to be honest about what's actually round-trippable:

• run + model + harness → ResultSet.run_id (e.g. "qwen3.6-plus__qwenpaw"), suite_id: "pawbench-v1.0"
• overall/automated/judge → ResultSet.summary.avg_score plus summary.metadata["pawbench.automated"] / ["pawbench.judge"] (EvalPort's built-in summary.by_grader is keyed by grader id, not by PawBench's five-dimension taxonomy, so the by_scenario/by_capability/by_complexity/by_modality/by_environment/by_source/by_category/by_subcategory blocks would live under ResultSet.metadata["pawbench.by_scenario"] etc. — preserved verbatim, not recomputed, the same <namespace>. metadata convention this repo's other adapters already use)
• tasks_errored/tasks_missing → summary.failed/summary.skipped (best-effort; PawBench doesn't distinguish per-task which specific tasks errored vs. went missing in the aggregate file, so per-Result fidelity is limited to what's actually in the file)

Comparable adapters already in the repo (verified, not asserted from memory)

• financebench-openeval-adapter — closest shape match: a fixed benchmark with real questions + gold answers + an llm_judge grader (because FinanceBench's free-text answers don't fit exact_match either) + real per-model result files with an existing human-annotated label, carried into EvalPort's Result/GraderResult verbatim rather than re-graded.
• lm-eval-harness-openeval-adapter — comparable on the "harness" side: documents two real discrepancies it found between lm-eval's docstrings and its actually-installed behavior, and is explicit about what does/doesn't round-trip (it can't reconstruct a live lm_eval.api.task.Task, same category of limitation I'm flagging above for PawBench's transcript+workspace grader signature).

What I'd propose

If this is a direction the maintainers are open to, I'd build pawbench-openeval-adapter under adapters/ in the EvalPort repo (not a PR against PawBench itself — no code changes needed here), following the same pattern: to_openeval()/from_openeval() for tasks, result_to_openeval() for submissions, real fixtures pulled from files actually in this repo, tests run against the real openeval.validate.validate_suite()/validate_result_set(). I'd rather get a maintainer's read on the grader-signature question above (and on whether by_scenario-style slice metadata under a pawbench. namespace is a reasonable round-trip target, versus something you'd want expressed differently) before writing it, rather than build first and ask later.

Happy to close this out if PawBench↔EvalPort interop isn't something you want to carry, no worries either way.

— Sahi, independent contributor (not affiliated with agentscope-ai)