Run-to-run variance, significance tests and minimum-runs planning for AI agent benchmarks (SWE-bench, WebArena, OSWorld, and anything else that scores tasks as success or failure).
Agent leaderboards rank systems by gaps of 1–3 pp, usually from one run. agentstats tells you
whether a gap is real, how many runs you need before it can be, and whether your noise is the kind
that breaks standard tests.
pip install "agentstats @ git+https://github.com/RishabhPande/agentstats.git" # core
pip install "agentstats[swebench] @ git+https://github.com/RishabhPande/agentstats.git" # + SWE-bench leaderboard loaderagentstats is not on PyPI: the agentstats name there belongs to an unrelated package. Install from this repo.
Real data: 12 configurations x 10 runs on SWE-bench Verified, from
ASSERT-KTH (CC-BY-4.0). "Repeated" means the same configurations, tasks and settings, run again; outcomes differ run to run. The two leaders sit in
the same statistical tier, and which one tops a published leaderboard depends on which run you happened to pick.
Regenerate with python examples/leaderboard_uncertainty.py (needs matplotlib, about 25 s).
One row per (agent, task, run). Extra columns are treated as nuisance factors.
| agent | task | run | success | api_date | env_version | latency |
|---|---|---|---|---|---|---|
| openhands | django-11099 | 0 | 1 | 2026-07-01 | v1.3 | p50 |
| openhands | django-11099 | 1 | 0 | 2026-08-01 | v1.3 | p95 |
success can be 0/1 or a score in [0, 1]. Each run should be one complete pass over the benchmark.
Missing values in agent, task, run or success are rejected instead of silently
changing which outcomes enter the analysis. Optional factor columns may contain missing values.
import agentstats as ast
df = ast.load("runs.csv")
ast.agent_scores(df) # score, SE from run noise, CI
ast.design_effect(df) # is noise correlated within runs? (DEFF >> 1 -> use level="run")
ast.compare(df, "A", "B") # studentized stratified run-permutation test
ast.compare(df, "A", "B", level="run") # run-level Welch test, robust to API drift / env breakage
ast.pairwise(df, adjacent_only=True) # Holm-corrected adjacent leaderboard gaps
ast.tiers(df) # statistically tied groups (report these, not ranks)
ast.flip_statistics(df) # Kendall tau / P(#1 changes) between single-run leaderboards
ast.decompose(df) # variance components: seed vs api_date vs env_version ...
ast.plan(pilot_df, delta=0.02) # minimum runs per agent to detect a 2 pp gap
print(ast.report(df)) # all of the above, as markdowncompare() validates its options before selecting a test: alpha must be a finite real number strictly between 0 and 1, n_perm a positive integer, studentize a boolean, and run_test either "welch" or "permutation". These checks apply even when the chosen test path does not use an option. Misspellings do not select a different test. Valid-input formulas and results are unchanged.
agentstats report runs.csv
agentstats compare runs.csv agentA agentB --level run --strata api_date
agentstats plan pilot.csv --delta 0.02
agentstats plan-generic --tasks 500 --delta 0.02 --flakiness 0.1
agentstats audit leaderboard.json --sd 1.5 # audit published numbers without re-running
# audit the live SWE-bench leaderboard (pip install agentstats[swebench])
git clone --depth 1 https://github.com/SWE-bench/experiments
agentstats swebench-audit experiments --pass1 --top 20- Pilot, then plan. Run each agent at least 3 times;
agentstats plangives k for the smallest gap you want to claim. With no pilot, use k = 5. - Spread runs over conditions you don't control: at least 2 API dates or environment builds. Interleave competing agents so you can block on those conditions (
strata=). - Diagnose correlation. If median DEFF > 1.5, use
level="run". - Test pairs and correct for multiplicity (Holm; built in).
- Report tiers, not ranks.
- Log conditions: snapshot ids, provider fingerprints, image digests, latency, and per-task outcomes.
- State the estimand: this benchmark (
inference="benchmark") or the task population ("population").
k_min_from_components() rejects unknown inference modes, non-finite inputs,
negative variance components, nonpositive gaps, alpha/power outside (0, 1), and
nonpositive or non-integer run floors/task counts. Booleans are not numeric
planner options. Population inference requires a task count. Valid inputs use
the same formula; an unattainable population gap still returns infinity.
| Situation | Use |
|---|---|
| 1 run per agent | inference="population" (paired sign-flip over tasks); collect more runs |
| ≥2 runs, DEFF ≈ 1 | default: studentized stratified run-permutation (level="task") |
| ≥3 runs, DEFF > 1.5 | level="run" (Welch on complete-run scores) |
| agents run on the same API dates / images | add strata=[...] to block on them |
pytest checks type-I error under sharp and weak nulls and under run-correlated noise, the power of the planner, variance-component recovery, Monte Carlo resolution under Holm correction, and the CLI.
See reproduce/.
R. Pande. Within the Noise: The Reproducibility Crisis in Agent Benchmarks. 2026.
MIT license.