Add confidence intervals to main code

[lindsaydbrin]

Every eva metrics run now emits 95% bootstrap confidence intervals for composite and main metrics (not sub-metrics) in metrics_summary.json. CIs are deterministic within a run (same run_dir.name → byte-identical bounds) and independent across runs.

Scope (purely additive — no existing field renamed or changed):

• overall_scores.{COMPOSITE}.{mean_ci_lower, mean_ci_upper, mean_ci_n_scenarios} - for 6 composites: [EVA-A, EVA-X, EVA-overall] x [_pass, _mean]
• overall_scores.pass_k.{COMPOSITE}.{stat}_ci_{lower,upper} - for pass_at_1, pass_at_k, pass_power_k_observed (multi-trial only). pass_power_k_theoretical does not have CIs (clearly).
• per_metric.{name}.{mean_ci_lower, mean_ci_upper, mean_ci_n_scenarios} for all metrics (typically/currently: task_completion, faithfulness, agent_speech_fidelity, conversation_progression, turn_taking, conciseness).
• per_metric.{name}.pass_k.{stat}_ci_{lower,upper} (k>1 only, again of course).

Statistical design (matches the approach on analysis/eva-bench-stats/ / in the paper): percentile bootstrap on per-scenario sample means, N_BOOT=2000, ALPHA=0.05, seed derived from run_dir.name via SHA-256 (cross-process-stable).

[lindsaydbrin]

Is this correct since it changes output fields?

[lindsaydbrin]

@tara-servicenow asked about handling missing/errored samples. Here's Claude's explanation:

How missing/errored trials are handled in CI computation
CI computation uses get_score() to filter raw trial data: errored trials (score.error set) and missing values return None and are dropped. Valid trials are grouped by scenario ID and averaged to produce one value per scenario; scenarios where every trial is None are dropped entirely. The bootstrap CI is then computed over those per-scenario means.

This matches what the main metric aggregation does — both exclude errors and missing values before computing statistics — with two differences worth noting:

1. Granularity. The main loop counts error_count and missing_count separately and tracks totals in the output. The CI path collapses both to None with no separate tracking; that distinction is only visible in the main aggregation stats.
2. Unit of analysis. The main loop treats each record/trial as an independent data point for the mean. The CI groups trials by scenario first and averages them, so each scenario contributes one value regardless of how many trials it has. This is the statistically correct unit for the bootstrap — a design choice, not a discrepancy.

You can see the get_score() on line 116 of aggregation.py:

    return _scenario_means(
        {record_id: record_metrics.get_score(metric_name) for record_id, record_metrics in all_metrics.items()}
    )

and then lines 96-97 in _scenario_means():

        if val is None:
            continue

@gabegma does this make sense to you? @tara-servicenow mentioned that you had thought about handling None vs. errors.

[tara-servicenow]

This looks good to me but will let Gab review