Skip to content

evals: remove misleading timing metrics - #33

Merged
alexmdac merged 2 commits into
mainfrom
remove-eval-timing
Mar 1, 2026
Merged

alexmdac merged 2 commits into
mainfrom
remove-eval-timing

Conversation

@alexmdac

@alexmdac alexmdac commented Mar 1, 2026

Copy link
Copy Markdown
Owner

Remove Duration and LLMTime tracking from eval results — these are misleading with concurrency control and retries (they measure queue wait and backoff time more than model performance).

Keep StarlarkTime since it measures local execution with no retry noise.

Also simplify the live display: remove duration stats from finished/in-progress lines, reduce refresh rate to 2/s.

Token counts and attempt counts are better metrics for comparing models.

alexmdac and others added 2 commits March 1, 2026 08:23
These timings are misleading with concurrency control and retries —
they measure queue wait and backoff time more than model performance.
Token counts and attempts are better metrics.

Keep StarlarkTime since it measures local execution with no retry noise.

Co-authored-by: Shelley <shelley@exe.dev>
Co-authored-by: Shelley <shelley@exe.dev>
@alexmdac
alexmdac merged commit 814c7d2 into main Mar 1, 2026
1 check passed
@alexmdac
alexmdac deleted the remove-eval-timing branch March 1, 2026 08:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant