Skip to content

fix: retain model names containing interval-like text in evaluation - #312

Closed
student-shivamm wants to merge 1 commit into
Nixtla:mainfrom
student-shivamm:codex/evaluation-model-names
Closed

student-shivamm wants to merge 1 commit into
Nixtla:mainfrom
student-shivamm:codex/evaluation-model-names

Conversation

@student-shivamm

Copy link
Copy Markdown

Fixes #311

When models is omitted, evaluate treats model-hi-2025-v2 as an interval column and silently omits its score. Explicit model selection returns the expected score. Anchor interval matching to the numeric suffix so interval-like text inside model names is retained, while integer, fractional and scientific-notation interval columns remain excluded.

Regression tests compare inferred and explicit selection in pandas and Polars and assert known MAE values. The same 24-case matrix has 16 failures and 8 passing controls on upstream, and passes with the patch.

Validation: evaluation/loss suites plus regression module: 190 passed, one skipped (upstream Windows distributed-test restriction). Separate data/validation/processing plus regression run: 399 passed, ten skipped. All configured pinned pre-commit hooks pass on the changed files. Direct no-incremental mypy with runtime dependencies reports the same 73 existing errors on upstream and the candidate; the isolated configured hook passes. No cross-platform or distributed CI result is claimed.

AI assistance was used for investigation, implementation and validation. The reproduction and recorded local results were checked; acceptance remains the maintainers' decision.

@CLAassistant

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Automatic evaluation silently omits models with interval-like text in their names

3 participants