Repository navigation
Automatic evaluation silently omits models with interval-like text in their names #311
Description
Automatic evaluation silently omits models with interval-like text in their names
At upstream commit 8576c45, evaluate drops a point-forecast column named model-hi-2025-v2 when models is omitted. Passing the two model names explicitly returns both scores.
import pandas as pd
from utilsforecast.evaluation import evaluate
from utilsforecast.losses import mae
df = pd.DataFrame({
"unique_id": ["a", "a"], "ds": [1, 2], "y": [10.0, 20.0],
"baseline": [10.0, 18.0], "model-hi-2025-v2": [12.0, 18.0],
})
print(evaluate(df, metrics=[mae]))
print(evaluate(df, metrics=[mae], models=["baseline", "model-hi-2025-v2"]))Expected: both calls return MAE 1.0 for baseline and 2.0 for model-hi-2025-v2.
Actual: automatic selection returns only baseline; no exception or warning occurs. The same omission occurs with Polars and when interval columns are supplied.
The interval-name regex in _get_model_cols matches inside a model name instead of only at its end. A local patch anchors the numeric interval suffix while retaining fractional/scientific-notation confidence levels. I can submit the focused fix with regression tests if welcome.
Environment: Windows, Python 3.12.14, utilsforecast 0.2.17 (editable checkout at the commit above), pandas 2.3.3, Polars 1.31.0, Narwhals 2.27.1.
AI assistance was used for investigation, implementation and local validation; the reproduction and recorded outcomes were checked locally.
Activity
nasaul commentedon Oct 15, 2026
Thanks for the detailed report and the clear reproduction, student-shivamm!
This is expected behavior. When
modelsis omitted,evaluateinfers the model columns with a heuristic: it drops the id, time and target columns, plus anything that looks like a prediction-interval column (<model>-lo-<level>/<model>-hi-<level>). A model whose name contains that pattern gets caught by the heuristic, which is the case you hit.The supported way to handle this is to pass the model names explicitly with the
modelsargument. Then no inference happens, and every column you list is evaluated:This works the same with Polars and with interval metrics. List only the point-forecast model names in
models; the<model>-lo-<level>/<model>-hi-<level>columns are found fromlevel.In general, we recommend passing
modelsexplicitly whenever your model names aren't plain identifiers, or whenever the dataframe has extra columns you don't want evaluated.Closing this one, since the
modelsargument covers it. Thanks again for taking the time to investigate!