Clarification on Calculation of UFSS

[reza-armuei]

Hi team,

I have been using METplus to calculate the Fractions Skill Score (FSS) for our ACCESS model rainfall forecasts at the Bureau of Meteorology. I noticed that the Uniform Fractions Skill Score (UFSS) in MET is computed over the neighbourhood scale (meaning the calculated UFSS varies with changes in neighbourhood size). However, if UFSS is intended to serve as a reference score as defined in Roberts and Lean 2008 to assess whether forecasts are "good" enough to be considered skilful based on the FSS value, it should be calculated at the grid scale (i.e., with a neighbourhood window size = 1).
For your reference, Roberts and Lean 2008 state:

The other dashed line (labeled “uniform”) represents the FSS that would be obtained at the grid scale ($n = 1$) from a forecast with a fraction/probability equal to $f_o$ at every point. It is given by $FSS_{uniform}$ = 0.5 + ${f_o}/{2}$ (i.e., halfway between random forecast skill and perfect skill);

I am aware of the concerns raised about this approach (use of uniform FSS as a reference score) mentioned in Antonio and Aitchison 2025, but regardless of whether this is considered a reasonable reference score, I wanted to clarify: whether you are introducing a new reference score by calculating UFSS over the neighbourhood scale?

Looking forward to your insights.

Thanks,
Reza

[j-opatz]

Thank you for your concern Reza,

This is a worthwhile topic I would like to get more attention from our lead engineers on before answering. I will make a note of what you've provided here and see if I can find the decisions behind our choice to allow UFSS calculation across neighborhoods >1.

[j-opatz]

Hi Reza,

Thank you again for bringing this discrepancy to our attention.

After consultations with our internal statistician, as well as the METplus team, it was deemed that our current calculation of UFSS is not aligned with the original intent of UFSS.

To correct this, I've created a MET issue here, that intends to update the name of the current method MET uses to calculate a neighborhood-relative UFSS, and create a new column of data that will reflect the Roberts and Lean calculation at n=1. Hopefully we can fit this work into the upcoming 7.0 release for METplus.