Bin edge handling for POD_i and POFD_i in the prc output

[rhuva]

Hello,

First time user of METplus (with help from a colleague who is more familiar with how to run METPlus.)
The question I really want answered involves real data, however, to make matters easier I created a simpler unit-test-like example. This simpler example revealed an error in the output from METplus that may help us resolve the issue with real data. We are not sure if it is an error by METplus, an error in our configuration of it or an error in our understanding of the output. The test I created involves a lat-lon grid of observations where the first half of the longitudes are all 0 and the next half are 0.0001 (rainfall in metres of accumulation). The forecast grid is in probability space and for all rainfall thresholds has 0.05 for the first 10 latitudes, 0.1 for the next 10, and so on until 0.9 for the last 8 latitudes (178 * 178 grid). However, my analysis focuses on the forecast threshold of 0.0001 as this is where the obs grid is either 0 or 1 when creating a contingency table.

Expected outcome vs the PODY_i and POFD_i output by METplus in the prc output is attached (
METplus_comparison.xlsx
). I also attach the GridStat conf file (
GridStat_singv_eps_pcpA01_GPM_late.txt
) and the raw prc text file (
grid_stat_SINGV-EPS_vs_GPM-Late_120000L_20220320_000000V_prc.txt
). There are a few values of i that differ with the manual python code, but the most glaring issue is for the PODY_0.85 and POFD_0.85, which are both 0 despite the forecast grid containing points with value 0.9 (by design). We suspect there could be an issue with the GridStat configuration file, or our understanding of what PODY_i and POFD_i are. I understand it is supposed to be the probability of detecting/false detecting when the forecast is greater than the ith probability threshold based on Table 7.13 (Format information for PRC output line type).

Many thanks for your help.

[j-opatz]

Hello,

And thank you for your question. With consultation from our lead MET engineer, I think there is a misunderstanding in how the PODY and POFD are calculated. I'll provide their edited response here:

When verifying probability forecasts, the Nx2ContingencyTable::value_to_row(double t) function converts the probability value to the correct row of the contingency table. Each Nx2 contingency table row has values for the left and right sides of the bin. By convention, the left side is inclusive and the right side is exclusive.

When we apply this logic to the results you're receiving, it makes sense why bin 18 (which holds values 0.85 <= x < 0.9) does not hold the forecast probability values of 0.9 -- those are inclusive in bin 19, which holds the values of 0.9 <= x < 0.95.

It's also worth noting that the calls to is_eq() in the Nx2ContingencyTable code are using the default tight_tol = 10e-10 tolerance level, which means it's checking for equality out to 10 decimal places.

Hopefully this helps explain the discrepancy you are seeing. If you still have concerns on this topic, you are open to continuing the discussion here. Please note that user support for METplus through GitHub Discussions is currently limited due to funding, so more complex or time-intensive topics may not receive timely attention.

[rhuva]

Hi @j-opatz ,

Thanks for your reply. Does this mean I should have values in bin 19 instead? The prc output from METplus shows 0 for both PODY and POFD for bins 18, 19 and 20.

Ultimately, we want to know the POD and POFD for probabilities on the grid greater than some bin values (i.e. for an RoC diagram), so within bin is less useful. I was hoping for this from the prc output (see Section 11.1.6 https://metplus.readthedocs.io/en/latest/Users_Guide/statistics_list.html#statistics-list-p-r). I understand the pct output (Section 11.1.5 https://metplus.readthedocs.io/en/latest/Users_Guide/statistics_list.html#statistics-list-n-o) gives OY_i, which is the within bin observation yes count.

If PODY_i and POFD_i are meant to be within bin as well, I did this calculation with my manual python code and the values returned should be about an order of magnitude lower due to all grid points now outside the bin that are now classified as misses or correct null (see
METplus_comparison_II.xlsx
).

Thanks,

[j-opatz]

Since your goal is to get the probabilities into a binary setup (i.e. all probabilities above the threshold in one bin, and all the probabilities below the threshold in another), I would suggest a slight change to your current threshold settings. You currently have the following in your config file:

FCST_GRID_STAT_PROB_THRESH = ==0.05

That will create bins of 0.05 width, centered at 0.025. Instead, I suggest something like the following:

FCST_GRID_STAT_PROB_THRESH = >0, >.25, >1.0

This would create two bins: one for probabilities from 0 to 0.25, and one for 0.25 to 1.0. You can update these values as you see fit, but they must satisfy MET's requirements of (1) including the range of 0 to 1, (2) be monotonically increasing, and (3) all be of the same inequality type.

If you're interested in continuing to try and see if/what the difference is for binning in MET with what you expect, I'd suggest requesting the matched pairs from GridStat (which is in the netCDF output when using the GRID_STAT_NC_PAIRS_FLAG_RAW flag) and making sure the sample size, the total number of matched pairs, is the same for your Excel calculations.

Unfortunately, if this guidance does not solve your question, our user support for METplus through GitHub Discussions is currently limited due to funding. This particular question is beginning to go beyond the level of support we are currently able to provide. We encourage users to consult the METplus documentation and existing discussions for possible solutions. We appreciate your understanding and continued interest in METplus. You are welcome to post follow-up questions here (we will leave the question open) in the hopes that the METplus community can chime in.

[rhuva]

Hi @j-opatz ,

Thanks for the reply. I managed to solve the issue using one of your suggestions. Analysing the pairs netCDF output revealed that actually METplus had been changing the observation field. I believe I am not the first person to discover the reversing of the latitude dimension behaviour by METplus. After pre-reversing the obs field in my python file that handles the obs reading I can now get the values I expect. I was not able to detect this with the unit-test data as that was identical when flipped, however, analysing the pairs output with a real data case revealed the issue.

Note that in the end I also did not need to run FCST_GRID_STAT_PROB_THRESH = >0, >.25, >1.0. I was able to get the output I expect from the original FCST_GRID_STAT_PROB_THRESH = ==0.05. Indicating that the description of PODY_i and POFD_i in the documentation is correct. These columns represent exceedance PODY and POFD (the hit rates and false-alarm rates of probability values greater than the ith bin).

Thanks!