grid_stat and interp to get neighborhoods for forecast grids and observation grids

[cbridge234]

Hi METplus Team,

I am writing to ask a question about verification with MET. I would like to verify gridded deterministic forecasts with gridded observations of a continuous variable using a neighborhood approach.

The approach I am thinking of is similar to the Fractions Skill Score where forecast and observed neighborhoods of grid cells are compared. The difference is I don't want to apply thresholds to each grid cell to produce binary grids. Instead, I would like to use the original forecast/observed grid values from each neighborhood in order to calculate a different neighborhood score.

An example of such a score is the "Scale-dependent error" by Rezacova et al. (https://doi.org/10.1016/j.atmosres.2005.08.011). In this work the authors sorted forecast values from a neighborhood from smallest to largest, They did the same for observations. Finally the RMSE for the neighborhood was calculated from the ordered series. Repeating this for different neighborhood sizes allowed the authors to see how RMSE changed with neighbourhood size, i.e. with spatial scale.

I don't believe I can currently achieve this with MET but I wondered if I might be able to get part way there. My question is:
Could I use MET to generate forecast neighborhoods and observation neighborhoods then output the values of the grid cells in each neighborhood to stat files?
If so I could then pick up the stat files and calculate verification scores like the one above separately, myself.

I looked at the grid_stat tool to see if I could use the "interp" dictionary to define neighbourhoods, but I think I also need to supply the "method" keyword which applies a smoothing operation to the grid cell values in the neighborhood. I don't think there is a way around the smoothing so I don't think it will work. I Also looked at the ensemble_stat tool. This also accepts the "interp" dictionary and appears to accept "HIRA" as a value for the method keyword for ensemble forecasts. I think this could return the values in my neighborhood that I am after but I would need to apply it to both my deterministic forecast grid and the observations grid, which is not really how the tool was intended to be used!

Any ideas would be great. I hope my question makes sense.

Thanks
Chris

[JohnHalleyGotway]

Hello Chris,

Can you please let me know the institution or organization with which you work? I'll add an appropriate "requestor" label to this discussion, which helps up keep track of the community support we provide.

This is really a great question, and thanks for linking to the paper. I've been thinking about it today and discussed it with other scientists and engineers during our weekly METplus team meeting. It's certainly appealing to use a neighborhood statistic that does not depend on a threshold. I can definitely see the benefit.

The good news is that I can see how/where we'd add this sort of computation in the MET library code and it certainly fits nicely within the context of existing functionality. The bad news is that I really can't think of a way to use existing output from MET to make its computation easier. For each grid point, you'd like to have a list of all the forecast and observation values in the surrounding neighborhood... so you could sort/pair/score them. And there just isn't a great way for MET to output that.

During our team meeting, there was some interest among project leads in this approach. But our funding and timelines are tight. We plan to talk with our resident statistics, Barb Brown, and Craig Schwartz, who's done a lot of work in neighborhood verification, to get their advice.

So the short answer is no, I don't have any immediate recommendations. But I am hopeful that this could make for a great enhancement to Grid-Stat itself.

Thanks,
John Halley Gotway

[cbridge234]

OFFICIAL Hi John, Thanks for the quick reply. I'm at the Bureau of Meteorology in Australia. It's great that the idea is of interest to the METplus team, I also understand the constraints you are facing. I'm not sure where I could contribute not being a C++ person but I suspect I'll prototype something in Python if that might be useful. Cheers, Chris

[JohnHalleyGotway]

Chris,

I'll note that I did add a "draft issue" about this to the METplus Idea Catcher project board, which we've recently started using to house good ideas for which we don't yet have a way of funding. If/when we find a way forward, we'd convert it to a full-fledged issue and assign it to a development project.

I've did mention this in an email to Marion Mittermaier (@mpm-meto) at the Met Office, and we've been discussing with Barb Brown (@bgbrowntollerud) and Craig Schwartz internally at NCAR. All, please feel free to chime in on this discussion.

Thanks,
John