Using MET V12.01 stat_analysis for UGRD and VGRD aggregation

[johnlwagner]

Greetings!
We recently switched from using MET V11.0.2 to MET V12.0.1 on WCOSS. When running stat_analysis to aggregate UGRD and VGRD into wind direction verification, we encountered many instances where the ME and MAE both returned NAs on the ROW_MEAN_WDIR line, despite there being plenty of cases for verification. For example:

JOB_LIST:      -job aggregate_stat -fcst_lead 240000 -fcst_valid_beg 20250601_000000 -fcst_valid_end 20250630_000000 -obs_valid_beg 20250601_000000 -obs_valid_end 20250630_000000 -li
ne_type MPR -by VX_MASK -out_line_type WDIR 
     COL_NAME: VX_MASK TOTAL      FBAR      OBAR       ME MAE
ROW_MEAN_WDIR: conus      22 107.97522 113.60505 NA        NA
    AGGR_WDIR: conus      22 106.36279 112.51111 -6.14832  NA

This seems to occur when both the UGRD and VGRD observations are both 0 in the MPR output from point_stat:

V12.0.1 NDFD  NA   240000    20250617_000000 20250617_000000 000000   20250617_000000 20250617_000000 UGRD     m/s        Z10      UGRD    m/s       Z10     ADPSFC conus     NEAREST     1           NA          NA         NA         NA    MPR       667  68 KBBW 41.4364  -99.6422 NA NA  -3.38289      0          NA NA NA NA NA NA
V12.0.1 NDFD  NA   240000    20250617_000000 20250617_000000 000000   20250617_000000 20250617_000000 VGRD     m/s        Z10      VGRD    m/s       Z10     ADPSFC conus     NEAREST     1           NA          NA         NA         NA    MPR       667  68 KBBW 41.4364  -99.6422 NA NA   1.23127      0          NA NA NA NA NA NA

If I remove these lines from the daily point_stat output files, I am then able to get the ME and MAE scores from stat_analysis, just with fewer cases:

JOB_LIST:      -job aggregate_stat -fcst_lead 240000 -fcst_valid_beg 20250601_000000 -fcst_valid_end 20250630_000000 -obs_valid_beg 20250601_000000 -obs_valid_end 20250630_000000 -li
ne_type MPR -by VX_MASK -out_line_type WDIR 
     COL_NAME: VX_MASK TOTAL      FBAR      OBAR       ME      MAE
ROW_MEAN_WDIR: conus      21 107.78408 113.60505  2.38096 21.42857
    AGGR_WDIR: conus      21 106.17138 112.51111 -6.33973 NA

I am using the standard config file for stat_analysis. My stat_analysis command looks like this:

   /apps/ops/para/libs/intel/19.1.3.304/met/12.0.1/bin/stat_analysis \
   -lookin $lookinFile \
   -job aggregate_stat \
   -line_type MPR \
   -out_line_type WDIR \
   -by VX_MASK \
   -fcst_lead 24 \
   -fcst_valid_beg 20250601 \
   -fcst_valid_end 20250630 \
   -obs_valid_beg 20250601 \
   -obs_valid_end 20250630 \
   -out $outFile

This was not an issue when running with MET V11.0.2. Please let me know if I don't have something configured correctly in point_stat or stat_analysis, if this is a bug, or if something else might be causing this behavior.
Thanks
John

[JohnHalleyGotway]

Hi @johnlwagner, thanks for pointing out this issue with STAT-Analysis.

I used our latest development version with a regression test sample data file (mpr.txt) to replicate the behavior you describe by running:

./stat_analysis -lookin mpr.txt -job aggregate_stat -vx_mask DTC165,DTC166 -line_type MPR -out_line_type WDIR -by VX_MASK -v 4 -log run_sa.log

Produces this output, where MAE is always NA and ME is NA in the ROW_MEAN_WDIR row:

     COL_NAME: VX_MASK TOTAL      FBAR      OBAR        ME MAE
ROW_MEAN_WDIR: DTC165   1175  91.1138  256.70314  NA        NA
    AGGR_WDIR: DTC165   1175 198.91955 218.53769 -19.61813  NA
ROW_MEAN_WDIR: DTC166   3527 293.247   290.54049  NA        NA
    AGGR_WDIR: DTC166   3527 294.8382  295.16897  -0.33077  NA

The STAT-Analysis job loops over all the input matched pair lines and computes the angle between the forecast and observation wind vectors. However, the wind direction is undefined for (u, v) = (0, 0) and that results in a -9999 bad data value. Here's the line of code where STAT-Analysis bails out when it encounters a bad angle value which results in NA values in the output. And here's the DEBUG(4) log message about it:

DEBUG 4: write_job_aggr_wind() -> ROW_MEAN_WDIR: [1] difference of forecast direction 144.447 - observed direction 139.635 = 4.81136 degrees
...
DEBUG 4: write_job_aggr_wind() -> ROW_MEAN_WDIR: [12] difference of forecast direction 17.4025 - observed direction 296.565 = 80.8374 degrees
DEBUG 4: write_job_aggr_wind() -> ROW_MEAN_WDIR: [13] difference of forecast direction 83.8565 - observed direction -9999 = -9999 degrees

Although version 11.0.2 includes exactly the same logic on this line. Are you sure that you're seeing different output from version 11.0.2?

You could try rerunning your original job with the options -out_wind_thresh gt0 -out_wind_logic INTERSECTION. That should discard from the analysis any points where the forecast or observation wind speed is 0, and thus, the wind direction is undefined.

I will note that in my testing I did find a minor bug in the log output that's written. This log message:

DEBUG 2: Processing Job 1: -job aggregate_stat -line_type MPR -by VX_MASK -out_line_type WDIR -out_wind_thresh >0 -out_wind_logic UNION 

Should actually say:

DEBUG 2: Processing Job 1: -job aggregate_stat -line_type MPR -by VX_MASK -out_line_type WDIR -out_wind_thresh >0 -out_wind_logic INTERSECTION 

The bug is that it's reporting the value for cnt_logic when it should be reporting the value for wind_logic. I'll patch that up and include it in the upcoming MET version 12.1.0 release.

Please let me know if adding -out_wind_thresh gt0 -out_wind_logic INTERSECTION performs the way you expect.

[johnlwagner]

Testing V12.0.1 with -out_wind_thresh gt0 -out_wind_logic INTERSECTION, I get the same results that I got when I removed the 0,0 lines from the point_stat output:

 JOB_LIST:      -job aggregate_stat -fcst_lead 240000 -fcst_valid_beg 20250601_000000 -fcst_valid_end 20250630_000000 -obs_valid_beg 20250601_000000 -obs_valid_end 20250630_000000 -l
ine_type MPR -by VX_MASK -out_line_type WDIR -out_wind_thresh >0 -out_wind_logic UNION 
     COL_NAME: VX_MASK TOTAL      FBAR      OBAR       ME      MAE
ROW_MEAN_WDIR: conus      21 107.78408 113.60505  2.38096 21.42857
    AGGR_WDIR: conus      21 106.17138 112.51111 -6.33973 NA

The out_wind_logic in the stat_analysis output file is listed as UNION, even though I set it to INTERSECTION (as you noted above). This does seem to solve the problem. Thanks!

Running with the same data for V11.0.2 with the 0,0 lines and without -out_wind_thresh gt0 -out_wind_logic INTERSECTION (which is how I had run this previously), I get the same results on the ROW_MEAN_WDIR line, but an extra case on the AGGR_WDIR line:

JOB_LIST:      -job aggregate_stat -fcst_lead 240000 -fcst_valid_beg 20250601_000000 -fcst_valid_end 20250630_000000 -obs_valid_beg 20250601_000000 -obs_valid_end 20250630_000000 -l
ine_type MPR -by VX_MASK -out_line_type WDIR 
     COL_NAME: VX_MASK TOTAL      FBAR      OBAR       ME      MAE
ROW_MEAN_WDIR: conus      21 107.78408 113.60505  2.38096 21.42857
    AGGR_WDIR: conus      22 106.36279 112.51111 -6.14832 NA

I also get the same results for V11.0.2 as V12.0.1 when I add -out_wind_thresh gt0 -out_wind_logic INTERSECTION back in, with only 21 cases on the AGGR_WDIR line:

JOB_LIST:      -job aggregate_stat -fcst_lead 240000 -fcst_valid_beg 20250601_000000 -fcst_valid_end 20250630_000000 -obs_valid_beg 20250601_000000 -obs_valid_end 20250630_000000 -l
ine_type MPR -by VX_MASK -out_line_type WDIR -out_wind_thresh >0 -out_wind_logic UNION 
     COL_NAME: VX_MASK TOTAL      FBAR      OBAR       ME      MAE
ROW_MEAN_WDIR: conus      21 107.78408 113.60505  2.38096 21.42857
    AGGR_WDIR: conus      21 106.17138 112.51111 -6.33973 NA

It does seem like something has changed here between V11.0.2 and V12.0.1. V11 (and previous versions) seems to drop the 0,0 obs automatically for ROW_MEAN_WDIR while V12 needs to be told to drop them.

[JohnHalleyGotway]

Thanks goodness for git and github. Made it pretty easy for me to track down the source of the difference.

This block of code starting on this line of version 11.0.2 checked for and discarded vectors of 0 length.

And that block of code is removed on this line of version 12.0.2.

And here is the actual merge commit where it was removed for issue dtcenter/MET#2395.

If you expand the history on that issue, there was a lot of back and forth and discussion of wind direction statistic updates. I can't find the explicit commit message to explain that one change but my guess is this...

• In earlier versions of MET, the STAT-Analysis -job aggregate_stat -line_type MPR -out_line_type WDIR job was the only direct way to get statistics about wind direction differences. And you've made extensive use of that.
• In subsequent versions, we added the VCNT line type for vector continuous statistics.
• Add new wind direction verification statistics for RMSE, Bias, and MAE MET#2395 added more wind direction statistics to that line type. This is a mixture of wind speed and direction statistics, where wind speed is always defined, but wind direction is undefined for 0 vectors. The TOTAL column indicates the number of vector matched pairs while the TOTAL_DIR column indicates the number of those for which wind direction statistics can be defined.
• The same code in STAT-Analysis handles the aggregation of pairs for both the older WDIR output and the newer VCNT output.
• I'm guessing that I wanted to make sure that Point-Stat and Stat-Analysis can produce EXACTLY the same output with the same input pairs.
• If STAT-Analysis always tossed the 0-vectors, then I couldn't replicate the TOTAL column in the VCNT line type created by the Point-Stat tool.

Seeing this in hindsight, we could add back in that block of code that discards the 0-vectors, but only when the output line type is WDIR. Seems like that would provide backward-compatibility.

[johnlwagner]

Thanks John! As always, I appreciate your help with this!

[JohnHalleyGotway]

@johnlwagner, are you satisfied with the existing functionality? Or would you like us to add back in that previous behavior from MET version 11.0.2?

I do need to fix that obvious bug where we're reporting -out_wind_logic using the value of cnt_logic when it should be wind_logic. Should I ALSO restore that logic that skips the 0-vector pairs, but only when the WDIR output line type is requested?

As the heaviest user of this functionality, would that be useful?

[johnlwagner]

Yes, I think it makes sense to restore the functionality from V11.0.2. I have updated our scripts with the workaround, so it shouldn't be an issue for us going forward, but I think its good to have the compatibility between versions if someone does miss the workaround.

[JohnHalleyGotway]

FYI, @johnlwagner, here's the dtcenter/MET#3210 bugfix issue and corresponding dtcenter/MET#3211 pull request to restore this functionality for the MET-12.1.0 release. I'll also push the same fix to the main_v12.0 branch (for any future v12.0.4 bugfix release) and the develop branch to make sure it is included in all future releases.

Thanks for paying close attention to the output!