Improve Ensemble-Stat to more efficiency apply observation error corrections and perturbations #3426

Describe the Enhancement

This issue is the result of @CPKalb observing significantly different performance for Ensemble-Stat when run with observation error logic enabled and disabled. A single call to Ensemble-Stat for 20 HAFS ensemble members with observation error logic disabled takes around 1 minute to run, but multiple hours with it enabled. This issue is to revisit and refine the implementation to make it more efficient.

Please find the example test case in seneca:/d1/projects/METplus/discussions/kalb_20260820/example.
Note that this compares a 20 member ensemble to GRIDDED CCPA analysis data for 6-hourly accumulated precipitation. Big picture, applying observation error to a CCPA analysis may not be necessary, but it provides a reasonable test case for the software.

Recommend making the following changes:

1. Enhance add_obs_error_bc() and add_obs_error_inc() by checking the observation error table to see if bias-correction and perturbation is indicated for the requested variable name (var_name). The default table applies no corrections to APCP_ data. It'd be faster to determine that before looping through all the grid points. Add functions like bool ObsErrorEntry::need_bc(), bool ObsErrorEntry::need_inc(), bool ObsErrorTable::need_bc(var_name), and bool ObsErrorTable::need_inc(var_name) to prevent unnecessary looping.
2. Remove const gsl_rng *r as an argument for add_obs_error_bc() functions since it's unused.
3. Add OpenMP parallelization in obs_error.cc. Note however that a separate random number generator is needed for each thread.
4. In obs_error.cc, replace calls to DataPlane::get() which waste time performing range checking each time with more efficient, direct looping over the points.
5. Reconsider the call to obs_error_table.lookup(). When applying to gridded data, we could subset to table to only those rows that could possibly match (e.g. add ObsErrorTable ObsErrorTable::subset(var_name)). And rather than always looking up at the first row of the table, consider starting at the row most recently matched.
6. After meeting with @mpm-meto, recommend that we improve the log messages in obs_error.cc to handle the case when an obs error table lookup finds no matching entry. This does result in a warning message from ObsErrorTable::lookup() but we should also update add_obs_error_bc() and add_obs_error_inc() to print consistent DEBUG(4) log messages similar to:

DEBUG 4: Applying no observation error bias correction to OBS value 4.32 since no table match found.

And

DEBUG 4: Applying no observation error update for FCST value 2.57 and OBS value 0 since no table match found.

Time Estimate

Estimate the amount of work required here.
Issues should represent approximately 1 to 3 days of work.

Sub-Issues

Consider breaking the enhancement down into sub-issues.

• Add a checkbox for each sub-issue here.

Relevant Deadlines

List relevant project deadlines here or state NONE.

Funding Source

PRJ014254 NOAA BENCHMARKING TNE

Define the Metadata

Assignee

• Select engineer(s) or no engineer required
• Select scientist(s) or no scientist required: @CPKalb should review

Labels

• Review default alert labels
• Select component(s)
• Select priority
• Select requestor(s)

Milestone and Projects

• Select Milestone as a MET-X.Y.Z version, Consider for Next Release, or Backlog of Development Ideas
• For a MET-X.Y.Z version, select the MET-X.Y Development project

Define Related Issue(s)

Consider the impact to the other METplus components.

• METplus, MET, METdataio, METviewer, METexpress, METcalcpy, METplotpy
  None

Enhancement Checklist

See the METplus Workflow for details.

• Complete the issue definition above, including the Time Estimate and Funding Source.
• Fork this repository or create a branch of develop.
  Branch name: feature_<Issue Number>_<Description>
• Complete the development and test your changes.
• Add/update log messages for easier debugging.
• Add/update unit tests.
• Add/update documentation.
• Push local changes to GitHub.
• Submit a pull request to merge into develop.
  Pull request: feature <Issue Number> <Description>
• Define the pull request metadata, as permissions allow.
  Select: Reviewer(s) and Development issue
  Select: Milestone as the next official version
  Select: MET-X.Y Development project for development toward the next coordinated release
• Iterate until the reviewer(s) accept and merge your changes.
• Delete your fork or branch.
• Close this issue.