Hires tracer optimizations

[wmputman]

These updates improve tracer communication performance for high resolution runs by enabling nonblocking reduce operations for determining the subcycling for advection.

[wmputman]

@mathomp4 can you verify this is zero diff for the GEOS-CAM_v12 branch? These updates have been running in GEOS-CAM since June, but have not been merged to the main develop branch yet

[mathomp4]

@mathomp4 can you verify this is zero diff for the GEOS-CAM_v12 branch? These updates have been running in GEOS-CAM since June, but have not been merged to the main develop branch yet

@wmputman Sure. Is there anything I'd need to change to "enable" the changes? Or are they all on by default?

or is there a specific type of experiment I'd need to do?

[mathomp4]

@wmputman Update. In my testing at C24 L181 with Intel Release flags this is not zero-diff. I'm asking Sol for an analysis

[mathomp4]

Source review of the nonzero-diff result: there are several changes here beyond communication scheduling or structural refactoring. They may be intentional and appropriate, but I would not expect this patch to be unconditionally bitwise identical.

Scope: I compared feature/sdrabenh/v11.11.0-rc5 (cf2453b) against PR head 8f6a2d65f1ed839400c65237d473531511b463a6. This isolates the optimization commit. As confirmed by @mathomp4, GEOS-CAM_v12 is identical to feature/sdrabenh/v11.11.0-rc5, so this comparison also represents the GEOS-CAM_v12 baseline. I have not reproduced the full model run or established which change first diverges in the reported C24 L181 Intel Release test. Line references below are at the PR head.

Strongest explanations for nonzero differences

1. R4/R4R8 tracer arithmetic now rounds at different points. The old caller narrowed the R8 accumulated fluxes and Courants into default-real arrays before transport (mfxL=mfxR8, cxL=cxR8, etc.). The new caller passes the R8 arrays directly (caller). Thus expressions such as xfx=cx*dxa*dy*sin_sg and cx2=cx*frac use R8 intermediates before assignment to default-real work arrays (geometry products, split inputs). round_R4(round_R4(C)*f_R4) is not generally equal to round_R4(C_R8*f_R4). This can change transport even with identical split counts and no limiter activation. Export accumulation also now uses the R8 arrays directly instead of narrowed copies (exports). These are explicit rounding-point changes, not merely a compiler-reassociation hypothesis.

2. Automatic tracer substep diagnosis loses the old upper-layer exception. Previously standard/1L transport omitted +1.-sin_sg for k < npz/6; nested transport omitted it for k < 4. The new producer applies max(abs(cx),abs(cy))+1.-sin_sg at every level (producer). With int(1.+cmax) this can change the integration substep count, not just when the reduction occurs. For example, a top-layer maximum of 0.95 gives one substep before, but two after a geometry correction of 0.10. Online diagnosis also now uses unrounded R8 Courants, which can matter near thresholds.

3. The nonhydrostatic height limiter changes direction and surface-velocity timing. Previously update_dz_d swept upward with zh(k)=max(zh(k),zh(k+1)+dz_min) and computed ws before limiting. Now it sweeps downward with zh(k)=min(zh(k),zh(k-1)-dz_min) and computes ws afterward (code). This is inactive if the thickness constraint never activates, but otherwise changes which interfaces move, can move the bottom interface, and can change the ws passed to the NH solver. Example: interface heights [100,5,0] with dz_min=10 become [100,10,0] before versus [100,5,-5] now.

4. Standard transport adds CFL clipping and fixed-split validation. Split Courants above one in magnitude are now capped at +/-1 and geometric fluxes reduced, but split mass fluxes remain unclipped (code). This changes transport whenever triggered. Fixed q_split also now warns or terminates based on new CFL thresholds (checks).

Additional conditional issues

• Offline transport with nonzero q_split: definite uninitialized read. Its local cmax_z is initialized only inside if (q_split == 0), but is passed to tracer_2d regardless; the new fixed-split CFL checks read it (initialization/call). This looks like a bug rather than an intentional numerical change.
• Other offline changes: thicknesses are newly floored to 1.0 at lines 1003 and 1038, and global mass rescaling changes product precision, area-weighting placement, summation order, and reduction API at lines 1094-1118. These can change outputs independently of the communication optimization.
• Nested exports: the old nested tracer routine scaled caller flux/Courant arrays in place; the new routine scales local copies (lines 747-775). Subsequent exports therefore no longer contain the old substep-scaled values. This might be a correction, but is not zero-diff.
• Inline vortex breeding: tracer halo exchange now starts at dyn_core.F90:869, before breeding calls at lines 1123-1127 can modify q. If breeding is enabled, this risks exchanging pre-breeding values while transport uses post-breeding interior values.

The large fv_mapz.F90 refactor appears source-arithmetically equivalent for valid finite monotone pressure grids; compiler-dependent rounding after loop fusion/hoisting remains possible, but the explicit changes above are stronger explanations. One validation caveat: the retained map_scalar_old and mapn_tracer_old routines call the new scalar_profile, so they are not independent full-baseline checks.

Suggested first-divergence checks: compare cmax/integer split counts and geometric fluxes before the first tracer transport call; for NH runs, also count height-limiter activations. If R4/R4R8 is used, item 1 alone is sufficient to invalidate a general zero-diff expectation. It would help to confirm which numerical changes are intentional and whether acceptance should be changed from bitwise equivalence to an appropriate scientific/regression comparison.

[mathomp4]

Baseline clarification from @mathomp4: GEOS-CAM_v12 is identical to feature/sdrabenh/v11.11.0-rc5. Therefore, the baseline used in the source analysis above also represents GEOS-CAM_v12; the scope caveat about not comparing directly against that branch does not imply a different baseline.