No way to request grids that are not decomposed in z but also do not span the full domain height (blocks implicit vertical diffusion)

[asalmgren]

Follow-on to #3997 / #3998.

The problem

The implicit vertical diffusion solves invert one tridiagonal system per column
(ImplicitDiffForStateLU_{N,S,T}, ImplicitDiffForMomLU_{N,S,T}), so a column of the domain
must lie entirely within a single grid. #3998 adds ERF::grids_are_split_in_z, which aborts
when the grids at any level are decomposed in the vertical while implicit vertical diffusion
is active, rather than silently solving each piece of a column separately with spurious
internal boundaries.

That guard is correct but it is a new failure mode for existing inputs, and right now there is
no way to ask the grid generator for grids that are not decomposed in z but also do not span
the full height of the domain. You can have one or the other, not both. That matters for
exactly the configurations ERF cares about: a refined level that covers the boundary layer but
not the free troposphere, or a refined region that is a staircase in z.

Concretely, Exec/CanonicalTests/DensityCurrent/inputs_amr now has to set
erf.vert_implicit_fac = 0 0 0 because the region tagged by lo_theta gets covered by boxes
stacked in z. There is no input setting that would let it keep the implicit solve.

What the existing knobs do

knob	what it prevents	sufficient?
amr.max_grid_size_z >= nz	BoxArray::maxSize chopping in z	no
amr.refine_grid_layout_z = 0	the load-balance ChopGrids pass chopping in z	no
amr.refine_whole_domain_dir = 2	all z-splitting	yes, but forces full height

Both of the first two together, on inputs_amr with the implicit solve enabled:

amr.max_grid_size_z=1024 amr.refine_grid_layout_z=0
  -> exit 6, "The grids at level 1 are decomposed in the vertical"

So the stacking comes from Berger-Rigoutsos clustering itself (ClusterList::chop), which
chops along whichever direction the signature favours and has no per-direction disable.

amr.refine_whole_domain_dir = 2 does avoid all z-splitting -- the same case then gives a
single level-1 box ((0,0,0) (127,0,63)) -- but by the mechanism we are trying to avoid:
ClusterList collapses every tag to domain.smallEnd(2) and re-expands each cluster to span
the full domain in z (AMReX_Cluster.cpp:441-471). Full height by construction.

Where ERF could intervene

AmrMesh::MakeNewGrids is not virtual (AMReX_AmrMesh.H:345). The virtual hooks are
ManualTagsPlacement (tags, before clustering) and PostProcessBaseGrids (level 0 only --
ERF already uses the equivalent at ERF_MakeNewLevel.cpp:43). So ERF would have to act either
in its own MakeNewLevelFromCoarse / RemakeLevel before SetBoxArray, or upstream.

Two designs

A -- collapse for clustering, restore afterwards. In ManualTagsPlacement, record the
tagged [kmin,kmax] per (i,j), then collapse the tags to full height. Berger-Rigoutsos then
produces full-height boxes, so their (i,j) footprints are disjoint rectangles and no column
can be split. Afterwards shrink each box back to the union of kmin/kmax over its own
footprint, blocking-factor aligned.

• Small amount of code; the footprint partition comes free from the clustering.
• Cost: each box's z-extent is the union over its footprint, so more cells are refined than
  were tagged, and grid_eff no longer controls vertical efficiency.

B -- columnize the generated BoxArray. Compute [kmin,kmax] per (i,j) from the grids
AMReX produced, partition the 2D footprint into maximal rectangles on which that pair is
constant, extrude each one, then apply max_grid_size in x and y only.

• Tighter fit than A.
• More code (a 2D rectangle decomposition, though BoxList::simplify() does most of the work),
  and it has to decide what to do when a column has a genuine vertical gap -- filling it adds
  cells to the level.

Caveats common to both

• Proper nesting with three or more levels. AmrCore computes nesting across all levels in
  one MakeNewGrids call, so shrinking or extending in z afterwards has to be done
  finest-level-first, taking the union with whatever the level above requires. For a two-level
  hierarchy it is straightforward.
• Load balance. Forbidding decomposition in z removes a degree of freedom from the grid
  layout; on large core counts that may produce fewer, larger grids than the machine wants.
  Worth measuring before making it a default.

Suggested upstream piece

A small addition to AMReX -- a virtual PostProcessGrids(int lev, BoxArray&) alongside the
existing PostProcessBaseGrids -- would give ERF a sanctioned place to do B without reaching
into the regrid callbacks.

Suggested ERF-side scope

Implement A behind an input flag (e.g. erf.no_vertical_grid_decomposition), so that a case
like DensityCurrent/inputs_amr can keep implicit vertical diffusion instead of turning it
off. Leave the default unchanged until the load-balance impact is understood.