`USE_FC_FACTORY=1` aborts at startup with EB terrain + anelastic / initial projection

[WeiqunZhang]

Summary

With USE_FC_FACTORY=1, ERF aborts during level creation whenever the EB terrain path is combined
with the anelastic solver or initial velocity projection:

Assertion `a_level.hasFCData(face_dir)' failed,
  file ".../amrex/Src/EB/AMReX_EBDataCollection.cpp", line 100,
  Msg: EBDataCollection: FC data not available for face_dir
SIGABRT

EB2::BuildFC() is called only on the non-multigrid branch of the constructor, but the face-centered
factories are built unconditionally for terrain_type = EB. When build_eb_for_multigrid is true the
FC data never exists, and the first FC factory construction trips an AMREX_ALWAYS_ASSERT (active in
Release as well as Debug).

This does not affect anyone today, since USE_FC_FACTORY defaults to 0
(Source/EB/ERF_EB.H:98). It blocks enabling the feature.

Affected configuration

erf.terrain_type = EB
# and either of:
erf.anelastic = 1
erf.project_initial_velocity = 1

built with USE_FC_FACTORY=1.

Cause

Source/ERF_Constructors.cpp:477-481:

const bool build_eb_for_multigrid = (solverChoice.terrain_type == TerrainType::EB &&
                                    ((solverChoice.project_initial_velocity[0] == 1) ||
                                      solverChoice.anelastic[0] == 1));
const int max_coarsening_level = (build_eb_for_multigrid) ? 100 : 0;

Source/ERF_Constructors.cpp:492-499 (and the same shape at 508, 524, 541):

if (build_eb_for_multigrid) {
    EB2::Build(gshop, geom[max_level], max_level, max_coarsening_level,
               ngrow_for_eb, build_coarse_level_by_coarsening);
    // <-- no EB2::BuildFC() here
} else {
    EB2::Build(gshop, this->Geom(), ngrow_for_eb);
#if USE_FC_FACTORY
    EB2::BuildFC();
#endif
}

Source/ERF_MakeNewLevel.cpp:80-81 — the terrain_type == EB branch is not guarded by
USE_FC_FACTORY, so it always goes through make_all_factories:

if (solverChoice.terrain_type == TerrainType::EB) {
    eb[lev]->make_all_factories(lev, geom[lev], grids[lev], dmap[lev], eb_level);
}

Source/EB/ERF_EB.cpp:54,63,72 — make_all_factories then builds the three FC factories when the
switch is on, and each of those asserts on hasFCData(face_dir) inside AMReX.

The same structure repeats at ERF_MakeNewLevel.cpp:335 and :633, so all three level-creation
paths (MakeNewLevelFromScratch, MakeNewLevelFromCoarse, RemakeLevel) are affected identically —
it is one bug, not three.

Why the obvious fix is a trap

Simply moving EB2::BuildFC() into the multigrid branch will make things much worse.
EB2::BuildFC() calls IndexSpaceImp<G>::buildAllFCData(), which loops over every level in
m_gslevel and, for each, builds a transient 2x-refined EB level once per face direction:

for (int face_dir = 0; face_dir < AMREX_SPACEDIM; ++face_dir)
    for (int ilev = 0; ilev < int(m_gslevel.size()); ++ilev)
        m_gslevel[ilev].buildFCData(m_gshop, face_dir, EB2::max_grid_size);

The multigrid branch builds with max_coarsening_level = 100. Each refined level is 8x the cells in
3D, and it is rebuilt three times per level rather than once. Measured cost of EB2::BuildFC()
against EB2::Build() for a single level (plane IF, ngrow_for_eb = 4):

grid	EB2::Build	EB2::BuildFC	ratio
32³	0.014 s	0.213 s	15.5x
64³	0.075 s	0.560 s	7.5x
128³	0.203 s	2.148 s	10.6x

Multiplying that across ~100 coarsening levels is not viable, in time or in peak memory.

Suggested directions

1. Guard the construction. Have make_all_factories check eb_level.hasFCData(idim) and either
   skip the FC factories or fall back to eb_aux_, so the configuration degrades instead of
   aborting. Cheapest fix, keeps both paths runnable.
2. Fail early and clearly. If FC factories are genuinely required whenever terrain_type = EB,
   detect the unsupported combination in the constructor and emit an ERF-level error naming
   anelastic / project_initial_velocity, rather than letting an AMReX assert fire during level
   creation.
3. Build FC data only where it is needed. If the multigrid path should support FC factories,
   it needs a way to build FC data for the finest level(s) only. That likely requires an upstream
   addition (AMReX EB2::BuildFC() currently offers no level selection).

Option 1 or 2 unblocks the switch; option 3 is the real feature.

Environment / provenance

• ERF 6d059dbeacb0491e0234dce0a03f6661ff10352e (2026-08-20)
• Found while reviewing AMReX PR
  AMReX-Codes/amrex#5548 ("Add face-centered EB
  factories"), which supplies EB2::BuildFC() and the FC EBFArrayBoxFactory constructor that ERF's
  USE_FC_FACTORY=1 path consumes.
• The abort itself was reproduced directly against AMReX: an EB level with no FC data reports
  hasFCData(0) = 0 and the FC factory constructor aborts at
  AMReX_EBDataCollection.cpp:100. The ERF call chain above was established by reading
  ERF_Constructors.cpp / ERF_MakeNewLevel.cpp / ERF_EB.cpp; ERF itself was not rebuilt with
  USE_FC_FACTORY=1 to observe the crash end to end.

Note that AMReX PR #5548 has separate open blockers of its own, so USE_FC_FACTORY=1 is not
production-ready regardless of this issue.

[skang67]

The anelastic solver with USE_FC_FACTORY=1 is currently not supported. The new face-centered EB factory build amrex::EB2::BuildFC() has been used only for compressible solver under the guard of USE_FC_FACTORY. The anelastic solver needs to be extended to use EB2::BuildFC().