Skip to content

erf.ibseb: store the checkpoint face state by face rather than as a cell field #4019

Description

@asalmgren

Follow-up to the surface energy balance on immersed-boundary building faces (#3960), recording work deliberately deferred from that PR so it is not lost when the PR merges.

Where it stands

IBSEBState is a cell-centred MultiFab written once per level into the checkpoint. It started as a field over the whole level with 6 * (2 + n_slab_layers) components at every cell; #3960 shrank it twice, first onto column blocks around the buildings and then onto 4 x 4 blocks clipped to the k-range that owns faces, with one slot per face of a cell rather than a fixed six.

Measured on Exec/CanonicalTests/SEB/FaceStorage (a 128^3 level, one skyscraper, 2056 faces, n_slab_layers = 4):

layout boxes cells components values
whole level - 2097152 36 75497472
8 x 8 blocks, ground to roof 8 15616 36 562176
4 x 4 blocks, clipped, n_slots slots 32 7872 18 141696
one record per face (ideal) - - - 12336

So the field is still about 11.5x larger than the data it carries. The faces are a one-cell shell, and a cell field covering that shell with whole boxes cannot be proportional to the face count.

What would fix it

Store the state by face: one record of 2 + n_slab_layers reals keyed by (i, j, k, dir, side), nface records rather than O(cells covered).

Two routes, each with a genuine cost:

  1. An AMReX particle container. Position at the fluid cell centre so each record lands in the grid that owns its face, dir/side as integer components. Checkpoint / Restart / Redistribute then give rank-count-independent restart for free, which is exactly the part Surface energy balance on immersed-boundary building faces (erf.ibseb), with immersed-boundary fixes for the forcing and the MRF/YSUNew schemes #3960 had to hand-roll with AllGatherBoxes plus a transfer MultiFab and a ParallelCopy. On restart, join the particles against the face list build() regenerates, per tile. The catch: ERF_USE_PARTICLES is an opt-in compile definition (CMake/BuildERFExe.cmake:199), so erf.ibseb would have to require particles or carry two checkpoint paths.
  2. A sorted COO side file. Avoids the dependency, but means reimplementing the redistribute-onto-a-different-rank-count logic that the particle container already has, which is the part most likely to harbour a bug.

Also worth knowing

state_ncomp() is n_slots * (2 + n_layers()), where n_slots is the globally reduced largest face count on any one cell of the level. It is 1 for a convex isolated box but 3 on the FaceStorage skyscraper, because three faces meet on a cell at the rim. A single one-cell slot anywhere on a level pushes it to 6 and erases that part of the win. It is never worse than the fixed six the layout used to assume, but the saving is data-dependent in a way face-indexed storage would not be.

Relevant code: Source/ImmersedBoundarySEB/ERF_IBFaceSet.{H,cpp} (state_ncomp, state_boxarray, make_state, save_state, load_state) and Source/ImmersedBoundarySEB/ERF_IBSEB.cpp (init_ibseb, ibseb_write_checkpoint). Exec/CanonicalTests/SEB/FaceStorage covers the round trip, the restart onto another rank count, and the two mismatch aborts, so it should carry any replacement straight across.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    challengingtask that would take ~ 1 week or moreenhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions