You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Follow-up to the surface energy balance on immersed-boundary building faces (#3960), recording work deliberately deferred from that PR so it is not lost when the PR merges.
Where it stands
IBSEBState is a cell-centred MultiFab written once per level into the checkpoint. It started as a field over the whole level with 6 * (2 + n_slab_layers) components at every cell; #3960 shrank it twice, first onto column blocks around the buildings and then onto 4 x 4 blocks clipped to the k-range that owns faces, with one slot per face of a cell rather than a fixed six.
Measured on Exec/CanonicalTests/SEB/FaceStorage (a 128^3 level, one skyscraper, 2056 faces, n_slab_layers = 4):
layout
boxes
cells
components
values
whole level
-
2097152
36
75497472
8 x 8 blocks, ground to roof
8
15616
36
562176
4 x 4 blocks, clipped, n_slots slots
32
7872
18
141696
one record per face (ideal)
-
-
-
12336
So the field is still about 11.5x larger than the data it carries. The faces are a one-cell shell, and a cell field covering that shell with whole boxes cannot be proportional to the face count.
What would fix it
Store the state by face: one record of 2 + n_slab_layers reals keyed by (i, j, k, dir, side), nface records rather than O(cells covered).
Two routes, each with a genuine cost:
An AMReX particle container. Position at the fluid cell centre so each record lands in the grid that owns its face, dir/side as integer components. Checkpoint / Restart / Redistribute then give rank-count-independent restart for free, which is exactly the part Surface energy balance on immersed-boundary building faces (erf.ibseb), with immersed-boundary fixes for the forcing and the MRF/YSUNew schemes #3960 had to hand-roll with AllGatherBoxes plus a transfer MultiFab and a ParallelCopy. On restart, join the particles against the face list build() regenerates, per tile. The catch: ERF_USE_PARTICLES is an opt-in compile definition (CMake/BuildERFExe.cmake:199), so erf.ibseb would have to require particles or carry two checkpoint paths.
A sorted COO side file. Avoids the dependency, but means reimplementing the redistribute-onto-a-different-rank-count logic that the particle container already has, which is the part most likely to harbour a bug.
Also worth knowing
state_ncomp() is n_slots * (2 + n_layers()), where n_slots is the globally reduced largest face count on any one cell of the level. It is 1 for a convex isolated box but 3 on the FaceStorage skyscraper, because three faces meet on a cell at the rim. A single one-cell slot anywhere on a level pushes it to 6 and erases that part of the win. It is never worse than the fixed six the layout used to assume, but the saving is data-dependent in a way face-indexed storage would not be.
Relevant code: Source/ImmersedBoundarySEB/ERF_IBFaceSet.{H,cpp} (state_ncomp, state_boxarray, make_state, save_state, load_state) and Source/ImmersedBoundarySEB/ERF_IBSEB.cpp (init_ibseb, ibseb_write_checkpoint). Exec/CanonicalTests/SEB/FaceStorage covers the round trip, the restart onto another rank count, and the two mismatch aborts, so it should carry any replacement straight across.
Follow-up to the surface energy balance on immersed-boundary building faces (#3960), recording work deliberately deferred from that PR so it is not lost when the PR merges.
Where it stands
IBSEBStateis a cell-centredMultiFabwritten once per level into the checkpoint. It started as a field over the whole level with6 * (2 + n_slab_layers)components at every cell; #3960 shrank it twice, first onto column blocks around the buildings and then onto 4 x 4 blocks clipped to the k-range that owns faces, with one slot per face of a cell rather than a fixed six.Measured on
Exec/CanonicalTests/SEB/FaceStorage(a 128^3 level, one skyscraper, 2056 faces,n_slab_layers = 4):n_slotsslotsSo the field is still about 11.5x larger than the data it carries. The faces are a one-cell shell, and a cell field covering that shell with whole boxes cannot be proportional to the face count.
What would fix it
Store the state by face: one record of
2 + n_slab_layersreals keyed by(i, j, k, dir, side),nfacerecords rather thanO(cells covered).Two routes, each with a genuine cost:
dir/sideas integer components.Checkpoint/Restart/Redistributethen give rank-count-independent restart for free, which is exactly the part Surface energy balance on immersed-boundary building faces (erf.ibseb), with immersed-boundary fixes for the forcing and the MRF/YSUNew schemes #3960 had to hand-roll withAllGatherBoxesplus a transferMultiFaband aParallelCopy. On restart, join the particles against the face listbuild()regenerates, per tile. The catch:ERF_USE_PARTICLESis an opt-in compile definition (CMake/BuildERFExe.cmake:199), soerf.ibsebwould have to require particles or carry two checkpoint paths.Also worth knowing
state_ncomp()isn_slots * (2 + n_layers()), wheren_slotsis the globally reduced largest face count on any one cell of the level. It is 1 for a convex isolated box but 3 on the FaceStorage skyscraper, because three faces meet on a cell at the rim. A single one-cell slot anywhere on a level pushes it to 6 and erases that part of the win. It is never worse than the fixed six the layout used to assume, but the saving is data-dependent in a way face-indexed storage would not be.Relevant code:
Source/ImmersedBoundarySEB/ERF_IBFaceSet.{H,cpp}(state_ncomp,state_boxarray,make_state,save_state,load_state) andSource/ImmersedBoundarySEB/ERF_IBSEB.cpp(init_ibseb,ibseb_write_checkpoint).Exec/CanonicalTests/SEB/FaceStoragecovers the round trip, the restart onto another rank count, and the two mismatch aborts, so it should carry any replacement straight across.