Fix IndexError in _pic_swarm_in_mesh on MPI ranks with empty cell partition#5078
Fix IndexError in _pic_swarm_in_mesh on MPI ranks with empty cell partition#5078hardik-corintis wants to merge 6 commits intofiredrakeproject:mainfrom
Conversation
|
|
||
|
|
||
| @pytest.mark.parallel(nprocs=2) | ||
| def test_submesh_interpolate_3Dcell_2Dfacet_empty_rank_2_processes(): |
There was a problem hiding this comment.
I've just had a look at this now (on the main branch). I don't get the IndexError you described. However, it does fail the assertion at the bottom, but this is because subm is an immersed manifold. We handle these by expanding the bounding boxes into an N-hypercube, where N is the geometric dimension of the mesh, and points not on x=1 were being found and evaluated on subm because they are within the default tolerance of the mesh. The test passes after setting subm.tolerance=0.1.
I'm also not sure that the reasoning for your fix is correct; see my comment here #5077
There was a problem hiding this comment.
Actually, if I run this on 8 ranks I do get the IndexError you described. The tests in test_swarm.py I mentioned don't actually input any points onto the rank which has zero cells, which is what is happening here. In that case I think this fix is good.
leo-collins
left a comment
There was a problem hiding this comment.
I think this is good. The same thing probably happens for the extruded case too - can you add a test for this (and a fix if necessary)?
|
|
||
|
|
||
| @pytest.mark.parallel(nprocs=2) | ||
| def test_submesh_interpolate_3Dcell_2Dfacet_empty_rank_2_processes(): |
There was a problem hiding this comment.
Actually, if I run this on 8 ranks I do get the IndexError you described. The tests in test_swarm.py I mentioned don't actually input any points onto the rank which has zero cells, which is what is happening here. In that case I think this fix is good.
Co-authored-by: Leo Collins <leocollins511@gmail.com>
Co-authored-by: Leo Collins <leocollins511@gmail.com>
Co-authored-by: Leo Collins <leocollins511@gmail.com>
|
Thanks @leo-collins for the quick review. I have applied all the inline suggestions and added the test and fix for the extruded meshes. |
Description
Fixes #5077.
In
_pic_swarm_in_mesh,parent_cell_nums_localcontains-1sentinels for points notowned by the current rank. On ranks with an empty cell partition,
cell_closurehas shape(0, k)and NumPy rejects-1as an axis-0 index, causing anIndexErrorwheninterpolating from a
Submeshonto its parent mesh in parallel.Fix: index
cell_closureonly onvisible_idxs(the non-sentinel entries), pre-fillingthe rest with
-1. The same issue exists in the extruded branch:_parent_extrusion_numberingfloor-divides
parent_cell_nums_local, and-1 // N == -1in Python, so sentinels passthrough into
base_parent_cell_numsand cause the same crash. Both branches are fixed.Two regression tests added in
tests/firedrake/submesh/test_submesh_interpolate.pyusing8 MPI ranks:
test_submesh_interpolate_3Dcell_2Dfacet_empty_rank_8_processes: interpolates from aSubmeshof the x=1 face ofUnitCubeMesh(2, 2, 2)onto the parent mesh.test_submesh_interpolate_3Dcell_extruded_empty_rank_8_processes: interpolates fromExtrudedMesh(UnitSquareMesh(1, 1), layers=3)(6 cells) onto aUnitCubeMesh(2, 2, 2),guaranteeing at least two ranks own zero extruded cells.