spack-stack/2.1.1: g-w jobs executing python scripts abort on Ursa

[RussTreadon-NOAA]

Description

g-w jobs that execute python scripts abort upon start. For example, g-w scripts/exglobal_atm_analysis_initialize.py aborts with

+ JGLOBAL_ATM_ANALYSIS_INITIALIZE[45]/scratch3/NCEPDEV/da/Russ.Treadon/git/global-workflow/ursa/ss200/scripts/exglobal_atm_analysis_initialize.py
Abort(1008352783) on node 0 (rank 0 in comm 0): Fatal error in internal_Init_thread: Other MPI error, error stack:
internal_Init_thread(43417): MPI_Init_thread(argc=(nil), argv=(nil), required=3, provided=0x7fff7a659c38) failed
MPII_Init_thread(118)......:
MPID_Init(1626)............:
MPIR_pmi_init(180).........: PMI2_Job_GetId returned 14
+ JGLOBAL_ATM_ANALYSIS_INITIALIZE[46]export err=1

To Reproduce

1. clone g-w develop. Let HOMEglobal be the directory in which g-w is cloned.
2. cd $HOMEglobal/sorc/gdas.cd
3. git checkout feature/ss2x
4. git submodule update --iniit --recursive
5. cd $HOMEglobal/sorc
6. export WORKFLOW_TESTS="ON"
7. ./build_all.sh -A da-cpu -c all
8. when all builds complete, execute $HOMEglobal/sorc/link_workflow.sh
9. module use $HOMEglobal/sorc/gdas.cd/modulefile
10. module load GDAS/ursa.intel
11. cd $HOMEglobal/sorc/gdas.cd/build
12. ctest test_gdasapp_setup_atm_jjob_cycled_exp This will pass.
13. ctest test_gdasapp_atm_jjob_var_init This will fail
14. examine log file in $HOMEglobal/sorc/gdas.cd/build/gdas/test/atm/global-workflow/testrun

Expected behavior

test_gdasapp_atm_jjob_var_init should successfully run to completion.

System, compiler, code, ...

System: Ursa
Compiler

-- The C compiler identification is IntelLLVM 2025.3.1
-- The CXX compiler identification is IntelLLVM 2025.3.1
-- The Fortran compiler identification is IntelLLVM 2025.3.0

Additional context

The following modules are loaded in the environment when test_gdasapp_atm_jjob_var_init is executed

Currently Loaded Modules:
  1) intel-oneapi-compilers/2025.3.1        49) ecmwf-atlas/0.44.1
  2) stack-intel-oneapi-compilers/2025.3.1  50) ip/5.4.0
  3) intel-oneapi-mpi/2021.17.1             51) gsl-lite/0.41.0
  4) stack-intel-oneapi-mpi/2021.17         52) libjpeg/3.0.4
  5) intel-oneapi-mkl/2025.3                53) krb5/1.21.1-1
  6) glibc/2.34                             54) libtirpc/1.3.3
  7) cmake/3.31.8                           55) hdf/4.2.15
  8) pigz/2.8                               56) jedi-cmake/1.4.0
  9) zstd/1.5.7                             57) libpng/1.6.37
 10) tar/1.35                               58) libxt/1.3.1
 11) gettext/0.23.1                         59) libxmu/1.2.1
 12) nghttp2/1.65.0                         60) libxpm/3.5.17
 13) curl/8.11.1                            61) libxaw/1.0.16
 14) zlib/1.2.13                            62) udunits/2.2.28
 15) git/2.43.5                             63) ncview/2.1.9
 16) hdf5/1.14.5                            64) netcdf-cxx4/4.3.1
 17) parallel-netcdf/1.12.3                 65) json/3.11.3
 18) snappy/1.2.1                           66) rocoto/1.3.7
 19) c-blosc/1.21.6                         67) bacio/2.6.0
 20) netcdf-c/4.9.2                         68) w3emc/2.13.0
 21) nccmp/1.9.0.1                          69) prod_util/2.1.2
 22) netcdf-fortran/4.6.1                   70) jasper/4.2.4
 23) antlr/2.7.7                            71) g2/3.5.1
 24) gsl/2.8                                72) grib-util/1.4.0
 25) nco/5.3.3                              73) py-markupsafe/2.1.3
 26) parallelio/2.6.2                       74) py-jinja2/3.1.4
 27) wget/1.21.1                            75) py-certifi/2025.4.26
 28) sqlite/3.46.0                          76) py-cftime/1.6.4
 29) util-linux-uuid/2.41                   77) py-netcdf4/1.7.2
 30) python/3.11.11                         78) py-pybind11/2.13.6
 31) boost/1.84.0                           79) py-setuptools/69.2.0
 32) python-venv/1.0                        80) py-pycodestyle/2.11.0
 33) py-numpy/1.26.4                        81) libyaml/0.2.5
 34) bufr/12.3.0                            82) py-pyyaml/6.0.2
 35) git-lfs/3.5.1                          83) py-scipy/1.16.0
 36) ecbuild/3.12.0                         84) py-packaging/25.0
 37) openjpeg/2.3.1                         85) py-six/1.17.0
 38) eccodes/2.33.0                         86) py-python-dateutil/2.8.2
 39) eigen/3.4.0                            87) py-pytz/2024.2
 40) openblas/0.3.27                        88) py-tzdata/2025.2
 41) eckit/1.32.3                           89) py-pandas/2.3.1
 42) fckit/0.14.1                           90) py-xarray/2024.7.0
 43) fiat/1.6.1                             91) py-f90nml/1.4.3
 44) ectrans/1.7.0                          92) py-pip/25.1.1
 45) fms/2025.03-gfs-constants              93) py-click/8.1.8
 46) py-mpi4py/4.0.1                        94) py-wheel/0.45.1
 47) esmf/8.8.0                             95) GDAS/ursa.intel
 48) qhull/2020.2

Add any other context about the problem here.
Test test_gdasapp_atm_jjob_var_init successfully runs to completion when using /contrib/spack-stack/spack-stack-1.9.2/envs/ue-oneapi-2024.2.1

[RussTreadon-NOAA]

I assume the failure with spack-stack/2.1.1 is due to loading an incorrect module or failing to load a necessary module. I have not yet figured out which module loads need to be removed, edited, or added to $HOMEglobal/sorc/gdas.cd/modulefiles/GDAS/ursa.intel.lua.

FYI: The working copy of g-w I am using is /scratch3/NCEPDEV/da/Russ.Treadon/git/global-workflow/ursa/ss200

[climbfuji]

Looks like this was compiled with MPI support. On a slurm system, this often requires running the execatuable with srun -n 1 or mpirun -np 1. Can you try?

[RussTreadon-NOAA]

Thank you @climbfuji .

I modified $HOMEglobal/dev/jobs/JGLOBAL_ATM_ANALYSIS_INITIALIZE as follows

@@ -42,7 +42,7 @@ declare -rx COMIN_ATMOS_HISTORY_PREV="${ROTDIR}/${GDUMP}.${gPDY}/${gcyc}/model/a
 # Run relevant script
 
 EXSCRIPT=${GDASATMINITPY:-${SCRglobal}/exglobal_atm_analysis_initialize.py}
-${EXSCRIPT} && true
+srun -n 1 ${EXSCRIPT} && true
 export err=$?
 if [[ ${err} -ne 0 ]]; then
     err_exit "Failed to initialize the atmospheric analysis"

The test passed

Test project /scratch3/NCEPDEV/da/Russ.Treadon/git/global-workflow/ursa/ss200/sorc/gdas.cd/build
    Start 316: test_gdasapp_atm_jjob_var_init
1/1 Test #316: test_gdasapp_atm_jjob_var_init ...   Passed   42.55 sec

100% tests passed, 0 tests failed out of 1

Label Time Summary:
gdasapp    =  42.55 sec*proc (1 test)
script     =  42.55 sec*proc (1 test)

Total Test time (real) =  42.69 sec

Why is the change needed with spack-stack/2.1.1? We don't need srun -n 1 when we build and run with spack-stack/1.9.2.

I only updated the GDASApp build to spack-stack/2.1.1. Other applications in g-w were build using spack-stack/1.9.2. Is this a problem?

[climbfuji]

I don't know what gdasapp contains (code) or does, how it is compiled, ... - impossible for me to answer.

IF there is no other change in gdasapp except compiling with spack-stack-2.1.1, then I suspect that one of the gdasapp dependencies (linked library - check ldd ...) is compiled with MPI support when it wasn't beforehand. That, on slurm systems, brings in links to libpmi etc.

[RussTreadon-NOAA]

This is interesting. test_gdasapp_atm_jjob_var_init does not use anything from GDASApp. It executes the g-w python script to stage data in a run directory. The next ctest, test_gdasapp_atm_jjob_var_run, runs GDASApp executable gdas.x.

As a test I did the following:

1. revert change to JGLOBAL_ATM_ANALYSIS_INITIALIZE (ie, remove srun -n 1 on line that executes the python script)
2. load a spack-stack/1.9.2 GDAS/ursa.intel.lua

With this change more GDASApp ctests (many of which are actually g-w CI jobs) successfully run to completion. Seems I should wait for g-w applications to build with spack-stack/2.1.1 so all g-w applications are built with the same spack-stack modules.

[climbfuji]

Maybe. I'd be interested in the content of the Python script. Does it load py-netcdf4?

[RussTreadon-NOAA]

Script $HOMEglobal/dev/scripts/exglobal_atm_analysis_initialize.py is relatively simple. It contains

#!/usr/bin/env python3                                                                                                                                     
# exglobal_atm_analysis_initialize.py                                                                                                                      
# This script creates an AtmAnalysis class                                                                                                                 
# and runs the initialize method                                                                                                                           
# which creates and stages the runtime directory                                                                                                           
# and creates the YAML configuration                                                                                                                       
# for a global atm variational analysis                                                                                                                    
import os

from wxflow import Logger, cast_strdict_as_dtypedict
from pygfs.task.atm_analysis import AtmAnalysis

# Initialize root logger                                                                                                                                   
logger = Logger(level='DEBUG', colored_log=True)


if __name__ == '__main__':

    # Take configuration from environment and cast it as python dictionary                                                                                 
    config = cast_strdict_as_dtypedict(os.environ)

    # Instantiate the atm analysis task                                                                                                                    
    AtmAnl = AtmAnalysis(config)

    # Initialize JEDI variational analysis                                                                                                                 
    AtmAnl.initialize()

wxflow and pygfs come from g-w develop .

The spack-stack/1.9.2 test loaded py-netcdf4/1.7.1.post2. The spack-stack/2.1.1 test loaded py-netcdf4/1.7.2

[climbfuji]

Yes, that's it. Compare 2.1.1 and develop where we removed the inadvertently added MPI features from py-netcdf4 because of exactly the same reason.

spack-stack/configs/common/packages.yaml

Line 410 in 32d0224

py-netcdf4:

spack-stack/configs/common/packages.yaml

Line 440 in af4bf7d

py-netcdf4:

[RussTreadon-NOAA]

Thank you for explaining what's going on. Do we have a spack-stack/2.1.2 with MPI features removed from py-netcdf4?

[climbfuji]

Thank you for explaining what's going on. Do we have a spack-stack/2.1.2 with MPI features removed from py-netcdf4?

We don't. We'll have spack-stack-2.2.0 in a while. It would be relatively easy to create chained environments for 2.1.1 however. Especially given that 2.1.1 is a limited release that is only available on selected platforms (only NOAA systems if I recall).

A quicker solution for 2.1.1 might be to run ctests as follows - can you try?

ctest --test-command-launcher "srun -n 1" ...

[RussTreadon-NOAA]

test_gdasapp_atm_jjob_var_init passed but test_gdasapp_atm_jjob_var_run failed. The var test runs GDASApp executable gdas.x with 6 tasks.

I don't need to update to spack-stack/2.1.1. GDASApp develop uses spack-stack/1.9.2. I can wait until spack-stack/2.2.0 is ready.

[RussTreadon-NOAA]

20260914 update

The Ursa spack-stack/2.1.1 issue regarding serial pythons scripts with py-netcdf4/1.7.2 requiring srun has cropped up again.

Interestingly, a spack-stack/2.1.1 build of GDASApp on Gaea C6 does not exhibit the same behavior. Test test_gdasapp_atm_jjob_var_init runs without srun on Gaea C6 as shown below

Gaea(gaea64):/gpfs/f6/ira-sti/scratch/Russ.Treadon/git/global-workflow/ss200/sorc/gdas.cd/build$ ctest -VV -R test_gdasapp_snow_create_ens
UpdateCTestConfiguration  from :/gpfs/f6/ira-sti/scratch/Russ.Treadon/git/global-workflow/ss200/sorc/gdas.cd/build/DartConfiguration.tcl
 Add coverage exclude regular expressions.
UpdateCTestConfiguration  from :/gpfs/f6/ira-sti/scratch/Russ.Treadon/git/global-workflow/ss200/sorc/gdas.cd/build/DartConfiguration.tcl
Test project /gpfs/f6/ira-sti/scratch/Russ.Treadon/git/global-workflow/ss200/sorc/gdas.cd/build
Constructing a list of tests
Done constructing a list of tests
Updating test list for fixtures
Added 0 tests to meet fixture requirements
Checking test dependency graph...
Checking test dependency graph end
test 300
    Start 300: test_gdasapp_snow_create_ens

300: Test command: /gpfs/f6/ira-sti/scratch/Russ.Treadon/git/global-workflow/ss200/sorc/gdas.cd/bundle/gdas/test/snow/create_bkg_ens.sh "/gpfs/f6/ira-sti/scratch/Russ.Treadon/git/global-workflow/ss200/sorc/gdas.cd/build/gdas" "/gpfs/f6/ira-sti/scratch/Russ.Treadon/git/global-workflow/ss200/sorc/gdas.cd/bundle/gdas"
300: Working Directory: /gpfs/f6/ira-sti/scratch/Russ.Treadon/git/global-workflow/ss200/sorc/gdas.cd/build/gdas/test/snow/
300: Environment variables: 
300:  GDASAPP_TESTDATA=/gpfs/f6/ira-sti/world-shared/GDASApp/testdata
300: Test timeout computed to be: 10000000
300: do_snowDA: calling create ensemble
300: adjusting 20210323.180000* by 21.213203435596423
1/1 Test #300: test_gdasapp_snow_create_ens .....   Passed    2.00 sec

The following tests passed:
        test_gdasapp_snow_create_ens

100% tests passed, 0 tests failed out of 1

Total Test time (real) =   2.18 sec

The following modules are loaded on Gaea C6 for the test above

Currently Loaded Modules:
  1) craype-x86-genoa                       22) zlib/1.2.13             43) intel-oneapi-mkl/2025.2    64) libjpeg/3.0.4         85) py-netcdf4/1.7.2
  2) perftools-base/25.03.0                 23) git/2.43.0              44) python-venv/1.0            65) krb5/1.20.1           86) py-pybind11/2.13.6
  3) xpmem/2.11.5-1.3_g73ade43320bc         24) hdf5/1.14.5             45) py-numpy/1.26.4            66) libtirpc/1.3.3        87) py-setuptools/69.2.0
  4) cray-pmi/6.1.15                        25) parallel-netcdf/1.12.3  46) bufr/12.3.0                67) hdf/4.2.15            88) py-pycodestyle/2.11.0
  5) CmrsEnv/default                        26) snappy/1.2.1            47) git-lfs/2.11.0             68) jedi-cmake/1.4.0      89) libyaml/0.2.5
  6) TimeZoneEDT/default                    27) zstd/1.5.7              48) ecbuild/3.12.0             69) libpng/1.6.37         90) py-pyyaml/6.0.2
  7) DefApps                                28) c-blosc/1.21.6          49) openjpeg/2.3.1             70) libxaw/1.10.13        91) py-scipy/1.16.0
  8) intel/2025.2                           29) netcdf-c/4.9.2          50) eccodes/2.33.0             71) udunits/2.2.28        92) py-packaging/25.0
  9) stack-intel-oneapi-compilers/2025.2.1  30) nccmp/1.9.0.1           51) eigen/3.4.0                72) ncview/2.1.9          93) py-six/1.17.0
 10) craype/2.7.34                          31) netcdf-fortran/4.6.1    52) openblas/0.3.26            73) netcdf-cxx4/4.3.1     94) py-python-dateutil/2.8.2
 11) cray-dsmml/0.3.1                       32) antlr/2.7.7             53) eckit/1.32.3               74) json/3.11.3           95) py-pytz/2024.2
 12) cray-libsci/25.03.0                    33) gsl/2.8                 54) fckit/0.14.1               75) bacio/2.6.0           96) py-tzdata/2025.2
 13) PrgEnv-intel/8.6.0                     34) nco/5.3.3               55) fiat/1.6.1                 76) w3emc/2.13.0          97) py-pandas/2.3.1
 14) cray-mpich/8.1.32                      35) parallelio/2.6.2        56) ectrans/1.7.0              77) prod_util/2.1.2       98) py-xarray/2024.7.0
 15) libfabric/1.22.0                       36) wget/1.21.3             57) fms/2025.03-gfs-constants  78) jasper/4.2.4          99) py-f90nml/1.4.3
 16) craype-network-ofi                     37) Core/24.11              58) py-mpi4py/4.0.1            79) g2/3.5.1             100) py-pip/25.1.1
 17) stack-cray-mpich/8.1.32                38) gettext/0.20.2          59) esmf/8.8.0                 80) grib-util/1.4.0      101) py-click/8.1.8
 18) glibc/2.38                             39) sqlite/3.46.0           60) qhull/2020.2               81) py-markupsafe/2.1.3  102) py-wheel/0.45.1
 19) cmake/3.31.8                           40) util-linux-uuid/2.41    61) ecmwf-atlas/0.44.1         82) py-jinja2/3.1.4      103) rocoto/1.3.7_fix
 20) nghttp2/1.65.0                         41) python/3.11.11          62) ip/5.4.0                   83) py-certifi/2025.4.26 104) hsi/9.3
 21) curl/8.11.1                            42) boost/1.84.0            63) gsl-lite/0.41.0            84) py-cftime/1.6.4      105) GDAS/gaeac6.intel

py-netcdf4/1.7.2 is loaded on both Gaea C6 and Ursa.

When will spack-stack/2.2.0 be released? If it's going to be a while, can we update Ursa spack-stack/2.1.1 so that it behaves like Gaea C6 spack-stack/2.1.1?

The current behavior of Ursa spack-stack/2.1.1 prevents us from upgrading GDASApp to a newer version of spack-stack.