I'm modifying theSaveForecast (or background) task where it needs to compress multiple model output files and store them under R2D2. Obvious benefit is saving disk space and no of IOnodes.
For MOM6 & CICE6 history and restarts are NetCDF-3 (64-bit offset and classic) and for certain workflows (such as S2S), there could be 20 total files that needs to be combined, 2.2GB each for ocean there are multiple 4D prognostic fields to save.
So I'm exploring potential (simple and efficient) options that can be used here.
AFAIK in GEOS this is done in a straightforward manner:
https://github.com/GEOS-ESM/GEOSgcm_App/blob/f6f7d231f0e42de0f250f20d0d9ea2ca4dc3ea63/gcm_run.j#L652-L653
GPT suggested using pigz or zstd via python, both of which would require using compute nodes. I used pigz before but I'm not familiar with zstd.
This could be a SWELL dev discussion as well but I'm curious to hear thoughts before that. Any other potential/better approaches?
FWIW, with my initial crude tar.gz test on a login node (sorry Discover) I got %90 compression for sea-ice files (9 x 110MB -> ~150MB) and %50 compression for ocean files (9 x 2.2GB -> ~10GB) which took about 13 minutes.
#790
I'm modifying the
SaveForecast(or background) task where it needs to compress multiple model output files and store them under R2D2. Obvious benefit is saving disk space and no of IOnodes.For MOM6 & CICE6 history and restarts are NetCDF-3 (64-bit offset and classic) and for certain workflows (such as S2S), there could be 20 total files that needs to be combined, 2.2GB each for ocean there are multiple 4D prognostic fields to save.
So I'm exploring potential (simple and efficient) options that can be used here.
AFAIK in GEOS this is done in a straightforward manner:
https://github.com/GEOS-ESM/GEOSgcm_App/blob/f6f7d231f0e42de0f250f20d0d9ea2ca4dc3ea63/gcm_run.j#L652-L653
GPT suggested using
pigzorzstdvia python, both of which would require using compute nodes. I usedpigzbefore but I'm not familiar withzstd.This could be a SWELL dev discussion as well but I'm curious to hear thoughts before that. Any other potential/better approaches?
FWIW, with my initial crude tar.gz test on a login node (sorry Discover) I got %90 compression for sea-ice files (9 x 110MB -> ~150MB) and %50 compression for ocean files (9 x 2.2GB -> ~10GB) which took about 13 minutes.
#790