Compress and SaveForecast with R2D2v3

[Dooruk]

New R2D2v3 approach to store and get forecast/background files.

I had to create geos_marine specific tasks as the requirements between marine model interface and the others are quite different. As I eluded to it on multiple occasions, varying DA window lengths and marine history file sizes made it a bit challenging to store files individually. In this PR, all state files are compressed under a single archive first and then gets stored afterwards. Because of R2D2's API calls, compression and store had to be separated into two different tasks. Hence, this PR introduces two major steps:

1. Listing and compressing all marine states, using pigz for parallel gunzip compression using compute nodes. Otherwise doing this on login node was quite inefficient and wasting other people's login source. This uses compress.py utility from @ftgoktas's PR: Compress and ingest background files via R2D2 #853

For milan nodes, using 8 threads proved to be most efficient so that is hardcoded. pigz compression level varies between 1-9, I picked 6 but that can be modified in the future. It takes ~80 seconds to compress 20GB worth netcdf files to ~9.7GB

2. Saving the same archives using store. Step is hardcoded as PT00, which is relative to the background (middle of the window for 3D case, beginning of the window for the FGAT case).

I also had to add a geos_marine specific get_background task that decompresses these files.

SaveAnalysis task will be a lot more straightforward for geos_marine as it is a single file for 3DVar or FGAT. Might not be the case for 4D analyses fields.

ALERT 🚨

⚠️ Important, with this PR, each time the marine cycling suites are executed, new files will be added to R2D2 datastore with unique experiment IDs. Until we sort the scrub and lifetime in R2D2, this will pose a problem on our swell project space, which is currently only 25TB with 500k IOnodes ⚠️ @mranst @ftgoktas

[Dooruk]

The last step I need to do is to add a toy model resolution compressed background for 3dvar_marine to work (@ftgoktas may be you can help?). Other than that it passes marine suite and is ready for a review, see the description for more details.

[ftgoktas]

Tested RunCompressForecast & SaveForecast on Discover and both tasks succeeded with the default 3dvar_marine_cycle config. They compressed and stored a couple of cycles fine and confirmed stored data in R2D2.

For GetBackground, I compressed and ingested manually and it pulled and unpacked it with no issues.

If anyone wants to try 3dvar_marine themselves without dealing with any of that, I stored it under r2d2_experiment_id: pr790_test, resolution 72x36, date 2021-07-01T12:00:00Z. Just set that in your experiment.yaml and GetBackground should be able to fetch it.

Heads up though, it's registered debug lifetime and it's just a borrowed file relabeled to that date, so treat it as a quick way to confirm the plumbing works.

[Dooruk]

If anyone wants to try 3dvar_marine themselves without dealing with any of that, I stored it under r2d2_experiment_id: pr790_test, resolution 72x36, date 2021-07-01T12:00:00Z. Just set that in your experiment.yaml and GetBackground should be able to fetch it.

Heads up though, it's registered debug lifetime and it's just a borrowed file relabeled to that date, so treat it as a quick way to confirm the plumbing works.

We don't have an active scrubber currently though, correct?

Thanks for testing. I'm running tier1 tests, if it passes I'm going to need approvals please. Trying to get this and his remaining PRs in before @ftgoktas leaves.

[Dooruk]

This is good to go in.

[ftgoktas]

If anyone wants to try 3dvar_marine themselves without dealing with any of that, I stored it under r2d2_experiment_id: pr790_test, resolution 72x36, date 2021-07-01T12:00:00Z. Just set that in your experiment.yaml and GetBackground should be able to fetch it.
Heads up though, it's registered debug lifetime and it's just a borrowed file relabeled to that date, so treat it as a quick way to confirm the plumbing works.

We don't have an active scrubber currently though, correct?

Thanks for testing. I'm running tier1 tests, if it passes I'm going to need approvals please. Trying to get this and his remaining PRs in before @ftgoktas leaves.

Yes, correct, no scrubber as of right now.