Rework to store-r2d2 and add fetch_from_r2d2

docs/additional_resources/faq.md

@@ -18,7 +18,7 @@
 
 - R2D2 is not working for me
 
-    > Please see [R2D2 settings](../configuration_reference/r2d2_v3_credentials.md). As of the writing of this documentation, the same shared gmao-user API key is used, but this will change soon. Alternatively, the `--skip-r2d2` or `-k` argument can be used for `geos_atmosphere` applications to bypass R2D2 and use Ricardo's x00** experiment directories.
+    > Please see [R2D2 settings](../configuration_reference/r2d2_v3_credentials.md). As of the writing of this documentation, the same shared gmao-user API key is used, but this will change soon. Alternatively, the `--skip-store-r2d2` or `-k` argument can be used for `geos_atmosphere` applications to bypass R2D2 and use Ricardo's x00** experiment directories.
 
 - Can I view Cylc logs with my own editor?
 

docs/introduction/prerequisites.md

@@ -5,7 +5,7 @@ Before installing Swell, make sure you have:
 - **A platform.** Swell is developed and tested primarily on NASA's Discover cluster, which has dedicated instructions — see [Supported Platforms](../installation_and_setup/platforms/README.md). Other platforms are supported in principle, but you're responsible for satisfying the dependencies below yourself.
 - **Python and matching dependencies.** Swell is a Python package installed with pip. It shares dependencies with the applications it drives (JEDI, GEOS), so these must be satisfied consistently. On Discover, loading the correct `spack-stack` modules before installing avoids this. See [Installing Swell](../installation_and_setup/installing_swell.md).
 - **Cylc.** Swell generates `flow.cylc` workflows but does not install or manage Cylc itself. You need a `cylc` executable on your `$PATH` and a `$HOME/.cylc/flow/global.cylc` file. On Discover this is handled automatically. See [Configuring Cylc](../installation_and_setup/configuring_cylc.md).
-- **R2D2 credentials.** A `~/.swell/r2d2_credentials.yaml` file is required for `swell create` to register experiments and for tasks to fetch/store data. See [R2D2 v3 Credentials](../configuration_reference/r2d2_v3_credentials.md). You can bypass registering your experiment and storing data in R2D2 by passing `-k`/`--skip-r2d2` to `swell create`, though suites that require fetching files from R2D2 (which is most) still require configured R2D2 credentials. — see [QuickStart](quickstart.md#skipping-r2d2-for-a-first-run).
+- **R2D2 credentials.** A `~/.swell/r2d2_credentials.yaml` file is required for `swell create` to register experiments and for tasks to fetch/store data. See [R2D2 v3 Credentials](../configuration_reference/r2d2_v3_credentials.md). You can bypass registering your experiment and storing data in R2D2 by passing `-k`/`--skip-store-r2d2` to `swell create`, though suites that require fetching files from R2D2 (which is most) still require configured R2D2 credentials. — see [QuickStart](quickstart.md#skipping-r2d2-for-a-first-run).
 - **Accounts for external data (if ingesting observations).** Some observation providers require their own authentication (e.g. an Earthdata account). See [Storing Observations and Backgrounds in R2D2](../practical_examples/r2d2/r2d2_ingest.md).
 
 Once these are in place, continue to [Installing Swell](../installation_and_setup/installing_swell.md).

docs/introduction/quickstart.md

@@ -36,18 +36,18 @@ This registers the experiment in R2D2 and creates a directory (default `/discove
 | `-o`, `--override <file.yaml>` | `swell create` | Override any `experiment.yaml` value, e.g. `experiment_root` (the run directory) or `experiment_id`. |
 | `-p`, `--platform <platform>` | `swell create` | Select platform-specific defaults (e.g. `nccs_discover_sles15`). |
 | `-s`, `--slurm <file.yaml>` | `swell create` | Override SLURM directives (account, nodes, qos, etc.), globally or per task/model. See [SLURM Configuration](../configuration_reference/slurm_configuration.md). |
-| `-k`, `--skip-r2d2` | `swell create` | Skip registering the experiment and storing products in R2D2. Useful if you don't have credentials set up yet — see [below](#skipping-r2d2-for-a-first-run). |
+| `-k`, `--skip-store-r2d2` | `swell create` | Skip registering the experiment and storing products in R2D2. Useful if you don't have credentials set up yet — see [below](#skipping-r2d2-for-a-first-run). |
 | `-l`, `--log_path <dir>` | `swell launch` | Directory to receive workflow manager (Cylc) logging output, instead of the default `$HOME/cylc-run/<suite_name>`. |
 
 Run `swell create --help` or `swell launch --help` for the full, up-to-date list.
 
 ### Skipping R2D2 for a first run
 
-Are you new to Swell and don't have R2D2 credentials set up yet? Pass `-k`/`--skip-r2d2` to
+Are you new to Swell and don't have R2D2 credentials set up yet? Pass `-k`/`--skip-store-r2d2` to
 `swell create` to skip registering the experiment and storing products in R2D2, though experiments needing to fetch from R2D2 still require credentials:
 
 ```bash
-swell create <suite> --skip-r2d2
+swell create <suite> --skip-store-r2d2
 ```
 
 This lets you complete a full create -> launch -> monitor cycle to see Swell working end to end
@@ -83,5 +83,5 @@ If the TUI gets closed you can reopen it by typing `cylc tui` in the terminal.
 
 Once tasks finish, the experiment directory will contain the generated files. Note that some tasks are required to succeed to trigger the execution of other tasks while some tasks can fail without impacting cycling, if the Cylc graph doesn't require completion of the task to move forward.
 
-By default `skip_r2d2` is `false`, so outputs (backgrounds, analyses, diagnostics) will also be registered and stored in R2D2 (see [Skipping R2D2 for a first run](#skipping-r2d2-for-a-first-run) if you don't have credentials set up yet). From here, see [Choosing a Workflow](../running_an_experiment/choosing_a_workflow.md) to explore other suites.
+By default `skip_store_r2d2` is `false` (note this is the name of the key in `experiment.yaml`, which can be set on the command line using `--skip-store-r2d2` upon experiment creation), so outputs (backgrounds, analyses, diagnostics) will also be registered and stored in R2D2 (see [Skipping R2D2 for a first run](#skipping-r2d2-for-a-first-run) if you don't have credentials set up yet). From here, see [Choosing a Workflow](../running_an_experiment/choosing_a_workflow.md) to explore other suites.
 

docs/practical_examples/geos_cf/3dvar_cf.md

@@ -136,7 +136,7 @@ For each cycle and each `model_component` (only `geos_cf` in this suite), the fo
 - `EvaObservations`: generate observation-space diagnostic plots.
 - `EvaJediLog`: generate JEDI log (cost function/iteration) diagnostic plots.
 - `EvaIncrement`: generate analysis increment diagnostic plots.
-- `SaveObsDiags` (unless `skip_r2d2` is `true`): store the observation diagnostics in R2D2.
+- `SaveObsDiags` (unless `skip_store_r2d2` is `true`): store the observation diagnostics in R2D2.
 - `CleanCycle`: remove large intermediate files matching `clean_patterns`.
 
 For the exact task dependency graph, look at the `3dvar_cf` suite's workflow definition in the Swell

docs/practical_examples/geos_cf/3dvar_cf_cycle.md

@@ -124,7 +124,7 @@ For each cycle and each `model_component` (only `geos_cf` in this suite), the fo
 - `EvaObservations`: generate observation-space diagnostic plots.
 - `EvaJediLog`: generate JEDI log (cost function/iteration) diagnostic plots.
 - `EvaIncrement`: generate analysis increment diagnostic plots.
-- `SaveObsDiags` (unless `skip_r2d2` is `true`): store the observation diagnostics in R2D2.
+- `SaveObsDiags` (unless `skip_store_r2d2` is `true`): store the observation diagnostics in R2D2.
 - `PrepForecastCf`: update RC files and create the forecast scratch directory from the analysis.
 - `GetRestartCf`: fetch the restart files (of the types in `rst_file_types`) from R2D2.
 - `RunForecast`: run the GEOS-CF forecast for `forecast_length`, writing output every

docs/practical_examples/geos_cf/hofx_cf.md

@@ -87,7 +87,7 @@ For each cycle and each `model_component` (only `geos_cf` in this suite), the fo
 - `RenderJediObservations`: render the JEDI observation configuration.
 - `RunJediHofxExecutable`: run the JEDI H(x) executable.
 - `EvaObservations`: generate observation-space diagnostic plots.
-- `SaveObsDiags` (unless `skip_r2d2` is `true`): store the observation diagnostics in R2D2.
+- `SaveObsDiags` (unless `skip_store_r2d2` is `true`): store the observation diagnostics in R2D2.
 - `CleanCycle`: remove large intermediate files matching `clean_patterns`.
 
 For the exact task dependency graph, look at the `hofx_cf` suite's workflow definition in the Swell

docs/running_an_experiment/understanding_configuration.md

@@ -46,7 +46,7 @@ Several options configure broad categories of behavior:
 - `-p/--platform` selects which platform defaults provide paths, accounts, and resources.
 - `-s/--slurm` applies [SLURM directives](../configuration_reference/slurm_configuration.md)
   globally, per task, or per task and model combination.
-- `-k/--skip-r2d2` disables R2D2 registration and storage for the experiment.
+- `-k/--skip-store-r2d2` disables R2D2 registration and storage for the experiment.
 
 These options are useful for settings that apply across the workflow. Experiment-specific values
 should normally remain in an override file.

src/swell/configuration/jedi/interfaces/geos_atmosphere/suite_questions.yaml

@@ -8,6 +8,9 @@ ensemble_hofx_strategy:
 ensemble_hofx_packets:
   default_value: 1
 
+fetch_from_r2d2:
+  default_value: false
+
 saber_central_block:
   default_value: None
 

src/swell/configuration/jedi/interfaces/geos_cf/suite_questions.yaml

@@ -8,6 +8,9 @@ ensemble_hofx_strategy:
 ensemble_hofx_packets:
   default_value: 1
 
+fetch_from_r2d2:
+  default_value: true
+
 saber_central_block:
   default_value: bump_nicas
   options:

src/swell/configuration/jedi/interfaces/geos_marine/suite_questions.yaml

@@ -11,6 +11,9 @@ ensemble_hofx_strategy:
 ensemble_hofx_packets:
   default_value: 1
 
+fetch_from_r2d2:
+  default_value: false
+
 saber_central_block:
   default_value: None
 

src/swell/deployment/create_experiment.py

@@ -165,8 +165,8 @@ def prepare_config(
 
     # Register the experiment in R2D2
     # -------------------------------
-    if 'r2d2_experiment_id' in experiment_dict and 'skip_r2d2' in experiment_dict \
-            and not experiment_dict['skip_r2d2']:
+    if 'r2d2_experiment_id' in experiment_dict and 'skip_store_r2d2' in experiment_dict \
+            and not experiment_dict['skip_store_r2d2']:
 
         from swell.utilities.r2d2 import load_r2d2_credentials, load_r2d2_module, unique_r2d2_id
 
@@ -226,7 +226,7 @@ def create_experiment_directory(
     override: dict,
     advanced: bool,
     slurm: str | None,
-    skip_r2d2: bool
+    skip_store_r2d2: bool
 ) -> None:
 
     # Get the base name of the suite
@@ -239,10 +239,10 @@ def create_experiment_directory(
 
     # Specify whether to skip registering and storing in R2D2
     # -------------------------------------------------------
-    if skip_r2d2:
+    if skip_store_r2d2:
 
         # Only override this if it is true, otherwise let the suite decide
-        override['skip_r2d2'] = skip_r2d2
+        override['skip_store_r2d2'] = skip_store_r2d2
 
     # Call the experiment config and suite generation
     # ------------------------------------------------

src/swell/suites/3dfgat_atmos/flow.cylc

@@ -60,12 +60,10 @@
             # Get observations
             {% if cycling_varbc %}
             # Cycling VarBC is active, biases from the previous cycle will be used
-
-            RunJediVariationalExecutable-{{model_component}}[-PT6H] => GetObsNotInR2d2-{{model_component}}?
+            RunJediVariationalExecutable-{{model_component}}[-PT6H] => fetch_obs-{{model_component}}
             {% else %}
-
             # Cycling VarBC is inactive, static bias files will be used
-            GetObsNotInR2d2-{{model_component}}: fail? => GetObservations-{{model_component}}?
+            fetch_obs-{{model_component}}
             {% endif %}
 
             # Perform staging that is cycle dependent
@@ -76,7 +74,6 @@
             CloneJedi[^] => StageJediCycle-{{model_component}}
             StageJediCycle-{{model_component}} => RunJediVariationalExecutable-{{model_component}}
             GetBackgroundGeosExperiment-{{model_component}}? | GetBackground-{{model_component}} => RunJediVariationalExecutable-{{model_component}}
-            GetObsNotInR2d2-{{model_component}}: fail? => GetObservations-{{model_component}}?
             GenerateObservingSystemRecords-{{model_component}} => RenderJediObservations-{{model_component}}
 
             RenderJediObservations-{{model_component}} => RunJediVariationalExecutable-{{model_component}}
@@ -90,7 +87,7 @@
             # EvaIncrement
             RunJediVariationalExecutable-{{model_component}} => EvaIncrement-{{model_component}}
 
-            {% if not skip_r2d2 %}            
+            {% if not skip_store_r2d2 %}
             # Save observations
             RunJediVariationalExecutable-{{model_component}} => SaveObsDiags-{{model_component}}
             SaveObsDiags-{{model_component}} => CleanCycle-{{model_component}}
@@ -141,9 +138,6 @@
     [[CloneGeosMksi-{{model_component}}]]
         script = "swell task CloneGeosMksi $config -m {{model_component}}"
 
-    [[GetObsNotInR2d2-{{model_component}}]]
-        script = "swell task GetObsNotInR2d2 $config -d $datetime -m {{model_component}}"
-
     [[GenerateObservingSystemRecords-{{model_component}}]]
         script = "swell task GenerateObservingSystemRecords $config -d $datetime -m {{model_component}}"
 
@@ -156,8 +150,15 @@
     [[GetBackgroundGeosExperiment-{{model_component}} ]]
         script = "swell task GetBackgroundGeosExperiment $config -d $datetime -m {{model_component}}"
 
-    [[GetObservations-{{model_component}}]]
+    {% if fetch_from_r2d2 %}
+    # Fetch from R2D2
+    [[fetch_obs-{{model_component}}]]
         script = "swell task GetObservations $config -d $datetime -m {{model_component}}"
+    {% else %}
+    # Copy from local filesystem
+    [[fetch_obs-{{model_component}}]]
+        script = "swell task GetObsNotInR2d2 $config -d $datetime -m {{model_component}}"
+    {% endif %}
 
     [[RenderJediObservations-{{model_component}}]]
         script = "swell task RenderJediObservations $config -d $datetime -m {{model_component}}"

src/swell/suites/3dfgat_atmos/suite_config.py

@@ -30,6 +30,7 @@ class SuiteConfig(QuestionContainer, Enum):
             qd.jedi_build_method("use_existing"),
             qd.model_components(['geos_atmosphere']),
             qd.runahead_limit("P2"),
+            qd.fetch_from_r2d2(False)
         ],
         geos_atmosphere=[
             qd.cycle_times([

src/swell/suites/3dfgat_marine_cycle/flow.cylc

@@ -105,7 +105,7 @@
             # Move restart to next cycle and then erase current forecast folder
             SaveRestart-{{model_component}} => MoveDaRestart-{{model_component}} => CleanCycle-{{model_component}}
 
-            {% if not skip_r2d2 %}
+            {% if not skip_store_r2d2 %}
             # Compress forecast on compute node, then store in R2D2 on login node
             LinkCoupledGeosOutput-{{model_component}} => RunCompressForecast-{{model_component}} => SaveForecast-{{model_component}} => MoveDaRestart-{{model_component}}
             RunJediFgatExecutable-{{model_component}} => SaveObsDiags-{{model_component}} => CleanCycle-{{model_component}}

src/swell/suites/3dvar_atmos/flow.cylc

@@ -59,11 +59,11 @@
             {% if cycling_varbc %}
             # Cycling VarBC is active, biases from the previous cycle will be used
 
-            RunJediVariationalExecutable-{{model_component}}[-PT6H] => GetObsNotInR2d2-{{model_component}}?
+            RunJediVariationalExecutable-{{model_component}}[-PT6H] => fetch_obs-{{model_component}}?
             {% else %}
 
             # Cycling VarBC is inactive, static bias files will be used
-            GetObsNotInR2d2-{{model_component}}: fail? => GetObservations-{{model_component}}?
+            fetch_obs-{{model_component}}
             {% endif %}
 
             # Perform staging that is cycle dependent
@@ -74,7 +74,7 @@
             CloneJedi[^] => StageJediCycle-{{model_component}}
             StageJediCycle-{{model_component}} => RunJediVariationalExecutable-{{model_component}}
             GetBackgroundGeosExperiment-{{model_component}}? | GetBackground-{{model_component}} => RunJediVariationalExecutable-{{model_component}}
-            GetObsNotInR2d2-{{model_component}}? | GetObservations-{{model_component}}? => RenderJediObservations-{{model_component}}
+            fetch_obs-{{model_component}} => RenderJediObservations-{{model_component}}
             GenerateObservingSystemRecords-{{model_component}} => RenderJediObservations-{{model_component}}
             RenderJediObservations-{{model_component}} => RunJediVariationalExecutable-{{model_component}}
 
@@ -87,7 +87,7 @@
             # EvaIncrement
             RunJediVariationalExecutable-{{model_component}} => EvaIncrement-{{model_component}}
 
-	    {% if not skip_r2d2 %}
+	    {% if not skip_store_r2d2 %}
             # Save observations
             RunJediVariationalExecutable-{{model_component}} => SaveObsDiags-{{model_component}} => CleanCycle-{{model_component}}
             {% endif %}
@@ -140,9 +140,6 @@
     [[GenerateObservingSystemRecords-{{model_component}}]]
         script = "swell task GenerateObservingSystemRecords $config -d $datetime -m {{model_component}}"
 
-    [[GetObsNotInR2d2-{{model_component}}]]
-        script = "swell task GetObsNotInR2d2 $config -d $datetime -m {{model_component}}"
-
     [[StageJediCycle-{{model_component}}]]
         script = "swell task StageJedi $config -d $datetime -m {{model_component}}"
 
@@ -152,8 +149,15 @@
     [[GetBackgroundGeosExperiment-{{model_component}} ]]
         script = "swell task GetBackgroundGeosExperiment $config -d $datetime -m {{model_component}}"
 
-    [[GetObservations-{{model_component}}]]
+    {% if fetch_from_r2d2 %}
+    # Fetch from R2D2
+    [[fetch_obs-{{model_component}}]]
         script = "swell task GetObservations $config -d $datetime -m {{model_component}}"
+    {% else %}
+    # Copy from local filesystem
+    [[fetch_obs-{{model_component}}]]
+        script = "swell task GetObsNotInR2d2 $config -d $datetime -m {{model_component}}"
+    {% endif %}
 
     [[RenderJediObservations-{{model_component}}]]
         script = "swell task RenderJediObservations $config -d $datetime -m {{model_component}}"

src/swell/suites/3dvar_atmos/suite_config.py

@@ -30,6 +30,7 @@ class SuiteConfig(QuestionContainer, Enum):
             qd.runahead_limit("P2"),
             qd.jedi_build_method("use_existing"),
             qd.model_components(['geos_atmosphere']),
+            qd.fetch_from_r2d2(False)
         ],
         geos_atmosphere=[
             qd.cycle_times([

src/swell/suites/3dvar_cf/flow.cylc

@@ -67,7 +67,7 @@
             # EvaIncrement
             RunJediVariationalExecutable-{{model_component}} => EvaIncrement-{{model_component}}
 
-            {% if not skip_r2d2 %}
+            {% if not skip_store_r2d2 %}
             # Save observations
             RunJediVariationalExecutable-{{model_component}} => SaveObsDiags-{{model_component}}
             SaveObsDiags-{{model_component}} => CleanCycle-{{model_component}}

src/swell/suites/3dvar_cf_cycle/flow.cylc

@@ -83,12 +83,16 @@
             # EvaIncrement
             RunJediVariationalExecutable-{{model_component}} => EvaIncrement-{{model_component}}
 
+            {% if not skip_store_r2d2 %}
             # Save observations
             RunJediVariationalExecutable-{{model_component}} => SaveObsDiags-{{model_component}}
+            {% endif %}
 
             # Clean up large files
-            EvaObservations-{{model_component}} & EvaJediLog-{{model_component}} & EvaIncrement-{{model_component}} & 
+            EvaObservations-{{model_component}} & EvaJediLog-{{model_component}} & EvaIncrement-{{model_component}} => CleanCycle-{{model_component}}
+            {% if not skip_store_r2d2 %}
             SaveObsDiags-{{model_component}} & SaveForecastCf-{{model_component}} & SaveRestartCf-{{model_component}} => CleanCycle-{{model_component}}
+            {% endif %}
 
         {% endif %}
         {% endfor %}

src/swell/suites/3dvar_marine/flow.cylc

@@ -75,16 +75,15 @@
             # EvaIncrement
             RunJediVariationalExecutable-{{model_component}} => EvaIncrement-{{model_component}}
 
-            {% if not skip_r2d2 %}
+            {% if not skip_store_r2d2 %}
             # Save observations
             RunJediVariationalExecutable-{{model_component}} => SaveObsDiags-{{model_component}}
             SaveObsDiags-{{model_component}} => CleanCycle-{{model_component}}
             {% endif %}
 
             # Clean up large files
             EvaJediLog-{{model_component}} & EvaIncrement-{{model_component}} &
-            EvaObservations-{{model_component}} & SaveObsDiags-{{model_component}} =>
-            CleanCycle-{{model_component}}
+            EvaObservations-{{model_component}} => CleanCycle-{{model_component}}
 
         {% endif %}
         {% endfor %}