diff --git a/docs/additional_resources/faq.md b/docs/additional_resources/faq.md index 090cca1ba..e70035678 100644 --- a/docs/additional_resources/faq.md +++ b/docs/additional_resources/faq.md @@ -18,7 +18,7 @@ - R2D2 is not working for me - > Please see [R2D2 settings](../configuration_reference/r2d2_v3_credentials.md). As of the writing of this documentation, the same shared gmao-user API key is used, but this will change soon. Alternatively, the `--skip-r2d2` or `-k` argument can be used for `geos_atmosphere` applications to bypass R2D2 and use Ricardo's x00** experiment directories. + > Please see [R2D2 settings](../configuration_reference/r2d2_v3_credentials.md). As of the writing of this documentation, the same shared gmao-user API key is used, but this will change soon. Alternatively, the `--skip-store-r2d2` or `-k` argument can be used for `geos_atmosphere` applications to bypass R2D2 and use Ricardo's x00** experiment directories. - Can I view Cylc logs with my own editor? diff --git a/docs/introduction/prerequisites.md b/docs/introduction/prerequisites.md index 9316f44e0..dfdda71d0 100644 --- a/docs/introduction/prerequisites.md +++ b/docs/introduction/prerequisites.md @@ -5,7 +5,7 @@ Before installing Swell, make sure you have: - **A platform.** Swell is developed and tested primarily on NASA's Discover cluster, which has dedicated instructions — see [Supported Platforms](../installation_and_setup/platforms/README.md). Other platforms are supported in principle, but you're responsible for satisfying the dependencies below yourself. - **Python and matching dependencies.** Swell is a Python package installed with pip. It shares dependencies with the applications it drives (JEDI, GEOS), so these must be satisfied consistently. On Discover, loading the correct `spack-stack` modules before installing avoids this. See [Installing Swell](../installation_and_setup/installing_swell.md). - **Cylc.** Swell generates `flow.cylc` workflows but does not install or manage Cylc itself. You need a `cylc` executable on your `$PATH` and a `$HOME/.cylc/flow/global.cylc` file. On Discover this is handled automatically. See [Configuring Cylc](../installation_and_setup/configuring_cylc.md). -- **R2D2 credentials.** A `~/.swell/r2d2_credentials.yaml` file is required for `swell create` to register experiments and for tasks to fetch/store data. See [R2D2 v3 Credentials](../configuration_reference/r2d2_v3_credentials.md). You can bypass registering your experiment and storing data in R2D2 by passing `-k`/`--skip-r2d2` to `swell create`, though suites that require fetching files from R2D2 (which is most) still require configured R2D2 credentials. — see [QuickStart](quickstart.md#skipping-r2d2-for-a-first-run). +- **R2D2 credentials.** A `~/.swell/r2d2_credentials.yaml` file is required for `swell create` to register experiments and for tasks to fetch/store data. See [R2D2 v3 Credentials](../configuration_reference/r2d2_v3_credentials.md). - **Accounts for external data (if ingesting observations).** Some observation providers require their own authentication (e.g. an Earthdata account). See [Storing Observations and Backgrounds in R2D2](../practical_examples/r2d2/r2d2_ingest.md). Once these are in place, continue to [Installing Swell](../installation_and_setup/installing_swell.md). diff --git a/docs/introduction/quickstart.md b/docs/introduction/quickstart.md index f44e707ac..20810b946 100644 --- a/docs/introduction/quickstart.md +++ b/docs/introduction/quickstart.md @@ -27,7 +27,7 @@ For example: swell create 3dvar_marine ``` -This registers the experiment in R2D2 and creates a directory (default `/discover/nobackup/$USER/SwellExperiments/-suite`) containing `experiment.yaml` and a generated `flow.cylc`. See [Creating an Experiment](../running_an_experiment/creating_an_experiment.md) for options like overrides and platform selection. +This creates a directory (default `/discover/nobackup/$USER/SwellExperiments/-suite`) containing `experiment.yaml` and a generated `flow.cylc`. See [Creating an Experiment](../running_an_experiment/creating_an_experiment.md) for options like overrides and platform selection. ### Useful Options @@ -36,24 +36,11 @@ This registers the experiment in R2D2 and creates a directory (default `/discove | `-o`, `--override ` | `swell create` | Override any `experiment.yaml` value, e.g. `experiment_root` (the run directory) or `experiment_id`. | | `-p`, `--platform ` | `swell create` | Select platform-specific defaults (e.g. `nccs_discover_sles15`). | | `-s`, `--slurm ` | `swell create` | Override SLURM directives (account, nodes, qos, etc.), globally or per task/model. See [SLURM Configuration](../configuration_reference/slurm_configuration.md). | -| `-k`, `--skip-r2d2` | `swell create` | Skip registering the experiment and storing products in R2D2. Useful if you don't have credentials set up yet — see [below](#skipping-r2d2-for-a-first-run). | +| `-r`, `--store-r2d2` | `swell create` | Enables registering the experiment and storing products in R2D2. Requires properly configured R2D2 credentials - see [R2D2 Credentials](../configuration_reference/r2d2_v3_credentials.md) | | `-l`, `--log_path ` | `swell launch` | Directory to receive workflow manager (Cylc) logging output, instead of the default `$HOME/cylc-run/`. | Run `swell create --help` or `swell launch --help` for the full, up-to-date list. -### Skipping R2D2 for a first run - -Are you new to Swell and don't have R2D2 credentials set up yet? Pass `-k`/`--skip-r2d2` to -`swell create` to skip registering the experiment and storing products in R2D2, though experiments needing to fetch from R2D2 still require credentials: - -```bash -swell create --skip-r2d2 -``` - -This lets you complete a full create -> launch -> monitor cycle to see Swell working end to end -before setting up your [R2D2 credentials](../configuration_reference/r2d2_v3_credentials.md). Tasks -that use R2D2 (fetching observations, storing backgrounds/analyses) will be skipped rather than run. - ## 3. Launch It `swell create` prints the exact command to run next, typically: @@ -83,5 +70,5 @@ If the TUI gets closed you can reopen it by typing `cylc tui` in the terminal. Once tasks finish, the experiment directory will contain the generated files. Note that some tasks are required to succeed to trigger the execution of other tasks while some tasks can fail without impacting cycling, if the Cylc graph doesn't require completion of the task to move forward. -By default `skip_r2d2` is `false`, so outputs (backgrounds, analyses, diagnostics) will also be registered and stored in R2D2 (see [Skipping R2D2 for a first run](#skipping-r2d2-for-a-first-run) if you don't have credentials set up yet). From here, see [Choosing a Workflow](../running_an_experiment/choosing_a_workflow.md) to explore other suites. +By default `store_r2d2` is `false` (note this is the name of the key in `experiment.yaml`, which can be set to true on the command line using `--store-r2d2` upon experiment creation), so outputs (backgrounds, analyses, diagnostics) will not be registered and stored in R2D2. From here, see [Choosing a Workflow](../running_an_experiment/choosing_a_workflow.md) to explore other suites. diff --git a/docs/practical_examples/geos_cf/3dvar_cf.md b/docs/practical_examples/geos_cf/3dvar_cf.md index 0e63e9411..a7bd9b568 100644 --- a/docs/practical_examples/geos_cf/3dvar_cf.md +++ b/docs/practical_examples/geos_cf/3dvar_cf.md @@ -136,7 +136,7 @@ For each cycle and each `model_component` (only `geos_cf` in this suite), the fo - `EvaObservations`: generate observation-space diagnostic plots. - `EvaJediLog`: generate JEDI log (cost function/iteration) diagnostic plots. - `EvaIncrement`: generate analysis increment diagnostic plots. -- `SaveObsDiags` (unless `skip_r2d2` is `true`): store the observation diagnostics in R2D2. +- `SaveObsDiags` (only if `store_r2d2` is `true`): store the observation diagnostics in R2D2. - `CleanCycle`: remove large intermediate files matching `clean_patterns`. For the exact task dependency graph, look at the `3dvar_cf` suite's workflow definition in the Swell diff --git a/docs/practical_examples/geos_cf/3dvar_cf_cycle.md b/docs/practical_examples/geos_cf/3dvar_cf_cycle.md index f0b4fc0b2..1cf3ec6a3 100644 --- a/docs/practical_examples/geos_cf/3dvar_cf_cycle.md +++ b/docs/practical_examples/geos_cf/3dvar_cf_cycle.md @@ -124,7 +124,7 @@ For each cycle and each `model_component` (only `geos_cf` in this suite), the fo - `EvaObservations`: generate observation-space diagnostic plots. - `EvaJediLog`: generate JEDI log (cost function/iteration) diagnostic plots. - `EvaIncrement`: generate analysis increment diagnostic plots. -- `SaveObsDiags` (unless `skip_r2d2` is `true`): store the observation diagnostics in R2D2. +- `SaveObsDiags` (only if `store_r2d2` is `true`): store the observation diagnostics in R2D2. - `PrepForecastCf`: update RC files and create the forecast scratch directory from the analysis. - `GetRestartCf`: fetch the restart files (of the types in `rst_file_types`) from R2D2. - `RunForecast`: run the GEOS-CF forecast for `forecast_length`, writing output every diff --git a/docs/practical_examples/geos_cf/hofx_cf.md b/docs/practical_examples/geos_cf/hofx_cf.md index ba3ef23f9..fd13ef3e1 100644 --- a/docs/practical_examples/geos_cf/hofx_cf.md +++ b/docs/practical_examples/geos_cf/hofx_cf.md @@ -87,7 +87,7 @@ For each cycle and each `model_component` (only `geos_cf` in this suite), the fo - `RenderJediObservations`: render the JEDI observation configuration. - `RunJediHofxExecutable`: run the JEDI H(x) executable. - `EvaObservations`: generate observation-space diagnostic plots. -- `SaveObsDiags` (unless `skip_r2d2` is `true`): store the observation diagnostics in R2D2. +- `SaveObsDiags` (only if `store_r2d2` is `true`): store the observation diagnostics in R2D2. - `CleanCycle`: remove large intermediate files matching `clean_patterns`. For the exact task dependency graph, look at the `hofx_cf` suite's workflow definition in the Swell diff --git a/docs/running_an_experiment/understanding_configuration.md b/docs/running_an_experiment/understanding_configuration.md index 797cd8ceb..5c850ed4b 100644 --- a/docs/running_an_experiment/understanding_configuration.md +++ b/docs/running_an_experiment/understanding_configuration.md @@ -46,7 +46,7 @@ Several options configure broad categories of behavior: - `-p/--platform` selects which platform defaults provide paths, accounts, and resources. - `-s/--slurm` applies [SLURM directives](../configuration_reference/slurm_configuration.md) globally, per task, or per task and model combination. -- `-k/--skip-r2d2` disables R2D2 registration and storage for the experiment. +- `-k/--skip-store-r2d2` disables R2D2 registration and storage for the experiment. These options are useful for settings that apply across the workflow. Experiment-specific values should normally remain in an override file. diff --git a/src/swell/configuration/jedi/interfaces/geos_atmosphere/suite_questions.yaml b/src/swell/configuration/jedi/interfaces/geos_atmosphere/suite_questions.yaml index a39e91c8c..8f03fdae8 100644 --- a/src/swell/configuration/jedi/interfaces/geos_atmosphere/suite_questions.yaml +++ b/src/swell/configuration/jedi/interfaces/geos_atmosphere/suite_questions.yaml @@ -8,6 +8,9 @@ ensemble_hofx_strategy: ensemble_hofx_packets: default_value: 1 +fetch_from_r2d2: + default_value: false + saber_central_block: default_value: None diff --git a/src/swell/configuration/jedi/interfaces/geos_cf/suite_questions.yaml b/src/swell/configuration/jedi/interfaces/geos_cf/suite_questions.yaml index 3a8bb4a09..fc2a50fe3 100644 --- a/src/swell/configuration/jedi/interfaces/geos_cf/suite_questions.yaml +++ b/src/swell/configuration/jedi/interfaces/geos_cf/suite_questions.yaml @@ -8,6 +8,9 @@ ensemble_hofx_strategy: ensemble_hofx_packets: default_value: 1 +fetch_from_r2d2: + default_value: true + saber_central_block: default_value: bump_nicas options: diff --git a/src/swell/configuration/jedi/interfaces/geos_marine/suite_questions.yaml b/src/swell/configuration/jedi/interfaces/geos_marine/suite_questions.yaml index f0e3ba553..305896534 100644 --- a/src/swell/configuration/jedi/interfaces/geos_marine/suite_questions.yaml +++ b/src/swell/configuration/jedi/interfaces/geos_marine/suite_questions.yaml @@ -11,6 +11,9 @@ ensemble_hofx_strategy: ensemble_hofx_packets: default_value: 1 +fetch_from_r2d2: + default_value: false + saber_central_block: default_value: None diff --git a/src/swell/deployment/create_experiment.py b/src/swell/deployment/create_experiment.py index 101849248..0660e5e49 100644 --- a/src/swell/deployment/create_experiment.py +++ b/src/swell/deployment/create_experiment.py @@ -165,8 +165,8 @@ def prepare_config( # Register the experiment in R2D2 # ------------------------------- - if 'r2d2_experiment_id' in experiment_dict and 'skip_r2d2' in experiment_dict \ - and not experiment_dict['skip_r2d2']: + if 'r2d2_experiment_id' in experiment_dict and 'store_r2d2' in experiment_dict \ + and experiment_dict['store_r2d2']: from swell.utilities.r2d2 import load_r2d2_credentials, load_r2d2_module, unique_r2d2_id @@ -226,7 +226,7 @@ def create_experiment_directory( override: dict, advanced: bool, slurm: str | None, - skip_r2d2: bool + store_r2d2: bool = False ) -> None: # Get the base name of the suite @@ -239,10 +239,10 @@ def create_experiment_directory( # Specify whether to skip registering and storing in R2D2 # ------------------------------------------------------- - if skip_r2d2: + if store_r2d2: # Only override this if it is true, otherwise let the suite decide - override['skip_r2d2'] = skip_r2d2 + override['store_r2d2'] = store_r2d2 # Call the experiment config and suite generation # ------------------------------------------------ diff --git a/src/swell/suites/3dfgat_atmos/flow.cylc b/src/swell/suites/3dfgat_atmos/flow.cylc index a6fdc1722..c82abd8a6 100644 --- a/src/swell/suites/3dfgat_atmos/flow.cylc +++ b/src/swell/suites/3dfgat_atmos/flow.cylc @@ -60,12 +60,10 @@ # Get observations {% if cycling_varbc %} # Cycling VarBC is active, biases from the previous cycle will be used - - RunJediVariationalExecutable-{{model_component}}[-PT6H] => GetObsNotInR2d2-{{model_component}}? + RunJediVariationalExecutable-{{model_component}}[-PT6H] => fetch_obs-{{model_component}} {% else %} - # Cycling VarBC is inactive, static bias files will be used - GetObsNotInR2d2-{{model_component}}: fail? => GetObservations-{{model_component}}? + fetch_obs-{{model_component}} {% endif %} # Perform staging that is cycle dependent @@ -76,7 +74,6 @@ CloneJedi[^] => StageJediCycle-{{model_component}} StageJediCycle-{{model_component}} => RunJediVariationalExecutable-{{model_component}} GetBackgroundGeosExperiment-{{model_component}}? | GetBackground-{{model_component}} => RunJediVariationalExecutable-{{model_component}} - GetObsNotInR2d2-{{model_component}}: fail? => GetObservations-{{model_component}}? GenerateObservingSystemRecords-{{model_component}} => RenderJediObservations-{{model_component}} RenderJediObservations-{{model_component}} => RunJediVariationalExecutable-{{model_component}} @@ -90,7 +87,7 @@ # EvaIncrement RunJediVariationalExecutable-{{model_component}} => EvaIncrement-{{model_component}} - {% if not skip_r2d2 %} + {% if store_r2d2 %} # Save observations RunJediVariationalExecutable-{{model_component}} => SaveObsDiags-{{model_component}} SaveObsDiags-{{model_component}} => CleanCycle-{{model_component}} @@ -141,9 +138,6 @@ [[CloneGeosMksi-{{model_component}}]] script = "swell task CloneGeosMksi $config -m {{model_component}}" - [[GetObsNotInR2d2-{{model_component}}]] - script = "swell task GetObsNotInR2d2 $config -d $datetime -m {{model_component}}" - [[GenerateObservingSystemRecords-{{model_component}}]] script = "swell task GenerateObservingSystemRecords $config -d $datetime -m {{model_component}}" @@ -156,8 +150,15 @@ [[GetBackgroundGeosExperiment-{{model_component}} ]] script = "swell task GetBackgroundGeosExperiment $config -d $datetime -m {{model_component}}" - [[GetObservations-{{model_component}}]] + {% if fetch_from_r2d2 %} + # Fetch from R2D2 + [[fetch_obs-{{model_component}}]] script = "swell task GetObservations $config -d $datetime -m {{model_component}}" + {% else %} + # Copy from local filesystem + [[fetch_obs-{{model_component}}]] + script = "swell task GetObsNotInR2d2 $config -d $datetime -m {{model_component}}" + {% endif %} [[RenderJediObservations-{{model_component}}]] script = "swell task RenderJediObservations $config -d $datetime -m {{model_component}}" diff --git a/src/swell/suites/3dfgat_atmos/suite_config.py b/src/swell/suites/3dfgat_atmos/suite_config.py index 58122cbf9..70574f63c 100644 --- a/src/swell/suites/3dfgat_atmos/suite_config.py +++ b/src/swell/suites/3dfgat_atmos/suite_config.py @@ -30,6 +30,7 @@ class SuiteConfig(QuestionContainer, Enum): qd.jedi_build_method("use_existing"), qd.model_components(['geos_atmosphere']), qd.runahead_limit("P2"), + qd.fetch_from_r2d2(False) ], geos_atmosphere=[ qd.cycle_times([ diff --git a/src/swell/suites/3dfgat_marine_cycle/flow.cylc b/src/swell/suites/3dfgat_marine_cycle/flow.cylc index a36d54a43..fb9ddb0a1 100644 --- a/src/swell/suites/3dfgat_marine_cycle/flow.cylc +++ b/src/swell/suites/3dfgat_marine_cycle/flow.cylc @@ -105,7 +105,7 @@ # Move restart to next cycle and then erase current forecast folder SaveRestart-{{model_component}} => MoveDaRestart-{{model_component}} => CleanCycle-{{model_component}} - {% if not skip_r2d2 %} + {% if store_r2d2 %} # Compress forecast on compute node, then store in R2D2 on login node LinkCoupledGeosOutput-{{model_component}} => RunCompressForecast-{{model_component}} => SaveForecast-{{model_component}} => MoveDaRestart-{{model_component}} RunJediFgatExecutable-{{model_component}} => SaveObsDiags-{{model_component}} => CleanCycle-{{model_component}} diff --git a/src/swell/suites/3dvar_atmos/flow.cylc b/src/swell/suites/3dvar_atmos/flow.cylc index e2ac0616b..7406073a7 100644 --- a/src/swell/suites/3dvar_atmos/flow.cylc +++ b/src/swell/suites/3dvar_atmos/flow.cylc @@ -59,11 +59,11 @@ {% if cycling_varbc %} # Cycling VarBC is active, biases from the previous cycle will be used - RunJediVariationalExecutable-{{model_component}}[-PT6H] => GetObsNotInR2d2-{{model_component}}? + RunJediVariationalExecutable-{{model_component}}[-PT6H] => fetch_obs-{{model_component}}? {% else %} # Cycling VarBC is inactive, static bias files will be used - GetObsNotInR2d2-{{model_component}}: fail? => GetObservations-{{model_component}}? + fetch_obs-{{model_component}} {% endif %} # Perform staging that is cycle dependent @@ -74,7 +74,7 @@ CloneJedi[^] => StageJediCycle-{{model_component}} StageJediCycle-{{model_component}} => RunJediVariationalExecutable-{{model_component}} GetBackgroundGeosExperiment-{{model_component}}? | GetBackground-{{model_component}} => RunJediVariationalExecutable-{{model_component}} - GetObsNotInR2d2-{{model_component}}? | GetObservations-{{model_component}}? => RenderJediObservations-{{model_component}} + fetch_obs-{{model_component}} => RenderJediObservations-{{model_component}} GenerateObservingSystemRecords-{{model_component}} => RenderJediObservations-{{model_component}} RenderJediObservations-{{model_component}} => RunJediVariationalExecutable-{{model_component}} @@ -87,7 +87,7 @@ # EvaIncrement RunJediVariationalExecutable-{{model_component}} => EvaIncrement-{{model_component}} - {% if not skip_r2d2 %} + {% if store_r2d2 %} # Save observations RunJediVariationalExecutable-{{model_component}} => SaveObsDiags-{{model_component}} => CleanCycle-{{model_component}} {% endif %} @@ -140,9 +140,6 @@ [[GenerateObservingSystemRecords-{{model_component}}]] script = "swell task GenerateObservingSystemRecords $config -d $datetime -m {{model_component}}" - [[GetObsNotInR2d2-{{model_component}}]] - script = "swell task GetObsNotInR2d2 $config -d $datetime -m {{model_component}}" - [[StageJediCycle-{{model_component}}]] script = "swell task StageJedi $config -d $datetime -m {{model_component}}" @@ -152,8 +149,15 @@ [[GetBackgroundGeosExperiment-{{model_component}} ]] script = "swell task GetBackgroundGeosExperiment $config -d $datetime -m {{model_component}}" - [[GetObservations-{{model_component}}]] + {% if fetch_from_r2d2 %} + # Fetch from R2D2 + [[fetch_obs-{{model_component}}]] script = "swell task GetObservations $config -d $datetime -m {{model_component}}" + {% else %} + # Copy from local filesystem + [[fetch_obs-{{model_component}}]] + script = "swell task GetObsNotInR2d2 $config -d $datetime -m {{model_component}}" + {% endif %} [[RenderJediObservations-{{model_component}}]] script = "swell task RenderJediObservations $config -d $datetime -m {{model_component}}" diff --git a/src/swell/suites/3dvar_atmos/suite_config.py b/src/swell/suites/3dvar_atmos/suite_config.py index 6dbe4cb6f..9364dcd32 100644 --- a/src/swell/suites/3dvar_atmos/suite_config.py +++ b/src/swell/suites/3dvar_atmos/suite_config.py @@ -30,6 +30,7 @@ class SuiteConfig(QuestionContainer, Enum): qd.runahead_limit("P2"), qd.jedi_build_method("use_existing"), qd.model_components(['geos_atmosphere']), + qd.fetch_from_r2d2(False) ], geos_atmosphere=[ qd.cycle_times([ diff --git a/src/swell/suites/3dvar_cf/flow.cylc b/src/swell/suites/3dvar_cf/flow.cylc index b23170d69..a6631409c 100644 --- a/src/swell/suites/3dvar_cf/flow.cylc +++ b/src/swell/suites/3dvar_cf/flow.cylc @@ -69,7 +69,7 @@ # EvaIncrement RunJediVariationalExecutable-{{model_component}} => EvaIncrement-{{model_component}} - {% if not skip_r2d2 %} + {% if store_r2d2 %} # Save observations RunJediVariationalExecutable-{{model_component}} => SaveObsDiags-{{model_component}} SaveObsDiags-{{model_component}} => CleanCycle-{{model_component}} diff --git a/src/swell/suites/3dvar_cf_cycle/flow.cylc b/src/swell/suites/3dvar_cf_cycle/flow.cylc index ff120077e..ad0652663 100644 --- a/src/swell/suites/3dvar_cf_cycle/flow.cylc +++ b/src/swell/suites/3dvar_cf_cycle/flow.cylc @@ -86,12 +86,16 @@ # EvaIncrement RunJediVariationalExecutable-{{model_component}} => EvaIncrement-{{model_component}} + {% if store_r2d2 %} # Save observations RunJediVariationalExecutable-{{model_component}} => SaveObsDiags-{{model_component}} + {% endif %} # Clean up large files - EvaObservations-{{model_component}} & EvaJediLog-{{model_component}} & EvaIncrement-{{model_component}} & + EvaObservations-{{model_component}} & EvaJediLog-{{model_component}} & EvaIncrement-{{model_component}} => CleanCycle-{{model_component}} + {% if store_r2d2 %} SaveObsDiags-{{model_component}} & SaveForecastCf-{{model_component}} & SaveRestartCf-{{model_component}} => CleanCycle-{{model_component}} + {% endif %} {% endif %} {% endfor %} diff --git a/src/swell/suites/3dvar_marine/flow.cylc b/src/swell/suites/3dvar_marine/flow.cylc index 95ab19c8f..052fe1f4b 100644 --- a/src/swell/suites/3dvar_marine/flow.cylc +++ b/src/swell/suites/3dvar_marine/flow.cylc @@ -75,7 +75,7 @@ # EvaIncrement RunJediVariationalExecutable-{{model_component}} => EvaIncrement-{{model_component}} - {% if not skip_r2d2 %} + {% if store_r2d2 %} # Save observations RunJediVariationalExecutable-{{model_component}} => SaveObsDiags-{{model_component}} SaveObsDiags-{{model_component}} => CleanCycle-{{model_component}} @@ -83,8 +83,7 @@ # Clean up large files EvaJediLog-{{model_component}} & EvaIncrement-{{model_component}} & - EvaObservations-{{model_component}} & SaveObsDiags-{{model_component}} => - CleanCycle-{{model_component}} + EvaObservations-{{model_component}} => CleanCycle-{{model_component}} {% endif %} {% endfor %} diff --git a/src/swell/suites/3dvar_marine_cycle/flow.cylc b/src/swell/suites/3dvar_marine_cycle/flow.cylc index d7619156a..9da5582fb 100644 --- a/src/swell/suites/3dvar_marine_cycle/flow.cylc +++ b/src/swell/suites/3dvar_marine_cycle/flow.cylc @@ -104,7 +104,7 @@ # Move restart to next cycle and then erase current forecast folder SaveRestart-{{model_component}} => MoveDaRestart-{{model_component}} => CleanCycle-{{model_component}} - {% if not skip_r2d2 %} + {% if store_r2d2 %} # Compress forecast on compute node, then store in R2D2 on login node LinkCoupledGeosOutput-{{model_component}} => RunCompressForecast-{{model_component}} => SaveForecast-{{model_component}} => MoveDaRestart-{{model_component}} RunJediVariationalExecutable-{{model_component}} => SaveObsDiags-{{model_component}} => CleanCycle-{{model_component}} diff --git a/src/swell/suites/build_geos/suite_config.py b/src/swell/suites/build_geos/suite_config.py index a61136668..dc4adecbd 100644 --- a/src/swell/suites/build_geos/suite_config.py +++ b/src/swell/suites/build_geos/suite_config.py @@ -10,6 +10,7 @@ from swell.utilities.swell_questions import QuestionContainer, QuestionList from swell.suites.suite_questions import SuiteQuestions as sq +# from swell.utilities.question_defaults import QuestionDefaults as qd from enum import Enum @@ -23,7 +24,7 @@ class SuiteConfig(QuestionContainer, Enum): build_geos = QuestionList( list_name="build_geos", questions=[ - sq.all_suites + sq.all_suites, ] ) diff --git a/src/swell/suites/build_jedi/suite_config.py b/src/swell/suites/build_jedi/suite_config.py index 4da92ab6b..6979e7df9 100644 --- a/src/swell/suites/build_jedi/suite_config.py +++ b/src/swell/suites/build_jedi/suite_config.py @@ -10,6 +10,7 @@ from swell.utilities.swell_questions import QuestionContainer, QuestionList from swell.suites.suite_questions import SuiteQuestions as sq +# from swell.utilities.question_defaults import QuestionDefaults as qd from enum import Enum @@ -23,7 +24,7 @@ class SuiteConfig(QuestionContainer, Enum): build_jedi = QuestionList( list_name="build_jedi", questions=[ - sq.all_suites + sq.all_suites, ] ) diff --git a/src/swell/suites/compare/suite_config.py b/src/swell/suites/compare/suite_config.py index 7847c366a..2b5bee45d 100644 --- a/src/swell/suites/compare/suite_config.py +++ b/src/swell/suites/compare/suite_config.py @@ -29,7 +29,7 @@ class SuiteConfig(QuestionContainer, Enum): qd.final_cycle_point(default_value=None, widget_type=WidgetType.STRING), qd.cycle_times(default_value=[None], widget_type=WidgetType.STRING_CHECK_LIST), qd.model_components(), - qd.runahead_limit(), + qd.runahead_limit() ] ) diff --git a/src/swell/suites/convert_bufr/suite_config.py b/src/swell/suites/convert_bufr/suite_config.py index 27e9d3e79..9e2c8afc3 100644 --- a/src/swell/suites/convert_bufr/suite_config.py +++ b/src/swell/suites/convert_bufr/suite_config.py @@ -28,7 +28,7 @@ class SuiteConfig(QuestionContainer, Enum): qd.start_cycle_point("2023-10-10T00:00:00Z"), qd.final_cycle_point("2023-10-10T06:00:00Z"), qd.jedi_build_method("use_existing"), - qd.model_components(['geos_atmosphere']), + qd.model_components(['geos_atmosphere']) ], geos_atmosphere=[ qd.cycle_times(['T00', 'T06', 'T12', 'T18']), diff --git a/src/swell/suites/convert_ncdiags/suite_config.py b/src/swell/suites/convert_ncdiags/suite_config.py index 268312c49..6db258f77 100644 --- a/src/swell/suites/convert_ncdiags/suite_config.py +++ b/src/swell/suites/convert_ncdiags/suite_config.py @@ -29,7 +29,7 @@ class SuiteConfig(QuestionContainer, Enum): qd.final_cycle_point("2025-12-30T06:00:00Z"), qd.jedi_build_method("use_existing"), qd.bundles("REMOVE"), - qd.model_components(['geos_atmosphere']), + qd.model_components(['geos_atmosphere']) ], geos_atmosphere=[ qd.cycle_times(['T00', 'T06']), diff --git a/src/swell/suites/eda_atmos/suite_config.py b/src/swell/suites/eda_atmos/suite_config.py index a632d8b86..5fe9fd379 100644 --- a/src/swell/suites/eda_atmos/suite_config.py +++ b/src/swell/suites/eda_atmos/suite_config.py @@ -30,6 +30,7 @@ class SuiteConfig(QuestionContainer, Enum): qd.runahead_limit("P2"), qd.jedi_build_method("use_existing"), qd.model_components(['geos_atmosphere']), + qd.fetch_from_r2d2(True) ], geos_atmosphere=[ qd.cycle_times([ diff --git a/src/swell/suites/hofx/flow.cylc b/src/swell/suites/hofx/flow.cylc index 82c560fa4..8258ff735 100644 --- a/src/swell/suites/hofx/flow.cylc +++ b/src/swell/suites/hofx/flow.cylc @@ -57,9 +57,6 @@ # Get background, provide a way to get background directly from GEOS experiment GetBackgroundGeosExperiment-{{model_component}} :fail? => GetBackground-{{model_component}} - # Get observations - GetObsNotInR2d2-{{model_component}}: fail? => GetObservations-{{model_component}} - # Perform staging that is cycle dependent StageJediCycle-{{model_component}} @@ -68,14 +65,14 @@ CloneJedi[^] => StageJediCycle-{{model_component}} StageJediCycle-{{model_component}} => RunJediHofxExecutable-{{model_component}} GetBackgroundGeosExperiment-{{model_component}}? | GetBackground-{{model_component}} => RunJediHofxExecutable-{{model_component}} - GetObsNotInR2d2-{{model_component}}? | GetObservations-{{model_component}} => RenderJediObservations-{{model_component}} + fetch_obs-{{model_component}} => RenderJediObservations-{{model_component}} GenerateObservingSystemRecords-{{model_component}} => RenderJediObservations-{{model_component}} RenderJediObservations-{{model_component}} => RunJediHofxExecutable-{{model_component}} # EvaObservations RunJediHofxExecutable-{{model_component}} => EvaObservations-{{model_component}} - {% if not skip_r2d2 %} + {% if store_r2d2 %} # Save observations RunJediHofxExecutable-{{model_component}} => SaveObsDiags-{{model_component}} => CleanCycle-{{model_component}} {% endif %} @@ -135,11 +132,15 @@ [[GetBackgroundGeosExperiment-{{model_component}} ]] script = "swell task GetBackgroundGeosExperiment $config -d $datetime -m {{model_component}}" - [[GetObservations-{{model_component}}]] + {% if fetch_from_r2d2 %} + # Fetch from R2D2 + [[fetch_obs-{{model_component}}]] script = "swell task GetObservations $config -d $datetime -m {{model_component}}" - - [[GetObsNotInR2d2-{{model_component}}]] + {% else %} + # Copy from local filesystem + [[fetch_obs-{{model_component}}]] script = "swell task GetObsNotInR2d2 $config -d $datetime -m {{model_component}}" + {% endif %} [[RenderJediObservations-{{model_component}}]] script = "swell task RenderJediObservations $config -d $datetime -m {{model_component}}" diff --git a/src/swell/suites/hofx/suite_config.py b/src/swell/suites/hofx/suite_config.py index ad48d0054..882d6d371 100644 --- a/src/swell/suites/hofx/suite_config.py +++ b/src/swell/suites/hofx/suite_config.py @@ -31,6 +31,7 @@ class SuiteConfig(QuestionContainer, Enum): qd.jedi_build_method("use_existing"), qd.save_geovals(True), qd.model_components(['geos_atmosphere']), + qd.fetch_from_r2d2(False) ], geos_atmosphere=[ qd.horizontal_resolution("91"), diff --git a/src/swell/suites/hofx_cf/flow.cylc b/src/swell/suites/hofx_cf/flow.cylc index bc95ab594..6b342c467 100644 --- a/src/swell/suites/hofx_cf/flow.cylc +++ b/src/swell/suites/hofx_cf/flow.cylc @@ -71,7 +71,7 @@ # EvaObservations RunJediHofxExecutable-{{model_component}} => EvaObservations-{{model_component}} - {% if not skip_r2d2 %} + {% if store_r2d2 %} # Save observations RunJediHofxExecutable-{{model_component}} => SaveObsDiags-{{model_component}} => CleanCycle-{{model_component}} {% endif %} diff --git a/src/swell/suites/localensembleda/flow.cylc b/src/swell/suites/localensembleda/flow.cylc index e62840326..27881bbcc 100644 --- a/src/swell/suites/localensembleda/flow.cylc +++ b/src/swell/suites/localensembleda/flow.cylc @@ -54,9 +54,7 @@ # Perform staging that is cycle dependent BuildJediByLinking[^]? | BuildJedi[^] => StageJediCycle-{{model_component}} => sync_point - GetObsNotInR2d2-{{model_component}}: fail? => GetObservations-{{model_component}} - - GetObsNotInR2d2-{{model_component}}? | GetObservations-{{model_component}} => RenderJediObservations-{{model_component}} + fetch_obs-{{model_component}} => RenderJediObservations-{{model_component}} RenderJediObservations-{{model_component}} => sync_point @@ -174,11 +172,15 @@ --{{key}} = {{value}} {%- endfor %} - [[GetObservations-{{model_component}}]] + {% if fetch_from_r2d2 %} + # Fetch from R2D2 + [[fetch_obs-{{model_component}}]] script = "swell task GetObservations $config -d $datetime -m {{model_component}}" - - [[GetObsNotInR2d2-{{model_component}}]] + {% else %} + # Copy from local filesystem + [[fetch_obs-{{model_component}}]] script = "swell task GetObsNotInR2d2 $config -d $datetime -m {{model_component}}" + {% endif %} {% if not skip_ensemble_hofx %} {% if ensemble_hofx_strategy == 'serial' %} diff --git a/src/swell/suites/localensembleda/suite_config.py b/src/swell/suites/localensembleda/suite_config.py index e80f67cb0..71a57727d 100644 --- a/src/swell/suites/localensembleda/suite_config.py +++ b/src/swell/suites/localensembleda/suite_config.py @@ -31,6 +31,7 @@ class SuiteConfig(QuestionContainer, Enum): qd.final_cycle_point("2023-10-10T12:00:00Z"), qd.jedi_build_method("use_existing"), qd.model_components(['geos_atmosphere']), + qd.fetch_from_r2d2(True) ], geos_atmosphere=[ qd.horizontal_resolution('91'), @@ -71,6 +72,7 @@ class SuiteConfig(QuestionContainer, Enum): qd.final_cycle_point("2023-10-10T12:00:00Z"), qd.jedi_build_method("use_existing"), qd.model_components(['geos_atmosphere']), + qd.fetch_from_r2d2(True) ], geos_atmosphere=[ qd.horizontal_resolution('91'), diff --git a/src/swell/suites/suite_questions.py b/src/swell/suites/suite_questions.py index 802ce9712..d141fa388 100644 --- a/src/swell/suites/suite_questions.py +++ b/src/swell/suites/suite_questions.py @@ -44,8 +44,8 @@ class SuiteQuestions(QuestionContainer, Enum): qd.r2d2_experiment_id(), qd.r2d2_server(), qd.r2d2_datastore(), - qd.skip_r2d2(), - qd.suite_to_run(), + qd.store_r2d2(), + qd.suite_to_run() ] ) diff --git a/src/swell/swell.py b/src/swell/swell.py index bfdabf285..b095b99e6 100644 --- a/src/swell/swell.py +++ b/src/swell/swell.py @@ -84,7 +84,7 @@ def swell_driver() -> None: or for task-model combinations. """ -skip_r2d2_help = """Skip registering this experiment and storing products in R2D2.""" +store_r2d2_help = """Register and store outputs from this experiment in R2D2?""" additional_parameter_help = ('Additional option to specify parameters to task, ' 'context-dependent on individual task.') @@ -104,7 +104,9 @@ def swell_driver() -> None: @click.option('-o', '--override', 'override', default=None, help=override_help) @click.option('-a', '--advanced', 'advanced', default=False, help=advanced_help) @click.option('-s', '--slurm', 'slurm', default=None, help=slurm_help) -@click.option('-k', '--skip-r2d2', 'skip_r2d2', is_flag=True, default=False, help=skip_r2d2_help) +@click.option('-r', '--store-r2d2', 'store_r2d2', + is_flag=True, default=False, help=store_r2d2_help) +@click.option('-k', '--skip-store-r2d2', 'skip_store_r2d2', is_flag=True, default=False) def create( suite: str, input_method: str, @@ -112,7 +114,8 @@ def create( override: Union[dict, str, None], advanced: bool, slurm: str, - skip_r2d2: bool + store_r2d2: bool, + skip_store_r2d2: bool ) -> None: """ Create a new experiment @@ -127,9 +130,16 @@ def create( # Read override file override_dict = read_override_file(override) + if skip_store_r2d2: + print() + print('Warning: As of PR #876, `-k`, or `skip-store-r2d2` is deprecated.') + print('By default, experiments will NOT be stored in R2D2.') + print('You should remove this flag from any existing scripts.') + print() + # Create the experiment directory create_experiment_directory(suite, input_method, platform, override_dict, - advanced, slurm, skip_r2d2) + advanced, slurm, store_r2d2) # -------------------------------------------------------------------------------------------------- diff --git a/src/swell/test/code_tests/suite_creation_test.py b/src/swell/test/code_tests/suite_creation_test.py index be11389a2..f93b39792 100644 --- a/src/swell/test/code_tests/suite_creation_test.py +++ b/src/swell/test/code_tests/suite_creation_test.py @@ -49,10 +49,10 @@ def suite_creation_test(self, suite: str) -> None: override_dict['experiment_id'] = experiment_id = f'{suite}-creation' override_dict['experiment_root'] = cache_location - override_dict['skip_r2d2'] = True + override_dict['store_r2d2'] = False create_experiment_directory(suite, 'defaults', 'nccs_discover_sles15', - override_dict, advanced=False, slurm=None, skip_r2d2=True) + override_dict, advanced=False, slurm=None, store_r2d2=False) experiment_yaml = os.path.join(cache_location, experiment_id, f'{experiment_id}-suite', 'experiment.yaml') diff --git a/src/swell/test/suite_tests/suite_tests.py b/src/swell/test/suite_tests/suite_tests.py index 327d53b5d..c418f26f1 100644 --- a/src/swell/test/suite_tests/suite_tests.py +++ b/src/swell/test/suite_tests/suite_tests.py @@ -32,7 +32,7 @@ def build_jedi_for_tier2(test_dir: str, experiment_id_root: str, platform: str, create_experiment_directory( "build_jedi", method="defaults", platform=platform, - override=override, advanced=False, slurm=None, skip_r2d2=True + override=override, advanced=False, slurm=None, store_r2d2=True ) suite_path = str(experiment_dir / f"{experiment_id}-suite") @@ -120,7 +120,7 @@ def run_suite(suite: str, platform: str, test_tier: TestSuite): create_experiment_directory( suite_config, "defaults", platform, - override, advanced=False, slurm=None, skip_r2d2=True + override, advanced=False, slurm=None, store_r2d2=False ) # TODO: Check some stuff about the experiment directory diff --git a/src/swell/utilities/mock_jedi_config.py b/src/swell/utilities/mock_jedi_config.py index c13f6463d..a7dd17f03 100644 --- a/src/swell/utilities/mock_jedi_config.py +++ b/src/swell/utilities/mock_jedi_config.py @@ -51,7 +51,8 @@ def mock_jedi_config(suite: str, override_dict['models'][model] = {'check_for_obs': False} create_experiment_directory(suite, method='defaults', platform='nccs_discover_sles15', - override=override_dict, advanced=False, slurm=None, skip_r2d2=True) + override=override_dict, advanced=False, slurm=None, + store_r2d2=False) experiment_yaml = os.path.join(tempdir, experiment_id, f'{experiment_id}-suite', 'experiment.yaml') diff --git a/src/swell/utilities/question_defaults.py b/src/swell/utilities/question_defaults.py index 1ed626a62..a083a5eec 100644 --- a/src/swell/utilities/question_defaults.py +++ b/src/swell/utilities/question_defaults.py @@ -106,6 +106,16 @@ class experiment_root(SuiteQuestion): # -------------------------------------------------------------------------------------------------- + @dataclass + class fetch_from_r2d2(SuiteQuestion): + default_value: str = True + question_name: str = "fetch_from_r2d2" + ask_question: bool = True + prompt: str = ("Fetch observations from r2d2, or local filesystem?") + widget_type: WType = WType.BOOLEAN + + # -------------------------------------------------------------------------------------------------- + @dataclass class final_cycle_point(SuiteQuestion): default_value: str = "2023-10-10T06:00:00Z" @@ -253,9 +263,9 @@ class skip_ensemble_hofx(SuiteQuestion): # -------------------------------------------------------------------------------------------------- @dataclass - class skip_r2d2(SuiteQuestion): + class store_r2d2(SuiteQuestion): default_value: bool = False - question_name: str = "skip_r2d2" + question_name: str = "store_r2d2" prompt: str = "Skip registering and storing results of this experiment in R2D2?" widget_type: WType = WType.BOOLEAN