Use ObsBuilder scripts from SPOC in bufr conversion

[mranst]

Description

#773

I've been experimenting a bit with using the spoc scripts in the bufr-to-ioda conversion to see if I could get it work. I believe it mostly works, but it is a somewhat crude implementation, so it will probably require some fine-tuning from people more familiar with obs handling

This PR adds the CloneSpoc task, which can grab a branch set by spoc_tag in experiment.yaml, or point to an existing spoc_source_directory. Bufr2Ioda then uses the scripts to run the conversion.

A few questions/notes:

• It seems like the scripts are expecting the corresponding yamls to live in the same directory they are being executed from? I may be misunderstanding something, but i added this process to CloneSpoc to accomodate:
  

  swell/src/swell/tasks/clone_spoc.py

  Lines 66 to 71 in eca7d5c

  	# Copy the mapping yaml's to the script path
  	# ------------------------------------------
  	config_glob = os.path.join(spoc_exp_path, 'dump', 'config', 'atmosphere', '*yaml')
  	script_path = os.path.join(spoc_exp_path, 'dump', 'scripts', 'atmosphere')
  	for config_file in glob.glob(config_glob):
  	shutil.copy(config_file, script_path)

• The output ioda files seem to be combined with all satId's in one file, not sure how significant this is

• There are some obs types that have separate yamls under the bufr2netcdf.x schema, that don't have individual python scripts (I think this is noted in Incorporating SPOC into swell convert_bufr suite #773)

[cohen-seth]

A few questions/notes:

1. "It seems like the scripts are expecting the corresponding yamls to live in the same directory they are being executed from? I may be misunderstanding something, but i added this process to CloneSpoc to accomodate:"

• This is correct the scripts are expecting the corresponding yamls to live in the same directory they are being executed from. This was done to preserve the yaml used for the conversion within the run directory for referring to later & for validation; that way, when we add the capability to customize the yamls and/or when spoc yamls are updated there will be a way of knowing how the files were converted. (See directory tree below)
• CloneSpoc will be very helpful. Previously, I had to manually add the spoc yamls to the configuration/jedi/bufr2ioda/bufr2netcdf_x/ directory - should be able to remove these going forward.

run/20231010T000000Z/geos_atmosphere/
│
├── bufr/
├── ioda/
├── spoc_radiance_1bamua.yaml
├── spoc_radiance_1bmhs.yaml
├── spoc_radiance_atms.yaml
├── spoc_radiance_avcsam.yaml
├── spoc_radiance_avcspm.yaml
├── spoc_radiance_gpsro.yaml
└── spoc_radiance_mtiasi.yaml

2. "The output ioda files seem to be combined with all satId's in one file, not sure how significant this is"

• I believe we want them in separate files. Double check with @rtodling
• this functionality of bufr2netcd.x is still being developed. There are 2 methods, one method that only partially works and one method that requires more steps. So I had to use the 2nd method, the --no-gather method. We will want to switch to the other method once it's functional.

[cohen-seth]

Are these obs builder python scripts placeholders?

[mranst]

Sort of? I tried to point to what was in the spoc scripts folder, mostly based on the name. I assume not all of them will line up correctly

[cohen-seth]

Notes from meeting with @jemccurr:

1. obs builder can't handle relative paths.
   -> will probably need help & guidance on how to handle this @mranst
2. yaml files for conv types - 'reference time' within yaml needs to be set to the cylce time.
   -> had to make edits within the yamls in the first version of bufr_to_ioda. should be straight forward.
3. need to be run in the same directory as the the yaml.
   -> this is how the current bufr_to_ioda.py works. should be ok as is.
4. where is spoc clone stored/linked?

[mranst]

@cohen-seth @jemccurr Changed this to copy scripts and yamls to a location in the cycle directory, should be better for making modifications that way.

For development purposes I'd suggest forking this branch, then opening PR's with this as the base when you want to submit changes