Add command to run ctests on select bundles

[mranst]

Description

Since the tier2 tests don't run the swell build_jedi suite anymore, I thought it would be best to absorb the ctest task into jedi_bundle directly (previously part of GEOS-ESM/swell#778. This can be configured using the option ctest_bundles in build.yaml, and the verbose output of the ctests will be placed in the build folder under ctests/<bundle>.yaml.

I'll remove that task from GEOS-ESM/swell#778, but keep the comparison suite. This can be added to tier2 tests as its own task that can run in parallel with the swell suites after jedi_bundle is built

[Dooruk]

I think running tier2 JEDI tests should be our main focus because JCSDA runs tier1 tests before any PR merges AFAIK. Other than that, a couple of things:

-Most ctests have to run on a compute note, is that handled with this?
-Any reason to use verbose -V for all? That will lead to a really long output file. Might be better to reserve that for failed ctests.
-Between PRs and all sometimes certain ctests fail and overlooked in soca or fv3-jedi. So it may take a few iterations to figure out which failures are telling which are not necessary. If we are considering tier1.
-The JEDI tier2 ctests require a special CFLAG during build to activate: GEOS-ESM/swell#699 (comment). I personally never used them but they could be relevant for @rtodling. @jeromebarre or @mer-a-o would know better about the tier2 JEDI tests and what type of tests they entail if we want to talk during the Swell Dev meeting.

[mranst]

I think running tier2 JEDI tests should be our main focus because JCSDA runs tier1 tests before any PR merges AFAIK. Other than that, a couple of things:

-Most ctests have to run on a compute note, is that handled with this? -Any reason to use verbose -V for all? That will lead to a really long output file. Might be better to reserve that for failed ctests. -Between PRs and all sometimes certain ctests fail and overlooked in soca or fv3-jedi. So it may take a few iterations to figure out which failures are telling which are not necessary. If we are considering tier1. -The JEDI tier2 ctests require a special CFLAG during build to activate: GEOS-ESM/swell#699 (comment). I personally never used them but they could be relevant for @rtodling. @jeromebarre or @mer-a-o would know better about the tier2 JEDI tests and what type of tests they entail if we want to talk during the Swell Dev meeting.

We can definitely tune it to run certain tests, and turn on the tier2 tests automatically. I guess my perspective for verbose is why not keep all the information on the first go rather than having to run certain tests again (unless I'm missing there's an easy option for running it only on failed tests). I'm still envisioning this working with the swell compare_jedi suite in the other PR, so it would show a pretty simple list of which have failed relative to the control build. The user could then consult the output file for the full verbose output. My basis for failure would be whether the same amount of tests pass for two different builds.

The user would be expected to salloc or sbatch this command, no different than the make step. Maybe we could add detection for compute nodes

[mer-a-o]

The tiering mechanism in jcsda repos has changed: JCSDA-internal/jedi-ci#57

Now, LABEL is used to mark tier1 and tier2 tests and both labels get built by default. ctest will run both labels and ctest -L tier2 can be used to execute only tier2 tests.

Also, ctest --output-on-failure will print logs for failed tests only. I'm not sure it's currently used or not.

[mranst]

Thanks. Added --output-on-failure and the label for fv3-jedi tier2 tests. soca doesn't seem to work with this, so I have it set to run all tests still, unless there's a specific range you think we should check @Dooruk