Add IODA and Increment comparisons by mranst · Pull Request #798 · GEOS-ESM/swell

mranst · 2026-06-01T15:25:08Z

Description

Add comparisons to hofx observations. Used to evaluate differences for observations. Allows comparing hofx suites, and comparison plots for all suites will only be run if a difference is detected for each individual obs. This works by reading the output obs file and averaging values along a set length to check if they are equivalent between the two experiments. I implemented it this way for memory concerns, we could switch to evaluating for point-by-point differences if that makes more sense.

…gov:/home/manstett/swell-main into feature/mranst/hofx_comparisons

mranst · 2026-06-15T21:21:10Z

I had to fight cylc to get this to work as intended, I was trying to get it to operate on each observation individually, but it would choose to wait for requisite tasks that were never triggered. I switched it back operating as one task.

Dooruk · 2026-06-16T15:01:00Z

Sorry, quick question. Is the main idea of this PR would be using hofx0 values if it's a variational and/or FGAT suite and use hofx if it's an hofx type suite (hofx or hofx_cf)?

mranst · 2026-06-16T16:06:23Z

Sorry, quick question. Is the main idea of this PR would be using hofx0 values if it's a variational and/or FGAT suite and use hofx if it's an hofx type suite (hofx or hofx_cf)?

Correct, is that sensible?

Dooruk · 2026-06-16T19:15:37Z

Correct, is that sensible?

Yeah. Maybe not necessarily for this PR but I was just wondering instead of hofx or hofx0 only if they could be chosen from a list from the following IODA outputs :

group: EffectiveError0 {
  variables:
        float absoluteDynamicTopography(Location) ;
                absoluteDynamicTopography:_FillValue = -3.368795e+38f ;
  } // group EffectiveError0

group: EffectiveError1 {
  variables:
        float absoluteDynamicTopography(Location) ;
                absoluteDynamicTopography:_FillValue = -3.368795e+38f ;
  } // group EffectiveError1

group: EffectiveQC0 {
  variables:
        int absoluteDynamicTopography(Location) ;
                absoluteDynamicTopography:_FillValue = -2147483643 ;
  } // group EffectiveQC0

group: EffectiveQC1 {
  variables:
        int absoluteDynamicTopography(Location) ;
                absoluteDynamicTopography:_FillValue = -2147483643 ;
  } // group EffectiveQC1

group: MetaData {
  variables:
        int64 dateTime(Location) ;
                dateTime:_FillValue = -9223372036854775801LL ;
                string dateTime:units = "seconds since 1970-01-01T00:00:00Z" ;
        float latitude(Location) ;
                latitude:_FillValue = -3.368795e+38f ;
                string latitude:units = "degrees_north" ;
        float longitude(Location) ;
                longitude:_FillValue = -3.368795e+38f ;
                string longitude:units = "degrees_east" ;
  } // group MetaData

group: ObsBias0 {
  variables:
        float absoluteDynamicTopography(Location) ;
                absoluteDynamicTopography:_FillValue = -3.368795e+38f ;
  } // group ObsBias0

group: ObsBias1 {
  variables:
        float absoluteDynamicTopography(Location) ;
                absoluteDynamicTopography:_FillValue = -3.368795e+38f ;
  } // group ObsBias1

group: ObsError {
  variables:
        float absoluteDynamicTopography(Location) ;
                absoluteDynamicTopography:_FillValue = -3.368795e+38f ;
                string absoluteDynamicTopography:units = "m" ;
  } // group ObsError

group: ObsValue {
  variables:
        float absoluteDynamicTopography(Location) ;
                absoluteDynamicTopography:_FillValue = -3.368795e+38f ;
                string absoluteDynamicTopography:units = "m" ;
  } // group ObsValue

group: PreQC {
  variables:
        int absoluteDynamicTopography(Location) ;
                absoluteDynamicTopography:_FillValue = -2147483643 ;
  } // group PreQC

group: hofx0 {
  variables:
        float absoluteDynamicTopography(Location) ;
                absoluteDynamicTopography:_FillValue = -3.368795e+38f ;
  } // group hofx0

group: hofx1 {
  variables:
        float absoluteDynamicTopography(Location) ;
                absoluteDynamicTopography:_FillValue = -3.368795e+38f ;
  } // group hofx1

group: oman {
  variables:
        float absoluteDynamicTopography(Location) ;
                absoluteDynamicTopography:_FillValue = -3.368795e+38f ;
  } // group oman

group: ombg {
  variables:
        float absoluteDynamicTopography(Location) ;
                absoluteDynamicTopography:_FillValue = -3.368795e+38f ;
  } // group ombg
}

mranst · 2026-06-16T21:58:00Z

Yeah. Maybe not necessarily for this PR but I was just wondering instead of hofx or hofx0 only if they could be chosen from a list from the following IODA outputs :

Done. Now goes through a list for those variables. I have it set to hofx for hofx suites, and hofx0 and hofx1 for variational, but it should be able to support all of them

mranst · 2026-07-01T16:34:49Z

Most recent commits add comparisons for increment files, as well

mer-a-o · 2026-07-01T22:43:40Z

@mranst can you please add a documentation on how to use this feature?
Let's say I want to compare obs values and hofx for tempo version V3 and V4. I can run the hofx_cf experiment twice to get the ObsDiag (or feedback) files for each version. But then the feedback files get deleted during the cleaning task. Also the second experiment writes over the first one. So I'm trying to understand the best practice to use the compare suite in my case. Or am I missing something?

mranst · 2026-07-02T13:41:31Z

@mranst can you please add a documentation on how to use this feature? Let's say I want to compare obs values and hofx for tempo version V3 and V4. I can run the hofx_cf experiment twice to get the ObsDiag (or feedback) files for each version. But then the feedback files get deleted during the cleaning task. Also the second experiment writes over the first one. So I'm trying to understand the best practice to use the compare suite in my case. Or am I missing something?

Thanks for the reminder on documentation. The clean_patterns field defines what will be deleted at the end of the suite. I have set this to not delete nc files for the other suites by default, and have added that to hofx_cf. As for not overwriting experiments, you must set different experiment id's using override.yaml before creating

Dooruk · 2026-07-02T13:51:45Z

Each time swell create is run a unique R2D2 ID is created, hence your V3 and V4 versions would get different ID's automatically and stored that way:

swell/src/swell/tasks/save_obs_diags.py

Line 89 in 9d6acfc

experiment=self.config.r2d2_experiment_id(),

So another comparison approach, or suite I should say, (not suggesting this PR) might be just comparing two stored R2D2 experiment feedbacks or ~~increments~~. The trick is finding the overlapping time period, but that might be just up to user to figure out.

Clarification; we will only store analyses and backgrounds so one would need to calculate increments.

Dooruk · 2026-07-02T13:53:44Z

+        script = """
+            cylc set $CYLC_WORKFLOW_ID//$datetime/EvaComparisonIncrement-{{model_component}} --out skip
+            cylc set $CYLC_WORKFLOW_ID//$datetime/EvaComparisonJediLog-{{model_component}} --out skip
+            cylc set $CYLC_WORKFLOW_ID//$datetime/EvaComparisonObservations-{{model_component}} --out skip
+            cylc set $CYLC_WORKFLOW_ID//$datetime/PublishComparisons --out skip
+            cylc set $CYLC_WORKFLOW_ID//$datetime/comparison_fail --out skip
+        """


Just one thing, what is the purpose of this here? I think I saw this on another PR as well.

These tasks are triggered by a fail state in the comparison tasks

JediLogComparison:fail? => EvaComparisonJediLog?

For some reason, cylc decides to stall the workflow (causing a failure state) if JediLogComparison succeeds, as it thinks it is waiting for EvaComparisonJediLog to be triggered. This is sort of a hacky workaround to skip those tasks if it succeeds. If anyone has any suggestions on how to do this better, I'm all ears. It seems like something that shouldn't be an issue, or have a more simple solution, but this is the only way I've been able to get it to work

mer-a-o · 2026-07-02T14:51:30Z

Each time swell create is run a unique R2D2 ID is created, hence your V3 and V4 versions would get different ID's automatically and stored that way:

swell/src/swell/tasks/save_obs_diags.py

Line 89 in 9d6acfc

experiment=self.config.r2d2_experiment_id(),

So another comparison approach, or suite I should say, (not suggesting this PR) might be just comparing two stored R2D2 experiment feedbacks or ~~increments~~. The trick is finding the overlapping time period, but that might be just up to user to figure out.

Clarification; we will only store analyses and backgrounds so one would need to calculate increments.

We need to keep track of all the cases where using R2D2 can help with simplifying the code. This is a great example.

mer-a-o · 2026-07-02T19:24:12Z

I was able to run compare_hofx_cf but not successfully and noticed a few things.
First, CompareIncrement ran (and failed) which I thought it shouldn't for hofx suite.
Second, Cylc shows JediLogComparison passed and also logs confirm this but I don't see anything in run directory. As a consequence EvaComparisonJediLog fails.

Also, what does EvaComparisonIncrement plot? Is it possible to have something like EvaComparisonHofx or EvaComparisonObs?

jeromebarre · 2026-07-30T15:29:03Z

                    observation_dict_2 = value.copy()

            if observation_dict_1 is None or observation_dict_2 is None:
-                continue


Would changing continue for return get out of the observation loop and skip all the other obs if such condition is met?
For example if this condition is met for the first obs this will get of the loop and misses all the other obs.

jeromebarre · 2026-07-30T15:29:10Z

            use_obs = use_obs_1 and use_obs_2

            if not use_obs:
-                continue


Would changing continue for return get out of the observation loop and skip all the other obs if such condition is met?
For example if this condition is met for the first obs this will get of the loop and misses all the other obs.

jeromebarre · 2026-07-30T15:29:59Z

+            'ObsBias1/{variable}',
+            'ObsValue/{variable}',
+            'PreQC/{variable}',
+            'hofx/{variable}'


There is a missing comma here

jeromebarre · 2026-07-30T15:32:21Z

+            compare,
+            qd.comparison_log_type('hofx'),
+            qd.ioda_fields_for_comparison(['hofx/{variable}']),
+            qd.model_components(['geos_cf']),


is

qd.observations([ "tempo_no2_tropo", "tropomi_s5p_no2_tropo", ])

missing here?

jeromebarre · 2026-07-30T15:35:31Z

-            JediLogComparison-{{model_component}}[^]:fail? => EvaComparisonIncrement-{{model_component}} => PublishComparisons => comparison_fail
-            JediLogComparison-{{model_component}}[^]:fail? => EvaComparisonJediLog-{{model_component}} => PublishComparisons => comparison_fail
-	    JediLogComparison-{{model_component}}[^]:fail? => EvaComparisonObservations-{{model_component}} => PublishComparisons => comparison_fail
+            CompareIodaObservations-{{model_component}}? & CompareIncrement-{{model_component}}? & JediLogComparison-{{model_component}}[^]? => Pass


There is no increments produced for hofx suites. And CompareIncrement will unconditionally run for hofx suites I think.

mranst added 13 commits May 21, 2026 16:03

Add task to compare hofx observations

12f6714

Add to questions

3a615c8

Operational

1ba3b73

Use observation parameter from command line

03e7d5e

comparison obs not in parallel

f680bfe

concurrent limit and evacompobs

650e748

fixes

56067e4

Merge branch 'feature/mranst/hofx_comparisons' of discover.nccs.nasa.…

abc0d93

…gov:/home/manstett/swell-main into feature/mranst/hofx_comparisons

Check for file

d52adb0

Switch back to single task

d084323

Working

144cf9b

Merge branch 'develop' into feature/mranst/hofx_comparisons

e98408d

Remove additional parameter

3c89cee

mranst marked this pull request as ready for review June 15, 2026 20:18

mranst added 2 commits June 15, 2026 17:04

Fix

60b47da

code test fix

2f93dc0

mranst requested review from Dooruk, jeromebarre and mer-a-o June 15, 2026 21:23

mranst added 3 commits June 16, 2026 17:28

Extend to all ioda fields

45d34f8

Extend to all ioda fields

66e0fc5

add task

7097c05

mranst changed the title ~~Add hofx comparisons~~ Add IODA comparisons Jun 17, 2026

mranst added 2 commits June 30, 2026 14:23

Add increment comparison task

fdccc56

Functional

026024d

mranst added 2 commits July 1, 2026 10:21

Add logging and pass through tasks

b4a850e

Merge branch 'develop' into feature/mranst/hofx_comparisons

3ed6b40

mranst changed the title ~~Add IODA comparisons~~ Add IODA and Increment comparisons Jul 1, 2026

mranst added 3 commits July 2, 2026 09:31

Add docs

24cc657

Add increment for cf

d2ae861

Adjust clean_patterns for hofx_cf

6d55867

Dooruk reviewed Jul 2, 2026

View reviewed changes

jeromebarre reviewed Jul 30, 2026

View reviewed changes

Uh oh!

Conversation

mranst commented Jun 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Uh oh!

mranst commented Jun 15, 2026

Uh oh!

Dooruk commented Jun 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mranst commented Jun 16, 2026

Uh oh!

Dooruk commented Jun 16, 2026

Uh oh!

mranst commented Jun 16, 2026

Uh oh!

mranst commented Jul 1, 2026

Uh oh!

mer-a-o commented Jul 1, 2026

Uh oh!

mranst commented Jul 2, 2026

Uh oh!

Dooruk commented Jul 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Dooruk Jul 2, 2026

Choose a reason for hiding this comment

Uh oh!

mranst Jul 2, 2026

Choose a reason for hiding this comment

Uh oh!

mer-a-o commented Jul 2, 2026

Uh oh!

mer-a-o commented Jul 2, 2026

Uh oh!

jeromebarre Jul 30, 2026

Choose a reason for hiding this comment

Uh oh!

jeromebarre Jul 30, 2026

Choose a reason for hiding this comment

Uh oh!

jeromebarre Jul 30, 2026

Choose a reason for hiding this comment

Uh oh!

jeromebarre Jul 30, 2026

Choose a reason for hiding this comment

Uh oh!

jeromebarre Jul 30, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

mranst commented Jun 1, 2026 •

edited

Loading

Dooruk commented Jun 16, 2026 •

edited

Loading

Dooruk commented Jul 2, 2026 •

edited

Loading