Explicit cyclic foreign vars

[jvail]

Sorry to open up a new 'battleground' - the last thing that haunts me for now:

What do you, @benbovy, think about the following idea, use case:

Let's say we have two processes A and B. A has a var x that is a foreign variable in B and B has a variable y that is a foreign variable in A. Wont work because you can not derive the DAG in that case.

What if I annotate the foreign variable B.y in A with something like is_cyclic=True . That would mean:

• I know this introduces a cycle but I don't care
• Just ignore me, A.y, when building the process graph, the ordering
• But please still draw the B -> y -> A relation in the visualization (maybe with a step - 1 label, dotted line...)
• And when executing A just pass in for y whatever is there (would be a NaN in step = 0 if y is not inout in B or the value set in B.initialize

It just happens that we have a lot of these "cycles" but typically it is just the value from step time - 1. So only quasi-cyclic.

[benbovy]

Could you provide a concrete example where such quasi-cyclic situation happens?

I wonder if you could refactor your problem into 3 processes A, B, C, or if user-defined process ordering in a Model (thus relaxing DAG constraints) would help here.

I think that foreign vars with something like is_cyclic=True would open a big can of worms. It is quite confusing and would complicate further xarray-simlab's internal logic for process sorting (it is already complex).

[jvail]

I think that foreign vars with something like is_cyclic=True would open a big can of worms. It is quite confusing and would complicate further xarray-simlab's internal logic for process sorting (it is already complex).

I don't know the internals but I thought - maybe naively - you could probably just ignore the variables with is_cyclic=True in the process ordering.

Could you provide a concrete example where such quasi-cyclic situation happens?

Fair enough

A - A process for a tree (here leaf) architecture - in practice all sorts of properties
B - Photosynthesis
C - Carbon allocation

A.leaf_area (out) --> B
B.assimilates (out) --> C
C.leaf_dry_matter (out) - - > A

In order to compute A.leaf_area A needs C.leaf_dry_matter from t-1 (or the initial value at t = 0). If there is a way to model that in xarray-simlab without cycle then I apologize for not having read the documentation more carefully or missing the obvious.

Of course we could just merge it all together in one process but that is where we come from and we would like to split it up in modules/processes.

[benbovy]

Would it be an option to declare C.leaf_dry_matter with intent='inout' and with some default value (i.e., some placeholder for the value at t = -1)? This way you could avoid the C -> A dependency thus remove cyclic dependencies.

[jvail]

Ok, a fat apology

I forgot to mention that circular imports where in my way when I first tried to get it working ;) This time I tried a combination of xs.group in A (avoid explicit linking and circular import of C) and an 'inout' in C with the same group.

So, moving everything into groups plus 'inout' is a solution but I loose a bit on the side of your nice model visualization/inspection features since I obscure where the variable comes from. But that's worth it!

Thank you for your outstanding support!

[benbovy]

Ok, a fat apology

No worries at all!

Thank you for your outstanding support!

You're welcome!

[jvail]

Sorry, need to bring this up again. I am still thinking about another way to get this "cyclic" dependencies working (this is indeed one of our three pain points apart from sparse and growing index variables):

I sketched a little prove of concept. Here is a notebook: https://github.com/jvail/xarray-simlab/blob/seemingly-cyclic/notebooks/seemingly_cyclic.ipynb

I am not entirely happy with the current work-around:

Declare an 'inout' rather than 'out' variable if you want to let previously run processes to use it) because it requires to deliver an input for each of them although initializing them in the initialize func would be sufficient; i.e. I can not just use intent 'out'.

Also, as far as I can see from the code where you compute the dependencies that you are filtering for 'out' variables to determine process deps and sorting. So simply allowing a "pseudo cyclic" variable with intent 'out' wont work because process ordering will be ambiguous. Therefore, I think, another property is required (for the sake of clarity I called it "seemingly_cyclic") to explicitly exclude this dependency from being evaluated.

It seems clear from my tests that if there is such a dependency the producer must declare a global name (or group - that's a bit confusing I have to say - a group is always global I guess) and the consumer use a global ref (group) in any case to avoid cycles at module import level. Therefore I added the property "seemingly_cyclic" only to "global_ref".

I also added a dashed arrow to the dot graph to visualize this dependency. I can not see why this might open a "pandora's box" and introduce chaos - but I may - most certainly - lack knowledge about the mechanics of your lib.

P.S.: I will try to make a suggestion for #163 as well but that one is really tricky - bear with me.

[jvail]

I kept playing a bit with this idea and tested if we could use it with xs.foreign as well. If we split it into base classes that implement the different aspects of a process I can get around the cyclic import issue and we could arrive at a pretty "plugable", modular model design (in our use case): i.e. just replace a process with some different math as long as it provides the same output.

https://github.com/jvail/xarray-simlab/blob/seemingly-cyclic/notebooks/seemingly_cyclic_foreign.ipynb

The advantage of xs.foreign is that the dependency is explicit - "global_ref" hides it. Makes the model more readable, I think.

[benbovy]

Thanks for sharing your notebooks, it gives a clear overview of the problem.

I admit that the 'inout' variable with a default value workaround is not ideal in the case where you just would use intialize. That said, conceptually it makes some sense IMO: in your example, I see b_v as a variable that pre-exists with some state before the simulation. If you need to set more advanced values as default, you could use a factory (although I'm not sure that the takes_self=True option would work out of the box in xarray-simlab).

TBH, I'm reluctant to implement "pseudo cyclic" variables in xarray-simlab for the main reason that it breaks a fundamental concept of this framework, i.e., that the process workflow may be represented as DAG. Allowing one exception like this is opening the door to more exceptions, which may end up with something conceptually very hard to understand. It also adds a parameter to xs.variable, xs.global_ref, etc., which have already (too) many options. In your notebooks, you need to rely on either global_ref or foreign with some base process classes, which both IMO are not simpler workarounds than 'inout' variables with a default value.

[jvail]

TBH, I'm reluctant to implement "pseudo cyclic" variables in xarray-simlab

Sure, understood and accepted - wont bring it up again - however :) , out of curiosity...

for the main reason that it breaks a fundamental concept of this framework

I don't understand why it would break it because you have it already (global/group + 'inout'). I think such a change would just bring this option to the surface and improve it because an input is no longer required. Is there a fundamental difference between input_vars values and values set in initialize? Isn't it both just the state at step -1?

It also adds a parameter to xs.variable, xs.global_ref, etc., which have already (too) many options

No, only to xs.global_ref and xs.foreign. Well, I could imagine to have an xs.cyclic(intent='in') instead ... But I admit, there are already quite a lot of different variables and options which could become even more confusing.

[benbovy]

I don't understand why it would break it because you have it already (global/group + 'inout'). I think such a change would just bring this option to the surface and improve it because an input is no longer required. Is there a fundamental difference between input_vars values and values set in initialize? Isn't it both just the state at step -1?

Ah I see, you're right. Probably it's not clear enough that the value of a 'inout' variable declared in a process should be updated in a different simulation stage in that case (that was clear only in my mind so far :-) ).

An 'inout' variable declared in a process A and that is accessed by a 'in' group or global_ref declared in process B would not be affected by process ordering (i.e., either A -> B or B -> A is correct) as long as the value of this variable is not updated in A and read in B during the same simulation stage.

This implicit condition is a bit unfortunate, actually, and I don't see how we could properly inspect the process classes to check this condition when building a new model :-/.

Your suggestion of a "seemingly_cyclic" reference at least makes the intention a bit clearer, but it doesn't solve this implicit condition which still applies here. I'm not very comfortable with adding another way to allow this.

[jvail]

I'm not very comfortable with adding another way to allow this.

D'accord. Fair enough.

Ah I see, you're right. Probably it's not clear enough that the value of a 'inout' variable declared in a process should be updated in a different simulation stage in that case (that was clear only in my mind so far :-) ).

An 'inout' variable declared in a process A and that is accessed by a 'in' group or global_ref declared in process B would not be affected by process ordering (i.e., either A -> B or B -> A is correct) as long as the value of this variable is not updated in A and read in B during the same simulation stage.

Wow - that's subtle. A bit hard to follow, I find. I think you could put a bit more faith in the modeler who is supposed to know what he is doing. Nevertheless - I regard this as settled and remain quite now and hope you keep up your excellent work! :)

[benbovy]

Nevertheless - I regard this as settled and remain quite now and hope you keep up your excellent work! :)

Thanks! Please continue to give helpful feedback, it's very much appreciated! In this case it made me realize that we can actually break the DAG in the current version!