Add a few model dependent questions to parse flow.cylc in jinja2

[metdyn]

Description

This PR addresses issue #831, which is required for EDA ControlPert framework.
It solves the issue in flow.cylc in case of complex rendering.

   {% for i in range( 1, models[model_component]['ensemble_num_chunks'] + 1 ) %}
     [[RunJediEdaControlPertExecutable_chunk{{i}}-{{model_component}}]]
   {% endfor %}

The code here replaces self.experiment_dict by render_dict, where the latter contains the former dictionary plus two key-value pairs involving ensemble_num_chunks amd ensemble_num_members needed by EDA.

Dependencies

Impact

[mranst]

I'm not against merging this if it solves the issue, but I'd like to maybe propose an alternative solution. I believe the issue you're encountering is that trying to use a task question in the runtime section of flow.cylc leads to swell not recognizing the task exists. This is because swell sweeps the partially filled flow.cylc for lines containing the phrase "swell task <task_name>", and these are not present if dependent on task questions. You can get around this by making sure that swell task <task_name> is present in the file in all cases.

(A) Either as a comment:

   # [[RunJediEdaControlPertExecutable-{{model_component}}]]
   #     script = "swell task RunJediEdiControlPertExecutable -m {{model_component}} -d $datetime experiment.yaml
   {% for i in range( 1, models[model_component]['ensemble_num_chunks'] + 1 ) %}
     [[RunJediEdaControlPertExecutable_chunk{{i}}-{{model_component}}]]
   {% endfor %}

(B) Or better yet, as a parent task. This has additional benefits such as being able to track the tasks execution as a group in cylc:

    [[RunJediEdaControlPertExecutable-{{model_component}}]]
        script = "swell task RunJediEdiControlPertExecutable -m {{model_component}} -d $datetime experiment.yaml
     {% if models[model_component]['ensemble_num_chunks'] is defined
     {% for i in range( 1, models[model_component]['ensemble_num_chunks'] + 1 ) %}
     [[RunJediEdaControlPertExecutable_chunk{{i}}-{{model_component}}]]
        script = "swell task RunJediEdiControlPertExecutable -m {{model_component}} -d $datetime -a {{i}} experiment.yaml
        inherit = RunJediEdaControlPertExecutable-{{model_component}}
     {% endfor %}
     {% endif %}

Admittedly, these strategies will not work if your control logic depends on tasks being in an adversarial relationship with each other. For example if you needed to have a case like:

{% if models[model_component]['x'] == 'A' %}
[[Task_A]]
{% else %}
[[Task_B]]
{% endif %}

[metdyn]

@mranst,
We are on the same page for this issue! The culprit is simply the few critical model-dependent questions that I need for EDA are not being parsed to or understood by flow.cylc, so that RunJediEdiControlPertExecutable is not registered as a task, hence its config/question variables are missing in experiment.yaml.

Let's not use tricks to bypass the scientific need. A simple fix like this will solve the problem.
Furthermore, I think the list of questions like this to be added to this code will go on and on, when more complex workflows are encountered. The initial design simply did not foresee future expansions.

BTW, thanks for the workaround you provided. I am sure case (A) will not work, case (B) will bypass the issue but leave scientific confusions to others, because the task RunJediEdiControlPertExecutable will fail without ichunk being passed as a parameter.