Hydra

[hcampe]

for now this is only a sketch. I have replaced the argparsers and old yaml files with a hydra setup for most scripts. Internally, however, the configs still get treated like before. Feel free to take a look at the current structure. :)

The remaining todos are

• merge Improved logging #152 into this to make everything consistent,
• properly refactor the code to use the configs more naturally,
• make sure all previous (default) configs are mirrored in the new setup,
• write a small guide for how to use this.

let me know if you have any thoughts

[stephengreen]

This is a lot of files! What do others think? Is there a way to have a more efficient storage of settings? @max-dax @nihargupte-ph

[hcampe]

the raw yaml files inside the configs/ directory are necessary, each of them corresponds to one main function somewhere in the code.

The folders inside configs/ are config groups that allow compartmentalization of the configs. I have organized them a little but perhaps there is a better way.

The config files in configs/experiment/ are special: they correspond to single experiments, overriding other config files. E.g., to train the toy model from the tutorial in the docs, one would run dingo_train experiment=train_toy. Probably some of these are superfluous.

[max-dax]

I agree with Stephen, feels like too many files to me.

Some main functions are only intended as quick tests and examples, not as proper production scripts. And for these, maybe it's ok to keep the default settings in the .py files.

I generally like the modularised setup, although some files seem too small, and often they would be fully overwritten in practice (e.g., configs/optimizer/adam.yaml or configs/scheduler/cosine.yaml).

[stephengreen]

I wonder if hydra is really the right approach for the whole thing. Hydra is great for composing settings for a large number of runs, but it doesn't provide validation of the settings. Would pydantic be the tool we are looking for here?

[hcampe]

I have worked a bit more on this. This is still intermediate but as an example I have refactored set_train_transforms() in train_builders.py to properly use hydra, simplifying the application of the transforms by moving them into the configs. It is not perfect but much simpler than before. Take a look. :)

My general impression is that it is definitely possible to move much of the complexity of the internal implicit handling of the configs to the external config files to make everything more explicit. Of course, this comes at the cost of introducing new complexity into the configs so I am not fully sure whether it would be less complex overall. To my mind, this separation would still be much cleaner but this is a question of personal taste.

At the same time, the proper refactor to get there would be quite large, I think. For me, Codex is struggling with this task (both in terms of the token consumption and the "intelligence"). Eg for the above example, I needed to prompt it repeatedly to remove individual if statements etc until the current state was reached. Perhaps Fable would work better? Still, probably much human intervention of an experienced Dingo developer would be needed to decide which parts of the code should become how modular etc.

Probably it is best to discuss again if the hydra refactor is really desired. I am really happy to work more on this but for now, I will switch back to working on the prior project.

[hcampe]

I have looked into pydantic. As far as I can tell, the (dis)advantages of both methods are

pydantic

• type checking of the config based on type hints just as if it was C code,
• even more powerful validation of the config (e.g. can check if num_workers if a power of two)
• would require some additional code
• no instantiate, configs resolving
• defaults seem possible but would live in the code

hydra

• no type checking or validation of the config at all
• instantiate -> would reduce the code but increase the no of lines in the config
• resolves configs taking into account defaults, command line overrides
• can move much of the logic into the configs

Both methods can serialize the configs equally well. Since the are slightly orthogonal, people have also combined both in the past. I personally would use hydra and perhaps add pydantic to do the validation.

[hcampe]

I've simplified the configs by removing many of the configs for smaller scripts and default lists. I have kept defaults lists for domain, waveform generator and the model (seemed the most sensible to me).

Users could add their modifications to the experiment/ folder. Eg, if one wanted to train with the default settings it would be just dingo_train, but to train the toy model it would by dingo_train +experiment=train_toy.