Train in DDP Multi GPU and launch Validation on Single GPU during training

[thad75]

Hi,

I was wondering if there was a way to launch a validation step on a single Gpu, while training_step is on multiple gpu ?

Thank you

[semaphore-egg]

Usually, we need to keep all GPUs working. Set replace_sampler_ddp=False for our trainer makes each validation step duplicated across all training GPUs. This is equivalent to using a single GPU regarding the validation results.

[bamfpga]

Ultralytics achieves this in their trainers script by creating the validation dataloader on rank zero only, not using distributed sampler for validation and performing the validation loop on rank 0 only.
Why isn't it possible for pytorch lightning to do so? It would save the user from the effort of reducing metrics every time at epoch end.