@ananthsub yes potentially in our parsing or alternatively also in the plugin which already gets the list of devices.

@justusschock sorry I'm not very familiar with how MPI works with GPUs intorch.distributed. However, if it relies on NCCL under the hood I'd guess it still wouldn't work.

@yifuwang I tried a bit more, and it actually worked withgloo. After addingPL_TORCH_DISTRIBUTED_BACKEND=gloo, I was able to run Lightning Training successfully with 2 replicas on a single GPU.

@tholop did you experience the same kind of speedup you'd expect if you were training on multiple separate physical devices?

PS: I haven't experimented yet, but I suspect you might be able to apply the fractional GPU capability in ray to achieve something like this.

@dmarx I did experience a speedup, but not as good as having separate physical devices. I didn't benchmark thoroughly though.

I totally agree regarding Ray's fractional GPUs! I mentioned it in the original issue as a possible workaround, but it might require a bit more work than just passing a string to Lightning.