Weird. A library that wraps SYCL within MPI, yet requires all processes to hold a copy of all the memory ? One of the main reasons to use MPI is to solve problems that do not fit within the memory available in a single cluster node. There is a presentation [0] from 2020-02-20 that's not very impressive. Particularly, they compare against MPI+OpenCL, but do not show a comparison against MPI-CUDA. Doing a distributed M…
> Weird. A library that wraps SYCL within MPI, yet requires all processes to hold a copy of all the memory ?
> One of the main reasons to use MPI is to solve problems that do not fit within the memory available in a single cluster node.
You are absolutely right; this is a problem that Celerity currently has. However, I'm happy to say that we are actively working on a solution that should cover a good portion of use cases and which will be available quite soon hopefully!
Of course, as was already pointed out in this thread (and on the website), this is a research project, and we only have limited resources. If you look a little closer, I'm sure you will find all sorts of issues ;-). However, we are committed to continuously improve Celerity, and ultimately strive to build a robust and modern HPC programming framework that is usable by non-experts (i.e., domain scientists) while also delivering solid performance.
> Doing a distributed MatMul using MPI-CUDA is trivial. I wonder how the cyclomatic complexity and performance compares for that case.
I would assume the programmability metrics to be somewhere in between OpenCL and SYCL. CUDA is a bit less verbose than OpenCL, but you still have to manually deal with MPI (which is also the main factor for MPI+SYCL having much higher cyclomatic complexity than Celerity). Performance-wise I would also not expect much difference, given that we are talking about naive matmul here.
In general, we're not trying to beat [insert your favorite BLAS library]. At this level of abstraction, that would be pretty much impossible. We are showing off results for matmul because everybody knows it and has at least somewhat of an understanding of the MPI operations required to do it in a distributed setting. You can see it as a stand-in for whatever domain-specific algorithm you want to run in a distributed setting.
> Insted, they only compare doing one MatMul per process using MPI-OpenCL... that's... a two liner with CUDA (just call MpiInit followed by a cuBLAS call).
That's not quite true. What we showed here are several successive matrix multiplications, each one being computed in distributed fashion across all participating GPUs. The splitting of work as well as all intermediate data transfers happen completely transparently to the user (save for having to provide a "range mapper"), which I think is one of the main selling points of the project!