Celerity – High-level C++ for Accelerator Clusters • Hacker News Board

Celerity – High-level C++ for Accelerator Clusters

#1

Post by pjmlp » Tue, Mar 31, 2020, 6:39 AM UTC

Celerity – High-level C++ for Accelerator Clusters
celerity.github.io

Re: Celerity – High-level C++ for Accelerator Clusters

#2

Post by fluffything » Tue, Mar 31, 2020, 8:37 AM UTC

Weird. A library that wraps SYCL within MPI, yet requires all processes to hold a copy of all the memory ?

One of the main reasons to use MPI is to solve problems that do not fit within the memory available in a single cluster node.

There is a presentation [0] from 2020-02-20 that's not very impressive. Particularly, they compare against MPI+OpenCL, but do not show a comparison against MPI-CUDA.

Doing a distributed MatMul using MPI-CUDA is trivial. I wonder how the cyclomatic complexity and performance compares for that case. Insted, they only compare doing one MatMul per process using MPI-OpenCL... that's... a two liner with CUDA (just call MpiInit followed by a cuBLAS call).

[0] https://www.uibk.ac.at/fz-hpc/events/resources/2020_02_20_ah...

Re: Celerity – High-level C++ for Accelerator Clusters

#3

Post by qayxc » Tue, Mar 31, 2020, 11:25 AM UTC

fluffything wrote: post #2

Weird. A library that wraps SYCL within MPI, yet requires all processes to hold a copy of all the memory ? One of the main reasons to use MPI is to solve problems that do not fit within the memory available in a single cluster node. There is a presentation [0] from 2020-02-20 that's not very impressive. Particularly, they compare against MPI+OpenCL, but do not show a comparison against MPI-CUDA. Doing a distributed M…

I'm honestly much more worried about NVIDIA's overwhelming success in buying their way into the entire scientific community.

CUDA is proprietary, not standardised, and tied to a single hardware vendor. Technical issues aside, I applaud any effort to break the dominance of CUDA. CUDA is a pest that needs to either open up or be replaced.

HPC is a very heterogeneous environment (on the whole) and making it easier to use the full potential of the hardware is understandable. Also keep in mind that it's a research project and very early in its development.

Re: Celerity – High-level C++ for Accelerator Clusters

#4

Post by adev_ » Tue, Mar 31, 2020, 1:04 PM UTC

qayxc wrote: post #3

fluffything wrote: post #2

Weird. A library that wraps SYCL within MPI, yet requires all processes to hold a copy of all the memory ? One of the main reasons to use MPI is to solve problems that do not fit within the memory available in a single cluster node. There is a presentation [0] from 2020-02-20 that's not very impressive. Particularly, they compare against MPI+OpenCL, but do not show a comparison against MPI-CUDA. Doing a distributed M…

I'm honestly much more worried about NVIDIA's overwhelming success in buying their way into the entire scientific community. CUDA is proprietary, not standardised, and tied to a single hardware vendor. Technical issues aside, I applaud any effort to break the dominance of CUDA. CUDA is a pest that needs to either open up or be replaced. HPC is a very heterogeneous environment (on the whole) and making it easier to us…

> CUDA is a pest that needs to either open up or be replaced.

My main problem with cuda is not that it is un-free, it is that the source code is unavailable and consequently you have to deal with a badly packaged, badly supported blob that turn only on a restricted set of compilers.

The consequences of that is that every software depending on cuda is a pain in the butt to distribute. Even major software like Pytorch or tensorflow need to have separated released due to cuda.

It is particularly sad that Nvidia in 2020 did not understand what Mellanox understood 10 years ago: that successful leadership in hardware is compatible OSS with an software stack.

Re: Celerity – High-level C++ for Accelerator Clusters

#5

Post by e40 » Tue, Mar 31, 2020, 2:56 PM UTC

Not to be confused with Celerity Computing: https://en.wikipedia.org/wiki/Celerity_Computing

Fun fact: I did a port of Franz Lisp[1] to the Celerity, having the compiler (liszt) generate code for the backend of the C compiler, since they told us: "in no way will you be able to generate assembly code out of your compiler for this architecture." They were an early Sun Microsystems competitor that didn't make it.

[1] Not to be confused with Franz's Common Lisp, this is the MacLisp compatible Lisp from UC Berkeley.

Re: Celerity – High-level C++ for Accelerator Clusters

#6

Post by pjmlp » Wed, Apr 01, 2020, 6:51 AM UTC

qayxc wrote: post #3

fluffything wrote: post #2

Weird. A library that wraps SYCL within MPI, yet requires all processes to hold a copy of all the memory ? One of the main reasons to use MPI is to solve problems that do not fit within the memory available in a single cluster node. There is a presentation [0] from 2020-02-20 that's not very impressive. Particularly, they compare against MPI+OpenCL, but do not show a comparison against MPI-CUDA. Doing a distributed M…

I'm honestly much more worried about NVIDIA's overwhelming success in buying their way into the entire scientific community. CUDA is proprietary, not standardised, and tied to a single hardware vendor. Technical issues aside, I applaud any effort to break the dominance of CUDA. CUDA is a pest that needs to either open up or be replaced. HPC is a very heterogeneous environment (on the whole) and making it easier to us…

Thank Khronos for being too focused on C and expecting partners to develop any kind of tooling or SDKs.

It took the CUDA bulldozer for them to consider more C++ love and introduction of bytecode formats that allow more languages to easily target Khronos standards.

Incidentally that were the biggest complains about Vulkan as well, on the last LunarG survey, tooling and documentation.

Re: Celerity – High-level C++ for Accelerator Clusters

#7

Post by queensnake » Wed, Apr 01, 2020, 7:34 AM UTC

Intel’s DPC++ seems similar - computation graph, communicating via buffers.

https://software.intel.com/en-us/oneapi/dpc-compiler

Anyone pick out differences?

Re: Celerity – High-level C++ for Accelerator Clusters

#8

Post by queensnake » Wed, Apr 01, 2020, 7:38 AM UTC

qayxc wrote: post #3

fluffything wrote: post #2

Weird. A library that wraps SYCL within MPI, yet requires all processes to hold a copy of all the memory ? One of the main reasons to use MPI is to solve problems that do not fit within the memory available in a single cluster node. There is a presentation [0] from 2020-02-20 that's not very impressive. Particularly, they compare against MPI+OpenCL, but do not show a comparison against MPI-CUDA. Doing a distributed M…

I'm honestly much more worried about NVIDIA's overwhelming success in buying their way into the entire scientific community. CUDA is proprietary, not standardised, and tied to a single hardware vendor. Technical issues aside, I applaud any effort to break the dominance of CUDA. CUDA is a pest that needs to either open up or be replaced. HPC is a very heterogeneous environment (on the whole) and making it easier to us…

Intel’s OneAPI promises to translate away from CUDA: https://software.intel.com/oneapi

Re: Celerity – High-level C++ for Accelerator Clusters

#9

Post by 14113 » Wed, Apr 01, 2020, 9:41 AM UTC

queensnake wrote: post #7

Intel’s DPC++ seems similar - computation graph, communicating via buffers. https://software.intel.com/en-us/oneapi/dpc-compiler Anyone pick out differences?

DPC++ is basically "just" SYCL, so no MPI.

Re: Celerity – High-level C++ for Accelerator Clusters

#10

Post by fluffything » Wed, Apr 01, 2020, 3:26 PM UTC

qayxc wrote: post #3

fluffything wrote: post #2

Weird. A library that wraps SYCL within MPI, yet requires all processes to hold a copy of all the memory ? One of the main reasons to use MPI is to solve problems that do not fit within the memory available in a single cluster node. There is a presentation [0] from 2020-02-20 that's not very impressive. Particularly, they compare against MPI+OpenCL, but do not show a comparison against MPI-CUDA. Doing a distributed M…

I'm honestly much more worried about NVIDIA's overwhelming success in buying their way into the entire scientific community. CUDA is proprietary, not standardised, and tied to a single hardware vendor. Technical issues aside, I applaud any effort to break the dominance of CUDA. CUDA is a pest that needs to either open up or be replaced. HPC is a very heterogeneous environment (on the whole) and making it easier to us…

> I'm honestly much more worried about NVIDIA's overwhelming success in buying their way into the entire scientific community.

They bought their way by providing software along with the hardware, and by paying developers and researchers to work full-time in improving the performance of user applications on their hardware.

If you are a medium-ly important HPC application, the chances that there is an NVIDIA engineer working full-time on making it perform good on GPUs is quite large.

What's worrying is that Intel and AMD haven't realized, 20 years later, that hardware without software is useless. Even if they were to make GPUs that outperform NVIDIA's hardware in raw numbers by a 10x factor, NVIDIA's GPUs would still probably be the better product to buy because of the software and support you get.