What is the terminal used for that demo? https://github.com/HigherOrderCO/Bend does it just skip commands it cannot execute or?
It was actually just me recording iTerm2 with OBS. The theme is Solarized Light. What do you mean by skip commands?
Bend: a high-level language that runs on GPUs (via HVM2)
21–30 of 269 posts
Re: Bend: a high-level language that runs on GPUs (via HVM2)
#22Fala Taelin, nice work! Does HVM2 compile interaction nets to e.g. spirv, or is this an interpreter (like the original HVM) that happens to run on the GPU?
I ask because a while back I was messing around with compiling interaction nets to C after reducing as much of the program as possible (without reducing the inputs), as a form of whole program optimization. Wouldn't be too much harder to target a shader language.
Edit: Oh I see...
> This repository provides a low-level IR language for specifying the HVM2 nets, and a compiler from that language to C and CUDA HVM
Will have to look at the code then!
https://github.com/HigherOrderCO/HVM
Edit: Wait nvm, it looks like the HVM2 cuda runtime is an interpreter, that traverses an in-memory graph and applies reductions.
https://github.com/HigherOrderCO/HVM/blob/5de3e7ed8f1fcee6f2...
I was talking about traversing an interaction net to recover a lambda-calculus-like term, which can be lowered to C a la lisp in small pieces with minimal runtime overhead.
Honestly the motivation is, you are unlikely to outperform a hand-written GPU kernel for like ML workloads using Bend. In theory, HVM could act as glue, stitching together and parallelizing the dispatch order of compute kernels, but you need a good FFI to do that. Interaction nets are hard to translate across FFI boundaries. But, if you compile nets to C, keeping track of FFI compute kernel nodes embedded in the interaction network, you can recover a sensible FFI with no translation overhead.
The other option is implementing HVM in hardware, which I've been messing around with on a spare FPGA.
Re: Bend: a high-level language that runs on GPUs (via HVM2)
#23This is cool! Is the idea to put Kind2 on top of this in some way?
I’d also love to find an example of writing a small interpreter in Bend - which runs on the GPU.
Re: Bend: a high-level language that runs on GPUs (via HVM2)
#24Fala Taelin, nice work! Does HVM2 compile interaction nets to e.g. spirv, or is this an interpreter (like the original HVM) that happens to run on the GPU? I ask because a while back I was messing around with compiling interaction nets to C after reducing as much of the program as possible (without reducing the inputs), as a form of whole program optimization. Wouldn't be too much harder to target a shader language.…
It is an interpreter that runs on GPUs, and a compiler to native C and CUDA. We don't target SPIR-V directly, but aim to. Sadly, while the C compiler results in the expected speedups (3x-4x, and much more soon), the CUDA runtime didn't achieve substantial speedups, compared to the non-compiled version. I believe this is due to warp-divergence: with non-compiled procedures, we can actually merge all function calls into a single "generic" interpreted function expander that can be reduced by warp threads without divergence. We'll be researching this more extensively looking forward.
Re: Bend: a high-level language that runs on GPUs (via HVM2)
#25This is cool! Is the idea to put Kind2 on top of this in some way? I’d also love to find an example of writing a small interpreter in Bend - which runs on the GPU.
Yes, Kind2 will be a type layer on top of Bend, with a similar relationship as in JavaScript / TypeScript (but much more integrated, less ad-hoc and with proofs!). I don't want Kind2 to compete directly with Lean though, as it is doing an amazing job and I'm rooting for it. So, Kind2 will be just a type system for Bend that happens to let you prove theorems about your programs, rather than a higher promise to digitalize all of maths and stuff.
Re: Bend: a high-level language that runs on GPUs (via HVM2)
#26What is the terminal used for that demo? https://github.com/HigherOrderCO/Bend does it just skip commands it cannot execute or?
They're probably hitting ctrl+c at the end of the lines they don't want to run, that's telling the terminal "cancel that" but it'll usually just go to the next line and leave what you typed in place, like in this video.
Re: Bend: a high-level language that runs on GPUs (via HVM2)
#27Fala Taelin, nice work! Does HVM2 compile interaction nets to e.g. spirv, or is this an interpreter (like the original HVM) that happens to run on the GPU? I ask because a while back I was messing around with compiling interaction nets to C after reducing as much of the program as possible (without reducing the inputs), as a form of whole program optimization. Wouldn't be too much harder to target a shader language.…
It is an interpreter that runs on GPUs, and a compiler to native C and CUDA. We don't target SPIR-V directly, but aim to. Sadly, while the C compiler results in the expected speedups (3x-4x, and much more soon), the CUDA runtime didn't achieve substantial speedups, compared to the non-compiled version. I believe this is due to warp-divergence: with non-compiled procedures, we can actually merge all function calls int…
Oh that's cool! Interested to see where your research leads. Could you drop me a link to where the interaction net → cuda compiler resides? I skimmed through the HVM2 repo and just read the .cu runtime file.
Edit: nvm, I read through the rest of the codebase. I see that HVM compiles the inet to a large static term and then links against the runtime.
https://github.com/HigherOrderCO/HVM/blob/5de3e7ed8f1fcee6f2...
Will have to play around with this and look at the generated assembly, see how much of the runtime a modern c/cu compiler can inline.
Btw, nice code, very compact and clean, well-organized easy to read. Rooting for you!
Re: Bend: a high-level language that runs on GPUs (via HVM2)
#28I've made a benchmark of Bend running a simple counter program on CPU vs GPU, vs Haskell,Node,Python,C that I plan to write a blogpost about, probably this Sunday:
https://docs.google.com/spreadsheets/d/1V_DZPpc7_BP3bmOR8Ees...
It's magical how the GPU version is basically flat (although with a high runtime init cost).
Re: Bend: a high-level language that runs on GPUs (via HVM2)
#29Looks cool but what's one toy problem that it can solve more efficiently than others?
Re: Bend: a high-level language that runs on GPUs (via HVM2)
#30Looks cool but what's one toy problem that it can solve more efficiently than others?
Here is an example of it summing a huge set of numbers 100x faster than in C.
https://github.com/HigherOrderCO/bend/blob/main/GUIDE.md#par...