It's pretty cool that you *actually* built this necessary Developer interface aimed towards accessibility of HPC!
Bend: a high-level language that runs on GPUs (via HVM2)
181–190 of 269 posts
Re: Bend: a high-level language that runs on GPUs (via HVM2)
#182Look, I understand the value proposition and how cool it is from a theoretical standpoint, but I honestly don't think this will ever become relevant. Here are some notes from my first impressions and after skimming through the paper. And yes, I am aware that this is very very early software. 1. Bend looks like an extremely limited DSL. No FFI. No way of interacting with raw buffers. Weird 24bit floating point format.…
Thanks for the feedback. Some corrections:
We do use multi-level caching, and you can achieve 5x higher performance by using it correctly. FFI is already implemented, just not published, because we want to release it with graphics rendering, which I think will be really cool. Haskell/GHC uses a graph and trees too, and nobody would say it is not practical of useful. And while it is true that arrays are king, there are many SOTA algorithms that are implemented in Haskell (including compilers, type-checkers, solvers) because they do not map well to arrays at all.
The main reason ICs are not fast is that nobody ever has done low-level optimization work over it. All previous implementations were terribly inefficient. And my own work is too, because I spent all time so far trying to get it to run *correctly* on GPUs, which was very hard. As you said yourself, there aren't even loops yet. So, how can we solve that? By adding the damn loops! Or do you think there is some inherent limitation preventing us to do that? If you do, you'll be surprised.
HVM2 is finally a correct algorithm that scales. Now we'll optimize it for the actual low-level performance.
Re: Bend: a high-level language that runs on GPUs (via HVM2)
#18324bit integers and floats, no array datatype, and a maximum 4GB heap of nodes are very harse restrictions, especially for any workloads that would actually want to be running on a GPU. The limitations in the HVM2 whitepaper about unsound evaluation around closures and infinite loops because it is evaluating both sides of a conditional without any short circuiting are also extremely concerning. Before you reply "these…
So use a metric that makes absolutely no sense on given domain, instead of one that is completely correct, sensible, accurate, stablished on the literature, and vastly superior in context? What even is a FLOPS in the context of Interaction Net evaluation? These things aren't even interchangeable.
The fact that you don’t know the answer to this question, and don’t even seem to think it is relevant, is chilling.
People want to be able to ground your work—which you are claiming is the “parallel future of computation”—in something familiar. Insulting them and telling them their concerns are irrelevant just isn’t going to work.
I would urge you to think about what a standard comparison versus Haskell would look like. Presumably it would be something that dealt with a large state space, but also top down computation (something you couldn’t easily do with matrices). Big examples might include simply taking a giant Haskell benchmark (given the setting of inets it seems like a natural fit) that is implemented in a fairly optimal way—-both algorithmically and also wrt performance—-and compare directly on large inputs.
Sorry to trash on you here, not trying to come across as insulting, but I agree that “reductions per second” is meaningless without a nuanced understanding of the potentially massive encoding blowup that compilation introduces.
We want to believe, but the claims here are big
Re: Bend: a high-level language that runs on GPUs (via HVM2)
#184Earlier quoted context omitted.
You're wrong. The Haskell code is compiled to a loop, which we didn't optimize for yet. I've edited the README to use the Bitonic Sort instead, on which allocations are unavoidable. Past N=20, HVM2 performs 4x faster than GHC -O2.
What? I ran your example, from your readme, where you promise a massive performance improvement, and you're accusing me of doing something wrong? This is exactly what a scammer would say. I guess that's the point here. Scam people who don't know anything about parallel computing by never comparing against any other method?
The insult against anyone who pushes back a little bit is not a good sign, I agree. From all we can see now, the massive speedups being claimed have zero optimal baselines. I badly would like to identify these.
Re: Bend: a high-level language that runs on GPUs (via HVM2)
#185Earlier quoted context omitted.
Where did he claim it is fast? As far as I can see the only claim is that it scales linearly with cores. Which it actually seems to do.
[flagged]
I'll give you the benefit of the doubt in case README changed, but here's the benchmark it claims, currently, against it's own execution modes:
CPU, Apple M3 Max, 1 thread: 12.15 seconds
CPU, Apple M3 Max, 16 threads: 0.96 seconds
GPU, NVIDIA RTX 4090, 16k threads: 0.21 seconds
The README mentions "fast" in 2 places, none of which is comparing to other languages.
Re: Bend: a high-level language that runs on GPUs (via HVM2)
#186Earlier quoted context omitted.
Yes and those benchmarks are real. Showing linear speed up in the number cores when writing standard code is a real achievement. If you assumed that somehow means this is a state of the art compiler with super blazing performance is on no one but you. The readme lays it out very clearly.
[flagged]
the irony in you blasting all over this thread is that you dont know how it even works. You have 0 idea if their claims of scaling linearly are causing bottlenecks in other places as you state, if you read actual docs on this its clear that the actaul "compiler" part of the compiler was put on the backburner while the parallellization was figured out and as that is now done a bunch of optimizations will come in the next year
Re: Bend: a high-level language that runs on GPUs (via HVM2)
#187Very cool idea - but unless I'm missing something, this seems very slow. I just wrote a simple loop in C++ to sum up 0 to 2^30. With a single thread without any optimizations it runs in 1.7s on my laptop -- matching Bend's performance on an RTX 4090! With -O3 it vectorizes the loop to run in less than 80ms. #include int main() { int sum = 0; for (int i = 0; i
Bend has no tail-call optimization yet. It is allocating a 1-billion long stack, while C is just looping. If you compare against a C program that does actual allocations, Bend will most likely be faster with a few threads. Bend's codegen is still abysmal, but these are all low-hanging fruits. Most of the work went into making the parallel evaluator correct (which is extremely hard!). I know that sounds "trust me", bu…
> (I wonder if I should have waited a little bit more before actually posting it)
No. You built something that’s pretty cool. It’s not done yet, but you’ve accomplished a lot! I’m glad you posted it. Thank you. Ignore the noise and keep cooking!
Re: Bend: a high-level language that runs on GPUs (via HVM2)
#188Earlier quoted context omitted.
The only claim I made is that it scales linearly with cores. Nothing else! I'm personally putting a LOT of effort to make our claims as accurate and truthful as possible, in every single place. Documentation, website, demos. I spent hours in meetings to make sure everything is correct. Yet, sometimes it feels that no matter how much effort I put, people will just find ways to misinterpret it. We published the real be…
Identifying what's parallelizable is valuable in the world of language theory, but pure functional languages are as trivial as it gets, so that research isn't exactly ground-breaking. And you're just not fast enough for anyone doing HPC, where the problem is not identifying what can be parallelized, but figuring out to make the most of the hardware, i.e. the codegen.
This approach is valuable because it abstracts away certain complexities for the user, allowing them to focus on the code itself. I found it especially beneficial for users who are not willing to learn functional languages or parallelize code in imperative languages. HPC specialists might not be the current target audience, and code generation can always be improved over time, and I trust based on the dev comments that it will be.
Re: Bend: a high-level language that runs on GPUs (via HVM2)
#189Earlier quoted context omitted.
[flagged]
the irony in you blasting all over this thread is that you dont know how it even works. You have 0 idea if their claims of scaling linearly are causing bottlenecks in other places as you state, if you read actual docs on this its clear that the actaul "compiler" part of the compiler was put on the backburner while the parallellization was figured out and as that is now done a bunch of optimizations will come in the n…
[deleted]
Re: Bend: a high-level language that runs on GPUs (via HVM2)
#190[flagged]