Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

CUDA-l2: Surpassing cuBLAS performance for matrix multiplication through RL

github.com

Post reply on HN
Print view View original

11–17 of 17 posts

Re: CUDA-l2: Surpassing cuBLAS performance for matrix multiplication through RL

#11

Post by slashdave »

alyxya wrote: post #8
j2kun wrote: post #5

They claim the algorithm "discovered" the new techniques, but the methods described in section 5 do not seem all that novel to me. It smells like it could be "laundering" the literature [1] and reshuffling existing techniques. This is not inherently a bad thing, but I would hope that if it is borrowing existing techniques, the appropriate citation would eventually make it into this paper. [1]: https://www.argmin.net/…

There generally aren't new techniques when optimizing something ubiquitous. Instead, there are a lot of ways to apply existing techniques to create new and better results. Most ideas are built on top of the same foundational principles.

I am not sure about that. However, what is clear is that if there is a new technique, it will not be found by this LLM.

Re: CUDA-l2: Surpassing cuBLAS performance for matrix multiplication through RL

#12

Post by CapsAdmin »

slashdave wrote: post #11
alyxya wrote: post #8

Earlier quoted context omitted.

There generally aren't new techniques when optimizing something ubiquitous. Instead, there are a lot of ways to apply existing techniques to create new and better results. Most ideas are built on top of the same foundational principles.

I am not sure about that. However, what is clear is that if there is a new technique, it will not be found by this LLM.

It's generally true, isn't it? Otherwise we'd have ground breaking discoveries every day about some new and fastest way to do X.

The way I see it, mathematicians have been trying (and somewhat succeeding every 5~ years) to prove faster ways to do matrix multiplications since the 1970s. But this is only in theory.

If you want to implement the theory, you suddenly have many variables you need to take care of such as memory speed, cpu instructions, bit precision, etc. So in practice, an actual implementation of some theory likely have more room to improve. It is also likely that LLM's can help figure out how to write a more optimal implementation.

Re: CUDA-l2: Surpassing cuBLAS performance for matrix multiplication through RL

#13

Post by hedgehog »

AlexCoventry wrote: post #6
j2kun wrote: post #5

They claim the algorithm "discovered" the new techniques, but the methods described in section 5 do not seem all that novel to me. It smells like it could be "laundering" the literature [1] and reshuffling existing techniques. This is not inherently a bad thing, but I would hope that if it is borrowing existing techniques, the appropriate citation would eventually make it into this paper. [1]: https://www.argmin.net/…

In the future, we will all be Jürgen Schmidhuber. :-)

I hate to break it to you but the original work on that topic was by Schmidhuber & Schmidhuber back in 1963.

Re: CUDA-l2: Surpassing cuBLAS performance for matrix multiplication through RL

#14

Post by konradha »

I've been trying my hand at RL envs for various sparse matrix algorithms in CUDA. It's easy to generate code that "looks good", "novel" and "fast". Escaping the distribution and actually creating novel sequences of instructions or even patterns (has any model come with something as useful as fan-in/fan-out or double buffering patterns that's now ubiquituous?) seems difficult to say the least.

Re: CUDA-l2: Surpassing cuBLAS performance for matrix multiplication through RL

#15

Post by josephg »

alyxya wrote: post #8
j2kun wrote: post #5

They claim the algorithm "discovered" the new techniques, but the methods described in section 5 do not seem all that novel to me. It smells like it could be "laundering" the literature [1] and reshuffling existing techniques. This is not inherently a bad thing, but I would hope that if it is borrowing existing techniques, the appropriate citation would eventually make it into this paper. [1]: https://www.argmin.net/…

There generally aren't new techniques when optimizing something ubiquitous. Instead, there are a lot of ways to apply existing techniques to create new and better results. Most ideas are built on top of the same foundational principles.

Yes. And there’s still lots of places where you can get significant speed ups by simply applying those old techniques in a new domain or a novel way. The difference between a naive implementation of an algorithm and an optimised one is often many orders of magnitude. Look at automerge - which went from taking 30 seconds on a simple example to tens of milliseconds.

I think about this regularly when I compile C++ or rust using llvm. It’s an excellent compiler backend. It produces really good code. But it is incredibly slow, and for no good technical reason. Plenty of other similar compilers run circles around it.

Imagine an llvm rewrite by the people who made V8, or chrome or the unreal engine. Or the guy who made luajit or the Go compiler team. I’d be shocked if we didn’t see an order of magnitude speed up overnight. They’d need some leeway to redesign llvm IR of course. And it would take years to port all of llvm’s existing optimisations. But my computer can retire billions of operations per second. And render cyberpunk at 60fps. It shouldn’t take seconds of cpu time to compile a small program.

Re: CUDA-l2: Surpassing cuBLAS performance for matrix multiplication through RL

#16

Post by roflmaostc »

> Q: What if I need matrix dimensions (M, N, K) not found in your configurations? >A: 1. You can find the nearest neighbor configuration (larger than yours) and pad with zeros. 2. Feel free to post your dimensions on GitHub issues. We are happy to release kernels for your configuration.

Lol, this will be potentially much slower than using the general matmul kernel.

However, I like this kind of research because it really exploits specific hardware configurations and makes it measurable faster (unlike some theoretical matmul improvements). Code specialization is cheap, and if it saves in the order of a few %, it quickly reimburses its price, especially for important things like matmul.

Re: CUDA-l2: Surpassing cuBLAS performance for matrix multiplication through RL

#17

Post by Bulat_Ziganshin »

stonogo wrote: post #2

Am I reading this wrong, or does this only support FP16 inputs, and compares its performance against an FP32 solver?

They compare HGEMM implementations. At least CUBLAS has HGEMM functions.

HGEMM means half-precision (i.e. FP16) general matrix multiplication

Post reply on HN