Is it worth implementing sub-cubic matrix multiplication algorithms like Strassen etc. for 4096x4096?
No
Optimizing Matrix Multiplication on RDNA3
21–30 of 30 posts
Re: Optimizing Matrix Multiplication on RDNA3
#22Is the author a genius or has AMD questionable software?
Considering the biggest difference between the kernels is the lack of dual issue instructions (an AMD specific innovation). I'd bet on the latter.
Re: Optimizing Matrix Multiplication on RDNA3
#23> Furthermore, performing custom ISA optimizations makes these changes RDNA3-specific this is overblown at least wrt forward compatibility - all of the instructions used are in RDNA4 and most of them are even in CDNA3 (CDNA4 isn't public yet?) and the ones that aren't exactly there are only slightly renamed (ds_load -> ds_read). Sure it's annoying but it's not the end of the world to have some `#ifdef`s in your code…
You're making the assumption that every kernel developer has enough AMD GPUs from different eras that they can test their ifdefs on all the possible ISAs.
Re: Optimizing Matrix Multiplication on RDNA3
#24Cuda has similar inefficiencies and many use cases can have equal uplifts by going lower level on the code. I think this is what deepseek had done to get their speedups on older hardware. Even way back in the days of GPU crypto mining - custom kernels hand built (mostly just unrolling loops) would yield 20% improvements over just running opencl and letting the drivers compile it down.
People have been trying to bypass CUDA and even PTX for a long time. One long rundown of optimizing gemm on NVIDIA hardware ( https://salykova.github.io/sgemm-gpu ) mentions 'maxas' ( https://github.com/NervanaSystems/maxas/wiki/Introduction ) - which was really a step forward in this space. I still blame Intel (buying NervanaSystems) for killing it...
> People have been trying to bypass CUDA and even PTX for a long time
i swear it's so funny when people talk about this stuff like it's all weird/surprising. y'all realize that there are hundreds (thousands?) of engineers across FAANG whose full time job is optimizing CUDA/ROCm/whatever code for their team/org/company's specific workloads? like do y'all think that serious shops really just go with whatever the vendor gives you? ie none of this is in the least surprising - it's completely expected that whatever the vendor designs generically for the entire market segment will fail to achieve peak perf for your use case.
Re: Optimizing Matrix Multiplication on RDNA3
#25I find it quite interesting that while vector instructions are present every other sort of "hardware level grouping" (wave, SIMD, thread) is hidden from the programmer. Why would vector instructions be the only thing the programmer ought to care about?
I wonder if there's untapped potential in a GPU language which made all of those implicit classes explicit in code, now that we've sort of stabilized on them. It wouldn't allow you to do anything that you can't already do with clever optimizations and a profiler, but it could have the potential to make the optimizations clearer.
In general I'm very curious as to why we don't have any new languages that are better aligned with current hardware. For some reason we collectively decided that it was more fun to make everything general, which is especially unfortunate considering the real world got increasingly homogeneous. Compiling to some intermediate language makes no sense when you're only ever going to run on x86 anyway.
Re: Optimizing Matrix Multiplication on RDNA3
#26Earlier quoted context omitted.
People have been trying to bypass CUDA and even PTX for a long time. One long rundown of optimizing gemm on NVIDIA hardware ( https://salykova.github.io/sgemm-gpu ) mentions 'maxas' ( https://github.com/NervanaSystems/maxas/wiki/Introduction ) - which was really a step forward in this space. I still blame Intel (buying NervanaSystems) for killing it...
> People have been trying to bypass CUDA and even PTX for a long time i swear it's so funny when people talk about this stuff like it's all weird/surprising. y'all realize that there are hundreds (thousands?) of engineers across FAANG whose full time job is optimizing CUDA/ROCm/whatever code for their team/org/company's specific workloads? like do y'all think that serious shops really just go with whatever the vendor…
>it's completely expected that whatever the vendor designs generically for the entire market segment will fail to achieve peak perf for your use case.
When Carmack left Meta I believe he claimed they were only getting around 20% utilization on their even then enormous GPU fleet. So I could see them also leaving a lot of perf headroom on the table.
Re: Optimizing Matrix Multiplication on RDNA3
#27Earlier quoted context omitted.
AMD isn’t losing sleep over the fact that J. Random Blogger is beating their GEMM by 60% on 4096x4096? What universe are you living in? This company is fighting for their life against CUDA and you’re telling me their software stack being so bad it can’t use a third of the hardware on the the first and literally only thing people want it to do is somehow not a problem?
The point of a platform is for software engineers to provide key functionality independently. Your issue here is you don't understand why CUDA has been so dominant over the last decade - a ~50% software performance gap isn't that material when hardware capacity doubles every generation. If we've reached the point where J. Random Blogger can solve their own problems then the CUDA moat has quite possibly been broken. I…
Though people may use CPUs several years old, they generally weren't at the moment they were bought, and the decision came from comparing with the competition. This argument of "my software will be faster when computers are faster" does not hold given that the competition is also benefiting from Moore's law. Nothing changes in relative terms, which is what matters, until you actually improve your slow software.
And while a possibility may exist to improve software on the user's end, do people not base their decisions on benchmarks involving existing (not potential) software? They find comparisons using the provided kernels, find AMD to be slower, unaware that they could (maybe, at that) find a 30% speedup to be had. Even if they stumbled on this article, would they trust they could pull it off, or simply go with the GPU that has the best performance with existing libraries?
These are machines sold for crunching numbers, they might as well crunch numbers as best they can...
Re: Optimizing Matrix Multiplication on RDNA3
#28Earlier quoted context omitted.
People have been trying to bypass CUDA and even PTX for a long time. One long rundown of optimizing gemm on NVIDIA hardware ( https://salykova.github.io/sgemm-gpu ) mentions 'maxas' ( https://github.com/NervanaSystems/maxas/wiki/Introduction ) - which was really a step forward in this space. I still blame Intel (buying NervanaSystems) for killing it...
> People have been trying to bypass CUDA and even PTX for a long time i swear it's so funny when people talk about this stuff like it's all weird/surprising. y'all realize that there are hundreds (thousands?) of engineers across FAANG whose full time job is optimizing CUDA/ROCm/whatever code for their team/org/company's specific workloads? like do y'all think that serious shops really just go with whatever the vendor…
Not saying it's surprising. My day job is doing exactly this, not in any FAANG.
Working on a platform that hides so many low-level details is a challenge, and the fact people have to go to such length to get access to it is noteworthy. 'maxas' was noteworthy and unneeded on many (most ?) other platforms.
Not saying Intelstuff or armstuff is 'easier' but at least you get access and are tooled to work on the actual low-level asm.
Re: Optimizing Matrix Multiplication on RDNA3
#29Earlier quoted context omitted.
> People have been trying to bypass CUDA and even PTX for a long time i swear it's so funny when people talk about this stuff like it's all weird/surprising. y'all realize that there are hundreds (thousands?) of engineers across FAANG whose full time job is optimizing CUDA/ROCm/whatever code for their team/org/company's specific workloads? like do y'all think that serious shops really just go with whatever the vendor…
Not saying it's surprising. My day job is doing exactly this, not in any FAANG. Working on a platform that hides so many low-level details is a challenge, and the fact people have to go to such length to get access to it is noteworthy. 'maxas' was noteworthy and unneeded on many (most ?) other platforms. Not saying Intelstuff or armstuff is 'easier' but at least you get access and are tooled to work on the actual low…
> and the fact people have to go to such length to get access to it is noteworthy
I'll repeat myself: no it's not. There's nothing noteworthy about it at all. In fact I literally cannot fathom why anyone ever expects or expected otherwise. Is it because the oft-repeated notion of "abstraction"? I guess I must be the sole programmer that has always known/understood, even from the first intro class, that abstractions are just assumptions and when those assumptions don't hold I will need to remove the abstraction.
Re: Optimizing Matrix Multiplication on RDNA3
#30Earlier quoted context omitted.
The point of a platform is for software engineers to provide key functionality independently. Your issue here is you don't understand why CUDA has been so dominant over the last decade - a ~50% software performance gap isn't that material when hardware capacity doubles every generation. If we've reached the point where J. Random Blogger can solve their own problems then the CUDA moat has quite possibly been broken. I…
Though people may use CPUs several years old, they generally weren't at the moment they were bought, and the decision came from comparing with the competition. This argument of "my software will be faster when computers are faster" does not hold given that the competition is also benefiting from Moore's law. Nothing changes in relative terms, which is what matters, until you actually improve your slow software. And w…
At risk of repeating myself, you're not anywhere close to grappling with how bad the situation has been on AMD cards. If they could consistently half-saturate the hardware they'd have a place in the AI revolution instead of being left out in the cold. The traditional achievement of an AMD card, in practice, is 0% hardware saturation because when they tried to multiply matricies then there was a good chance that the system would crash.
The type of commercial logic you're talking about isn't the important factor in the real world. 50% saturation with the option to fully saturate is amazing by AMDs standards and they have much bigger problems than this affecting people's buying decisions. If they had been able to achieve this standard in 2020 I would still be buying AMD.