Making AMD GPUs competitive for LLM inference (2023)
1–10 of 221 posts
Re: Making AMD GPUs competitive for LLM inference (2023)
#2Intriguing. I thought AMD GPUs didn't have tensor cores (or matrix multiplication units) like NVidia. I believe they are only dot product / fused multiply and accumulate instructions.
Are these LLMs just absurdly memory bound so it doesn't matter?
Re: Making AMD GPUs competitive for LLM inference (2023)
#3[2023]
Btw, this is from MLC-LLM which makes WebLLM and other good stuff.
Re: Making AMD GPUs competitive for LLM inference (2023)
#4> Aug 9, 2023
Ignoring the very old (in ML time) date of the article...
What's the catch? People are still struggling with this a year later so I have to assume it doesn't work as well as claimed.
I'm guessing this is buggy in practice and only works for the HF models they chose to test with?
Re: Making AMD GPUs competitive for LLM inference (2023)
#5Intriguing. I thought AMD GPUs didn't have tensor cores (or matrix multiplication units) like NVidia. I believe they are only dot product / fused multiply and accumulate instructions. Are these LLMs just absurdly memory bound so it doesn't matter?
They don’t, but GPUs were designed for doing matrix multiplications even without the special hardware instructions for doing matrix multiplication tiles. Also, the forward pass for transformers is memory bound, and that is what does token generation.
Re: Making AMD GPUs competitive for LLM inference (2023)
#6Intriguing. I thought AMD GPUs didn't have tensor cores (or matrix multiplication units) like NVidia. I believe they are only dot product / fused multiply and accumulate instructions. Are these LLMs just absurdly memory bound so it doesn't matter?
> Are these LLMs just absurdly memory bound so it doesn't matter?
During inference? Definitely. Training is another story.
Re: Making AMD GPUs competitive for LLM inference (2023)
#7Intriguing. I thought AMD GPUs didn't have tensor cores (or matrix multiplication units) like NVidia. I believe they are only dot product / fused multiply and accumulate instructions. Are these LLMs just absurdly memory bound so it doesn't matter?
They don’t, but GPUs were designed for doing matrix multiplications even without the special hardware instructions for doing matrix multiplication tiles. Also, the forward pass for transformers is memory bound, and that is what does token generation.
Well sure, but in other GPU tasks, like Raytracing, the difference between these GPUs is far more pronounced.
And AMD has passable Raytracing units (NVidias are better but the difference is bigger than these LLM results).
If RAM is the main bottleneck then CPUs should be on the table.
Re: Making AMD GPUs competitive for LLM inference (2023)
#8> Aug 9, 2023 Ignoring the very old (in ML time) date of the article... What's the catch? People are still struggling with this a year later so I have to assume it doesn't work as well as claimed. I'm guessing this is buggy in practice and only works for the HF models they chose to test with?
[deleted]
Re: Making AMD GPUs competitive for LLM inference (2023)
#9I have come across quite few startups who are trying a similar idea: break the nvidia monopoly by utilizing AMD GPUs (for inference at least): Felafax, Lamini, tensorwave (partially), SlashML. Even saw optimistic claims like CUDA moat is only 18 months deep from some of them [1]. Let's see.
[1] https://www.linkedin.com/feed/update/urn:li:activity:7275885...
Re: Making AMD GPUs competitive for LLM inference (2023)
#10> Aug 9, 2023 Ignoring the very old (in ML time) date of the article... What's the catch? People are still struggling with this a year later so I have to assume it doesn't work as well as claimed. I'm guessing this is buggy in practice and only works for the HF models they chose to test with?
It’s not terribly hard to port ML inference to alternative GPU APIs. I did it for D3D11 and the performance is pretty good too: https://github.com/Const-me/Cgml
The only catch is, for some reason developers of ML libraries like PyTorch aren’t interested in open GPU APIs like D3D or Vulkan. Instead, they focus on proprietary ones i.e. CUDA and to lesser extent ROCm. I don’t know why that is.
D3D-based videogames are heavily using GPU compute for more than a decade now. Since Valve shipped SteamDeck, the same now applies to Vulkan on Linux. By now, both technologies are stable, reliable and performant.