This benchmark doest look right. Is it using the tensor cores in the Nvidia gpu? AMD does not have AI cores so should run noticeably slower.
Making AMD GPUs competitive for LLM inference (2023)
41–50 of 221 posts
Re: Making AMD GPUs competitive for LLM inference (2023)
#42[deleted]
Re: Making AMD GPUs competitive for LLM inference (2023)
#43Intriguing. I thought AMD GPUs didn't have tensor cores (or matrix multiplication units) like NVidia. I believe they are only dot product / fused multiply and accumulate instructions. Are these LLMs just absurdly memory bound so it doesn't matter?
They absolutely do have similar cores to tensor cores, it's called matrix cores. And they have particular instructions to utilize them (MFMA).
Note I'm talking about DC compute chips, like MI300.
LLMs aren't memory bound in production loads, they are pretty much compute bound too, at least in prefill phase, but in practice in general too.
Re: Making AMD GPUs competitive for LLM inference (2023)
#44A used 3090 is $600-900, performs better than 7900, and is much more versatile because CUDA
Reality check for anyone considering this: I just got a used 3090 for $900 last month. It works great.
I would not recommend buying one for $600, it probably either won’t arrive or will be broken. Someone will reply saying they got one for $600 and it works, that doesn’t mean it will happen if you do it.
I’d say the market is realistically $900-1100, maybe $800 if you know the person or can watch the card running first.
All that said, this advice will expire in a month or two when the 5090 comes out.
Re: Making AMD GPUs competitive for LLM inference (2023)
#45Intriguing. I thought AMD GPUs didn't have tensor cores (or matrix multiplication units) like NVidia. I believe they are only dot product / fused multiply and accumulate instructions. Are these LLMs just absurdly memory bound so it doesn't matter?
They absolutely do have similar cores to tensor cores, it's called matrix cores. And they have particular instructions to utilize them (MFMA). Note I'm talking about DC compute chips, like MI300. LLMs aren't memory bound in production loads, they are pretty much compute bound too, at least in prefill phase, but in practice in general too.
Ya people in these comments don't know what they're talking about (no one ever does in these threads). AMDGPU has had MMA and WMMA for a while now
https://rocm.docs.amd.com/projects/rocWMMA/en/latest/what-is...
Re: Making AMD GPUs competitive for LLM inference (2023)
#46Earlier quoted context omitted.
Is there no hope for AMD anymore? After George Hotz/Tinygrad gave up on AMD I feel there’s no realistic chance of using their chips to break the CUDA dominance.
Maybe from Modular (the company Chris Lattner is working for). In this recent announcement they said they had achieved competitive ML performance… on NVIDIA GPUs, but with their own custom stack completely replacing CUDA. And they’re targeting AMD next. https://www.modular.com/blog/introducing-max-24-6-a-gpu-nati...
Ah yes, the programming language (Mojo) that requires an account before I can use it...
Re: Making AMD GPUs competitive for LLM inference (2023)
#47Earlier quoted context omitted.
Peculiar business model, at a glance. It seems like they're doing work that AMD ought to be doing, and is probably doing behind the scenes. Who is the customer for a third-party GPU driver shim?
Could be trying to make themselves a target for a big acquihire.
Cynical take: Try to get acquired by Intel for Arc.
Re: Making AMD GPUs competitive for LLM inference (2023)
#48Re: Making AMD GPUs competitive for LLM inference (2023)
#49Earlier quoted context omitted.
Peculiar business model, at a glance. It seems like they're doing work that AMD ought to be doing, and is probably doing behind the scenes. Who is the customer for a third-party GPU driver shim?
It would be interesting to find out AMD is funding these other companies to ensure the shim happens while they focus on not doing it.
AMD is kind of doing that funding by pricing its GPUs low and/or giving them away at cost to these startups
Re: Making AMD GPUs competitive for LLM inference (2023)
#50Earlier quoted context omitted.
Peculiar business model, at a glance. It seems like they're doing work that AMD ought to be doing, and is probably doing behind the scenes. Who is the customer for a third-party GPU driver shim?
Could be trying to make themselves a target for a big acquihire.
> Could be trying to make themselves a target for a big acquihire.
Is this something anyone sets out to do?