Earlier quoted context omitted.
Could be trying to make themselves a target for a big acquihire.
> Could be trying to make themselves a target for a big acquihire. Is this something anyone sets out to do?
Yes.
Making AMD GPUs competitive for LLM inference (2023)
51–60 of 221 posts
Re: Making AMD GPUs competitive for LLM inference (2023)
#52I have come across quite few startups who are trying a similar idea: break the nvidia monopoly by utilizing AMD GPUs (for inference at least): Felafax, Lamini, tensorwave (partially), SlashML. Even saw optimistic claims like CUDA moat is only 18 months deep from some of them [1]. Let's see. [1] https://www.linkedin.com/feed/update/urn:li:activity:7275885...
Tinygrad was another one, but they ended up getting frustrated with AMD and semi-pivoted to Nvidia.
This is discussed in the lex Friedman episode. AMD’s own demo would kernel panic when run in a loop [1].
Re: Making AMD GPUs competitive for LLM inference (2023)
#53Re: Making AMD GPUs competitive for LLM inference (2023)
#54This benchmark doest look right. Is it using the tensor cores in the Nvidia gpu? AMD does not have AI cores so should run noticeably slower.
AMD has WMMA.
Re: Making AMD GPUs competitive for LLM inference (2023)
#55Earlier quoted context omitted.
Is there no hope for AMD anymore? After George Hotz/Tinygrad gave up on AMD I feel there’s no realistic chance of using their chips to break the CUDA dominance.
Not really. AMD is constitutionally incapable of shipping anything but mid range hardware that requires no innovation. The only reason why they are doing so well in CPUs right now is that Intel has basically destroyed itself without any outside help.
Everything is comparative. AMD isn't perfect. As an Ex Shareholder I have argued they did well partly because of Intel's downfall. In terms of execution it is far from perfect.
But Nvidia is a different beast. It is a bit like Apple in the late 00s where you take business, forecast, marketing, operation, software, hardware, sales etc You take any part of it and they are all industry leading. And having industry leading capability is only part of the game, having it all work together is completely another thing. And unlike Apple where they lost direction once Steve Jobs passed away and weren't sure about how to deploy capital. Jensen is still here, and they have more resources now making Nvidia even more competitive.
It is often most people underestimate the magnitude of the task required, ( I like to tell the story again about an Intel GPU engineer in 2016 arguing they could take dGPU market shares by 2020, and we are now 2025 ), over estimate the capability of an organisation, under estimate the rival's speed of innovation and execution. These three thing combined is why most people are often off the estimate by an order of magnitude.
Re: Making AMD GPUs competitive for LLM inference (2023)
#56Earlier quoted context omitted.
Not really. AMD is constitutionally incapable of shipping anything but mid range hardware that requires no innovation. The only reason why they are doing so well in CPUs right now is that Intel has basically destroyed itself without any outside help.
And I'm supposed to believe that HN is this amazing platform for technology and science discussions, totally unlike its peers...
I don't really care what you believe.
Everyone whose dug deep into what AMD is doing has left in disgust if they are lucky and bankruptcy if they are not.
If I can save someone else from wasting $100,000 on hardware and six months of their life then my post has done more good than the AMD marketing department ever will.
Re: Making AMD GPUs competitive for LLM inference (2023)
#57Earlier quoted context omitted.
Unlike training, ML inference is almost always bound by memory bandwidth as opposed to computations. For this reason, tensor cores, cuDNN, and other advanced shenanigans make very little sense for the use case. OTOH, general-purpose compute instead of fixed-function blocks used by cuDNN enables custom compression algorithms for these weights which does help, by saving memory bandwidth. For example, I did custom 5 bit…
Only local (read batch size 1) ML inference is memory bound, production loads are pretty much compute bound. Prefill phase is very compute bound, and with continuous batching generation phase is getting mixed with prefill, which makes whole process altogether to be compute bound too. So no, tensor cores and all other shenanigans absolutely critical for performant inference infrastructure.
PyTorch is a project by Linux foundation. The about page with the mission of the foundation contains phrases like “empowering generations of open source innovators”, “democratize code”, and “removing barriers to adoption”.
I would argue running local inference with batch size=1 is more useful for empowering innovators compared to running production loads on shared servers owned by companies. Local inference increases count of potential innovators by orders of magnitude.
BTW, in the long run it may also benefit these companies because in theory, an easy migration path from CUDA puts a downward pressure on nVidia’s prices.
Re: Making AMD GPUs competitive for LLM inference (2023)
#58Earlier quoted context omitted.
Not really. AMD is constitutionally incapable of shipping anything but mid range hardware that requires no innovation. The only reason why they are doing so well in CPUs right now is that Intel has basically destroyed itself without any outside help.
Everything is comparative. AMD isn't perfect. As an Ex Shareholder I have argued they did well partly because of Intel's downfall. In terms of execution it is far from perfect. But Nvidia is a different beast. It is a bit like Apple in the late 00s where you take business, forecast, marketing, operation, software, hardware, sales etc You take any part of it and they are all industry leading. And having industry leadi…
Yeah, no.
We are in the middle of a monopoly squeeze by NVidia on the most innovative part of the economy right now. I expect the DOJ to hit them harder than they did MS in the 90s given the bullshit they are pulling and the drag on the economy they are causing.
By comparison if AMD could write a driver that didn't shit itself when it had to multiply more than two matrices in a row they'd be selling cards faster than they can make them. You don't need to sell the best shovels in a gold rush to make mountains of money, but you can't sell teaspoons as premium shovels and expect people to come back.
Re: Making AMD GPUs competitive for LLM inference (2023)
#59Note: this is old work, and much of the team working on TVM, and MLC were from OctoAI and we have all recently joined NVIDIA.
Is there no hope for AMD anymore? After George Hotz/Tinygrad gave up on AMD I feel there’s no realistic chance of using their chips to break the CUDA dominance.
The world is bigger than AMD and Nvidia. Plenty of interesting new AI-tuned non-GPU accelerators coming online.
Re: Making AMD GPUs competitive for LLM inference (2023)
#60Earlier quoted context omitted.
Could be trying to make themselves a target for a big acquihire.
Cynical take: Try to get acquired by Intel for Arc.
More cynical take: this would be a bad strategy, because Intel hasn't shown much competence in its leadership for a long time, especially in regards to GPUs.