Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Making AMD GPUs competitive for LLM inference (2023)

blog.mlc.ai

Post reply on HN
Print view View original

61–70 of 221 posts

Re: Making AMD GPUs competitive for LLM inference (2023)

#61

Post by teleforce »

schmidtleonard wrote: post #38
teleforce wrote: post #29

Earlier quoted context omitted.

Compute Express Link (CXL) should mostly solve limited RAM with CPU: 1) Compute Express Link (CXL): https://en.wikipedia.org/wiki/Compute_Express_Link PCIe vs. CXL for Memory and Storage: https://news.ycombinator.com/item?id=38125885

Gigabytes per second? What is this, bandwidth for ants? My years old pleb tier non-HBM GPU has more than 4 times the bandwidth you would get from a PCIe Gen 7 x16 link, which doesn't even officially exist yet.

Yes CXL will soon benefit from PCIe Gen 7 x16 with expected 64GB/s in 2025 and the non-HBM bandwidth I/O alternative is increasing rapidly by the day. For most inferences of near real-time LLM it will be feasible. For majority of SME companies and other DIY users (humans or ants) with their localized LLM should not be any issues [1],[2]. In addition new techniques for more efficient LLM are being discover to reduce the memory consumption [3].

[1] Forget ChatGPT: why researchers now run small AIs on their laptops:

https://news.ycombinator.com/item?id=41609393

[2] Welcome to LLMflation – LLM inference cost is going down fast:

https://a16z.com/llmflation-llm-inference-cost/

[3] New LLM optimization technique slashes memory costs up to 75%:

https://news.ycombinator.com/item?id=42411409

Re: Making AMD GPUs competitive for LLM inference (2023)

#62

Post by mattfrommars »

Great, I have yet to understand why does not the ML community really push or move away from CUDA? To me, it feel like a dinosaur move to build on top of CUDA which is screaming proprietary nothing about it is open source or cross platform.

The reason why I say its dinosaur is, imagine, we as a dev community continued to build on top of Flash or Microsoft Silverlight...

LLM and ML has been out for quiet a while, with AI/LLM advancement, the transition must have been much quicker to move cross platform. But this hasn't yet and not sure when it will happen.

Building a translation layer on top CUDA is not the answer either to this problem.

Re: Making AMD GPUs competitive for LLM inference (2023)

#63

Post by shiroiushi »

llm_trw wrote: post #58
ksec wrote: post #55

Earlier quoted context omitted.

Everything is comparative. AMD isn't perfect. As an Ex Shareholder I have argued they did well partly because of Intel's downfall. In terms of execution it is far from perfect. But Nvidia is a different beast. It is a bit like Apple in the late 00s where you take business, forecast, marketing, operation, software, hardware, sales etc You take any part of it and they are all industry leading. And having industry leadi…

Yeah, no. We are in the middle of a monopoly squeeze by NVidia on the most innovative part of the economy right now. I expect the DOJ to hit them harder than they did MS in the 90s given the bullshit they are pulling and the drag on the economy they are causing. By comparison if AMD could write a driver that didn't shit itself when it had to multiply more than two matrices in a row they'd be selling cards faster than…

>I expect the DOJ to hit them harder than they did MS in the 90s given the bullshit they are pulling and the drag on the economy they are causing.

It sounds like you're expecting extreme competence from the DOJ. Given their history with regulating big tech companies, and even worse, the incoming administration, I think this is a very unrealistic expectation.

Re: Making AMD GPUs competitive for LLM inference (2023)

#64

Post by dwood_dev »

mattfrommars wrote: post #62

Great, I have yet to understand why does not the ML community really push or move away from CUDA? To me, it feel like a dinosaur move to build on top of CUDA which is screaming proprietary nothing about it is open source or cross platform. The reason why I say its dinosaur is, imagine, we as a dev community continued to build on top of Flash or Microsoft Silverlight... LLM and ML has been out for quiet a while, with…

Except I never hear complaints about CUDA from a quality perspective. The complaints are always about lock in to the best GPUs on the market. The desire to shift away is to make cheaper hardware with inferior software quality more usable. Flash was an abomination, CUDA is not.

Re: Making AMD GPUs competitive for LLM inference (2023)

#65

Post by xedrac »

dwood_dev wrote: post #64
mattfrommars wrote: post #62

Great, I have yet to understand why does not the ML community really push or move away from CUDA? To me, it feel like a dinosaur move to build on top of CUDA which is screaming proprietary nothing about it is open source or cross platform. The reason why I say its dinosaur is, imagine, we as a dev community continued to build on top of Flash or Microsoft Silverlight... LLM and ML has been out for quiet a while, with…

Except I never hear complaints about CUDA from a quality perspective. The complaints are always about lock in to the best GPUs on the market. The desire to shift away is to make cheaper hardware with inferior software quality more usable. Flash was an abomination, CUDA is not.

Maybe the situation has gotten better in recent years, but my experience with Nvidia toolchains was a complete nightmare back in 2018.

Re: Making AMD GPUs competitive for LLM inference (2023)

#66

Post by idonotknowwhy »

mattfrommars wrote: post #62

Great, I have yet to understand why does not the ML community really push or move away from CUDA? To me, it feel like a dinosaur move to build on top of CUDA which is screaming proprietary nothing about it is open source or cross platform. The reason why I say its dinosaur is, imagine, we as a dev community continued to build on top of Flash or Microsoft Silverlight... LLM and ML has been out for quiet a while, with…

For me personally, hacking together projects as a hobbiest, 2 reasons :

1. It just works. When i tried to build things on Intel Arcs, i spent way more hours bikeshedding ipex and driver issues than developing

2. LLMs seem to have more cuda code in their training data. I can leverage claude and 4o to help me build things with cuda, but trying to get them to help me do the same things on ipex just doesn't work.

I'd very much love a translation layer for Cuda, like a dxvk or wine equivalent.

Would save a lot of money since Arc gpus are in the bargain bin and nvidia cloud servers are double the price of AMD.

As it stands now, my dual Intel Arc rig is now just a llama.cpp inference server for the family to use.

Re: Making AMD GPUs competitive for LLM inference (2023)

#67

Post by idonotknowwhy »

Uehreka wrote: post #44
lxe wrote: post #33

A used 3090 is $600-900, performs better than 7900, and is much more versatile because CUDA

Reality check for anyone considering this: I just got a used 3090 for $900 last month. It works great. I would not recommend buying one for $600, it probably either won’t arrive or will be broken. Someone will reply saying they got one for $600 and it works, that doesn’t mean it will happen if you do it. I’d say the market is realistically $900-1100, maybe $800 if you know the person or can watch the card running fir…

I've bought 5 used and they're all perfect. But that's what buyer protection on ebay is for. Had to send back an Epyc mobo with bent pins and ebay handled it fine.

Re: Making AMD GPUs competitive for LLM inference (2023)

#68

Post by idonotknowwhy »

Const-me wrote: post #57
boroboro4 wrote: post #39

Earlier quoted context omitted.

Only local (read batch size 1) ML inference is memory bound, production loads are pretty much compute bound. Prefill phase is very compute bound, and with continuous batching generation phase is getting mixed with prefill, which makes whole process altogether to be compute bound too. So no, tensor cores and all other shenanigans absolutely critical for performant inference infrastructure.

PyTorch is a project by Linux foundation. The about page with the mission of the foundation contains phrases like “empowering generations of open source innovators”, “democratize code”, and “removing barriers to adoption”. I would argue running local inference with batch size=1 is more useful for empowering innovators compared to running production loads on shared servers owned by companies. Local inference increases…

Most people running local inference do so thorough quants with llamacpp (which runs on everything) or awq/exl2/mlx with vllm/tabbyAPI/lmstudio which are much faster to than using pytorch directly

Re: Making AMD GPUs competitive for LLM inference (2023)

#69

Post by pinsiang »

shihab wrote: post #9

I have come across quite few startups who are trying a similar idea: break the nvidia monopoly by utilizing AMD GPUs (for inference at least): Felafax, Lamini, tensorwave (partially), SlashML. Even saw optimistic claims like CUDA moat is only 18 months deep from some of them [1]. Let's see. [1] https://www.linkedin.com/feed/update/urn:li:activity:7275885...

AMD GPUs are becoming a serious contender for LLM inference. vLLM is already showing impressive performance on AMD [1], even with consumer-grade Radeon cards (even support GGUF) [2]. This could be a game-changer for folks who want to run LLMs without shelling out for expensive NVIDIA hardware.

[1] https://blog.vllm.ai/2024/10/23/vllm-serving-amd.html [2] https://embeddedllm.com/blog/vllm-now-supports-running-gguf-...

Re: Making AMD GPUs competitive for LLM inference (2023)

#70

Post by kadoban »

llm_trw wrote: post #58
ksec wrote: post #55

Earlier quoted context omitted.

Everything is comparative. AMD isn't perfect. As an Ex Shareholder I have argued they did well partly because of Intel's downfall. In terms of execution it is far from perfect. But Nvidia is a different beast. It is a bit like Apple in the late 00s where you take business, forecast, marketing, operation, software, hardware, sales etc You take any part of it and they are all industry leading. And having industry leadi…

Yeah, no. We are in the middle of a monopoly squeeze by NVidia on the most innovative part of the economy right now. I expect the DOJ to hit them harder than they did MS in the 90s given the bullshit they are pulling and the drag on the economy they are causing. By comparison if AMD could write a driver that didn't shit itself when it had to multiply more than two matrices in a row they'd be selling cards faster than…

What effect did the DOJ have on MS in the 90s? Didn't all of that get rolled back before they had to pay a dime, and all it amounted to was that browser choice screen that was around for a while? Hardly a crippling blow. If anything that showed the weakness of regulators in fights against big tech, just outlast them and you're fine.
Post reply on HN