Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Making AMD GPUs competitive for LLM inference (2023)

blog.mlc.ai

Post reply on HN
Print view View original

131–140 of 221 posts

Re: Making AMD GPUs competitive for LLM inference (2023)

#131

Post by moffkalast »

lhl wrote: post #123
moffkalast wrote: post #119

Earlier quoted context omitted.

I've recently been trying to get IPEX working as well, apparently picking Ubuntu 24.04 was a mistake, because while things compile, everything fails at runtime. I've tried native, docker, different oneAPI versions, threw away a solid week of afternoons for nothing. SYCL with llama.cpp is great though, at least at FP16 since it supports nothing else but even Arc iGPUs easily give 2-4x performance compared to CPU infer…

My testing has been w/ a Lunar Lake Core 258V chip (Xe2 - Arc 140V) on Arch Linux. It sounds like you've tried a lot of things already, but case it helps, my notes for installing llama.cpp and PyTorch: https://llm-tracker.info/howto/Intel-GPUs I have some benchmarks as well, and the IPEX-LLM backend performed a fair bit better than the SYCL llama.cpp backend for me (almost +50% pp512 and almost 2X tg128) so worth get…

Well that's funny, I think we already spoke on Reddit. I'm the guy who was testing the 125H recently. I guess there's like 5 of us who have intel hardware in total and we keep running into each other :P

Honestly I think there's just something seriously broken with the way IPEX expects the GPU driver to be on 24.04 and there's nothing I can really do about it except wait for them to fix it if I want to keep using this OS.

I am vaguely considering adding another drive and installing 22.04 or 20.04 with the exact kernel they want to see if that might finally work in the meantime, but honestly I'm fairly satisfied with the speed I get from SYCL already. The problem is more that it's annoying to integrate it directly through the server endpoint, every projects expects a damn ollama api or llama-cpp-python these days and I'm a fan of neither since it's just another layer of headaches to get those compiled with SYCL.

> I found a lot of stuff in that repo that depend on versions of oneKit that aren't even downloadable on Intel's site. I couldn't help but smirk when I heard someone say "Intel has their software nailed down."

Yeah well the fact that oneAPI 2025 got released, broke IPEX, and they still haven't figured out a way to patch it for months makes me think it's total chaos internally, where teams work against each other instead of talking and coordinating.

Re: Making AMD GPUs competitive for LLM inference (2023)

#132

Post by highwaylights »

mpreda wrote: post #125
llm_trw wrote: post #97

Earlier quoted context omitted.

The problem is that the specs of AMD consumer-grade GPUs do not translate to computer performance when you try and chain more than one together. I have 7 NVidia 4090s under my desk happily chugging along on week long training runs. I once managed to get a Radeon VII to run for six hours without shitting itself.

> I have 7 NVidia 4090s under my desk I have 6 Radeon Pro VII under my desk (in a single system BTW), and they run hard for weeks until I choose to reboot e.g. for Linux kernel updates. I bought them "new old stock" for $300 apiece. So that's $1800 for all six.

How does the compute performance compare to 4090’s for these workloads?

(I release it will be significantly lower, just try to get as much of a comparison as is possible).

Re: Making AMD GPUs competitive for LLM inference (2023)

#133

Post by MrBuddyCasino »

pinsiang wrote: post #69
shihab wrote: post #9

I have come across quite few startups who are trying a similar idea: break the nvidia monopoly by utilizing AMD GPUs (for inference at least): Felafax, Lamini, tensorwave (partially), SlashML. Even saw optimistic claims like CUDA moat is only 18 months deep from some of them [1]. Let's see. [1] https://www.linkedin.com/feed/update/urn:li:activity:7275885...

AMD GPUs are becoming a serious contender for LLM inference. vLLM is already showing impressive performance on AMD [1], even with consumer-grade Radeon cards (even support GGUF) [2]. This could be a game-changer for folks who want to run LLMs without shelling out for expensive NVIDIA hardware. [1] https://blog.vllm.ai/2024/10/23/vllm-serving-amd.html [2] https://embeddedllm.com/blog/vllm-now-supports-running-gguf-...

Fun fact: Nvidia H200 are currently half the price/hr of H100 bc people can’t get vLLM to work on it.

https://x.com/nisten/status/1871325538335486049

Re: Making AMD GPUs competitive for LLM inference (2023)

#134

Post by cainxinth »

highwaylights wrote: post #132
mpreda wrote: post #125

Earlier quoted context omitted.

> I have 7 NVidia 4090s under my desk I have 6 Radeon Pro VII under my desk (in a single system BTW), and they run hard for weeks until I choose to reboot e.g. for Linux kernel updates. I bought them "new old stock" for $300 apiece. So that's $1800 for all six.

How does the compute performance compare to 4090’s for these workloads? (I release it will be significantly lower, just try to get as much of a comparison as is possible).

The 4090 offers 82.58 teraflops of single-precision performance compared to the Radeon Pro VII's 13.06 teraflops.

Re: Making AMD GPUs competitive for LLM inference (2023)

#135

Post by bitmasher9 »

kimixa wrote: post #101
rockskon wrote: post #71

Earlier quoted context omitted.

They've actually been making positive moves with GPUs lately along with a success story for the B580.

B580 being a "success" is purely a business decision as a loss leader to get their name into the market. A larger die on a newer node than either Nvidia or AMD means their per-unit costs are higher, and are selling it at a lower price. That's not a long-term success strategy. Maybe good for getting your name in the conversation, but not sustainable.

It’s a long term strategy to release a hardware platform with minimal margins in the beginning to attract software support needed for long term viability.

One of the benefits of being Intel.

Re: Making AMD GPUs competitive for LLM inference (2023)

#136

Post by adrian_b »

cainxinth wrote: post #134
highwaylights wrote: post #132

Earlier quoted context omitted.

How does the compute performance compare to 4090’s for these workloads? (I release it will be significantly lower, just try to get as much of a comparison as is possible).

The 4090 offers 82.58 teraflops of single-precision performance compared to the Radeon Pro VII's 13.06 teraflops.

On the other hand, for double precision a Radeon Pro VII is many times faster than a RTX 4090 (due to 1:2 vs. 1:64 FP64:FP32 ratio).

Moreover, for workloads limited by the memory bandwidth, a Radeon Pro VII and a RTX 4090 will have about the same speed, regardless what kind of computations are performed. It is said that speed limitation by memory bandwidth happens frequently for ML/AI inferencing.

Re: Making AMD GPUs competitive for LLM inference (2023)

#137

Post by melodyogonna »

Modular claims that it achieves 93% GPU utilization on AMD GPUs [1], official preview release coming early next year, we'll see. I must say I'm bullish because of feedback I've seen people give about the performance on Nvidia GPUs

1.https://www.modular.com/max

Re: Making AMD GPUs competitive for LLM inference (2023)

#138

Post by crest »

highwaylights wrote: post #132
mpreda wrote: post #125

Earlier quoted context omitted.

> I have 7 NVidia 4090s under my desk I have 6 Radeon Pro VII under my desk (in a single system BTW), and they run hard for weeks until I choose to reboot e.g. for Linux kernel updates. I bought them "new old stock" for $300 apiece. So that's $1800 for all six.

How does the compute performance compare to 4090’s for these workloads? (I release it will be significantly lower, just try to get as much of a comparison as is possible).

The Radeon VII is special compared to most older (and current) affordable GPUs in that it used HBM giving it memory bandwidth comparable to modern cards ~1TB/s and has reasonable FP64 (1:4) throughput instead of (1:64). So this card can still be pretty interesting for running memory bandwidth intensive FP64 workloads. Anything affordable afterward by either AMD or Nvidia crippled realistic FP64 throughput to below what a AVX-512 many-core CPU can do.

Re: Making AMD GPUs competitive for LLM inference (2023)

#139

Post by adrian_b »

MrBuddyCasino wrote: post #133
pinsiang wrote: post #69

Earlier quoted context omitted.

AMD GPUs are becoming a serious contender for LLM inference. vLLM is already showing impressive performance on AMD [1], even with consumer-grade Radeon cards (even support GGUF) [2]. This could be a game-changer for folks who want to run LLMs without shelling out for expensive NVIDIA hardware. [1] https://blog.vllm.ai/2024/10/23/vllm-serving-amd.html [2] https://embeddedllm.com/blog/vllm-now-supports-running-gguf-...

Fun fact: Nvidia H200 are currently half the price/hr of H100 bc people can’t get vLLM to work on it. https://x.com/nisten/status/1871325538335486049

That seems like a CPU problem, not a GPU problem (due to Aarch64 replacing x86-64).

Re: Making AMD GPUs competitive for LLM inference (2023)

#140

Post by adrian_b »

Dylan16807 wrote: post #79
schmidtleonard wrote: post #38

Earlier quoted context omitted.

Gigabytes per second? What is this, bandwidth for ants? My years old pleb tier non-HBM GPU has more than 4 times the bandwidth you would get from a PCIe Gen 7 x16 link, which doesn't even officially exist yet.

> 4 times the bandwidth you would get from a PCIe Gen 7 x16 link So you have a full terabyte per second of bandwidth? What GPU is that? (The 64GB/s number is an x4 link. If you meant you have over four times that, then it sounds like CXL would be pretty competitive.)

Already an ancient Radeon VII from 5 years ago had 1 terabyte per second of memory bandwidth.

Later consumer GPUs have regressed and only RTX 4090 offers the same memory bandwidth in the current NVIDIA generation.

Post reply on HN