Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AMD GPU Inference

github.com

Post reply on HN
Print view View original

81–90 of 102 posts

Re: AMD GPU Inference

#81

Post by snep »

freeqaz wrote: post #15

What's the best bang-for-your-buck AMD GPU these days? I just bought 2 used 3090s for $750ish refurb'd on eBay. Curious what others are using for running LLMs locally.

I bought an MI100 recently for $650. 32GB of HBM2 and it performs around around 0-5% faster than a 3090 on the default flash attention 2 benchmarks. Performance on actual applications can be mixed though, as many are not well optimised for CDNA's matrix cores - even where work has been done for RDNA, which is not that often, it doesn't necessarily carry over. It's also frustrating when efforts to improve performance get turned back by maintainers: llama.cpp closing PR for flash attention on AMD because the requisite (header-only) lib is supposedly adding an unneeded dependency (https://github.com/ggerganov/llama.cpp/pull/7011).

There's also a few tricks/updates I'd like to try which may improve performance, e.g. hipblaslt support being added next rocm release - of course these are "maybes".

To give you a rough idea of practical performance, default SDXL with xformers is around 4.5-5it/s (between 3090 and 4090 from my understanding), and exllamav2 with qwen 72B at 3bpw is around 7t/s (slower than a 3090, though a 3090 has to use a lower precision to fit).

As others have pointed out, I can't really see what this project offers for AMD users over existing options like llama.cpp, exllamav2, mlc-ai, etc. Most projects work relatively easily these days.

Re: AMD GPU Inference

#82

Post by Chordless »

phkahler wrote: post #30

Does it work with an APU? I just put 64GB in my system and gonna drop in a 5700G. Will that be enough? SFF inference if so.

I'm able to run Ollama and llama.cpp on my Ryzen 4600G APU following this guide: https://agieverywhere.com/apuguide/AMDAPU/APU_Linux

Your APU should be similar, just faster.

There are some magic environment variables you want to set to get ROCM to work with this technically unsupported APU: HSA_OVERRIDE_GFX_VERSION=9.0.0 HSA_ENABLE_SDMA=0

Performance is not great, but slightly better than running inference on the CPU, with the bonus that your CPU is essentially free for other tasks even while running LLMs.

Re: AMD GPU Inference

#83

Post by danielEM »

It has been like 8 months since I got Ryzen 8700G with NPU just for the purpose of inferencing NN, and so far only acceleration I'm getting is through vulkan on iGPU, not NPU (I'm using Linux only). On the bright side, with 64GB of RAM had no isues with trying models over 32GB. Kudos to llama.cpp for supporting vulkan backend!

Re: AMD GPU Inference

#84

Post by horsebridge »

sangnoir wrote: post #50
3abiton wrote: post #16

Earlier quoted context omitted.

Personal experience: It's not even worth it. AMD (i)GPU breaks with every pytorch, ROCm, xformers, or ollama updates. You'll sleep more compfortably at night.

When dealing with ROCM, it's critical that once you have a working configuration, you freeze everything in place (except your application). Docker is one way to achieve this if your host machine is subject to kernel or package updates

I don't really have any problem with ROCm these days, although I only use system packages. It used to be quite wonky though, and I've totally given up on custom ROCm installs.

Re: AMD GPU Inference

#86

Post by tpoacher »

mdaniel wrote: post #78
tcdent wrote: post #60

The rise of generated slop ml libraries is staggering. This library is 50% print statements. And where it does branch, it doesn't even need to. Defines two environment variables and sets two flags on torch.

I also had to go to therapy to cure myself of the misunderstanding that data scientists and machine learning folks are software engineers, and expecting the same work product from those disparate audiences only raises your blood pressure Expectation management is a huge part of any team/organization, I think

They can be the same or different, given how you define them. People throw these words around with little thought, especially ones superficial to or outside the field.

I wouldn't disparage an entire field for lack of a clear definition in the buzzwords people use to refer to it.

Re: AMD GPU Inference

#87

Post by lhl »

danielEM wrote: post #83

It has been like 8 months since I got Ryzen 8700G with NPU just for the purpose of inferencing NN, and so far only acceleration I'm getting is through vulkan on iGPU, not NPU (I'm using Linux only). On the bright side, with 64GB of RAM had no isues with trying models over 32GB. Kudos to llama.cpp for supporting vulkan backend!

You should have ROCm/HIP support on the iGPU as well, be sure to compile llama.cpp w/ the LLAMA_HIP_UMA=1 flag. If you take a look at https://github.com/amd/RyzenAI-SW you can see there's a fair amount of software to play with on the NPU now, but Phoenix is only 16 TOPS, so I've never bothered testing it.

Re: AMD GPU Inference

#88

Post by driverdan »

tcdent wrote: post #60

The rise of generated slop ml libraries is staggering. This library is 50% print statements. And where it does branch, it doesn't even need to. Defines two environment variables and sets two flags on torch.

I thought you were being overly harsh until I looked at the repo. You're not kidding, there's very little to it.

Re: AMD GPU Inference

#90

Post by EmilyHATFIELD »

saagarjha wrote: post #75
lhl wrote: post #63

Earlier quoted context omitted.

Here's the big files in my /opt/rocm/lib which is most of it: 4.8G hipblaslt 1.6G libdevice_conv_operations.a 2.0G libdevice_gemm_operations.a 1.4G libMIOpen.so.1.0.60200 1.1G librocblas.so.4.2.60200 1.6G librocsolver.so.0.2.60200 1.4G librocsparse.so.1.0.60200 1.5G llvm 3.5G rocblas 2.0G rocfft The biggest one just to pick on one is hipblaslt is "a library that provides general matrix-matrix operations. It has a fle…

Ok so like four of those files literally just do matrix multiplications

"just"
Post reply on HN