Does it work with GGUF files?
AMD GPU Inference
71–80 of 102 posts
Re: AMD GPU Inference
#72On Ubuntu 24.04 (and Debian Unstable¹), the OS-provided packages should be able to get llama.cpp running on ROCm on just about any discrete AMD GPU from Vega onwards²³⁴. No docker or HSA_OVERRIDE_GFX_VERSION required. The performance might not be ideal in every case⁵, but I've tested a wide variety of cards: # install dependencies sudo apt -y update sudo apt -y upgrade sudo apt -y install git wget hipcc libhipblas-de…
are there AMD cards with more than 24GB VRAM on the market right now at consumer friendly prices?
Re: AMD GPU Inference
#73What's the best bang-for-your-buck AMD GPU these days? I just bought 2 used 3090s for $750ish refurb'd on eBay. Curious what others are using for running LLMs locally.
Personal experience: It's not even worth it. AMD (i)GPU breaks with every pytorch, ROCm, xformers, or ollama updates. You'll sleep more compfortably at night.
It's not the experience I have. I've been using ollama for 6 months on mine and never had any issues with ROCm breaking.
Re: AMD GPU Inference
#74What's the best bang-for-your-buck AMD GPU these days? I just bought 2 used 3090s for $750ish refurb'd on eBay. Curious what others are using for running LLMs locally.
I got my radeon pro vii for €300 new. Was not a bad deal IMO especially since it comes with HBM2 and has the same memory bandwidth as the 4090 (1TB/s). It's got only 16GB though.
Re: AMD GPU Inference
#75Earlier quoted context omitted.
I suspected that, but any binaries being that large just seems wrong, I mean the whole thing is 35 time larger than my entire OS install. Do you know what is included in ROCm that could be so big? Does it include training datasets or something?
Here's the big files in my /opt/rocm/lib which is most of it: 4.8G hipblaslt 1.6G libdevice_conv_operations.a 2.0G libdevice_gemm_operations.a 1.4G libMIOpen.so.1.0.60200 1.1G librocblas.so.4.2.60200 1.6G librocsolver.so.0.2.60200 1.4G librocsparse.so.1.0.60200 1.5G llvm 3.5G rocblas 2.0G rocfft The biggest one just to pick on one is hipblaslt is "a library that provides general matrix-matrix operations. It has a fle…
Ok so like four of those files literally just do matrix multiplications
Re: AMD GPU Inference
#76On Ubuntu 24.04 (and Debian Unstable¹), the OS-provided packages should be able to get llama.cpp running on ROCm on just about any discrete AMD GPU from Vega onwards²³⁴. No docker or HSA_OVERRIDE_GFX_VERSION required. The performance might not be ideal in every case⁵, but I've tested a wide variety of cards: # install dependencies sudo apt -y update sudo apt -y upgrade sudo apt -y install git wget hipcc libhipblas-de…
are there AMD cards with more than 24GB VRAM on the market right now at consumer friendly prices?
It sort of depends on how you define "consumer friendly prices". AFAIK, in the $1000 - "slightly over or under $1000" range, 24GB is all you can get. But there are Radeon Pro boards with 32GB or 48GB of RAM for various prices between around $2000 to about $3500. So not "cheap" but possibly within reach for a serious hobbyist who doesn't mind spending a little bit more.
Re: AMD GPU Inference
#77If you're interested in how much the AMD graphics cards cost compared to the NVidia ones, I have https://gpuquicklist.com/ which gives you a quick table view of lowest prices available on Amazon that I can find.
Re: AMD GPU Inference
#78The rise of generated slop ml libraries is staggering. This library is 50% print statements. And where it does branch, it doesn't even need to. Defines two environment variables and sets two flags on torch.
I also had to go to therapy to cure myself of the misunderstanding that data scientists and machine learning folks are software engineers, and expecting the same work product from those disparate audiences only raises your blood pressure
Expectation management is a huge part of any team/organization, I think
Re: AMD GPU Inference
#79Earlier quoted context omitted.
related: https://www.nonbios.ai/post/deploying-large-405b-models-in-f... tldr: uses the latest rocm 6.2 to run full precision inference for llama 405b on a single node 8 x MI300x AMD GPU How mature do you think Rocm 6.2-AMD stack is compared to Nvidia ?
this uses vllm?
Yes.
Re: AMD GPU Inference
#80On Ubuntu 24.04 (and Debian Unstable¹), the OS-provided packages should be able to get llama.cpp running on ROCm on just about any discrete AMD GPU from Vega onwards²³⁴. No docker or HSA_OVERRIDE_GFX_VERSION required. The performance might not be ideal in every case⁵, but I've tested a wide variety of cards: # install dependencies sudo apt -y update sudo apt -y upgrade sudo apt -y install git wget hipcc libhipblas-de…
are there AMD cards with more than 24GB VRAM on the market right now at consumer friendly prices?
Amazon prices:
$3,600 - 61 TFLOPS - AMD Radeon Pro W7900
$4,200 - 38.7 TFLOPS - NVidia RTX A6000 48GB Ampere
$7,200 - 91.1 TFLOPS - NVidia RTX A6000 48GB Ada