Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AMD GPU Inference

github.com

Post reply on HN
Print view View original

51–60 of 102 posts

Re: AMD GPU Inference

#51

Post by qamononep »

lhl wrote: post #4

For inference, if you have a supported card (or probably architecture if you are on Linux and can use HSA_OVERRIDE_GFX_VERSION), then you can probably run anything with (upstream) PyTorch and transformers. Also, compiling llama.cpp is has been pretty trouble-free for me for at least a year. (If you are on Windows, there is usually a win-hip binary of llama.cpp in the project's releases or if things totally refuse to…

It would be great if you included a section on running with Docker on Linux. The only one that worked out of the box was Ollama, and it had an example. https://github.com/ollama/ollama/blob/main/docs/docker.md

has a docker image but no examples to run it https://github.com/ggerganov/llama.cpp/blob/master/docs/dock...

has a docker image but no examples to run it https://github.com/LostRuins/koboldcpp?tab=readme-ov-file#do...

docker image was broken for me on 7800xt running rhel9 https://github.com/Atinoda/text-generation-webui-docker

Re: AMD GPU Inference

#52

Post by freeqaz »

elorant wrote: post #35
freeqaz wrote: post #15

What's the best bang-for-your-buck AMD GPU these days? I just bought 2 used 3090s for $750ish refurb'd on eBay. Curious what others are using for running LLMs locally.

Probably the 7900xtx. $1k for 24GB of RAM.

That's about the same price as a 3090 and it's also 24GB. Are they faster at inference?

Re: AMD GPU Inference

#53

Post by elorant »

freeqaz wrote: post #52
elorant wrote: post #35

Earlier quoted context omitted.

Probably the 7900xtx. $1k for 24GB of RAM.

That's about the same price as a 3090 and it's also 24GB. Are they faster at inference?

I doubt it, but the 3090 is a four year old card which means it might have a lot of mileage from the previous owner. A lot of them are from mining rigs.

Re: AMD GPU Inference

#54

Post by andyferris »

steeve wrote: post #46
tomxor wrote: post #8

I almost tried to install AMD rocm a while ago after discovering the simplicity of llamafile. sudo apt install rocm Summary: Upgrading: 0, Installing: 203, Removing: 0, Not Upgrading: 0 Download size: 2,369 MB / 2,371 MB Space needed: 35.7 GB / 822 GB available I don't understand how 36 GB can be justified for what amounts to a GPU driver.

You can look us up at https://github.com/zml/zml , we fix that.

Wait, looking at that link I don't see how it avoids downloading CUDA or ROCM. Do you use MLIR to compile to GPU without using the vendor provided tooling at all?

Re: AMD GPU Inference

#55

Post by fazkan »

qamononep wrote: post #51
lhl wrote: post #4

For inference, if you have a supported card (or probably architecture if you are on Linux and can use HSA_OVERRIDE_GFX_VERSION), then you can probably run anything with (upstream) PyTorch and transformers. Also, compiling llama.cpp is has been pretty trouble-free for me for at least a year. (If you are on Windows, there is usually a win-hip binary of llama.cpp in the project's releases or if things totally refuse to…

It would be great if you included a section on running with Docker on Linux. The only one that worked out of the box was Ollama, and it had an example. https://github.com/ollama/ollama/blob/main/docs/docker.md has a docker image but no examples to run it https://github.com/ggerganov/llama.cpp/blob/master/docs/dock... has a docker image but no examples to run it https://github.com/LostRuins/koboldcpp?tab=readme-ov-fil…

good feedback thanks, would you be able to open an issue

Re: AMD GPU Inference

#56

Post by slavik81 »

a2128 wrote: post #2

It seems to use an old, 2 year old version of ROCm (5.4.2) which I'm doubtful would support my RX 7900 XTX. I personally found it easiest to just use the latest `rocm/pytorch` image and run what I need from there

The RX 7900 XTX (gfx1100) was first enabled in the math libraries (e.g. rocBLAS) for ROCm 5.4, but I don't think the AI libraries (e.g. MIOpen) had it enabled until ROCm 5.5. I believe the performance improved significantly in later releases, as well.

Re: AMD GPU Inference

#59

Post by bongodongobob »

leonheld wrote: post #5

People use "Docker-based" all the time but what they mean is that they ship $SOFTWARE in a Docker image. "Docker-based" reads, to me, as if you were doing Inference on AMD cards with Docker somehow, which doesn't make sense.

Yeah, they're using docker to wrap up the software packages, which is what Docker is used for. I don't understand why that confuses you or what you think Docker is otherwise used for.

Re: AMD GPU Inference

#60

Post by tcdent »

The rise of generated slop ml libraries is staggering.

This library is 50% print statements. And where it does branch, it doesn't even need to.

Defines two environment variables and sets two flags on torch.

Post reply on HN