Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AMD GPU Inference

github.com

Post reply on HN
Print view View original

41–50 of 102 posts

Re: AMD GPU Inference

#41

Post by fazkan »

Const-me wrote: post #34
phkahler wrote: post #30

Does it work with an APU? I just put 64GB in my system and gonna drop in a 5700G. Will that be enough? SFF inference if so.

The integrated GPU of the 5700G uses old architecture from 2017, this one: https://en.wikipedia.org/wiki/Radeon_RX_Vega_series Pretty sure it does not support ROCm. BTW if you just want to play with a local LLM, you can try my old port of Mistral: https://github.com/Const-me/Cgml/tree/master/Mistral/Mistral... Unlike CUDA or ROCm my port is based on Direct3D 11 GPU API, runs on all GPUs regardless of the brand.

@Const-me according to this it should work, https://github.com/ROCm/ROCm/issues/2216

Re: AMD GPU Inference

#42

Post by dhruvdh »

fazkan wrote: post #27
dhruvdh wrote: post #13

Why would you use this over vLLM?

we have vllm in certin production instances, it is a pain for most non-nvidia related architectures. A bit of digging around and we realized that most of it is just a wrapper on top of pytorch function calls. If we can do away with batch processing with vllm supports, we can be good, this is what we did here.

Batching is how you get ~350 tokens/sec on Qwen 14b on vLLM (7900XTX). By running 15 requests at once.

Also, there is a Dockerfile.rocm at the root of vLLM's repo. How is it a pain?

Re: AMD GPU Inference

#43

Post by burnte »

tomxor wrote: post #8

I almost tried to install AMD rocm a while ago after discovering the simplicity of llamafile. sudo apt install rocm Summary: Upgrading: 0, Installing: 203, Removing: 0, Not Upgrading: 0 Download size: 2,369 MB / 2,371 MB Space needed: 35.7 GB / 822 GB available I don't understand how 36 GB can be justified for what amounts to a GPU driver.

CPU drivers are complete OSes that run on the GPUs now.

Re: AMD GPU Inference

#44

Post by tannhaeuser »

mikepurvis wrote: post #9
leonheld wrote: post #5

People use "Docker-based" all the time but what they mean is that they ship $SOFTWARE in a Docker image. "Docker-based" reads, to me, as if you were doing Inference on AMD cards with Docker somehow, which doesn't make sense.

Docker became part of the standard toolkit for ML because deploying Python that links to underlying system libraries is a gong show unless you ship that layer too.

Even Docker doesn't guarantee reproducible results due to sensitivity towards host GPU drivers, and ML frontends/integrations bringing their own "helpful" newby-friendly all-in-one dependency checks and updater services.

Re: AMD GPU Inference

#45

Post by steeve »

fazkan wrote: post #39
steeve wrote: post #38

Earlier quoted context omitted.

Hi, we (ZML), fix that: https://github.com/zml/zml

This is pretty cool. Is there a document that shows which AMD drivers are supported out of the box?

We are in line with ROCm 6.2 support. We actually just opened a PR to bump to 6.2.2: https://github.com/zml/zml/pull/39

Re: AMD GPU Inference

#46

Post by steeve »

tomxor wrote: post #8

I almost tried to install AMD rocm a while ago after discovering the simplicity of llamafile. sudo apt install rocm Summary: Upgrading: 0, Installing: 203, Removing: 0, Not Upgrading: 0 Download size: 2,369 MB / 2,371 MB Space needed: 35.7 GB / 822 GB available I don't understand how 36 GB can be justified for what amounts to a GPU driver.

You can look us up at https://github.com/zml/zml, we fix that.

Re: AMD GPU Inference

#47

Post by fazkan »

dhruvdh wrote: post #42
fazkan wrote: post #27

Earlier quoted context omitted.

we have vllm in certin production instances, it is a pain for most non-nvidia related architectures. A bit of digging around and we realized that most of it is just a wrapper on top of pytorch function calls. If we can do away with batch processing with vllm supports, we can be good, this is what we did here.

Batching is how you get ~350 tokens/sec on Qwen 14b on vLLM (7900XTX). By running 15 requests at once. Also, there is a Dockerfile.rocm at the root of vLLM's repo. How is it a pain?

driver mismatch issues, we mostly use publicly available instances, so the drivers change as the instances change, according to their base image. Not saying it won't work, but it was more painful to figure out vllm, than to write a simple inference script and do it ourselves.

Re: AMD GPU Inference

#48

Post by skirmish »

tomxor wrote: post #37
atq2119 wrote: post #33

Earlier quoted context omitted.

So no doubt modern software is ridiculously bloated, but ROCm isn't just a GPU driver. It includes all sorts of tools and libraries as well. By comparison, if you go and download the CUDA toolkit as a single file, you get a download file that's over 4GB, so quite a bit larger than the download size you quoted. I haven't checked how much that expands to (it seems the ROCm install has a lot of redundancy given how well…

I suspected that, but any binaries being that large just seems wrong, I mean the whole thing is 35 time larger than my entire OS install. Do you know what is included in ROCm that could be so big? Does it include training datasets or something?

My understanding is that ROCm contains all included kernels for each supported architecture, so it would have (made up):

  -- matrix multiply 2048x2048 for Navi 31,
  -- same for Navi 32,
  -- same for Navi 33,
  -- same for Navi 21,
  -- same for Navi 22,
  -- same for Navi 23,
  -- same for Navi 24, etc.
  -- matrix multiply 4096x4096 for Navi 31,
  -- ...

Re: AMD GPU Inference

#50

Post by sangnoir »

3abiton wrote: post #16
freeqaz wrote: post #15

What's the best bang-for-your-buck AMD GPU these days? I just bought 2 used 3090s for $750ish refurb'd on eBay. Curious what others are using for running LLMs locally.

Personal experience: It's not even worth it. AMD (i)GPU breaks with every pytorch, ROCm, xformers, or ollama updates. You'll sleep more compfortably at night.

When dealing with ROCM, it's critical that once you have a working configuration, you freeze everything in place (except your application). Docker is one way to achieve this if your host machine is subject to kernel or package updates
Post reply on HN