AMD GPU Inference • Hacker News Board

Re: AMD GPU Inference

#11

Post by karamanolev » Wed, Oct 02, 2024, 5:56 PM UTC

khurdula wrote: post #6

Are we supposed to use AMD GPUs for this to work? Or Does it work on any GPU?

> This project provides a Docker-based inference engine for running Large Language Models (LLMs) on AMD GPUs.

First sentence of the README in the repo. Was it somehow unclear?

Re: AMD GPU Inference

#12

Post by sandGorgon » Wed, Oct 02, 2024, 5:59 PM UTC

is anyone using the new HX370 based laptops for any LLM work ? i mean the ipex-llm libraries of Intel's new Lunar Lake is already supporting Llama 3.2 (https://www.intel.com/content/www/us/en/developer/articles/t...), but AMD's new Zen5 chips dont seem to be much active here.

Re: AMD GPU Inference

#13

Post by dhruvdh » Wed, Oct 02, 2024, 6:02 PM UTC

Why would you use this over vLLM?

Re: AMD GPU Inference

#14

Post by white_waluigi » Wed, Oct 02, 2024, 6:06 PM UTC

Isn't this just a wrapper for huggingface-transformers?

Re: AMD GPU Inference

#15

Post by freeqaz » Wed, Oct 02, 2024, 6:07 PM UTC

What's the best bang-for-your-buck AMD GPU these days? I just bought 2 used 3090s for $750ish refurb'd on eBay. Curious what others are using for running LLMs locally.

Re: AMD GPU Inference

#16

Post by 3abiton » Wed, Oct 02, 2024, 6:11 PM UTC

freeqaz wrote: post #15

What's the best bang-for-your-buck AMD GPU these days? I just bought 2 used 3090s for $750ish refurb'd on eBay. Curious what others are using for running LLMs locally.

Personal experience: It's not even worth it. AMD (i)GPU breaks with every pytorch, ROCm, xformers, or ollama updates. You'll sleep more compfortably at night.

Re: AMD GPU Inference

#17

Post by woodrowbarlow » Wed, Oct 02, 2024, 6:14 PM UTC

lhl wrote: post #4

For inference, if you have a supported card (or probably architecture if you are on Linux and can use HSA_OVERRIDE_GFX_VERSION), then you can probably run anything with (upstream) PyTorch and transformers. Also, compiling llama.cpp is has been pretty trouble-free for me for at least a year. (If you are on Windows, there is usually a win-hip binary of llama.cpp in the project's releases or if things totally refuse to…

i also have been playing with inference on the amd 7900xtx, and i agree. there are no hoops to jump through these days. just make sure to install the rocm version of torch (if using a1111 or similar, don't trust requirements.txt), as shown clearly on the pytorch homepage. obsidian is a similar story. hip is straightforward, at least on arch and ubuntu (fedora still requires some twiddling, though). i didn't realize xformers is also functional! that's good news.

Re: AMD GPU Inference

#18

Post by stefan_ » Wed, Oct 02, 2024, 6:15 PM UTC

This seems to be some AI generated wrapper around a wrapper of a wrapper.

> # Other AMD-specific optimizations can be added here

> # For example, you might want to set specific flags or use AMD-optimized libraries

What are we doing here, then?

Re: AMD GPU Inference

#19

Post by fazkan » Wed, Oct 02, 2024, 6:25 PM UTC

white_waluigi wrote: post #14

Isn't this just a wrapper for huggingface-transformers?

yes, but handles all the dependencies for AMD architecture. So technically its just a requirements file :). Author of the repo above.

Re: AMD GPU Inference

#20

Post by greenavocado » Wed, Oct 02, 2024, 6:25 PM UTC

tomxor wrote: post #8

I almost tried to install AMD rocm a while ago after discovering the simplicity of llamafile. sudo apt install rocm Summary: Upgrading: 0, Installing: 203, Removing: 0, Not Upgrading: 0 Download size: 2,369 MB / 2,371 MB Space needed: 35.7 GB / 822 GB available I don't understand how 36 GB can be justified for what amounts to a GPU driver.

It's not just you; AMD manages to completely shit-up the Linux kernel with their drivers: https://www.phoronix.com/news/AMD-5-Million-Lines