AMD GPU Inference • Hacker News Board

Re: AMD GPU Inference

#91

Post by tcdent » Thu, Oct 03, 2024, 3:45 PM UTC

tpoacher wrote: post #86

mdaniel wrote: post #78

Earlier quoted context omitted.

I also had to go to therapy to cure myself of the misunderstanding that data scientists and machine learning folks are software engineers, and expecting the same work product from those disparate audiences only raises your blood pressure Expectation management is a huge part of any team/organization, I think

They can be the same or different, given how you define them. People throw these words around with little thought, especially ones superficial to or outside the field. I wouldn't disparage an entire field for lack of a clear definition in the buzzwords people use to refer to it.

As suggested by my original comment, I read the source of a lot of the open source projects around ML.

mdaniel is absolutely correct. They are not software engineers.

I'm not vocal about the naïve stuff, poor design, sloppy formatting, bad english. I am vocal about projects that have no place in the ecosystem.

Re: AMD GPU Inference

#92

Post by lenova » Thu, Oct 03, 2024, 3:56 PM UTC

Why ROCm 5.4, and not the latest (6.2)?

https://github.com/slashml/amd_inference/blob/main/Dockerfil...

Also looks like the Docker image provided by this project doesn't successfully build: https://github.com/slashml/amd_inference/issues/2

Re: AMD GPU Inference

#93

Post by skirmish » Thu, Oct 03, 2024, 4:53 PM UTC

anthonix1 wrote: post #67

skirmish wrote: post #49

Earlier quoted context omitted.

They prefer you pay $3,600 for AMD Radeon Pro W7900, 48GB VRAM.

... which also has a much lower power cap

Not that much lower, 295W vs 355W, and for LLM inference VRAM bandwidth is the main bottleneck. But the price is ridiculous.

Re: AMD GPU Inference

#94

Post by fazkan » Thu, Oct 03, 2024, 5:13 PM UTC

lenova wrote: post #92

Why ROCm 5.4, and not the latest (6.2)? https://github.com/slashml/amd_inference/blob/main/Dockerfil... Also looks like the Docker image provided by this project doesn't successfully build: https://github.com/slashml/amd_inference/issues/2

tested with the old Rocm, created a PR with the latest.

Re: AMD GPU Inference

#95

Post by FeepingCreature » Thu, Oct 03, 2024, 6:16 PM UTC

horsebridge wrote: post #84

sangnoir wrote: post #50

Earlier quoted context omitted.

When dealing with ROCM, it's critical that once you have a working configuration, you freeze everything in place (except your application). Docker is one way to achieve this if your host machine is subject to kernel or package updates

I don't really have any problem with ROCm these days, although I only use system packages. It used to be quite wonky though, and I've totally given up on custom ROCm installs.

Same here on my 7900 XTX. Used to be terrible, now it's (seemingly) fine (for now).

Re: AMD GPU Inference

#96

Post by steeve » Thu, Oct 03, 2024, 6:24 PM UTC

andyferris wrote: post #54

steeve wrote: post #46

Earlier quoted context omitted.

You can look us up at https://github.com/zml/zml , we fix that.

Wait, looking at that link I don't see how it avoids downloading CUDA or ROCM. Do you use MLIR to compile to GPU without using the vendor provided tooling at all?

We do use ROCm and CUDA. Only we sandbox it with the model and download only the needed parts which are about 1/10th of the size.

Re: AMD GPU Inference

#97

Post by TechDebtDevin » Thu, Oct 03, 2024, 7:24 PM UTC

tcdent wrote: post #60

The rise of generated slop ml libraries is staggering. This library is 50% print statements. And where it does branch, it doesn't even need to. Defines two environment variables and sets two flags on torch.

While I see where you are coming from, these are the types of comments that keep people from sharing their code, contributing to OSS or continuing to program in general.

Re: AMD GPU Inference

#98

Post by saagarjha » Fri, Oct 04, 2024, 6:48 AM UTC

EmilyHATFIELD wrote: post #90

saagarjha wrote: post #75

Earlier quoted context omitted.

Ok so like four of those files literally just do matrix multiplications

"just"

Ok some of them do tensor contractions too my bad

Re: AMD GPU Inference

#99

Post by slavik81 » Fri, Oct 04, 2024, 7:01 AM UTC

ekianjo wrote: post #72

slavik81 wrote: post #62

On Ubuntu 24.04 (and Debian Unstable¹), the OS-provided packages should be able to get llama.cpp running on ROCm on just about any discrete AMD GPU from Vega onwards²³⁴. No docker or HSA_OVERRIDE_GFX_VERSION required. The performance might not be ideal in every case⁵, but I've tested a wide variety of cards: # install dependencies sudo apt -y update sudo apt -y upgrade sudo apt -y install git wget hipcc libhipblas-de…

are there AMD cards with more than 24GB VRAM on the market right now at consumer friendly prices?

The Radeon Pro W6800, W7800 or W7900 would be the standard answer. A hacker-spirited alternative would be to purchase a used MI50, MI60 or MI100 and 3d print a fan adapter. There are versions of all of those cards with 32GB of VRAM and they can be found on ebay for between 350 USD and 1200 USD. Plus twenty bucks for a fan adapter and a fan.

Those old gfx906 or gfx908 cards are more competitive for fp64 than for low-precision AI workloads, but they have the memory and the price is right. I'm not sure I would recommend the hacker approach to the average user, but it is what I've done for some of the continuous integration servers I host for the Debian project.

Re: AMD GPU Inference

#100

Post by latchkey » Fri, Oct 04, 2024, 5:09 PM UTC

steeve wrote: post #38

leonheld wrote: post #5

People use "Docker-based" all the time but what they mean is that they ship $SOFTWARE in a Docker image. "Docker-based" reads, to me, as if you were doing Inference on AMD cards with Docker somehow, which doesn't make sense.

Hi, we (ZML), fix that: https://github.com/zml/zml

Works out of the box on our MI300x. Fantastic work steeve!

https://x.com/HotAisle/status/1842245896085356949