Ask HN: Replacing Nvidia GPU with AMD

#1

Post by textcortex » Sun, Dec 24, 2023, 10:55 PM UTC

Dear community I am currently exploring the feasibility of replacing Nvidia GPUs with AMD GPUs for LLM inference. Can anyone share their experience or point me to any relevant research on the performance differences between these GPUs for this task? Are there any particular hardware or software limitations that may affect the feasibility of such a switch? Thank you for your insights!

Re: Ask HN: Replacing Nvidia GPU with AMD

#2

Post by InitEnabler » Sun, Dec 24, 2023, 11:15 PM UTC

I'd take a look ROCm docs: https://www.amd.com/en/developer/resources/rocm-hub.html

Re: Ask HN: Replacing Nvidia GPU with AMD

#3

Post by textcortex » Sun, Dec 24, 2023, 11:44 PM UTC

InitEnabler wrote: post #2

I'd take a look ROCm docs: https://www.amd.com/en/developer/resources/rocm-hub.html

I heard lot of bad things about ROCm, hope things improved since then.

Re: Ask HN: Replacing Nvidia GPU with AMD

#4

Post by gvd » Mon, Dec 25, 2023, 1:44 AM UTC

textcortex wrote: post #3

InitEnabler wrote: post #2

I'd take a look ROCm docs: https://www.amd.com/en/developer/resources/rocm-hub.html

I heard lot of bad things about ROCm, hope things improved since then.

TGI has ROCm support.

Re: Ask HN: Replacing Nvidia GPU with AMD

#5

Post by Havoc » Mon, Dec 25, 2023, 2:10 AM UTC

My understanding is inference on latest amds (79*) is mostly fine but training is still shaky

Re: Ask HN: Replacing Nvidia GPU with AMD

#6

Post by LorenDB » Mon, Dec 25, 2023, 2:30 AM UTC

Ollama just had AMD support merged. I haven't got it working with my 6700 XT eGPU yet, but I anticipate getting there soon.

Re: Ask HN: Replacing Nvidia GPU with AMD

#7

Post by cjbprime » Mon, Dec 25, 2023, 2:39 AM UTC

Since the amount of VRAM is low (max 24GB), one question to look into before investing might be whether there's support for chaining multiple cards together.

Re: Ask HN: Replacing Nvidia GPU with AMD

#8

Post by htrp » Mon, Dec 25, 2023, 3:38 AM UTC

You'll be fine for inference but probably struggle to run any large models requiring multi gpu ops.

Re: Ask HN: Replacing Nvidia GPU with AMD

#9

Post by textcortex » Mon, Dec 25, 2023, 12:32 PM UTC

LorenDB wrote: post #6

Ollama just had AMD support merged. I haven't got it working with my 6700 XT eGPU yet, but I anticipate getting there soon.

Awesome! Looking into it. Thanks

Re: Ask HN: Replacing Nvidia GPU with AMD

#10

Post by textcortex » Mon, Dec 25, 2023, 12:38 PM UTC

cjbprime wrote: post #7

Since the amount of VRAM is low (max 24GB), one question to look into before investing might be whether there's support for chaining multiple cards together.

I would be interested in running inference on Instinct GPUs(MI250 with 128gb)BUT I can’t find any cloud provider to spin up a machine. It seems they are not yet available or cloud providers are not interested in supporting AMD hardware..