Ollama now supports AMD graphics cards • Hacker News Board

Re: Ollama now supports AMD graphics cards

#61

Post by karmakaze » Fri, Mar 15, 2024, 7:47 PM UTC

reilly3000 wrote: post #6

I’m curious as to how they pulled this off. OpenCL isn’t that common in the wild relative to Cuda. Hopefully it can become robust and widespread soon enough. I personally succumbed to the pressure and spent a relative fortune on a 4090 but wish I had some choice in the matter.

It would serve Nvidia right if their insistence on only running CUDA workloads on their hardware results in adoption of ROCm/OpenCL.

Re: Ollama now supports AMD graphics cards

#62

Post by taminka » Fri, Mar 15, 2024, 7:47 PM UTC

moffkalast wrote: post #47

reilly3000 wrote: post #6

I’m curious as to how they pulled this off. OpenCL isn’t that common in the wild relative to Cuda. Hopefully it can become robust and widespread soon enough. I personally succumbed to the pressure and spent a relative fortune on a 4090 but wish I had some choice in the matter.

OpenCL is as dead as OpenGL and the inference implementations that exist are very unperformant. The only real options are CUDA, ROCm, Vulkan and CPU. And Vulkan is a proper pain too, takes forver to build compute shaders and has to do so for each model. It only makes sense on Intel Arc since there's nothing else there.

why though? except for apple, most vendors still actively support it and newer versions of OpenCL are released…

Re: Ollama now supports AMD graphics cards

#63

Post by shmerl » Fri, Mar 15, 2024, 7:53 PM UTC

reilly3000 wrote: post #6

I’m curious as to how they pulled this off. OpenCL isn’t that common in the wild relative to Cuda. Hopefully it can become robust and widespread soon enough. I personally succumbed to the pressure and spent a relative fortune on a 4090 but wish I had some choice in the matter.

May be Vulkan compute? But yeah, interesting how.

Re: Ollama now supports AMD graphics cards

#64

Post by KeplerBoy » Fri, Mar 15, 2024, 7:53 PM UTC

karmakaze wrote: post #61

reilly3000 wrote: post #6

I’m curious as to how they pulled this off. OpenCL isn’t that common in the wild relative to Cuda. Hopefully it can become robust and widespread soon enough. I personally succumbed to the pressure and spent a relative fortune on a 4090 but wish I had some choice in the matter.

It would serve Nvidia right if their insistence on only running CUDA workloads on their hardware results in adoption of ROCm/OpenCL.

OpenCL is fine on Nvidia Hardware. Of course it's a second class citizen next to CUDA, but then again everything is a second class citizen on AMD hardware.

Re: Ollama now supports AMD graphics cards

#65

Post by washadjeffmad » Fri, Mar 15, 2024, 7:55 PM UTC

refulgentis wrote: post #12

Apofis wrote: post #9

Earlier quoted context omitted.

I'm surprised they didn't speak about the implementation at all. Anyone got more intel?

They're open source and based on llama.cpp so nothings secret. My money, looking at nothing, would be on one of the two Vulkan backends added in Jan/Feb. I continue to be flummoxed by a mostly-programmer-forum treating ollama like a magical new commercial entity breaking new ground. It's a CLI wrapper around llama.cpp so you don't have to figure out how to compile it

I tried it recently and couldn't figure out why it existed. It's just a very feature limited app that doesn't require you to know anything or be able to read a model card to "do AI".

And that more or less answered it.

Re: Ollama now supports AMD graphics cards

#66

Post by rahimnathwani » Fri, Mar 15, 2024, 7:57 PM UTC

Here is the commit that added ROCm support to llama.cpp back in August:

https://github.com/ggerganov/llama.cpp/commit/6bbc598a632560...

Re: Ollama now supports AMD graphics cards

#67

Post by tarruda » Fri, Mar 15, 2024, 7:58 PM UTC

Does this work with integrated Radeon graphics? If so it might be worth getting one of those Ryzen mini PCs to use as a local LLM server.

Re: Ollama now supports AMD graphics cards

#68

Post by renewiltord » Fri, Mar 15, 2024, 8:00 PM UTC

duskwuff wrote: post #60

nottorp wrote: post #29

Earlier quoted context omitted.

Used crypto mining parts not available any more?

Crypto mining didn't require significant bandwidth to the card. Mining-oriented motherboards typically only provisioned a single lane of PCIe to each card, and often used anemic host CPUs (like Celeron embedded parts).

Exactly. They'd use PCIe x1 to PCIe x16 risers with power adapters. These require high-bandwidth.

Re: Ollama now supports AMD graphics cards

#69

Post by SparkyMcUnicorn » Fri, Mar 15, 2024, 8:02 PM UTC

isoprophlex wrote: post #45

hugozap wrote: post #39

Earlier quoted context omitted.

8gb

Thanks, wow, amazing that you can already run a small model with so little ram. I need to buy a new laptop, guess more than 16 gb on a macbook isn't really needed

I would advise getting as much RAM as you possibly can. You can't upgrade later, so get as much as you can afford.

Mine is 64GB, and my memory pressure goes into the red when running a quantized 70B model with a dozen Chrome tabs open.

Re: Ollama now supports AMD graphics cards

#70

Post by aseipp » Fri, Mar 15, 2024, 8:02 PM UTC

karmakaze wrote: post #61

reilly3000 wrote: post #6

I’m curious as to how they pulled this off. OpenCL isn’t that common in the wild relative to Cuda. Hopefully it can become robust and widespread soon enough. I personally succumbed to the pressure and spent a relative fortune on a 4090 but wish I had some choice in the matter.

It would serve Nvidia right if their insistence on only running CUDA workloads on their hardware results in adoption of ROCm/OpenCL.

You can use OpenCL just fine on Nvidia, but CUDA is just a superior compute programming model overall (both in features and design.) Pretty much every vendor offers something superior to OpenCL (HIP, OneAPI, etc), because it simply isn't very nice to use.