Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ollama now supports AMD graphics cards

ollama.com

Post reply on HN
Print view View original

21–30 of 234 posts

Re: Ollama now supports AMD graphics cards

#21

Post by bravetraveler »

simon83 wrote: post #10

Does anyone know how the AMD consumer GPU support on Linux has been implemented? Must use something else than ROCm I assume? Because ROCm only supports the 7900 XTX on Linux[1], while on Windows[2] support is from RX 6600 and upwards. [1]: https://rocblas.readthedocs.io/en/rocm-6.0.0/about/compatibi... [2]: https://rocblas.readthedocs.io/en/rocm-6.0.0/about/compatibi...

I wouldn't read too much into support. It's more in terms of business/warranty/promises than what can actually do things

I've had a 6900XT since launch and this is the first I'm hearing "unsupported", having played with ROCM plenty over the years with Fedora Linux.

I think, at most, it's taken a couple key environment variables

Re: Ollama now supports AMD graphics cards

#22

Post by Zambyte »

SushiHippie wrote: post #20

Can anyone (maybe ollama contributors) explain to me the relationship between llama.cpp and ollama? I always thought that ollama basically just was a wrapper (i.e. not much changes to inference code, and only built on top) around llama.cpp, but this makes it seem like it is more than that?

llama.cpp also supports AMD cards: https://github.com/ggerganov/llama.cpp?tab=readme-ov-file#hi...

Re: Ollama now supports AMD graphics cards

#23

Post by hugozap »

KronisLV wrote: post #16

Feels like all of this local LLM stuff is definitely pushing people in the direction of getting new hardware, since nothing like RX 570/580 or other older cards sees support. On one hand, the hardware nowadays is better and more powerful, but on the other, the initial version of CUDA came out in 2007 and ROCm in 2016. You'd think that compute on GPUs wouldn't require the latest cards.

I'm a happy user of Mistral on my Mac Air M1.

Re: Ollama now supports AMD graphics cards

#25

Post by programmarchy »

reilly3000 wrote: post #6

I’m curious as to how they pulled this off. OpenCL isn’t that common in the wild relative to Cuda. Hopefully it can become robust and widespread soon enough. I personally succumbed to the pressure and spent a relative fortune on a 4090 but wish I had some choice in the matter.

Apple killed off OpenCL for their platforms when they created Metal which was disappointing. Sounds like ROCm will keep it alive but the fragmentation sucks. Gotta support CUDA, OpenCL, and Metal now to be cross-platform.

Re: Ollama now supports AMD graphics cards

#27

Post by mysteria »

KronisLV wrote: post #16

Feels like all of this local LLM stuff is definitely pushing people in the direction of getting new hardware, since nothing like RX 570/580 or other older cards sees support. On one hand, the hardware nowadays is better and more powerful, but on the other, the initial version of CUDA came out in 2007 and ROCm in 2016. You'd think that compute on GPUs wouldn't require the latest cards.

The Ollama backend llama.cpp definitely supports those older cards with the OpenCL and Vulkan backends, though performance is worse than ROCm or CUDA. In their Vulkan thread for instance I see people getting it working with Polaris and even Hawaii cards.

https://github.com/ggerganov/llama.cpp/pull/2059

Personally I just run it on CPU and several tokens/s is good enough for my purposes.

Re: Ollama now supports AMD graphics cards

#29

Post by nottorp »

renewiltord wrote: post #24

Wow, that's a huge feature. Thank you, guys. By the way, does anyone have a preferred case where they can put 4 AMD 7900XTX? There's a lot of motherboards and CPUs that support 128 lanes. It's the physical arrangement that I have trouble with.

Used crypto mining parts not available any more?

Re: Ollama now supports AMD graphics cards

#30

Post by isoprophlex »

hugozap wrote: post #23
KronisLV wrote: post #16

Feels like all of this local LLM stuff is definitely pushing people in the direction of getting new hardware, since nothing like RX 570/580 or other older cards sees support. On one hand, the hardware nowadays is better and more powerful, but on the other, the initial version of CUDA came out in 2007 and ROCm in 2016. You'd think that compute on GPUs wouldn't require the latest cards.

I'm a happy user of Mistral on my Mac Air M1.

How many gbs of RAM do you have in your M1 machine?
Post reply on HN