Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ollama now supports AMD graphics cards

ollama.com

Post reply on HN
Print view View original

11–20 of 234 posts

Re: Ollama now supports AMD graphics cards

#11

Post by ixaxaar »

sofixa wrote: post #2

This is great news. The more projects do this, the less of a moat CUDA is, and the less of a competitive advantage Nvidia has.

Hey I did, and sorry for the self promo,

Please check out https://github.com/geniusrise - tool for running llms and other stuff, behaves like docker compose, works with whatever is supported by underlying engines:

Huggingface - MPS, cuda VLLM - cuda, ROCm llama.cpp, whisper.cpp - cuda, mps, rocm

Also coming up integration with spark (TorchDistributor), kafka and airflow.

Re: Ollama now supports AMD graphics cards

#12

Post by refulgentis »

Apofis wrote: post #9
reilly3000 wrote: post #6

I’m curious as to how they pulled this off. OpenCL isn’t that common in the wild relative to Cuda. Hopefully it can become robust and widespread soon enough. I personally succumbed to the pressure and spent a relative fortune on a 4090 but wish I had some choice in the matter.

I'm surprised they didn't speak about the implementation at all. Anyone got more intel?

They're open source and based on llama.cpp so nothings secret.

My money, looking at nothing, would be on one of the two Vulkan backends added in Jan/Feb.

I continue to be flummoxed by a mostly-programmer-forum treating ollama like a magical new commercial entity breaking new ground.

It's a CLI wrapper around llama.cpp so you don't have to figure out how to compile it

Re: Ollama now supports AMD graphics cards

#13

Post by j33zusjuice »

Apofis wrote: post #9
reilly3000 wrote: post #6

I’m curious as to how they pulled this off. OpenCL isn’t that common in the wild relative to Cuda. Hopefully it can become robust and widespread soon enough. I personally succumbed to the pressure and spent a relative fortune on a 4090 but wish I had some choice in the matter.

I'm surprised they didn't speak about the implementation at all. Anyone got more intel?

Ahhhh, I see what you did there.

Re: Ollama now supports AMD graphics cards

#14

Post by harwoodr »

Apofis wrote: post #9
reilly3000 wrote: post #6

I’m curious as to how they pulled this off. OpenCL isn’t that common in the wild relative to Cuda. Hopefully it can become robust and widespread soon enough. I personally succumbed to the pressure and spent a relative fortune on a 4090 but wish I had some choice in the matter.

I'm surprised they didn't speak about the implementation at all. Anyone got more intel?

ROCm: https://github.com/ollama/ollama/commit/6c5ccb11f993ccc88c47...

Re: Ollama now supports AMD graphics cards

#15

Post by choilive »

deadalus wrote: post #7

I wish AMD did well in the Stable Diffusion front because AMD is never greedy on VRAM. The 4060Ti 16GB(minimum required for Stable Diffusion in 2024) starts at $450. AMD with ROCm is decent on Linux but pretty bad on Windows.

They bump up VRAM because they can't compete on raw compute.

Re: Ollama now supports AMD graphics cards

#16

Post by KronisLV »

Feels like all of this local LLM stuff is definitely pushing people in the direction of getting new hardware, since nothing like RX 570/580 or other older cards sees support.

On one hand, the hardware nowadays is better and more powerful, but on the other, the initial version of CUDA came out in 2007 and ROCm in 2016. You'd think that compute on GPUs wouldn't require the latest cards.

Re: Ollama now supports AMD graphics cards

#17

Post by Symmetry »

simon83 wrote: post #10

Does anyone know how the AMD consumer GPU support on Linux has been implemented? Must use something else than ROCm I assume? Because ROCm only supports the 7900 XTX on Linux[1], while on Windows[2] support is from RX 6600 and upwards. [1]: https://rocblas.readthedocs.io/en/rocm-6.0.0/about/compatibi... [2]: https://rocblas.readthedocs.io/en/rocm-6.0.0/about/compatibi...

The newest release, 6.0.2, supports a number of other cards[1] and in general people are able to get a lot more cards to work than are officially supported. My 7900 XT worked on 6.0.0 for instance.

[1]https://rocm.docs.amd.com/projects/install-on-linux/en/lates...

Re: Ollama now supports AMD graphics cards

#18

Post by Zambyte »

It's pretty funny to see this blog post, when I have been running Ollama on my AMD RX 6650 for weeks :D

They have shipped ROCm containers since 0.1.27 (21 days ago). This blog post seems to be published along with the latest release, 0.1.29. I wonder what they actually changed in this release with regards to AMD support.

Also: see this issue[0] that I made where I worked through running Ollama on an AMD card that they don't "officially" support yet. It's just a matter of setting an environment variable.

[0] https://github.com/ollama/ollama/issues/2870

Edit: I did notice one change, now the starcoder2[1] model works now. Before that would crash[2].

[1] https://ollama.com/library/starcoder2

[2] https://github.com/ollama/ollama/issues/2953

Re: Ollama now supports AMD graphics cards

#19

Post by Adverblessly »

deadalus wrote: post #7

I wish AMD did well in the Stable Diffusion front because AMD is never greedy on VRAM. The 4060Ti 16GB(minimum required for Stable Diffusion in 2024) starts at $450. AMD with ROCm is decent on Linux but pretty bad on Windows.

I run A1111, ComfyUI and kohya-ss on an AMD (6900XT which has 16GB, the minimum required for Stable Diffusion in 2024 ;)), though on Linux. Is it a Windows specific Issue for you?

Edit to add: Though apparently I still don't run ollama on AMD since it seems to disagree with my setup.

Re: Ollama now supports AMD graphics cards

#20

Post by SushiHippie »

Can anyone (maybe ollama contributors) explain to me the relationship between llama.cpp and ollama?

I always thought that ollama basically just was a wrapper (i.e. not much changes to inference code, and only built on top) around llama.cpp, but this makes it seem like it is more than that?

Post reply on HN