Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ollama now supports AMD graphics cards

ollama.com

Post reply on HN
Print view View original

71–80 of 234 posts

Re: Ollama now supports AMD graphics cards

#71

Post by mey »

freedomben wrote: post #50

I'm thrilled to see support for RX 6800/6800 XT / 6900 XT. I bought one of those for an outrageous amount during the post-covid shortage in hopes that I could use it for ML stuff, and thus far it hasn't been very successful, which is a shame because it's a beast of a card! Many thanks to ollama project and llama.cpp!

Sad to see that the cut off is just after 6700 XT which is what is in my desktop. They indicate more devices are coming, hopefully that includes some of the more modern all in one chips with RDNA 2/3 from AMD as well.

Re: Ollama now supports AMD graphics cards

#72

Post by dartos »

washadjeffmad wrote: post #65
refulgentis wrote: post #12

Earlier quoted context omitted.

They're open source and based on llama.cpp so nothings secret. My money, looking at nothing, would be on one of the two Vulkan backends added in Jan/Feb. I continue to be flummoxed by a mostly-programmer-forum treating ollama like a magical new commercial entity breaking new ground. It's a CLI wrapper around llama.cpp so you don't have to figure out how to compile it

I tried it recently and couldn't figure out why it existed. It's just a very feature limited app that doesn't require you to know anything or be able to read a model card to "do AI". And that more or less answered it.

It’s because most devs nowadays are new devs and probably aren’t very familiar with native compilation.

So compiling the correct version of llama.cpp for their hardware is confusing.

Compound that with everyone’s relative inexperience with configuring any given model and you have prime grounds for a simple tool to exist.

That’s what ollama and their Modelfiles accomplish.

Re: Ollama now supports AMD graphics cards

#74

Post by dartos »

isoprophlex wrote: post #45
hugozap wrote: post #39

Earlier quoted context omitted.

8gb

Thanks, wow, amazing that you can already run a small model with so little ram. I need to buy a new laptop, guess more than 16 gb on a macbook isn't really needed

Mistral is _very_ small when quantized.

I’d still go with 16gbs

Re: Ollama now supports AMD graphics cards

#75

Post by refulgentis »

washadjeffmad wrote: post #65
refulgentis wrote: post #12

Earlier quoted context omitted.

They're open source and based on llama.cpp so nothings secret. My money, looking at nothing, would be on one of the two Vulkan backends added in Jan/Feb. I continue to be flummoxed by a mostly-programmer-forum treating ollama like a magical new commercial entity breaking new ground. It's a CLI wrapper around llama.cpp so you don't have to figure out how to compile it

I tried it recently and couldn't figure out why it existed. It's just a very feature limited app that doesn't require you to know anything or be able to read a model card to "do AI". And that more or less answered it.

Edited it out of my original comment because I didn't want to seem ranty/angry/like I have some personal vendatta, as opposed to just being extremely puzzled, but it legit took me months to realize it wasn't a GUI because of how it's discussed on HN, i.e. as key to democratizing, as a large, unique, entity, etc.

Hadn't thought about it recently. After seeing it again here, and being gobsmacked by the # of genuine, earnest, comments assuming there's extensive independent development of large pieces going on in it, I'm going with:

- "The puzzled feeling you have is simply because llama.cpp is a challenge on the best of days, you need to know a lot to get to fully accelerated on ye average MacBook. and technical users don't want a GUI for an LLM, they want a way to call an API, so that's why there isn't content extalling the virtues of GPT4All*. So TL;DR you're old and have been on computer too much :P"

but I legit don't know and still can't figure it out.

* picked them because they're the most recent example of a genuinely democratizing tool that goes far beyond llama.cpp and also makes large contributions back to llama.cpp, ex. GPT4All landed 1 of the 2 vulkan backends

Re: Ollama now supports AMD graphics cards

#76

Post by mey »

mey wrote: post #71
freedomben wrote: post #50

I'm thrilled to see support for RX 6800/6800 XT / 6900 XT. I bought one of those for an outrageous amount during the post-covid shortage in hopes that I could use it for ML stuff, and thus far it hasn't been very successful, which is a shame because it's a beast of a card! Many thanks to ollama project and llama.cpp!

Sad to see that the cut off is just after 6700 XT which is what is in my desktop. They indicate more devices are coming, hopefully that includes some of the more modern all in one chips with RDNA 2/3 from AMD as well.

It appears that the cut off lines up with HIP SDK support from AMD, https://rocm.docs.amd.com/projects/install-on-windows/en/lat...

Re: Ollama now supports AMD graphics cards

#77

Post by eclectic29 »

I'm not sure why Ollama garners so much attention. It has limited value - used for only experimenting with models + cannot support more than 1 model at a time. It's not meant for production deployments. Granted that it makes the experimentation process super easy but for something that relies on llama.cpp completely and whose main value proposition is easy model management I'm not sure it deserves the brouhaha people are giving it.

Edit: what do you do after the initial experimentation? you need to deploy these models eventually to production. I'm not even talking about giving credit to llama.cpp, just mentioning that this product is gaining disproportionate attention and kudos compared to the value it delivers. Not denying that it's a great product.

Re: Ollama now supports AMD graphics cards

#78

Post by JonChesterfield »

skipants wrote: post #57
harwoodr wrote: post #14

Earlier quoted context omitted.

ROCm: https://github.com/ollama/ollama/commit/6c5ccb11f993ccc88c47...

Another giveaway that it's ROCm is that it doesn't support the 5700 series... I'm really salty because I "upgraded" to a 5700XT from a Nvidia GTX 1070 and can't do AI on the GPU anymore, purely because the software is unsupported. But, as a dev, I suppose I should feel some empathy that there's probably some really difficult problem causing 5700XT to be unsupported by ROCm.

I wrote a bunch of openmp code on a 5700XT a couple of years ago, if you're building from source it'll probably run fine

Re: Ollama now supports AMD graphics cards

#79

Post by tracerbulletx »

dartos wrote: post #72
washadjeffmad wrote: post #65

Earlier quoted context omitted.

I tried it recently and couldn't figure out why it existed. It's just a very feature limited app that doesn't require you to know anything or be able to read a model card to "do AI". And that more or less answered it.

It’s because most devs nowadays are new devs and probably aren’t very familiar with native compilation. So compiling the correct version of llama.cpp for their hardware is confusing. Compound that with everyone’s relative inexperience with configuring any given model and you have prime grounds for a simple tool to exist. That’s what ollama and their Modelfiles accomplish.

It's just because it's convenient. I wrote a rich text editor front end for llama.cpp and I originally wrote a quick go web server with streaming using the go bindings, but now I just use ollama because it's just simpler and the workflow for pulling down models with their registry and packaging new ones in containers is simpler. Also most people who want to play around with local models aren't developers at all.
Post reply on HN