Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ollama now supports AMD graphics cards

ollama.com

Post reply on HN
Print view View original

81–90 of 234 posts

Re: Ollama now supports AMD graphics cards

#81

Post by reustle »

eclectic29 wrote: post #77

I'm not sure why Ollama garners so much attention. It has limited value - used for only experimenting with models + cannot support more than 1 model at a time. It's not meant for production deployments. Granted that it makes the experimentation process super easy but for something that relies on llama.cpp completely and whose main value proposition is easy model management I'm not sure it deserves the brouhaha people…

Even for just running a model locally, Ollama provided a much simpler "one click install" earlier than most tools. That in itself is worth the support.

Re: Ollama now supports AMD graphics cards

#82

Post by crooked-v »

eclectic29 wrote: post #77

I'm not sure why Ollama garners so much attention. It has limited value - used for only experimenting with models + cannot support more than 1 model at a time. It's not meant for production deployments. Granted that it makes the experimentation process super easy but for something that relies on llama.cpp completely and whose main value proposition is easy model management I'm not sure it deserves the brouhaha people…

> Granted that it makes the experimentation process super easy

That's the answer to your question. It may have less space than a Zune, but the average person doesn't care about technically superior alternatives that are much harder to use.

Re: Ollama now supports AMD graphics cards

#83

Post by davidhariri »

eclectic29 wrote: post #77

I'm not sure why Ollama garners so much attention. It has limited value - used for only experimenting with models + cannot support more than 1 model at a time. It's not meant for production deployments. Granted that it makes the experimentation process super easy but for something that relies on llama.cpp completely and whose main value proposition is easy model management I'm not sure it deserves the brouhaha people…

As it turns out, making it faster and better to manage things tends to get people’s attention. I think it’s well deserved.

Re: Ollama now supports AMD graphics cards

#84

Post by vikramkr »

eclectic29 wrote: post #77

I'm not sure why Ollama garners so much attention. It has limited value - used for only experimenting with models + cannot support more than 1 model at a time. It's not meant for production deployments. Granted that it makes the experimentation process super easy but for something that relies on llama.cpp completely and whose main value proposition is easy model management I'm not sure it deserves the brouhaha people…

It's nice for personal use which is what I think it was built for, has some nice frontend options too. The tooling around it is nice, and there are projects building in rag etc. I don't think people are intending to deploy days services through these tools

Re: Ollama now supports AMD graphics cards

#85

Post by andrewstuart »

eclectic29 wrote: post #77

I'm not sure why Ollama garners so much attention. It has limited value - used for only experimenting with models + cannot support more than 1 model at a time. It's not meant for production deployments. Granted that it makes the experimentation process super easy but for something that relies on llama.cpp completely and whose main value proposition is easy model management I'm not sure it deserves the brouhaha people…

Sounds like you are dismissing Ollama as a "toy".

Refer:

https://paulgraham.com/startupideas.html

Re: Ollama now supports AMD graphics cards

#86

Post by mypalmike »

dartos wrote: post #72
washadjeffmad wrote: post #65

Earlier quoted context omitted.

I tried it recently and couldn't figure out why it existed. It's just a very feature limited app that doesn't require you to know anything or be able to read a model card to "do AI". And that more or less answered it.

It’s because most devs nowadays are new devs and probably aren’t very familiar with native compilation. So compiling the correct version of llama.cpp for their hardware is confusing. Compound that with everyone’s relative inexperience with configuring any given model and you have prime grounds for a simple tool to exist. That’s what ollama and their Modelfiles accomplish.

Eh, I've been building native code for decades and hit quite a few roadblocks trying to get llama.cpp building with cuda support on my Ubuntu box. Library version issues and such. Ended up down a rabbit hole related to codenames for the various Nvidia architectures... It's a project on hold for now.

Weirdly, the Python bindings built without issue with pip.

Re: Ollama now supports AMD graphics cards

#87

Post by TylerE »

isoprophlex wrote: post #45
hugozap wrote: post #39

Earlier quoted context omitted.

8gb

Thanks, wow, amazing that you can already run a small model with so little ram. I need to buy a new laptop, guess more than 16 gb on a macbook isn't really needed

I've run LLMs and some of the various image models on my M1 Studio 32GB without issue. Not as fast as my old 3080 card, but considering the Mac all in has about a 5th the power draw, it's a lot closer than I expected. I'm not sure of the exact details but there is clearly some secret sauce that allows it to leverage the onboard NN hardware.

Re: Ollama now supports AMD graphics cards

#88

Post by observationist »

There's a thing somewhat conspicuous in its absence - why isn't llama.cpp more directly credited and thanked for providing the base technology powering this tool?

All the other cool "run local" software seems to have the appropriate level of credit. You can find llama.cpp references in the code, being set up in a kind of "as is" fashion such that it might be OK as far as MIT licensing goes, but it seems kind of petty to have no shout out or thank you anywhere in the repository or blog or ollama website.

GPT4All - https://gpt4all.io/index.html

LM Studio - https://lmstudio.ai/

Both of these projects credit and attribute appropriately, Ollama seems to bend over backwards so they don't have to?

Re: Ollama now supports AMD graphics cards

#89

Post by nerdix »

bornfreddy wrote: post #41
sevagh wrote: post #8

Earlier quoted context omitted.

Spoiler alert: not good enough to break CUDA's moat

Not sure why you're downvoted, but as far as I've heard AMD cards can't beat 4090 - yet. Still, I think AMD will catch or overtake NVidia in hardware soon, but software is a bigger problem. Hopefully the opensource strategy will pay off for them.

A RTX 4090 is about twice the price of and 50%-ish faster than AMD's most expensive consumer card so I'm not sure anyone really expects it to ever surpass a 4090.

A 7900 XTX beating a RTX 4080 at inference is probably a more realistic goal though I'm not sure how they compare right now.

Re: Ollama now supports AMD graphics cards

#90

Post by jart »

observationist wrote: post #88

There's a thing somewhat conspicuous in its absence - why isn't llama.cpp more directly credited and thanked for providing the base technology powering this tool? All the other cool "run local" software seems to have the appropriate level of credit. You can find llama.cpp references in the code, being set up in a kind of "as is" fashion such that it might be OK as far as MIT licensing goes, but it seems kind of petty…

ollama has made a lot of nice contributions of their own. It's a good look to give a hat tip to the great work llama.cpp is also doing, but they're strictly speaking not required to do that in their advertising any more than llama.cpp is required to give credit to Google Brain, and I think that's because llama.cpp has pulled off tricks in the execution that Brain never could have accomplished, just as ollama has had great success focusing on things that wouldn't make sense for llama.cpp. Besides everyone who wants to know what's up can read the source code, research papers, etc. then make their own judgements about who's who. It's all in the open.
Post reply on HN