Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Llama.cpp supports Vulkan. why doesn't Ollama?

github.com

Post reply on HN
Print view View original

71–80 of 235 posts

Re: Llama.cpp supports Vulkan. why doesn't Ollama?

#71

Post by himhckr »

zophiana wrote: post #21
turnsout wrote: post #8

This is going to sound like a troll, but it's an honest question: Why do people use Ollama over llama.cpp? llama.cpp has added a ton of features, is about as user-friendly as Ollama, and is higher-performance. Is there some key differentiator for Ollama that I'm missing?

Honestly I just didn't know it was this easy to use, maybe because of the name... But ramalama seems to be a full replacement for ollama

ramalama still needs users to be able to install docker first, no? That’s a barrier to entry for many users esp. Windows where I have had my struggles running Docker not to mention a massive resource hog.

Re: Llama.cpp supports Vulkan. why doesn't Ollama?

#72

Post by zozbot234 »

buyucu wrote: post #2

llama.cpp has supported vulkan for more than a year now. For more than 6 months now there has been an open PR to add vulkan backend support for Ollama. However, Ollama team has not even looked at it or commented on it. Vulkan backends are existential for running LLMs on consumer hardware (iGPUs especially). It's sad to see Ollama miss this opportunity.

The PR has been legitimately out-of-date and unmergeable for many months. It was forward-ported a few weeks ago, and is now still awaiting formal review and merging. (To be sure, Vulkan support in Ollama will likely stay experimental for some time even if the existing PR is merged, and many setups will need manual adjustment of the number of GPU layers and such. It's far from 100% foolproof even in the best-case scenario!)

For that matter, some people are still having issues building and running it, as seen from the latest comments on the linked GitHub page. It's not clear that it's even in a fully reviewable state just yet.

Re: Llama.cpp supports Vulkan. why doesn't Ollama?

#73

Post by zozbot234 »

buyucu wrote: post #31
paradite wrote: post #13

Could it be that supporting multiple platforms open up more support tickets and adds more work to keep the software working on those new platforms? As someone who built apps for Windows, Linux, macOS, iOS and Android, it is not trivial to ensure your new features or updates work on all platforms, and you have to deal with deprecations.

ollama is not doing anything. llama cpp does all that work. ollama is just a small wrapper on top.

This is not quite correct. Ollama must assess the state of Vulkan support and amount of available memory, then pick the fraction of the model to be hosted on GPU. This is not totally foolproof and will likely always need manual adjustment in some cases.

Re: Llama.cpp supports Vulkan. why doesn't Ollama?

#74

Post by buyucu »

2-3-7-43-1807 wrote: post #68

can someone please give a quick summary of the criticism towards ollama? as far as my intel goes it's a mozilla project shouldered mostly by one 10x programmer. i found ollama through hn and last time i didn't notice any lack of trust or suspected sketchiness ... so what changed?

ollama has been advertising llama.cpp features as their own, which I find very dishonest.

Re: Llama.cpp supports Vulkan. why doesn't Ollama?

#75

Post by diggan »

7thpower wrote: post #24
the_mitsuhiko wrote: post #15

Ollama needs competition. I’m not sure what drives the people that maintain it but some of their actions imply that there are ulterior motives at play that do not have the benefit of their users in mind. However such projects require a lot of time and effort and it’s not clear if this project can be forked and kept alive.

Can you please explain why you think they may be operating in bad faith?

Not parent, but same feeling.

First I got the feeling because of how they store things on disk and try to get all models rehosted in their own closed library.

Second time I got the feeling is when it's not obvious at all about what their motives are, and that it's a for-profit venture.

Third time is trying to discuss things in their Discord and the moderators there constantly shut down a lot of conversation citing "Misinformation" and rewrites your messages. You can ask a honest question, it gets deleted and you get blocked for a day.

Just today I asked why the R1 models they're shipping that are the distilled ones, doesn't have "distilled" in the name, or even any way of knowing which tag is which model, and got the answer "if you don't like how things are done on Ollama, you can run your own object registry" which doesn't exactly inspire confidence.

Another thing I noticed after a while is that there are bunch of people with zero knowledge of terminals that want to run Ollama, even though Ollama is a project for developers (since you do need to know how to run a terminal). Just making the messaging clearer would help a lot in this regarding, but somehow the Ollama team thinks thats gatekeeping and it's better to teach people basic terminal operations.

Re: Llama.cpp supports Vulkan. why doesn't Ollama?

#76

Post by fkyoureadthedoc »

buyucu wrote: post #57
paradite wrote: post #51

Earlier quoted context omitted.

Ok assuming what you said is correct, why wouldn't Ollama then be able to support Vulkan by default out of the box? Sorry I'm not sure what's the relationship exactly between the two projects. This is a genuine questions, not a troll question.

check the PR, it's a very short one. It's not more complicated than setting a compile time flag. I have no idea why they have been ignoring it. Ollama is just a friendly front end for llama.cpp. It doesn't have to do any of those things you mentioned. Llama.cpp does all that.

If it's "just" a friendly front and, why doesn't llama.cpp just drop one themselves? Do they actually care about the situation, or are random people just mad on their behalf?

Re: Llama.cpp supports Vulkan. why doesn't Ollama?

#77

Post by jdright »

andy_ppp wrote: post #12
Kubuxu wrote: post #3

Earlier quoted context omitted.

Don’t be sad for commercial entity that is not a good player https://github.com/ggerganov/llama.cpp/pull/11016#issuecomme...

This is great, I did not know about RamaLama and I'll be using and recommending that in future and if I see people using Ollama in instructions I'll recommend they move to RamaLama in the future. Cheers.

Yeah, I would love an actual alternative to Ollama, but RamaLama is not it unfortunately. As the other commenter said, onboarding is important. I just want one operation install and it needs to work and the simple fact RamaLama is written in Python, assures it will never be that easy, and this is even more true with LLM stuff when using AMD gpu.

I know there will be people that disagree with this, that's ok. This is my personal experience with Python in general, and 10x worse when I need to figure out all compatible packages with specifc ROCm support for my GPU. This is madness, even C and C++ setup and build is easier than this Python hell.

Re: Llama.cpp supports Vulkan. why doesn't Ollama?

#78

Post by buyucu »

zozbot234 wrote: post #73
buyucu wrote: post #31

Earlier quoted context omitted.

ollama is not doing anything. llama cpp does all that work. ollama is just a small wrapper on top.

This is not quite correct. Ollama must assess the state of Vulkan support and amount of available memory, then pick the fraction of the model to be hosted on GPU. This is not totally foolproof and will likely always need manual adjustment in some cases.

the work involved is tiny compared to the work llama.cpp did to get vulkan up and running.

this is not rocket science.

Re: Llama.cpp supports Vulkan. why doesn't Ollama?

#79

Post by bloomingkales »

buyucu wrote: post #28
cedws wrote: post #11

The way Ollama has basically been laundering llama.cpp’s features as its own felt dodgy, this appears to confirm there’s something underhanded going on.

I did not assume the worst when submitting the post, but that is also my suspicion. The whole thing is very dodgy.

Are closer to the metal AI developers an under tracked bottle neck? AMD and Intel can barely get off the ground due to lagging software developers.

Re: Llama.cpp supports Vulkan. why doesn't Ollama?

#80

Post by ethbr1 »

moffkalast wrote: post #64
cedws wrote: post #11

The way Ollama has basically been laundering llama.cpp’s features as its own felt dodgy, this appears to confirm there’s something underhanded going on.

Ollama is a private for profit company, of course there's something shady going on.

Ollama is a private for profit AI company, of course there's something shady going on.

Because apparently you can take unethical business practices, add AI, and suddenly it's a whole new thing that no one can judge!

Post reply on HN