Live data from Hacker News

Llama.cpp supports Vulkan. why doesn't Ollama?

github.com

41–50 of 235 posts

Re: Llama.cpp supports Vulkan. why doesn't Ollama?

#41

Ollama needs competition. I’m not sure what drives the people that maintain it but some of their actions imply that there are ulterior motives at play that do not have the benefit of their users in mind. However such projects require a lot of time and effort and it’s not clear if this project can be forked and kept alive.

LM Studio has been around for a long time and does a lot of similar things but with a more UI-based approach. I used to use it before Ollama, and seems it's still going strong. https://lmstudio.ai/

Re: Llama.cpp supports Vulkan. why doesn't Ollama?

#42
post #33
post #29

Ollama is sketchy enough that I run it in a VM. Which is odd because it would probably take less effort to just run Llama.cpp directly, but VMs are pretty easy so just went that route. When I see people bring up the sketchiness most of the time the creator responds with the equivalent of shrugs , which imo increases the sketchiness.

ollama advertising llama.cpp features as their own is very dishonest in my opinion.

That’s the curse and blessing of open source I guess? I have billion dollar companies running my oss software without giving me anything - but do I gripe about it in public forums? Yea maybe sometimes but it never helps to improve the situation.

Re: Llama.cpp supports Vulkan. why doesn't Ollama?

#43
post #9

Earlier quoted context omitted.

For starters: - It doesn't have a website - It doesn't have a download page, you have to build it yourself

> - It doesn't have a download page, you have to build it yourself I'd wager that anyone capable enough to run a command line tool like Ollama should also be able to download prebuilt binaries from the llama.cpp releases page[1]. Also, prebuilt binaries are available on things like homebrew[2]. [1]: https://github.com/ggerganov/llama.cpp/releases [2]: https://formulae.brew.sh/formula/llama.cpp

I'm perfectly capable of compiling my own software but why bother if I can curl | sh into ollama.

Re: Llama.cpp supports Vulkan. why doesn't Ollama?

#44
post #18

Can someone explain what the point of ollama is? Every time I look at it, it seems like it's a worse llama.cpp that removes options to make things "easier".

yeah that's literally the point. you're listing something that you think is a disadvantage and some people think exactly the opposite.

Re: Llama.cpp supports Vulkan. why doesn't Ollama?

#45

Ollama tries to appeal to a lowest common denominator user base, who does not want to worry about stuff like configuration and quants, or which binary to download. I think they want their project to be smart enough to just 'figure out what to do' on behalf of the user. That appeals to a lot of people, but I think them stuffing all backends into one binary and auto-detecting at runtime which to use and is actually a s…

I disagree with this, but it's a reasonable argument. The problem is that the Ollama team has basically ignored the PR, instead of engaging the community. The least they can do is to explain their reasoning.

This PR is #1 on their repo based on multiple metrics (comments, iterations, what have you)

Re: Llama.cpp supports Vulkan. why doesn't Ollama?

#47

I don't know why one would use Ollama instead of llama.cpp. llama.cpp is so easy to use and the maintainer is pretty famous and active in the community.

Llama.cpp dropped support for multimodal vlms. That is why I am using ollama. I would happily switch back if I could.

Re: Llama.cpp supports Vulkan. why doesn't Ollama?

#48
post #3

Earlier quoted context omitted.

Don’t be sad for commercial entity that is not a good player https://github.com/ggerganov/llama.cpp/pull/11016#issuecomme...

This is great, I did not know about RamaLama and I'll be using and recommending that in future and if I see people using Ollama in instructions I'll recommend they move to RamaLama in the future. Cheers.

This is fascinating. I’ve been using ollama with no knowledge of this because it just works without a ton of knobs I don’t feel like spending the time to mess with.

As usual, the real work seems to be appropriated by people who do the last little bit — put an acceptable user experience and some polish on it — and they take all the money and credit.

It’s shitty but it also happens because the vast majority of devs, especially in the FOSS world, do not understand or appreciate user experience. It is bar none the most important thing in the success of most things in computing.

My rule is: every step a user has to do to install or set up something halves adoption. So if 100 people enter and there are two steps, 25 complete the process.

For a long time Apple was the most valuable corporation on Earth on the basis of user experience alone. Apple doesn’t invent much. They polish it, and that’s where like 99% of the value is as far as the market is concerned.

The reason is that computers are very confusing and hard to use. Computer people, which most of us are, don’t see that because it’s second nature to us. But even for computer people you get to the point where you’re busy and don’t have time to nerd out on every single thing you use, so it even matters to computer people in the end.

Re: Llama.cpp supports Vulkan. why doesn't Ollama?

#49
post #29

Ollama is sketchy enough that I run it in a VM. Which is odd because it would probably take less effort to just run Llama.cpp directly, but VMs are pretty easy so just went that route. When I see people bring up the sketchiness most of the time the creator responds with the equivalent of shrugs , which imo increases the sketchiness.

> but VMs are pretty easy so just went that route.

Don’t you need at least 2 GPUs in that case and put kernel level passthrough?

Re: Llama.cpp supports Vulkan. why doesn't Ollama?

#50
post #32
post #29

Ollama is sketchy enough that I run it in a VM. Which is odd because it would probably take less effort to just run Llama.cpp directly, but VMs are pretty easy so just went that route. When I see people bring up the sketchiness most of the time the creator responds with the equivalent of shrugs , which imo increases the sketchiness.

It's fully open source. I mean yes it uses llama.cpp without giving it credit. But why run it in a VM?

It severely over-permissions itself on my Mac.
Post reply on HN