Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Llama.cpp supports Vulkan. why doesn't Ollama?

github.com

Post reply on HN
Print view View original

161–170 of 235 posts

Re: Llama.cpp supports Vulkan. why doesn't Ollama?

#161

Post by lolinder »

axegon_ wrote: post #154
lolinder wrote: post #97

So many here are trashing on Ollama, saying it's "just" nice porcelain around llama.cpp and it's not doing anything complicated. Okay. Let's stipulate that. So where's the non-sketchy, non-for-profit equivalent? Where's the nice frontend for llama.cpp that makes it trivial for anyone who wants to play around with local LLMs without having to know much about their internals? If Ollama isn't doing anything difficult, w…

I think you are missing the point. To get things straight: llama.cpp is not hard to setup and get running. It was a bit of a hassle in 2023 but even then it was not catastrophically complicated if you were willing to read the errors you were getting. People are dissatisfied for two, very valid reasons: ollama gives little to no credit to llama.cpp. The second one is the point of the post: a PR has been open for over…

> To get things straight: llama.cpp is not hard to setup and get running. It was a bit of a hassle in 2023 but even then it was not catastrophically complicated if you were willing to read the errors you were getting.

It's made a lot of progress in that the README [0] now at least has instructions for how to download pre-built releases or docker images, but that requires actually reading the section entitled "Building the Project" to realize that it provides more than just building instructions. That is not accessible to the masses, and it's hard for me to not see that placement and prioritization as an intentional choice to be inaccessible (which is a perfectly valid choice for them!)

And that's aside from the fact that Ollama provides a ton of convenience features that are simply missing, starting with the fact that it looks like with llama.cpp I still have to pick a model at startup time, which means switching models requires SSHing into my server and restarting it.

None of this is meant to disparage llama.cpp: what they're doing is great and they have chosen to not prioritize user convenience as their primary goal. That's a perfectly valid choice. And I'm also not defending Ollama's lack of acknowledgment. I'm responding to a very specific set of ideas that have been prevalent in this thread: that not only does Ollama not give credit, they're not even really doing very much "real work". To me that is patently nonsense—the last mile to package something in a way that is user friendly is often at least as much work, it's just not the kind of work that hackers who hang out on forums like this appreciate.

[0] https://github.com/ggerganov/llama.cpp

Re: Llama.cpp supports Vulkan. why doesn't Ollama?

#162

Post by [deleted] »

traverseda wrote: post #134
lolinder wrote: post #97

So many here are trashing on Ollama, saying it's "just" nice porcelain around llama.cpp and it's not doing anything complicated. Okay. Let's stipulate that. So where's the non-sketchy, non-for-profit equivalent? Where's the nice frontend for llama.cpp that makes it trivial for anyone who wants to play around with local LLMs without having to know much about their internals? If Ollama isn't doing anything difficult, w…

>So where's the non-sketchy, non-for-profit equivalent? Serving models is currently expensive. I'd argue that some big cloud providers have conspired to make egress bandwidth expensive. That, coupled with the increasing scale of the internet, make it harder and harder for smaller groups to do these kinds of things. At least until we get some good content addressed distributed storage system.

[deleted]

Re: Llama.cpp supports Vulkan. why doesn't Ollama?

#163

Post by woadwarrior01 »

traverseda wrote: post #134
lolinder wrote: post #97

So many here are trashing on Ollama, saying it's "just" nice porcelain around llama.cpp and it's not doing anything complicated. Okay. Let's stipulate that. So where's the non-sketchy, non-for-profit equivalent? Where's the nice frontend for llama.cpp that makes it trivial for anyone who wants to play around with local LLMs without having to know much about their internals? If Ollama isn't doing anything difficult, w…

>So where's the non-sketchy, non-for-profit equivalent? Serving models is currently expensive. I'd argue that some big cloud providers have conspired to make egress bandwidth expensive. That, coupled with the increasing scale of the internet, make it harder and harder for smaller groups to do these kinds of things. At least until we get some good content addressed distributed storage system.

> Serving models is currently expensive. I'd argue that some big cloud providers have conspired to make egress bandwidth expensive.

Cloudflare R2 has unlimited egress, and AFAIK, that's what ollama uses for hosting quantized model weights.

Re: Llama.cpp supports Vulkan. why doesn't Ollama?

#164

Post by Havoc »

dingocat wrote: post #122
Havoc wrote: post #102

ollama was good initially in that it made LLMs more accessible for non-technical people while everyone was figuring things out. Lately they seem to be contributing mostly confusion to the conversation. The #1 model the entire world is talking about is literally mislabeled their side. There is no such thing as R1-1.5b. Quantization without telling users also confuses noobs as to what is possible. Setting up an api dif…

What do you mean there is no such thing as R1-1.5b? DeepSeek released a distilled version based on a 1.5B Qwen model with the full name DeepSeek-R1-Distill-Qwen-1.5B, see chapter 3.2 on page 14 of their research article [0]. [0] https://arxiv.org/abs/2501.12948

ollama labels the qwen models R1, while the "R1" moniker standing on its own in deepseek world means the full model that has nothing to do with qwen.

https://ollama.com/library/deepseek-r1

That may have been ok if it was just same model at different sizes but they're completely different things here & it's created confusion out of thin air for absolutely no reason other than ollama being careless.

Re: Llama.cpp supports Vulkan. why doesn't Ollama?

#166

Post by sroecker »

himhckr wrote: post #71
zophiana wrote: post #21

Earlier quoted context omitted.

Honestly I just didn't know it was this easy to use, maybe because of the name... But ramalama seems to be a full replacement for ollama

ramalama still needs users to be able to install docker first, no? That’s a barrier to entry for many users esp. Windows where I have had my struggles running Docker not to mention a massive resource hog.

Yes, but ramalama defaults to podman. Podman Desktop is very easy to install and use.

Re: Llama.cpp supports Vulkan. why doesn't Ollama?

#167

Post by homebrewer »

bestcoder69 wrote: post #113
lolinder wrote: post #97

So many here are trashing on Ollama, saying it's "just" nice porcelain around llama.cpp and it's not doing anything complicated. Okay. Let's stipulate that. So where's the non-sketchy, non-for-profit equivalent? Where's the nice frontend for llama.cpp that makes it trivial for anyone who wants to play around with local LLMs without having to know much about their internals? If Ollama isn't doing anything difficult, w…

Llamafile: https://github.com/Mozilla-Ocho/llamafile

It's actually more difficult to use on linux (compared to ollama) because of the weird binfmt contortions you have to go through.

Re: Llama.cpp supports Vulkan. why doesn't Ollama?

#168

Post by dinosaurdynasty »

turnsout wrote: post #8

This is going to sound like a troll, but it's an honest question: Why do people use Ollama over llama.cpp? llama.cpp has added a ton of features, is about as user-friendly as Ollama, and is higher-performance. Is there some key differentiator for Ollama that I'm missing?

Can you even use bare llama.cpp with OpenWebUI? Especially when they are running on two different computers?

Re: Llama.cpp supports Vulkan. why doesn't Ollama?

#169

Post by evilduck »

washadjeffmad wrote: post #149
lolinder wrote: post #97

So many here are trashing on Ollama, saying it's "just" nice porcelain around llama.cpp and it's not doing anything complicated. Okay. Let's stipulate that. So where's the non-sketchy, non-for-profit equivalent? Where's the nice frontend for llama.cpp that makes it trivial for anyone who wants to play around with local LLMs without having to know much about their internals? If Ollama isn't doing anything difficult, w…

>So where's the non-sketchy, non-for-profit equivalent llama.cpp, kobold.cpp, oobabooga, llmstudio, etc. There are dozens at this point. And while many chalk the attachment to ollama up to a "skill issue", that's just venting frustration that all something has to do to win the popularity contest is to repackage and market it as an "app". I prefer first-party tools, I'm comfortable managing a build environment and cal…

> llama.cpp, kobold.cpp, oobabooga

None of these three are remotely as easy to install or use. They could be, but none of them are even trying.

> lmstudio

This is a closed source app with a non-free license from a business not making money. Enshittification is just a matter of when.

Re: Llama.cpp supports Vulkan. why doesn't Ollama?

#170

Post by av_conk »

I tried using ollama because I couldn't get ROCm working on my system with llama-cpp. Ollama bundles the ROCm libraries for you. I got around 50 tokens per second with that setup.

I tried llama-cpp with the Vulkan backend and doubled the amount of tokens per second. I was under the impression ROCm is superior to Vulkan, so I was confused about the result.

In any case, I've stuck with llama-cpp.

Post reply on HN