Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Llama.cpp supports Vulkan. why doesn't Ollama?

github.com

Post reply on HN
Print view View original

171–180 of 235 posts

Re: Llama.cpp supports Vulkan. why doesn't Ollama?

#171

Post by buyucu »

cmm wrote: post #150
buyucu wrote: post #130

Earlier quoted context omitted.

This is incorrect. The effort it took to enable Vulkan was relatively minor. The PR is short and to be honest it doesn't do much, because it doesn't need to.

that PR doesn't actually work though -- it finds the Vulkan libraries and has some memory accounting logic, but the bits to actually build a Vulkan llama.cpp runner are not there. I'm not sure why its author deems it ready for inclusion. (I mean, the missing work should not be much, but it still has to be done)

the pr was working 6 months ago and it has been rebased multiple times as the ollama team kept ignoring it and mainline moved. I'm using it right now.

Re: Llama.cpp supports Vulkan. why doesn't Ollama?

#172

Post by Aurornis »

lolinder wrote: post #97

So many here are trashing on Ollama, saying it's "just" nice porcelain around llama.cpp and it's not doing anything complicated. Okay. Let's stipulate that. So where's the non-sketchy, non-for-profit equivalent? Where's the nice frontend for llama.cpp that makes it trivial for anyone who wants to play around with local LLMs without having to know much about their internals? If Ollama isn't doing anything difficult, w…

It’s so hard to decipher the complaints about ollama in this comment section. I keep reading comments from people saying they don’t trust it, but then they don’t explain why they don’t trust it and don’t answer any follow up questions.

As someone who doesn’t follow this space, it’s hard to tell if there’s actually something sketchy going on with ollama or if it’s the usual reactionary negativity that happens when a tool comes along and makes someone’s niche hobby easier and more accessible to a broad audience.

Re: Llama.cpp supports Vulkan. why doesn't Ollama?

#173

Post by imtringued »

the_mitsuhiko wrote: post #15

Ollama needs competition. I’m not sure what drives the people that maintain it but some of their actions imply that there are ulterior motives at play that do not have the benefit of their users in mind. However such projects require a lot of time and effort and it’s not clear if this project can be forked and kept alive.

Ollama doesn't really need competition. Llama.cpp just needs a few usability updates to the gguf format so that you can specify a hugging face repository like you can do in vLLM already.

Re: Llama.cpp supports Vulkan. why doesn't Ollama?

#174

Post by bloomingkales »

Aurornis wrote: post #172
lolinder wrote: post #97

So many here are trashing on Ollama, saying it's "just" nice porcelain around llama.cpp and it's not doing anything complicated. Okay. Let's stipulate that. So where's the non-sketchy, non-for-profit equivalent? Where's the nice frontend for llama.cpp that makes it trivial for anyone who wants to play around with local LLMs without having to know much about their internals? If Ollama isn't doing anything difficult, w…

It’s so hard to decipher the complaints about ollama in this comment section. I keep reading comments from people saying they don’t trust it, but then they don’t explain why they don’t trust it and don’t answer any follow up questions. As someone who doesn’t follow this space, it’s hard to tell if there’s actually something sketchy going on with ollama or if it’s the usual reactionary negativity that happens when a t…

they don’t explain why they don’t trust it

We need to know a few things:

1) Show me the lines of code that log things and how it handles temp files and storage.

2) No remote calls at all.

3) No telemetry at all.

This is the feature list I would want to begin trusting. I use this stuff, but I also don’t trust it.

Re: Llama.cpp supports Vulkan. why doesn't Ollama?

#175

Post by buyucu »

av_conk wrote: post #170

I tried using ollama because I couldn't get ROCm working on my system with llama-cpp. Ollama bundles the ROCm libraries for you. I got around 50 tokens per second with that setup. I tried llama-cpp with the Vulkan backend and doubled the amount of tokens per second. I was under the impression ROCm is superior to Vulkan, so I was confused about the result. In any case, I've stuck with llama-cpp.

It depends on your GPU. Vulkan is well-supported by essentially all GPUs. AMD support ROCm well for their datacenter GPUs, but support for consumer hardware has not been as good.

Re: Llama.cpp supports Vulkan. why doesn't Ollama?

#176

Post by portaouflop »

axegon_ wrote: post #154
lolinder wrote: post #97

So many here are trashing on Ollama, saying it's "just" nice porcelain around llama.cpp and it's not doing anything complicated. Okay. Let's stipulate that. So where's the non-sketchy, non-for-profit equivalent? Where's the nice frontend for llama.cpp that makes it trivial for anyone who wants to play around with local LLMs without having to know much about their internals? If Ollama isn't doing anything difficult, w…

I think you are missing the point. To get things straight: llama.cpp is not hard to setup and get running. It was a bit of a hassle in 2023 but even then it was not catastrophically complicated if you were willing to read the errors you were getting. People are dissatisfied for two, very valid reasons: ollama gives little to no credit to llama.cpp. The second one is the point of the post: a PR has been open for over…

llama.ccp is hard to set up - I develop software for a living and it wasn’t trivial for me. ollama I can give to my non-technical family members and they know how to use it.

As for not merging the PR - why are you entitled to have a PR merged? This attitude of entitlement around contributions is very disheartening as oss maintainer - it’s usually more work to review/merge/maintain a feature etc than to open a PR. Also no one is entitled to comments / discussion or literally one second of my time as an OSS maintainer. This is imo the cancer that is eating open source.

Re: Llama.cpp supports Vulkan. why doesn't Ollama?

#177

Post by aduffy »

bloomingkales wrote: post #174
Aurornis wrote: post #172

Earlier quoted context omitted.

It’s so hard to decipher the complaints about ollama in this comment section. I keep reading comments from people saying they don’t trust it, but then they don’t explain why they don’t trust it and don’t answer any follow up questions. As someone who doesn’t follow this space, it’s hard to tell if there’s actually something sketchy going on with ollama or if it’s the usual reactionary negativity that happens when a t…

they don’t explain why they don’t trust it We need to know a few things: 1) Show me the lines of code that log things and how it handles temp files and storage. 2) No remote calls at all. 3) No telemetry at all. This is the feature list I would want to begin trusting. I use this stuff, but I also don’t trust it.

The code is literally open source with MIT license to boot https://github.com/ollama/ollama

Re: Llama.cpp supports Vulkan. why doesn't Ollama?

#178

Post by bloomingkales »

aduffy wrote: post #177
bloomingkales wrote: post #174

Earlier quoted context omitted.

they don’t explain why they don’t trust it We need to know a few things: 1) Show me the lines of code that log things and how it handles temp files and storage. 2) No remote calls at all. 3) No telemetry at all. This is the feature list I would want to begin trusting. I use this stuff, but I also don’t trust it.

The code is literally open source with MIT license to boot https://github.com/ollama/ollama

Oh.

Re: Llama.cpp supports Vulkan. why doesn't Ollama?

#179

Post by romperstomper »

lolinder wrote: post #142
bestcoder69 wrote: post #113

Earlier quoted context omitted.

Llamafile: https://github.com/Mozilla-Ocho/llamafile

Llamafile is great but solves a slightly different problem very well: how do I easily download and run a single model without having any infrastructure in place first? Ollama solves the problem of how I run many models without having to deal with many instances of infrastructure.

You don't need any infrastructure for llamafiles, you just download and run them (everywhere).

Re: Llama.cpp supports Vulkan. why doesn't Ollama?

#180

Post by cge »

jdright wrote: post #77
andy_ppp wrote: post #12

Earlier quoted context omitted.

This is great, I did not know about RamaLama and I'll be using and recommending that in future and if I see people using Ollama in instructions I'll recommend they move to RamaLama in the future. Cheers.

Yeah, I would love an actual alternative to Ollama, but RamaLama is not it unfortunately. As the other commenter said, onboarding is important. I just want one operation install and it needs to work and the simple fact RamaLama is written in Python, assures it will never be that easy, and this is even more true with LLM stuff when using AMD gpu. I know there will be people that disagree with this, that's ok. This is…

RamaLama's use of Python is different: it appears to just be using Python for scripting its container management. It doesn't need ROCm to work with Python or anything else. It has no difficult dependencies or anything else: I just installed it with `uv tool install ramalama` and it worked fine.

I'd agree that Python packaging is generally bad, and that within an LLM context it's a disastrous mess (especially for ROCm), but that doesn't appear to be how RamaLama is using it at all.

Post reply on HN