Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Llama.cpp supports Vulkan. why doesn't Ollama?

github.com

Post reply on HN
Print view View original

51–60 of 235 posts

Re: Llama.cpp supports Vulkan. why doesn't Ollama?

#51

Post by paradite »

buyucu wrote: post #31
paradite wrote: post #13

Could it be that supporting multiple platforms open up more support tickets and adds more work to keep the software working on those new platforms? As someone who built apps for Windows, Linux, macOS, iOS and Android, it is not trivial to ensure your new features or updates work on all platforms, and you have to deal with deprecations.

ollama is not doing anything. llama cpp does all that work. ollama is just a small wrapper on top.

Ok assuming what you said is correct, why wouldn't Ollama then be able to support Vulkan by default out of the box?

Sorry I'm not sure what's the relationship exactly between the two projects. This is a genuine questions, not a troll question.

Re: Llama.cpp supports Vulkan. why doesn't Ollama?

#52

Post by a12k »

nicce wrote: post #49
a12k wrote: post #29

Ollama is sketchy enough that I run it in a VM. Which is odd because it would probably take less effort to just run Llama.cpp directly, but VMs are pretty easy so just went that route. When I see people bring up the sketchiness most of the time the creator responds with the equivalent of shrugs , which imo increases the sketchiness.

> but VMs are pretty easy so just went that route. Don’t you need at least 2 GPUs in that case and put kernel level passthrough?

I don’t use GPU. Works fine, but the large Mixtral models are slow.

Re: Llama.cpp supports Vulkan. why doesn't Ollama?

#53

Post by buyucu »

blixt wrote: post #41
the_mitsuhiko wrote: post #15

Ollama needs competition. I’m not sure what drives the people that maintain it but some of their actions imply that there are ulterior motives at play that do not have the benefit of their users in mind. However such projects require a lot of time and effort and it’s not clear if this project can be forked and kept alive.

LM Studio has been around for a long time and does a lot of similar things but with a more UI-based approach. I used to use it before Ollama, and seems it's still going strong. https://lmstudio.ai/

isn't lm stuido closed source?

Re: Llama.cpp supports Vulkan. why doesn't Ollama?

#54

Post by paradite »

woadwarrior01 wrote: post #26
paradite wrote: post #9

Earlier quoted context omitted.

For starters: - It doesn't have a website - It doesn't have a download page, you have to build it yourself

> - It doesn't have a download page, you have to build it yourself I'd wager that anyone capable enough to run a command line tool like Ollama should also be able to download prebuilt binaries from the llama.cpp releases page[1]. Also, prebuilt binaries are available on things like homebrew[2]. [1]: https://github.com/ggerganov/llama.cpp/releases [2]: https://formulae.brew.sh/formula/llama.cpp

Ok I was looking at the repo from mobile and missed the releases.

Still it's not immediate obvious from README that there is an option to download it. There are instructions on how to build it, but not how to download it. Or maybe I'm blind, please correct me.

Re: Llama.cpp supports Vulkan. why doesn't Ollama?

#55

Post by rfoo »

buyucu wrote: post #38
benxh wrote: post #34

My biggest gripe with Ollama is the badly named models, e.g. under deepseek-r1, it defaults to the distill models.

I agree they should rename them. But defaulting to a 671b model is also evil.

No. If you can't run it and most people can never run the model on their laptop, it's fine, let people know the fact, instead of giving them illusion.

Re: Llama.cpp supports Vulkan. why doesn't Ollama?

#56

Post by sitkack »

portaouflop wrote: post #42
buyucu wrote: post #33

Earlier quoted context omitted.

ollama advertising llama.cpp features as their own is very dishonest in my opinion.

That’s the curse and blessing of open source I guess? I have billion dollar companies running my oss software without giving me anything - but do I gripe about it in public forums? Yea maybe sometimes but it never helps to improve the situation.

Are they a wrapper with a similar name? You, like I, do gripe in public forums.

Re: Llama.cpp supports Vulkan. why doesn't Ollama?

#57

Post by buyucu »

paradite wrote: post #51
buyucu wrote: post #31

Earlier quoted context omitted.

ollama is not doing anything. llama cpp does all that work. ollama is just a small wrapper on top.

Ok assuming what you said is correct, why wouldn't Ollama then be able to support Vulkan by default out of the box? Sorry I'm not sure what's the relationship exactly between the two projects. This is a genuine questions, not a troll question.

check the PR, it's a very short one. It's not more complicated than setting a compile time flag.

I have no idea why they have been ignoring it.

Ollama is just a friendly front end for llama.cpp. It doesn't have to do any of those things you mentioned. Llama.cpp does all that.

Re: Llama.cpp supports Vulkan. why doesn't Ollama?

#58

Post by SkyPuncher »

turnsout wrote: post #8

This is going to sound like a troll, but it's an honest question: Why do people use Ollama over llama.cpp? llama.cpp has added a ton of features, is about as user-friendly as Ollama, and is higher-performance. Is there some key differentiator for Ollama that I'm missing?

Ollama - `brew install ollama`

llama.cpp - Read the docs, with loads of information and unclear use cases. Question if it has API compatibility and secondary features that a bunch of tools expect. Decide it's not worth your effort when `ollama` is already running by the time you've read the docs

Re: Llama.cpp supports Vulkan. why doesn't Ollama?

#59

Post by stuaxo »

turnsout wrote: post #8

This is going to sound like a troll, but it's an honest question: Why do people use Ollama over llama.cpp? llama.cpp has added a ton of features, is about as user-friendly as Ollama, and is higher-performance. Is there some key differentiator for Ollama that I'm missing?

The server in llama-cpp is documented as being only for demonstration, but ollama supports it as a model to run it.

For work, we are given Macs and so the GPU can't be passed through to docker.

I wanted a client/server where the server has the LLM and runs outside of Docker, but without me having to write the client/server part.

I run my model in ollama, then inside the code use litellm to speak to it during local development.

Re: Llama.cpp supports Vulkan. why doesn't Ollama?

#60

Post by Gracana »

buyucu wrote: post #47
your_challenger wrote: post #46

I don't know why one would use Ollama instead of llama.cpp. llama.cpp is so easy to use and the maintainer is pretty famous and active in the community.

Llama.cpp dropped support for multimodal vlms. That is why I am using ollama. I would happily switch back if I could.

llama.cpp readme still lists multimodal models.. Qwen2-VL and others. Is that inaccurate, or something different?

[edit] Oh I see, here's an issue about it: https://github.com/ggerganov/llama.cpp/issues/8010

Post reply on HN