Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Llama.cpp supports Vulkan. why doesn't Ollama?

github.com

Post reply on HN
Print view View original

211–220 of 235 posts

Re: Llama.cpp supports Vulkan. why doesn't Ollama?

#211

Post by zozbot234 »

chown wrote: post #114
lolinder wrote: post #97

So many here are trashing on Ollama, saying it's "just" nice porcelain around llama.cpp and it's not doing anything complicated. Okay. Let's stipulate that. So where's the non-sketchy, non-for-profit equivalent? Where's the nice frontend for llama.cpp that makes it trivial for anyone who wants to play around with local LLMs without having to know much about their internals? If Ollama isn't doing anything difficult, w…

I totally agree with this. I wanted to make it really easy for non-technical users with an app that hid all the complexities. I basically just wanted to embed the engine without making users open their terminal, let alone make them configure. I started with llama.cpp amd almost gave up on the idea before I stumbled upon Ollama, which made the app happen[1] There are many flaws in Ollama but it makes many things much…

> They do take a long time to merge any PRs though.

I guess you have a point there, seeing as after many months of waiting we finally have a comment on this PR from someone with real involvement in Ollama - see https://github.com/ollama/ollama/pull/5059#issuecomment-2628... . Of course this is very welcome news.

Re: Llama.cpp supports Vulkan. why doesn't Ollama?

#212

Post by buyucu »

bdhcuidbebe wrote: post #209
buyucu wrote: post #117

Earlier quoted context omitted.

If you support Vulkan, you support almost every GPU out there in the consumer market across all hardware vendors. It's an amazing fallback option. I agree they should also support OpenVINO, but compared to Vulkan OpenVINO is a tiny market.

I made an argument for performance, not for compatibility. If you run your local llm in the least performant way possible on tour overly expensive GPU, then you are not making value of your purchase. Vulkan is a fallback option is all. I even see people running on their CPU because some apps dont support their hardware and llama.cpp made it even possible. It is still a really bad idea. Its just goes to show there’s s…

I'm willing to bet that Vulkan will outperform OpenVINO.

Vulkan is the API right now in the graphics world. It's very well supported and actively being improved on. Everyone is pouring resources into making Vulkan better.

OpenVINO feels barely developed. Intel never made it a proper backend for Pytorch like AMD did with ROCm. It's hard to see where it is going, or if it is going anywhere at all. Between Sycl and OneApi it's hard to see how much interest Intel has developing it.

Re: Llama.cpp supports Vulkan. why doesn't Ollama?

#213

Post by 7thpower »

diggan wrote: post #75
7thpower wrote: post #24

Earlier quoted context omitted.

Can you please explain why you think they may be operating in bad faith?

Not parent, but same feeling. First I got the feeling because of how they store things on disk and try to get all models rehosted in their own closed library. Second time I got the feeling is when it's not obvious at all about what their motives are, and that it's a for-profit venture. Third time is trying to discuss things in their Discord and the moderators there constantly shut down a lot of conversation citing "M…

This is helpful, thank you.

Re: Llama.cpp supports Vulkan. why doesn't Ollama?

#214

Post by macki0 »

dingocat wrote: post #186
trissi1996 wrote: post #135

Earlier quoted context omitted.

Which is not the same model, it's not R1 it's R1-Distill-Qwen-1.5B....

A distinction they make clear and write extensively about on the model page, yes?

wheres that made clear in "ollama run deepseek-r1” the command to download/run the model?

Re: Llama.cpp supports Vulkan. why doesn't Ollama?

#215

Post by bdhcuidbebe »

buyucu wrote: post #212
bdhcuidbebe wrote: post #209

Earlier quoted context omitted.

I made an argument for performance, not for compatibility. If you run your local llm in the least performant way possible on tour overly expensive GPU, then you are not making value of your purchase. Vulkan is a fallback option is all. I even see people running on their CPU because some apps dont support their hardware and llama.cpp made it even possible. It is still a really bad idea. Its just goes to show there’s s…

I'm willing to bet that Vulkan will outperform OpenVINO. Vulkan is the API right now in the graphics world. It's very well supported and actively being improved on. Everyone is pouring resources into making Vulkan better. OpenVINO feels barely developed. Intel never made it a proper backend for Pytorch like AMD did with ROCm. It's hard to see where it is going, or if it is going anywhere at all. Between Sycl and OneA…

> I'm willing to bet that Vulkan will outperform OpenVINO.

> Vulkan is the API right now in the graphics world.

YUP, Vulkan is all the rage in the graphics world, and for good reasons. But we arent discussing graphics now are we?

Vulkan is a general graphics API with some computing capabilities.

OpenVINO is a toolkit for inference neural networks, by intel built to make use of their GPUs and NPUs for this specific task.

Using vulkan, first you need to translate your payload to shaders, then they need to be compiled to SPIR-V, then they can use a subset of the cards capabilities.

How could this even remotely match something written specifically for the task?

Also, it is dead easy to benchmark if you still think otherwise.

Or just read up on it..

Re: Llama.cpp supports Vulkan. why doesn't Ollama?

#216

Post by diggan »

lolinder wrote: post #205
diggan wrote: post #199

Earlier quoted context omitted.

Ollama is also written for technical/developer users, by accident (it seems), even though they don't want it to be strictly for technical users. I've opened a issue asking them to make it more clear that Ollama is for technical users, but they seem confident people with no terminal experience can and will also use Ollama: https://github.com/ollama/ollama/issues/7116

Why do you care? They're the ones who will deal with the support burden of people who don't understand how to use it—if that support burden is low enough that they're happy with where they're at, what motivation do you have to tell them to deliberately restrict their audience?

> Why do you care?

Like many in FOSS I care about making the experience better for everyone. Slightly weird question, why do you care that I care?

> what motivation do you have to tell them to deliberately restrict their audience?

I don't have any motivation to say any such thing, and I wouldn't either. Is that really your take away from reading that issue?

Stating something like "Ollama is a daemon/cli for running LLMs in your terminal" on your website isn't a restriction whatsoever, it's just being clear up front what the tool is. Currently, the website literally doesn't say what Ollama actually is.

Re: Llama.cpp supports Vulkan. why doesn't Ollama?

#217

Post by justinmayer »

the_mitsuhiko wrote: post #15

Ollama needs competition. I’m not sure what drives the people that maintain it but some of their actions imply that there are ulterior motives at play that do not have the benefit of their users in mind. However such projects require a lot of time and effort and it’s not clear if this project can be forked and kept alive.

Benefiting users is definitely not Ollama’s first priority, as seen when this pull request was summarily closed: https://github.com/jmorganca/ollama/pull/395

Those README changes only served to provide greater transparency to would-be users.

Ulterior motives, indeed.

Re: Llama.cpp supports Vulkan. why doesn't Ollama?

#218

Post by sebazzz »

buyucu wrote: post #69
trash_cat wrote: post #66

I use Ollama because I am a casual user and can't be bothered to read the docs on how to setup llama.cpp. I just want to run a simple llm locally. Why would I care about Vulkan?

with vulkan it runs much much faster on consumer hardware, especially opn igpus like intel or amd.

How is the performance of Vulkan vs ROCm on AMD iGPUs? Ollama can be persuaded to run on iGPUs with ROCm.

Re: Llama.cpp supports Vulkan. why doesn't Ollama?

#219

Post by wkat4242 »

That's a weird thing about Ollama yes.

It took very long for them to support KV cache quantisation too (which drastically reduces the amount of VRAM needed for context!). Even though the underlying llama.cpp had offered it for ages. And they had it handed to them on a platter, someone had developed everything and submitted a patch.

The developer of that patch even was about to give up as he had to constantly keep it up to date with upstream even though he was constantly being ignored. So he had no idea if it would ever be merged.

They just seem to be really hesitant to offer new features.

Eventually it was merged and it made a huge difference to people with low VRAM cards.

Re: Llama.cpp supports Vulkan. why doesn't Ollama?

#220

Post by wkat4242 »

zozbot234 wrote: post #211
chown wrote: post #114

Earlier quoted context omitted.

I totally agree with this. I wanted to make it really easy for non-technical users with an app that hid all the complexities. I basically just wanted to embed the engine without making users open their terminal, let alone make them configure. I started with llama.cpp amd almost gave up on the idea before I stumbled upon Ollama, which made the app happen[1] There are many flaws in Ollama but it makes many things much…

> They do take a long time to merge any PRs though. I guess you have a point there, seeing as after many months of waiting we finally have a comment on this PR from someone with real involvement in Ollama - see https://github.com/ollama/ollama/pull/5059#issuecomment-2628... . Of course this is very welcome news.

It's not really welcome news, he is just saying they're putting it on the long finger because they think other stuff is more important. He's the same guy that kept ignoring the KV cache quant merge.

And the actual patch is tiny..

I think it's about time for a bleeding-edge fork of ollama. These guys are too static and that is not what AI development is all about.

Post reply on HN