Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Llama.cpp supports Vulkan. why doesn't Ollama?

github.com

Post reply on HN
Print view View original

221–230 of 235 posts

Re: Llama.cpp supports Vulkan. why doesn't Ollama?

#221

Post by zozbot234 »

wkat4242 wrote: post #220
zozbot234 wrote: post #211

Earlier quoted context omitted.

> They do take a long time to merge any PRs though. I guess you have a point there, seeing as after many months of waiting we finally have a comment on this PR from someone with real involvement in Ollama - see https://github.com/ollama/ollama/pull/5059#issuecomment-2628... . Of course this is very welcome news.

It's not really welcome news, he is just saying they're putting it on the long finger because they think other stuff is more important. He's the same guy that kept ignoring the KV cache quant merge. And the actual patch is tiny.. I think it's about time for a bleeding-edge fork of ollama. These guys are too static and that is not what AI development is all about.

He specifically says that they're reworking the Ollama server implementation in order to better support other kinds of models, and that such work has priority and is going to be a roadblock for this patch. This is not even news to those who were following the project, and it seems reasonable in many ways - users will want Vulkan to work across the board if it's made available at all, not for it to be limited to the kinds of models that exist today.

Re: Llama.cpp supports Vulkan. why doesn't Ollama?

#222

Post by lolinder »

diggan wrote: post #216
lolinder wrote: post #205

Earlier quoted context omitted.

Why do you care? They're the ones who will deal with the support burden of people who don't understand how to use it—if that support burden is low enough that they're happy with where they're at, what motivation do you have to tell them to deliberately restrict their audience?

> Why do you care? Like many in FOSS I care about making the experience better for everyone. Slightly weird question, why do you care that I care? > what motivation do you have to tell them to deliberately restrict their audience? I don't have any motivation to say any such thing, and I wouldn't either. Is that really your take away from reading that issue? Stating something like "Ollama is a daemon/cli for running L…

> Is that really your take away from reading that issue?

Yes. You went to them with a definition of who they're trying to serve and they wrote back that they didn't agree with your relatively narrow scope. Now you're out in random threads about Ollama complaining that they didn't like your definition of their target audience.

Am I missing something?

Re: Llama.cpp supports Vulkan. why doesn't Ollama?

#223

Post by discardedrefuse »

jdright wrote: post #77
andy_ppp wrote: post #12

Earlier quoted context omitted.

This is great, I did not know about RamaLama and I'll be using and recommending that in future and if I see people using Ollama in instructions I'll recommend they move to RamaLama in the future. Cheers.

Yeah, I would love an actual alternative to Ollama, but RamaLama is not it unfortunately. As the other commenter said, onboarding is important. I just want one operation install and it needs to work and the simple fact RamaLama is written in Python, assures it will never be that easy, and this is even more true with LLM stuff when using AMD gpu. I know there will be people that disagree with this, that's ok. This is…

I gave Ramalama shot today. I'm very impressed. `uvx ramalama run deepseek-r1:1.5b` just works™ for me. And that's saying A LOT, because I'm running Fedora Kinoite (KDE spin of Silverblue) with nothing layered on the ostree. That means no ROCm or extra AMDGPU stuff on the base layer. Prior to this, I was running llamafile in a podman/toolbox container with ROCm installed inside. Looks like the container ramalama is using has that stuff in there and amdgpu_top tells me the gpu is cooking when I run a query.

Side note: `uv` is a new package manager for python that replaces the pips, the virtualenvs and more. It's quite good. https://github.com/astral-sh/uv

Re: Llama.cpp supports Vulkan. why doesn't Ollama?

#225

Post by dingocat »

macki0 wrote: post #214
dingocat wrote: post #186

Earlier quoted context omitted.

A distinction they make clear and write extensively about on the model page, yes?

wheres that made clear in "ollama run deepseek-r1” the command to download/run the model?

Which you have to go to the model page to find.

Re: Llama.cpp supports Vulkan. why doesn't Ollama?

#226

Post by tharant »

portaouflop wrote: post #176
axegon_ wrote: post #154

Earlier quoted context omitted.

I think you are missing the point. To get things straight: llama.cpp is not hard to setup and get running. It was a bit of a hassle in 2023 but even then it was not catastrophically complicated if you were willing to read the errors you were getting. People are dissatisfied for two, very valid reasons: ollama gives little to no credit to llama.cpp. The second one is the point of the post: a PR has been open for over…

llama.ccp is hard to set up - I develop software for a living and it wasn’t trivial for me. ollama I can give to my non-technical family members and they know how to use it. As for not merging the PR - why are you entitled to have a PR merged? This attitude of entitlement around contributions is very disheartening as oss maintainer - it’s usually more work to review/merge/maintain a feature etc than to open a PR. Als…

> As for not merging the PR - why are you entitled to have a PR merged?

I didn’t get entitlement vibes from the comment; I think the author believes the PR could have wide benefit, and believes that others support his position, thus the post to HN.

I don’t mean to be preach-y; I’m learning to interpret others by using a kinder mental model of society. Wish me luck!

Re: Llama.cpp supports Vulkan. why doesn't Ollama?

#227

Post by diggan »

lolinder wrote: post #222
diggan wrote: post #216

Earlier quoted context omitted.

> Why do you care? Like many in FOSS I care about making the experience better for everyone. Slightly weird question, why do you care that I care? > what motivation do you have to tell them to deliberately restrict their audience? I don't have any motivation to say any such thing, and I wouldn't either. Is that really your take away from reading that issue? Stating something like "Ollama is a daemon/cli for running L…

> Is that really your take away from reading that issue? Yes. You went to them with a definition of who they're trying to serve and they wrote back that they didn't agree with your relatively narrow scope. Now you're out in random threads about Ollama complaining that they didn't like your definition of their target audience. Am I missing something?

Yes, I'm not asking them to adopt what I think is the target audience, I'm asking them to define any target audience, then add at least one sentence on their website describing what Ollama is, not sure why that's controversial.

Basically the only information on the website right now is "Get up and running with large language models.", do you think that's helping people? Could mean anything.

Re: Llama.cpp supports Vulkan. why doesn't Ollama?

#228

Post by ecurtin »

cge wrote: post #180
jdright wrote: post #77

Earlier quoted context omitted.

Yeah, I would love an actual alternative to Ollama, but RamaLama is not it unfortunately. As the other commenter said, onboarding is important. I just want one operation install and it needs to work and the simple fact RamaLama is written in Python, assures it will never be that easy, and this is even more true with LLM stuff when using AMD gpu. I know there will be people that disagree with this, that's ok. This is…

RamaLama's use of Python is different: it appears to just be using Python for scripting its container management. It doesn't need ROCm to work with Python or anything else. It has no difficult dependencies or anything else: I just installed it with `uv tool install ramalama` and it worked fine. I'd agree that Python packaging is generally bad, and that within an LLM context it's a disastrous mess (especially for ROCm…

@cge you have this right, the main python script has no dependancies, it just uses python3 stdlib stuff. So if you have a python3 executable on your system you are good to go. All the stuff with dependancies runs in a container. On macOS, using no containers works well also, as we basically just install brew llama.cpp

There's really no major python dependancy problems people have been running this on many Linux distros, macOS, etc.

We deliberately don't use python libraries because of the packaging problems.

Re: Llama.cpp supports Vulkan. why doesn't Ollama?

#229

Post by ecurtin »

exe34 wrote: post #93
jdright wrote: post #77

Earlier quoted context omitted.

Yeah, I would love an actual alternative to Ollama, but RamaLama is not it unfortunately. As the other commenter said, onboarding is important. I just want one operation install and it needs to work and the simple fact RamaLama is written in Python, assures it will never be that easy, and this is even more true with LLM stuff when using AMD gpu. I know there will be people that disagree with this, that's ok. This is…

I get the impression the important stuff is done in a container rather than on the host system, so having python/pip might be all you need.

This is true.

Re: Llama.cpp supports Vulkan. why doesn't Ollama?

#230

Post by ecurtin »

zozbot234 wrote: post #72
buyucu wrote: post #2

llama.cpp has supported vulkan for more than a year now. For more than 6 months now there has been an open PR to add vulkan backend support for Ollama. However, Ollama team has not even looked at it or commented on it. Vulkan backends are existential for running LLMs on consumer hardware (iGPUs especially). It's sad to see Ollama miss this opportunity.

The PR has been legitimately out-of-date and unmergeable for many months. It was forward-ported a few weeks ago, and is now still awaiting formal review and merging. (To be sure, Vulkan support in Ollama will likely stay experimental for some time even if the existing PR is merged, and many setups will need manual adjustment of the number of GPU layers and such. It's far from 100% foolproof even in the best-case scen…

It's gets out of date with conflicts, etc. Because it's ignored, if this was the upstream project of Ollama, llama.cpp the maintainers would have got this merged months ago.
Post reply on HN