Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Llama.cpp supports Vulkan. why doesn't Ollama?

github.com

Post reply on HN
Print view View original

61–70 of 235 posts

Re: Llama.cpp supports Vulkan. why doesn't Ollama?

#61

Post by woadwarrior01 »

a12k wrote: post #36
woadwarrior01 wrote: post #26

Earlier quoted context omitted.

> - It doesn't have a download page, you have to build it yourself I'd wager that anyone capable enough to run a command line tool like Ollama should also be able to download prebuilt binaries from the llama.cpp releases page[1]. Also, prebuilt binaries are available on things like homebrew[2]. [1]: https://github.com/ggerganov/llama.cpp/releases [2]: https://formulae.brew.sh/formula/llama.cpp

I am very technically inclined and use Ollama (in a VM, but still) because of all the steps and non-obviousness of how to run Llama.cpp. This framing feels a bit like the “Dropbox won’t succeed because rsync is easy” thinking.

> This framing feels a bit like the “Dropbox won’t succeed because rsync is easy” thinking.

No this isn't. There are plenty of end user GUI apps that make it far easier than Ollama to download and run local LLMs (disclaimer: I build one of them). That's an entirely different market.

IMO, the intersection between the set of people who use a command line tool, and the set of people who are incapable of running `brew install llama.cpp` (or its Windows or Linux equivalents) is exceedingly small.

Re: Llama.cpp supports Vulkan. why doesn't Ollama?

#62

Post by parineum »

andy_ppp wrote: post #14
cedws wrote: post #11

The way Ollama has basically been laundering llama.cpp’s features as its own felt dodgy, this appears to confirm there’s something underhanded going on.

It would be extremely unsurprising if Nvidia was funding this embrace and extend behind the scenes.

It would be pretty surprising to their shareholders if Nvidia was hiding where it was spending it's money.

Re: Llama.cpp supports Vulkan. why doesn't Ollama?

#63

Post by bearjaws »

Kubuxu wrote: post #3
buyucu wrote: post #2

llama.cpp has supported vulkan for more than a year now. For more than 6 months now there has been an open PR to add vulkan backend support for Ollama. However, Ollama team has not even looked at it or commented on it. Vulkan backends are existential for running LLMs on consumer hardware (iGPUs especially). It's sad to see Ollama miss this opportunity.

Don’t be sad for commercial entity that is not a good player https://github.com/ggerganov/llama.cpp/pull/11016#issuecomme...

It's hilarious that docker guys are trying to take another OSS and monetize it. Hey if it worked once?...

Re: Llama.cpp supports Vulkan. why doesn't Ollama?

#64

Post by moffkalast »

cedws wrote: post #11

The way Ollama has basically been laundering llama.cpp’s features as its own felt dodgy, this appears to confirm there’s something underhanded going on.

Ollama is a private for profit company, of course there's something shady going on.

Re: Llama.cpp supports Vulkan. why doesn't Ollama?

#65

Post by anewhnaccount2 »

api wrote: post #48
andy_ppp wrote: post #12

Earlier quoted context omitted.

This is great, I did not know about RamaLama and I'll be using and recommending that in future and if I see people using Ollama in instructions I'll recommend they move to RamaLama in the future. Cheers.

This is fascinating. I’ve been using ollama with no knowledge of this because it just works without a ton of knobs I don’t feel like spending the time to mess with. As usual, the real work seems to be appropriated by people who do the last little bit — put an acceptable user experience and some polish on it — and they take all the money and credit. It’s shitty but it also happens because the vast majority of devs, es…

I think it's understood, but someone needs to work on the actual infrastructure.

Re: Llama.cpp supports Vulkan. why doesn't Ollama?

#68

Post by 2-3-7-43-1807 »

can someone please give a quick summary of the criticism towards ollama?

as far as my intel goes it's a mozilla project shouldered mostly by one 10x programmer. i found ollama through hn and last time i didn't notice any lack of trust or suspected sketchiness ... so what changed?

Re: Llama.cpp supports Vulkan. why doesn't Ollama?

#69

Post by buyucu »

trash_cat wrote: post #66

I use Ollama because I am a casual user and can't be bothered to read the docs on how to setup llama.cpp. I just want to run a simple llm locally. Why would I care about Vulkan?

with vulkan it runs much much faster on consumer hardware, especially opn igpus like intel or amd.
Post reply on HN