Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Llama.cpp supports Vulkan. why doesn't Ollama?

github.com

Post reply on HN
Print view View original

91–100 of 235 posts

Re: Llama.cpp supports Vulkan. why doesn't Ollama?

#91

Post by 9cb14c1ec0 »

buyucu wrote: post #2

llama.cpp has supported vulkan for more than a year now. For more than 6 months now there has been an open PR to add vulkan backend support for Ollama. However, Ollama team has not even looked at it or commented on it. Vulkan backends are existential for running LLMs on consumer hardware (iGPUs especially). It's sad to see Ollama miss this opportunity.

The PR at issue here blocks iGPUs. My fork of the PR changes removes that:

https://github.com/9cb14c1ec0/ollama-vulkan

I successfully ran Phi4 on my AMD Ryzen 7 PRO 5850U iGPU with it.

Re: Llama.cpp supports Vulkan. why doesn't Ollama?

#92

Post by bdhcuidbebe »

nicce wrote: post #49
a12k wrote: post #29

Ollama is sketchy enough that I run it in a VM. Which is odd because it would probably take less effort to just run Llama.cpp directly, but VMs are pretty easy so just went that route. When I see people bring up the sketchiness most of the time the creator responds with the equivalent of shrugs , which imo increases the sketchiness.

> but VMs are pretty easy so just went that route. Don’t you need at least 2 GPUs in that case and put kernel level passthrough?

i pass through my dGPU to VM and use iGPU for desktop

Re: Llama.cpp supports Vulkan. why doesn't Ollama?

#93

Post by exe34 »

jdright wrote: post #77
andy_ppp wrote: post #12

Earlier quoted context omitted.

This is great, I did not know about RamaLama and I'll be using and recommending that in future and if I see people using Ollama in instructions I'll recommend they move to RamaLama in the future. Cheers.

Yeah, I would love an actual alternative to Ollama, but RamaLama is not it unfortunately. As the other commenter said, onboarding is important. I just want one operation install and it needs to work and the simple fact RamaLama is written in Python, assures it will never be that easy, and this is even more true with LLM stuff when using AMD gpu. I know there will be people that disagree with this, that's ok. This is…

I get the impression the important stuff is done in a container rather than on the host system, so having python/pip might be all you need.

Re: Llama.cpp supports Vulkan. why doesn't Ollama?

#94

Post by lxgr »

zozbot234 wrote: post #90
bloomingkales wrote: post #79

Earlier quoted context omitted.

Are closer to the metal AI developers an under tracked bottle neck? AMD and Intel can barely get off the ground due to lagging software developers.

Metal is an Apple thing, not Intel or AMD. (And Ollama supports that.)

I assume they mean closer to the figurative metal, not literal Metal.

Re: Llama.cpp supports Vulkan. why doesn't Ollama?

#95

Post by moffkalast »

ethbr1 wrote: post #80
moffkalast wrote: post #64

Earlier quoted context omitted.

Ollama is a private for profit company, of course there's something shady going on.

Ollama is a private for profit AI company, of course there's something shady going on. Because apparently you can take unethical business practices, add AI, and suddenly it's a whole new thing that no one can judge!

Well yes, though I was thinking more that they have no clear way to get income besides VCs and need to figure out a way to monetize in some weird way eventually. I would not have predicted them taking Nvidia money to axe AMD compatibility though lol.

Re: Llama.cpp supports Vulkan. why doesn't Ollama?

#96

Post by 2-3-7-43-1807 »

woadwarrior01 wrote: post #84
2-3-7-43-1807 wrote: post #68

can someone please give a quick summary of the criticism towards ollama? as far as my intel goes it's a mozilla project shouldered mostly by one 10x programmer. i found ollama through hn and last time i didn't notice any lack of trust or suspected sketchiness ... so what changed?

> as far as my intel goes it's a mozilla project shouldered mostly by one 10x programmer. That's completely off the mark. https://www.ycombinator.com/companies/ollama

seems like i confused llamafile with ollama ... this whole llm biotope is a huge mess

Re: Llama.cpp supports Vulkan. why doesn't Ollama?

#97

Post by lolinder »

So many here are trashing on Ollama, saying it's "just" nice porcelain around llama.cpp and it's not doing anything complicated. Okay. Let's stipulate that.

So where's the non-sketchy, non-for-profit equivalent? Where's the nice frontend for llama.cpp that makes it trivial for anyone who wants to play around with local LLMs without having to know much about their internals? If Ollama isn't doing anything difficult, why isn't llama.cpp as easy to use?

Making local LLMs accessible to the masses is an essential job right now—it's important to normalize owning your data as much as it can be normalized. For all of its faults, Ollama does that, and it does it far better than any alternative. Maybe wait to trash it for being "just" a wrapper until someone actually creates a viable alternative.

Re: Llama.cpp supports Vulkan. why doesn't Ollama?

#98

Post by Mashimo »

rfoo wrote: post #55
buyucu wrote: post #38

Earlier quoted context omitted.

I agree they should rename them. But defaulting to a 671b model is also evil.

No. If you can't run it and most people can never run the model on their laptop, it's fine, let people know the fact, instead of giving them illusion.

Letting people download 400GB just to find that out is also .. not optimal.

But yes, I have been "yelled" at on reddit for telling people you need vram in the hundreds of GB.

Re: Llama.cpp supports Vulkan. why doesn't Ollama?

#99

Post by michaelt »

llm_trw wrote: post #18

Can someone explain what the point of ollama is? Every time I look at it, it seems like it's a worse llama.cpp that removes options to make things "easier".

Open-weights LLMs provide a dizzying array of options.

You'd have Llama, Mistral, Gemma, Phi, Yi.

You'd have Llama, Llama 2, Llama 3, Llama 3.2...

And those offer with 8B, 13B or 70B parameters

And you can get it quantised to GGUF, AWQ, exl2...

And quantised to 2, 3, 4, 6 or 8 bits.

And that 4-bit quant is available as Q4_0, Q4_K_S, Q4_K_M...

And on top of that there are a load of fine-tunes that score better on some benchmarks.

Sometimes a model is split into 30 files and you need all 30, other times there's 15 different quants in the same release and you only need a single one. And you have to download from huggingface and put the files in the right place yourself.

ollama takes a lot of that complexity and hides it. You run "ollama run llama3.1" and the selection and download all gets taken care of.

Re: Llama.cpp supports Vulkan. why doesn't Ollama?

#100

Post by bdhcuidbebe »

buyucu wrote: post #69
trash_cat wrote: post #66

I use Ollama because I am a casual user and can't be bothered to read the docs on how to setup llama.cpp. I just want to run a simple llm locally. Why would I care about Vulkan?

with vulkan it runs much much faster on consumer hardware, especially opn igpus like intel or amd.

For Intel, OpenVINO should be the preferred route. I dont follow AMD, but Vulkan is just the common denominator here.
Post reply on HN