Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Llama.cpp supports Vulkan. why doesn't Ollama?

github.com

Post reply on HN
Print view View original

191–200 of 235 posts

Re: Llama.cpp supports Vulkan. why doesn't Ollama?

#191

Post by dingocat »

Havoc wrote: post #164
dingocat wrote: post #122

Earlier quoted context omitted.

What do you mean there is no such thing as R1-1.5b? DeepSeek released a distilled version based on a 1.5B Qwen model with the full name DeepSeek-R1-Distill-Qwen-1.5B, see chapter 3.2 on page 14 of their research article [0]. [0] https://arxiv.org/abs/2501.12948

ollama labels the qwen models R1, while the "R1" moniker standing on its own in deepseek world means the full model that has nothing to do with qwen. https://ollama.com/library/deepseek-r1 That may have been ok if it was just same model at different sizes but they're completely different things here & it's created confusion out of thin air for absolutely no reason other than ollama being careless.

And their documentation makes that distinction clear, having dedicated a section specifically to the distilled models.

Re: Llama.cpp supports Vulkan. why doesn't Ollama?

#192

Post by diggan »

diggan wrote: post #75
7thpower wrote: post #24

Earlier quoted context omitted.

Can you please explain why you think they may be operating in bad faith?

Not parent, but same feeling. First I got the feeling because of how they store things on disk and try to get all models rehosted in their own closed library. Second time I got the feeling is when it's not obvious at all about what their motives are, and that it's a for-profit venture. Third time is trying to discuss things in their Discord and the moderators there constantly shut down a lot of conversation citing "M…

Ollama team's response (verbatim) when asking what they think of the comments about Ollama in this HN submission: "Who cares? It's the internet... everybody has an opinion... and they're usually bad". Not exactly the response you'd expect from people who should ideally learn from what others think (correct or not) about your project.

Re: Llama.cpp supports Vulkan. why doesn't Ollama?

#193

Post by bdavbdav »

v9v wrote: post #182
evilduck wrote: post #169

Earlier quoted context omitted.

> llama.cpp, kobold.cpp, oobabooga None of these three are remotely as easy to install or use. They could be, but none of them are even trying. > lmstudio This is a closed source app with a non-free license from a business not making money. Enshittification is just a matter of when.

I would argue that kobold.cpp is even easier to use than Ollama. You click on the link in the README to download an .exe and doubleclick it and select your model file. No command line involved. Which part of the user experience did you have problems with when using it?

You’re coming at it from a point of knowledge. Read the first sentence of the Ollama website against the first paragraph of kobold’s GitHub. Newcomers don’t have a clue what “running a GGUF model..” means. It’s written by tech folk without an understanding of the audience.

Re: Llama.cpp supports Vulkan. why doesn't Ollama?

#194

Post by diggan »

pornel wrote: post #67
llm_trw wrote: post #18

Can someone explain what the point of ollama is? Every time I look at it, it seems like it's a worse llama.cpp that removes options to make things "easier".

Ollama : llama.cpp :: Dropbox : rsync

Not sure this is a good analogy. LM Studio is closer to Dropbox as both takes X and makes it easier for users who don't necessarily are very technical. Ollama is a developer-oriented tool (used via terminal + a daemon), so wouldn't compare it to what Dropbox is/did for file syncing.

Re: Llama.cpp supports Vulkan. why doesn't Ollama?

#195

Post by zozbot234 »

paradite wrote: post #185
buyucu wrote: post #57

Earlier quoted context omitted.

check the PR, it's a very short one. It's not more complicated than setting a compile time flag. I have no idea why they have been ignoring it. Ollama is just a friendly front end for llama.cpp. It doesn't have to do any of those things you mentioned. Llama.cpp does all that.

At the risk of being pedantic (I don't know much about C++ and I'm genuinely curious), if Ollama is really just a wrapper around Llama.cpp, why would it need the Vulkan specific flags? Shouldn't it just call Llama.cpp and let Llama.cpp handle the flags internally within Llama.cpp? I'm thinking from an abstraction layer perspective.

The Vulkan-specific flags are needed (1) to set up the llama.cpp build options when building Ollama w/ Vulkan support - which apparently is still a challenge with the current PR, if the latest comments on the GitHub page are accurate; also (2) to pick how many model layers should be run on the GPU, depending on available GPU memory. Llama.cpp doesn't do that for you, you have to set that option yourself or just tell it to move "everything", which often fails with an error. (Finding the right amount is actually a trial-and-error process which depends on the model, quantization and also varies depending on how much context you have in the current conversation. If you have too many layers loaded and too little GPU memory, a large context can result in unpredictable breakage.)

Re: Llama.cpp supports Vulkan. why doesn't Ollama?

#196

Post by diggan »

quibono wrote: post #6

Is Ollama just the porcelain around llama.cpp? Or is there more to it than that?

They also decided to rehost the model files in their own (closed) library/repository + store the files split into layers on disk, so you cannot easily reuse model-files between applications. I think the point is that models can share layers, I'm not sure how much space you actually save, I just know that if you use both LM Studio + Ollama you cannot share models but if you use LM Studio + llama.cpp you can share the same files between them, no need to download duplicate model weights.

Re: Llama.cpp supports Vulkan. why doesn't Ollama?

#198

Post by diggan »

Mashimo wrote: post #98
rfoo wrote: post #55

Earlier quoted context omitted.

No. If you can't run it and most people can never run the model on their laptop, it's fine, let people know the fact, instead of giving them illusion.

Letting people download 400GB just to find that out is also .. not optimal. But yes, I have been "yelled" at on reddit for telling people you need vram in the hundreds of GB.

> Letting people download 400GB just to find that out is also .. not optimal.

Letting people download any amount of bytes just to find out they got something else isn't optimal. So what to do? Highlight the differences when you reference them so people understand.

Tweets like these: https://x.com/ollama/status/1881427522002506009

> DeepSeek's first-generation reasoning models are achieving performance comparable to OpenAI's o1 across math, code, and reasoning tasks! Give it a try! 7B distilled: ollama run deepseek-r1:7b

Are really misleading. Reading the first part, you think the second part is that model that gives "performance comparable to OpenAI's o1" but it's not, it's a distilled model with way worse performance. Yes, they do say it's the distilled model, but I hope I'm not alone in seeing how people less careful would confuse the two.

If they're doing this on purpose, I'd leave a very bad taste in my mouth. If they're doing this accidentally, it also gives me reason to pause and re-evaluate what they're doing.

Re: Llama.cpp supports Vulkan. why doesn't Ollama?

#199

Post by diggan »

bdavbdav wrote: post #193
v9v wrote: post #182

Earlier quoted context omitted.

I would argue that kobold.cpp is even easier to use than Ollama. You click on the link in the README to download an .exe and doubleclick it and select your model file. No command line involved. Which part of the user experience did you have problems with when using it?

You’re coming at it from a point of knowledge. Read the first sentence of the Ollama website against the first paragraph of kobold’s GitHub. Newcomers don’t have a clue what “running a GGUF model..” means. It’s written by tech folk without an understanding of the audience.

Ollama is also written for technical/developer users, by accident (it seems), even though they don't want it to be strictly for technical users. I've opened a issue asking them to make it more clear that Ollama is for technical users, but they seem confident people with no terminal experience can and will also use Ollama: https://github.com/ollama/ollama/issues/7116

Re: Llama.cpp supports Vulkan. why doesn't Ollama?

#200

Post by paradite »

zozbot234 wrote: post #195
paradite wrote: post #185

Earlier quoted context omitted.

At the risk of being pedantic (I don't know much about C++ and I'm genuinely curious), if Ollama is really just a wrapper around Llama.cpp, why would it need the Vulkan specific flags? Shouldn't it just call Llama.cpp and let Llama.cpp handle the flags internally within Llama.cpp? I'm thinking from an abstraction layer perspective.

The Vulkan-specific flags are needed (1) to set up the llama.cpp build options when building Ollama w/ Vulkan support - which apparently is still a challenge with the current PR, if the latest comments on the GitHub page are accurate; also (2) to pick how many model layers should be run on the GPU, depending on available GPU memory. Llama.cpp doesn't do that for you, you have to set that option yourself or just tell…

Thanks a lot for the explanation.

If I can ask one more question, why don't Ollama use binaries of pre-built llama.cpp with Vulkan support directly?

Post reply on HN