Earlier quoted context omitted.
It's fully open source. I mean yes it uses llama.cpp without giving it credit. But why run it in a VM?
It severely over-permissions itself on my Mac.
I have never had it request any permissions
Llama.cpp supports Vulkan. why doesn't Ollama?
131–140 of 235 posts
Re: Llama.cpp supports Vulkan. why doesn't Ollama?
#132Earlier quoted context omitted.
Isn't there a clause in MIT that says you're required to give credit? Also, I didn't know a YC company which started it: https://www.ycombinator.com/companies/ollama .
The project existed in the open source and then subsequently the creators sought funding to work on it full time.
Makes sense!
Re: Llama.cpp supports Vulkan. why doesn't Ollama?
#133Earlier quoted context omitted.
As has been pointed out in this thread in a comment that you replied to (so I know you saw it) [0], Ollama goes to a lot of contortions to support multiple llama.cpp backends. Yes, their solution is a bit of a hack, but it means that the effort to adding a new back end is substantial. And again, they're doing those contortions to make it easy for people. Making it easy involves trade-offs. Yes, Ollama has flaws. They…
This is incorrect. The effort it took to enable Vulkan was relatively minor. The PR is short and to be honest it doesn't do much, because it doesn't need to.
This is a change from your response to the comment that I linked to, where you said it was a good point. Why the difference?
Maybe I should clarify that I'm not saying that the effort to enable a new backend is substantial, I'm saying that my understanding of that comment (the one you acknowledged made a good argument) is that the maintenance burden of having a new backend is substantial.
Re: Llama.cpp supports Vulkan. why doesn't Ollama?
#134So many here are trashing on Ollama, saying it's "just" nice porcelain around llama.cpp and it's not doing anything complicated. Okay. Let's stipulate that. So where's the non-sketchy, non-for-profit equivalent? Where's the nice frontend for llama.cpp that makes it trivial for anyone who wants to play around with local LLMs without having to know much about their internals? If Ollama isn't doing anything difficult, w…
>So where's the non-sketchy, non-for-profit equivalent?
Serving models is currently expensive. I'd argue that some big cloud providers have conspired to make egress bandwidth expensive.
That, coupled with the increasing scale of the internet, make it harder and harder for smaller groups to do these kinds of things. At least until we get some good content addressed distributed storage system.
Re: Llama.cpp supports Vulkan. why doesn't Ollama?
#135ollama was good initially in that it made LLMs more accessible for non-technical people while everyone was figuring things out. Lately they seem to be contributing mostly confusion to the conversation. The #1 model the entire world is talking about is literally mislabeled their side. There is no such thing as R1-1.5b. Quantization without telling users also confuses noobs as to what is possible. Setting up an api dif…
What do you mean there is no such thing as R1-1.5b? DeepSeek released a distilled version based on a 1.5B Qwen model with the full name DeepSeek-R1-Distill-Qwen-1.5B, see chapter 3.2 on page 14 of their research article [0]. [0] https://arxiv.org/abs/2501.12948
Which is not the same model, it's not R1 it's R1-Distill-Qwen-1.5B....
Re: Llama.cpp supports Vulkan. why doesn't Ollama?
#136Earlier quoted context omitted.
This is incorrect. The effort it took to enable Vulkan was relatively minor. The PR is short and to be honest it doesn't do much, because it doesn't need to.
This is a change from your response to the comment that I linked to, where you said it was a good point. Why the difference? Maybe I should clarify that I'm not saying that the effort to enable a new backend is substantial, I'm saying that my understanding of that comment (the one you acknowledged made a good argument) is that the maintenance burden of having a new backend is substantial.
I didn't say it was a good point. I said I disagree, but it's a respectable opinion I could imagine someone having.
Re: Llama.cpp supports Vulkan. why doesn't Ollama?
#137llama.cpp has supported vulkan for more than a year now. For more than 6 months now there has been an open PR to add vulkan backend support for Ollama. However, Ollama team has not even looked at it or commented on it. Vulkan backends are existential for running LLMs on consumer hardware (iGPUs especially). It's sad to see Ollama miss this opportunity.
The PR at issue here blocks iGPUs. My fork of the PR changes removes that: https://github.com/9cb14c1ec0/ollama-vulkan I successfully ran Phi4 on my AMD Ryzen 7 PRO 5850U iGPU with it.
this is great! I think pufferfish is taking PRs to his fork as well.
Re: Llama.cpp supports Vulkan. why doesn't Ollama?
#138Earlier quoted context omitted.
This is a change from your response to the comment that I linked to, where you said it was a good point. Why the difference? Maybe I should clarify that I'm not saying that the effort to enable a new backend is substantial, I'm saying that my understanding of that comment (the one you acknowledged made a good argument) is that the maintenance burden of having a new backend is substantial.
I didn't say it was a good point. I said I disagree, but it's a respectable opinion I could imagine someone having.
Okay, now we're playing semantics. "Reasonable argument" were your words. What changed between then and now to where the same argument is now "incorrect"?
Re: Llama.cpp supports Vulkan. why doesn't Ollama?
#139[deleted]
Re: Llama.cpp supports Vulkan. why doesn't Ollama?
#140So many here are trashing on Ollama, saying it's "just" nice porcelain around llama.cpp and it's not doing anything complicated. Okay. Let's stipulate that. So where's the non-sketchy, non-for-profit equivalent? Where's the nice frontend for llama.cpp that makes it trivial for anyone who wants to play around with local LLMs without having to know much about their internals? If Ollama isn't doing anything difficult, w…
Here you go: https://github.com/hecrj/icebreaker