Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Llama.cpp supports Vulkan. why doesn't Ollama?

github.com

Post reply on HN
Print view View original

151–160 of 235 posts

Re: Llama.cpp supports Vulkan. why doesn't Ollama?

#151

Post by lolinder »

airstrike wrote: post #140
lolinder wrote: post #97

So many here are trashing on Ollama, saying it's "just" nice porcelain around llama.cpp and it's not doing anything complicated. Okay. Let's stipulate that. So where's the non-sketchy, non-for-profit equivalent? Where's the nice frontend for llama.cpp that makes it trivial for anyone who wants to play around with local LLMs without having to know much about their internals? If Ollama isn't doing anything difficult, w…

Here you go: https://github.com/hecrj/icebreaker

> No pre-built binaries yet! Use cargo to try it out

Not an equivalent yet, sorry.

Re: Llama.cpp supports Vulkan. why doesn't Ollama?

#152

Post by adastra22 »

zozbot234 wrote: post #125
adastra22 wrote: post #116

Earlier quoted context omitted.

I'm not sure that's fair, given that the distilled models are almost as good. Do you really think Deepseek's web interface is giving you access to 671b? They're going to be running distilled models there too.

Given that the 671B model is reportedly MoE-based, it definitely could be powering the web interface and API. MoE slashes the per-inference compute cost - and when serving the model for multiple users you only have to host a single copy of the model params in memory, so the bulk doesn't hurt you as much.

They can still run a lot more users on the same number of GPUs (and they don't have a lot) using distilled models.

Re: Llama.cpp supports Vulkan. why doesn't Ollama?

#153

Post by dogma1138 »

adastra22 wrote: post #108
zozbot234 wrote: post #90

Earlier quoted context omitted.

Metal is an Apple thing, not Intel or AMD. (And Ollama supports that.)

"Close to the metal" is an expression.

Also the name of AMD’s (well ATI) previous attempt at a CUDA alternative ;)

Re: Llama.cpp supports Vulkan. why doesn't Ollama?

#154

Post by axegon_ »

lolinder wrote: post #97

So many here are trashing on Ollama, saying it's "just" nice porcelain around llama.cpp and it's not doing anything complicated. Okay. Let's stipulate that. So where's the non-sketchy, non-for-profit equivalent? Where's the nice frontend for llama.cpp that makes it trivial for anyone who wants to play around with local LLMs without having to know much about their internals? If Ollama isn't doing anything difficult, w…

I think you are missing the point. To get things straight: llama.cpp is not hard to setup and get running. It was a bit of a hassle in 2023 but even then it was not catastrophically complicated if you were willing to read the errors you were getting. People are dissatisfied for two, very valid reasons: ollama gives little to no credit to llama.cpp. The second one is the point of the post: a PR has been open for over 6 months and not a huge PR at that has been completely ignored. Perhaps the ollama maintainers personally don't have use for it so they shrugged it off but this is the equivalent of "it works on my computer". Imagine if all kernel devs used Intel CPUs and ignored every non-intel CPU-related PR. I am not saying that the kernel mailing list is not a large scale version of a countryside pub on a Friday night - it is. But the maintainers do acknowledge the efforts of people making PRs and do a decent job at addressing them. While small, the PR here is not trivial and should have been, at the very least, discussed. Yes, the workstation/server I use for running models uses two Nvidia GPU's. But my desktop computer uses an Intel Arc and in some scenarios, hypothetically, this pr might have been useful.

Re: Llama.cpp supports Vulkan. why doesn't Ollama?

#155

Post by trissi1996 »

LorenDB wrote: post #126
SkyPuncher wrote: post #58

Earlier quoted context omitted.

Ollama - `brew install ollama` llama.cpp - Read the docs, with loads of information and unclear use cases. Question if it has API compatibility and secondary features that a bunch of tools expect. Decide it's not worth your effort when `ollama` is already running by the time you've read the docs

Additionally, Ollama makes model installation a single command. With llama.cpp, you have to download the raw models from Huggingface and handle storage for them yourself.

Not really, llama.cpp can download for quite some time, not as elegant as ollama but:

    llama-server --model-url "https://huggingface.co/bartowski/DeepSeek-R1-Distill-Qwen-32B-GGUF/resolve/main/DeepSeek-R1-Distill-Qwen-32B-IQ4_XS.gguf"
Will get you up and running in one single command.

Re: Llama.cpp supports Vulkan. why doesn't Ollama?

#156

Post by portaouflop »

weinzierl wrote: post #109
portaouflop wrote: post #42

Earlier quoted context omitted.

That’s the curse and blessing of open source I guess? I have billion dollar companies running my oss software without giving me anything - but do I gripe about it in public forums? Yea maybe sometimes but it never helps to improve the situation.

It's the curse of permissively licensed open source. Copyleft is not the answer to everything but against companies leeching and not giving back it is effective.

Well we had this conversation but once you change the license to something like that you lose the trust of the oss community instantly and as elasticsearch etc. show it also doesn’t really help with monetisation

Re: Llama.cpp supports Vulkan. why doesn't Ollama?

#157

Post by Deathmax »

adastra22 wrote: post #116
Deathmax wrote: post #87

Earlier quoted context omitted.

The most recent one of the top of my head is their horrendous aliasing of DeepSeek R1 on their model hub, misleading users into thinking they are running the full model but really anything but the 671b alias is one of the distilled models. This has already led to lots of people claiming that they are running R1 locally when they are not.

I'm not sure that's fair, given that the distilled models are almost as good. Do you really think Deepseek's web interface is giving you access to 671b? They're going to be running distilled models there too.

It's simple enough to test the tokenizer to determine the base model in use (DeepSeek V3, or a Llama 3/Qwen 2.5 distill).

Using the text "സ്മാർട്ട്", Qwen 2.5 tokenizes as 10 tokens, Llama 3 as 13, and DeepSeek V3 as 8.

Using DeepSeek's chat frontend, both DeepSeek V3 and R1 returns the following response (SSE events edited for brevity):

  {"content":"സ","type":"text"},"chunk_token_usage":1
  {"content":"്മ","type":"text"},"chunk_token_usage":2
  {"content":"ാ","type":"text"},"chunk_token_usage":1
  {"content":"ർ","type":"text"},"chunk_token_usage":1
  {"content":"ട","type":"text"},"chunk_token_usage":1
  {"content":"്ട","type":"text"},"chunk_token_usage":1
  {"content":"്","type":"text"},"chunk_token_usage":1
which totals to 8, as expected for DeepSeek V3's tokenizer.

Re: Llama.cpp supports Vulkan. why doesn't Ollama?

#158

Post by portaouflop »

sitkack wrote: post #56
portaouflop wrote: post #42

Earlier quoted context omitted.

That’s the curse and blessing of open source I guess? I have billion dollar companies running my oss software without giving me anything - but do I gripe about it in public forums? Yea maybe sometimes but it never helps to improve the situation.

Are they a wrapper with a similar name? You, like I, do gripe in public forums.

No it’s totally different from this case - but the software is a super important part of their stack - if it were to stop working they can shut down their business

Re: Llama.cpp supports Vulkan. why doesn't Ollama?

#159

Post by smcleod »

chown wrote: post #114
lolinder wrote: post #97

So many here are trashing on Ollama, saying it's "just" nice porcelain around llama.cpp and it's not doing anything complicated. Okay. Let's stipulate that. So where's the non-sketchy, non-for-profit equivalent? Where's the nice frontend for llama.cpp that makes it trivial for anyone who wants to play around with local LLMs without having to know much about their internals? If Ollama isn't doing anything difficult, w…

I totally agree with this. I wanted to make it really easy for non-technical users with an app that hid all the complexities. I basically just wanted to embed the engine without making users open their terminal, let alone make them configure. I started with llama.cpp amd almost gave up on the idea before I stumbled upon Ollama, which made the app happen[1] There are many flaws in Ollama but it makes many things much…

That qkv PR was mine! Small world.

Re: Llama.cpp supports Vulkan. why doesn't Ollama?

#160

Post by woadwarrior01 »

washadjeffmad wrote: post #149
lolinder wrote: post #97

So many here are trashing on Ollama, saying it's "just" nice porcelain around llama.cpp and it's not doing anything complicated. Okay. Let's stipulate that. So where's the non-sketchy, non-for-profit equivalent? Where's the nice frontend for llama.cpp that makes it trivial for anyone who wants to play around with local LLMs without having to know much about their internals? If Ollama isn't doing anything difficult, w…

>So where's the non-sketchy, non-for-profit equivalent llama.cpp, kobold.cpp, oobabooga, llmstudio, etc. There are dozens at this point. And while many chalk the attachment to ollama up to a "skill issue", that's just venting frustration that all something has to do to win the popularity contest is to repackage and market it as an "app". I prefer first-party tools, I'm comfortable managing a build environment and cal…

> llmstudio

ICYMI, you might want to read their terms of use:

https://lmstudio.ai/terms

Post reply on HN