Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ollama and gguf

github.com

Post reply on HN
Print view View original

51–60 of 97 posts

Re: Ollama and gguf

#51

Post by sunnycoder5 »

for folks wrestling with Ollama, llama.cpp or local LLM versioning - did you guys check out Docker's new feature - Docker Model Runner?

Docker Model Runner makes it easy to manage, run, and deploy AI models using Docker. Designed for developers, Docker Model Runner streamlines the process of pulling, running, and serving large language models (LLMs) and other AI models directly from Docker Hub or any OCI-compliant registry.

Whether you're building generative AI applications, experimenting with machine learning workflows, or integrating AI into your software development lifecycle, Docker Model Runner provides a consistent, secure, and efficient way to work with AI models locally.

For more details check this out : https://docs.docker.com/ai/model-runner/

Re: Ollama and gguf

#52

Post by vlovich123 »

jychang wrote: post #44
MarkSweep wrote: post #33

Earlier quoted context omitted.

The Ollama distribution size is already pretty big (at least on Windows) due to all the GPU support libraries and whatnot. Having to multiple that by the number of llama.cpp versions supported would not be great.

? llamacpp> ls -l \*llama\* -rwxr-xr-x 1 root root 2505480 Aug 7 05:06 libllama.so -rwxr-xr-x 1 root root 5092024 Aug 7 05:23 llama-server That's a terrible excuse, Llama.cpp is just 7.5 megabytes. You can easily ship a couple copies of that. The current ollama for windows download is 700MB. I don't buy it. They're not willing to make an 700MB download a few megabytes bigger to ~730MB, but they are willing to support…

It’s 700mib because they’re likely redistributing the CUDA libraries so that users don’t need to separately run that installer. Llama.cpp is a bit more “you are expected to know what you’re doing” on that front. But yeah, you could plausibly ship multiple versions of the inference engine although from a maintenance perspective that sounds like hell for any number of reasons

Re: Ollama and gguf

#53

Post by pduggishetti »

am17an wrote: post #48

There’s a GitHub link which is open from last year, about the missing license in ollama. They have not bothered to reply, which goes to show how much they care. Also it’s a YC company, I see more and more morally bankrupt companies making the cut recently, why is that?

I think most of them were morally bankrupt, you might just be realizing now.

Re: Ollama and gguf

#54

Post by indigodaddy »

iamshrimpy wrote: post #43

This title makes no sense and it links nowhere helpful. It's "Ollama's forked ggml is incompatible with other gpt-oss GGUFs" and it should link to GG's comment[0] [0] https://github.com/ollama/ollama/issues/11714#issuecomment-3...

HN strips any "?|#" etc at the end of URLs, and you cannot edit the URL after submissions (like you can the title).

As far as the title, yeah I didn't work too hard on making it good, sorry

Re: Ollama and gguf

#55

Post by hodgehog11 »

anoncareer0212 wrote: post #41
hodgehog11 wrote: post #36

Earlier quoted context omitted.

It's clear you have a better handle on the situation than I do, so it's a shame you weren't the one to talk to them face-to-face. > llama.cpp has been just fine for me. Of course, so you really shouldn't use Ollama then. Ollama isn't a hobby project anymore, they were the only ones at the table with OpenAI many months before the release of GPT-OSS. I honestly don't think they care one bit about the community drama at…

> ...they were the only ones at the table with OpenAI many months before the release of GPT-OSS In the spirit of TFA: This isn't true, at all. I don't know where the idea comes from. You've been repeating this claim frequently. You were corrected on this 2 hours ago . llama.cpp had early access to it just as well. It's bizarre for several reasons: 1. It is a fantasy that engineering involves seats at tables and bands…

Wow, I wasn't expecting this. These are fair critiques, as I am only somewhat informed about what is clearly a very sensitive topic.

For transparency, I attended ICML2025 where Ollama had set up a booth and had a casual conversation with the representatives there (one of whom turned out to lead the Ollama project) before they went to their 2nd birthday celebration. I'm repeating what I can remember from the conversation, about ten minutes or so. I am a researcher not affiliated with the development of llama.cpp or Ollama.

> for a project you root for

I don't use Ollama, and I certainly don't root for it. I'm a little disappointed that people would assume this. I also don't use llama.cpp and it seems that is the problem. I'm not really interested in the drama, I just want to understand what these projects want to do. I work in theory and try to stay up to date on how the general public can run LLMs locally.

> no one realized until after it shipped.

I'm not sensing that the devs at Ollama are particularly competent, especially when compared to the behemoths at llama.cpp. To me, this helps explain why their actions differ from their claimed motivation, but this is probably because I prefer to assume incompetence over something sinister.

> as promised previously...

I don't think I made any such promises. I can't cite those claimed breakages, because I do not remember further details from the conversation needed to find them. The guy made a strong point to claim they had happened and there was enough frustrated rambling there to believe him. If I had more, I would have cited them. I remember seeing news regarding the deprecation of multimodal support hence the "I could swear that" comment (although I regret this language, and wish I could edit the comment to tone it down a bit), but I do not think this was what the rep cited. I had hoped that someone could fill in the blanks there, but if knowledgeable folks claim this is impossible (which is hard to believe for a project of this size, but I digress), I defer to their expert opinion here.

> llama.cpp had early access to it just as well.

I knew this from the conversation, but was told Ollama had even earlier discussions with OpenAI as the initial point of contact. Again, this is what I was told, so feel free to critique it. At that time, the rep could not explicitly disclose that it was OpenAI, but it was pretty obvious from the timing due to the delay.

> avoiding information pollution

I'm a big believer in free speech and that the truth will always come out eventually.

> I'm seeing more of a gish-gallop than an attempt to recalibrate your technical understanding...

> I sense a passionate but youthful spirit, not an astro-turfer,

This is pretty humbling, and frankly comes off a little patronizing, but I suppose this is what happens when I step out of my lane. My objective was to stimulate further conversation and share a perspective I thought was unique. I can see this was not welcome, my apologies.

Re: Ollama and gguf

#58

Post by buyucu »

hodgehog11 wrote: post #13
tarruda wrote: post #8

I recently discovered that ollama no longer uses llama.cpp as a library, and instead they link to the low level library (ggml) which requires them to reinvent a lot of wheel for absolutely no benefit (if there's some benefit I'm missing, please let me know). Even using llama.cpp as a library seems like an overkill for most use cases. Ollama could make its life much easier by spawning llama-server as a subprocess list…

I got to speak with some of the leads at Ollama and asked more or less this same question. The reason they abandoned llama.cpp is because it does not align with their goals. llama.cpp is designed to rapidly adopt research-level optimisations and features, but the downside is that reported speeds change all the time (sometimes faster, sometimes slower) and things break really often. You can't hope to establish contrac…

this looks ok on paper, but isn't realized in reality. ollama is full of bugs, problems and issues llama.cpp has solved ages ago. this thread is a good example of that.

Re: Ollama and gguf

#60

Post by zozbot234 »

hodgehog11 wrote: post #23
jychang wrote: post #22

Earlier quoted context omitted.

That's a dumb answer from them. What's wrong with using an older well-tested build of llama.cpp, instead of reinventing the wheel? Like every linux distro ever who's ever ran into this issue? Red Hat doesn't ship the latest build of the linux kernel to production. And Red Hat didn't reinvent the linux kernel for shits and giggles.

The Linux kernel does not break userspace. > What's wrong with using an older well-tested build of llama.cpp, instead of reinventing the wheel? Yeah, they tried this, this was the old setup as I understand it. But every time they needed support for a new model and had to update llama.cpp, an old model would break and one of their partners would go ape on them. They said it happened more than once, but one particular…

> Yeah, they tried this, this was the old setup as I understand it. But every time they needed support for a new model and had to update llama.cpp, an old model would break and one of their partners would go ape on them.

Shouldn't any such regressions be regarded as bugs in llama.cpp and fixed there? Surely the Ollama folks can test and benchmark the main models that people care about before shipping the update in a stable release. That would be a lot easier than trying to reimplement major parts of llama.cpp from scratch.

Post reply on HN