Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ollama and gguf

github.com

Post reply on HN
Print view View original

41–50 of 97 posts

Re: Ollama and gguf

#41

Post by anoncareer0212 »

hodgehog11 wrote: post #36
refulgentis wrote: post #35

Earlier quoted context omitted.

This is a good handwave-y answer for them but truth is they've always been allergic to ever mentioning llama.cpp, even when legally required, they made a political decision instead of an engineering one, and now justify it to themselves and you by handwaving about it somehow being less stable than the core of it, which they still depend on. A lot of things happened to get to the point they're getting called out aggre…

It's clear you have a better handle on the situation than I do, so it's a shame you weren't the one to talk to them face-to-face. > llama.cpp has been just fine for me. Of course, so you really shouldn't use Ollama then. Ollama isn't a hobby project anymore, they were the only ones at the table with OpenAI many months before the release of GPT-OSS. I honestly don't think they care one bit about the community drama at…

> ...they were the only ones at the table with OpenAI many months before the release of GPT-OSS

In the spirit of TFA:

This isn't true, at all. I don't know where the idea comes from.

You've been repeating this claim frequently. You were corrected on this 2 hours ago. llama.cpp had early access to it just as well.

It's bizarre for several reasons:

1. It is a fantasy that engineering involves seats at tables and bands of brothers growing from a hobby to a ???, one I find appealing and romantic. But, fantasy nonetheless. Additionally, no one mentioned or implied anything about it being a hobby or unserious.

2. Even if it wasn't a fantasy, it's definitely not what happened here. That's what TFA is about, ffs.

No heroics, they got the ultimate embarrassing thing that can happen to a project piggybacking on FOSS: ollama can't work with the materials OpenAI put out to help ollama users because llama.cpp and ollama had separate day 1 landings of code, and ollama has 0 path to forking literally the entire community to use their format. They were working so loosely with OpenAI that OpenAI assumed they were being sane and weren't attempting to use it as an excuse to force a community fork of GGUF and no one realized until after it shipped.

3. I've seen multiple comments from you this afternoon spiking out odd narratives about Ollama and llama.cpp, that don't make sense at their face from the perspective of someone who also deps on llama.cpp. AFAICT you understood the GGML fork as some halcyon moment of freedom / not-hobbiness for a project you root for. That's fine. Unfortunately, reality is intruding, hence TFA. Given you're aware, it makes your humbleness re: knowing whats going on here sound very fake, especially when it precedes another rush of false claims.

4. I think at some point you owe it to even yourself, if not the community, to take a step back and slow down on the misleading claims. I'm seeing more of a gish-gallop than an attempt to recalibrate your technical understanding.

It's been almost 2 hours since you claimed you were sure there were multiple huge breakages due to bad code quality in llama.cpp, and here, we see you reframe that claim as a much weaker one someone else made to you vaguely.

Maybe a good first step to avoiding information pollution here would be to invest time spent repeating other peoples technical claims you didn't understand, and find some of those breakages you know for sure happened, as promised previously.

In general, I sense a passionate but youthful spirit, not an astro-turfer, and this isn't a group of professionals being disrespected because people still think they're a hobby project. Again, that's what the article is about.

Re: Ollama and gguf

#42

Post by [deleted] »

hodgehog11 wrote: post #13
tarruda wrote: post #8

I recently discovered that ollama no longer uses llama.cpp as a library, and instead they link to the low level library (ggml) which requires them to reinvent a lot of wheel for absolutely no benefit (if there's some benefit I'm missing, please let me know). Even using llama.cpp as a library seems like an overkill for most use cases. Ollama could make its life much easier by spawning llama-server as a subprocess list…

I got to speak with some of the leads at Ollama and asked more or less this same question. The reason they abandoned llama.cpp is because it does not align with their goals. llama.cpp is designed to rapidly adopt research-level optimisations and features, but the downside is that reported speeds change all the time (sometimes faster, sometimes slower) and things break really often. You can't hope to establish contrac…

[deleted]

Re: Ollama and gguf

#44

Post by jychang »

MarkSweep wrote: post #33
tarruda wrote: post #27

Earlier quoted context omitted.

> every time they needed support for a new model and had to update llama.cpp, an old model would break and one of their partners would go ape on them. They said it happened more than once, but one particular case (wish I could remember what it was) was so bad they felt they had no choice but to reimplement. It's the lowest risk strategy. A much lower risk strategy would be using multiple versions of llama-server to k…

The Ollama distribution size is already pretty big (at least on Windows) due to all the GPU support libraries and whatnot. Having to multiple that by the number of llama.cpp versions supported would not be great.

?

    llamacpp> ls -l \*llama\*
    -rwxr-xr-x 1 root root 2505480 Aug  7 05:06 libllama.so
    -rwxr-xr-x 1 root root 5092024 Aug  7 05:23 llama-server
That's a terrible excuse, Llama.cpp is just 7.5 megabytes. You can easily ship a couple copies of that. The current ollama for windows download is 700MB.

I don't buy it. They're not willing to make an 700MB download a few megabytes bigger to ~730MB, but they are willing to support a fork/rewrite indefinitely (and the fork is outside of their core competency, as seen by the current issues)? What kind of decisionmaking is that?

Re: Ollama and gguf

#45

Post by rjzzleep »

hodgehog11 wrote: post #13
tarruda wrote: post #8

I recently discovered that ollama no longer uses llama.cpp as a library, and instead they link to the low level library (ggml) which requires them to reinvent a lot of wheel for absolutely no benefit (if there's some benefit I'm missing, please let me know). Even using llama.cpp as a library seems like an overkill for most use cases. Ollama could make its life much easier by spawning llama-server as a subprocess list…

I got to speak with some of the leads at Ollama and asked more or less this same question. The reason they abandoned llama.cpp is because it does not align with their goals. llama.cpp is designed to rapidly adopt research-level optimisations and features, but the downside is that reported speeds change all the time (sometimes faster, sometimes slower) and things break really often. You can't hope to establish contrac…

Georgi gave a response to some of the issues ollama has in the attached thread[1]

> Looking at ollama's modifications in ggml, they have too much branching in their MXFP4 kernels and the attention sinks implementation is really inefficient. Along with other inefficiencies, I expect the performance is going to be quite bad in ollama.

ollama responded to that

> Ollama has worked to correctly implement MXFP4, and for launch we've worked to validate correctness against the reference implementations against OpenAI's own. > Will share more later, but here is some testing from the public (@ivanfioravanti ) not done by us - and not paid or

leading to another response

> I am sure you worked hard and did your best. > But, this ollama TG graph makes no sense - speed cannot increase at larger context. Do you by any chance limit the context to 8k tokens? > Why is 16k total processing time less than 8k?

Whether or not Ollama's claim is right, I find this "we used your thing, but we know better, we'll share details later" behaviour a bit weird.

[1] https://x.com/ggerganov/status/1953088008816619637

Re: Ollama and gguf

#46

Post by gcr »

wubrr wrote: post #16
tarruda wrote: post #8

I recently discovered that ollama no longer uses llama.cpp as a library, and instead they link to the low level library (ggml) which requires them to reinvent a lot of wheel for absolutely no benefit (if there's some benefit I'm missing, please let me know). Even using llama.cpp as a library seems like an overkill for most use cases. Ollama could make its life much easier by spawning llama-server as a subprocess list…

> Does ollama support strict structured output or strict tool calls adhering to a json schema? As far as I understand this is generally not possible at the model level. Best you can do is wrap the call in a (non-llm) json schema validator, and emit an error json in case the llm output does not match the schema, which is what some APIs do for you, but not very complicated to do yourself. Someone correct me if I'm wron…

This is misinformation. Ollama’s supported structured outputs that conform to a given JSON-schema for months. Here’s a post about this from last year: https://ollama.com/blog/structured-outputs

This is absolutely possible to do at the model level via logit shaping. Llama-cpp’s functionality for this is called GBNF. It’s tightly integrated into the token sampling infrastructure, and is what ollama builds upon for their json schema functionality.

Re: Ollama and gguf

#47

Post by gcr »

12345hn6789 wrote: post #11

Just days ago ollama devs claimed[0] that ollama no longer relies on ggml / llama.cpp. here is their pull request(+165,966 −47,980) to reimplement (copy) llama.cpp code in their repository. https://github.com/ollama/ollama/pull/11823 [0] https://news.ycombinator.com/item?id=44802414#44805396

The PR you linked to says “thanks to the amazing work done by ggml-org” and doesn’t remove GGML code, it instead updates the vendored version and seems to throw away ollama’s custom changes. That’s the opposite of disentangling.

Here’s the maintainer of ggml explaining the context behind this change: https://github.com/ollama/ollama/issues/11714#issuecomment-3...

Re: Ollama and gguf

#48

Post by am17an »

There’s a GitHub link which is open from last year, about the missing license in ollama. They have not bothered to reply, which goes to show how much they care. Also it’s a YC company, I see more and more morally bankrupt companies making the cut recently, why is that?

Re: Ollama and gguf

#49

Post by Ycros »

rjzzleep wrote: post #45
hodgehog11 wrote: post #13

Earlier quoted context omitted.

I got to speak with some of the leads at Ollama and asked more or less this same question. The reason they abandoned llama.cpp is because it does not align with their goals. llama.cpp is designed to rapidly adopt research-level optimisations and features, but the downside is that reported speeds change all the time (sometimes faster, sometimes slower) and things break really often. You can't hope to establish contrac…

Georgi gave a response to some of the issues ollama has in the attached thread[1] > Looking at ollama's modifications in ggml, they have too much branching in their MXFP4 kernels and the attention sinks implementation is really inefficient. Along with other inefficiencies, I expect the performance is going to be quite bad in ollama. ollama responded to that > Ollama has worked to correctly implement MXFP4, and for la…

ollama has always had a weird attitude towards upstream, and then they wonder why many in the community don't like them

Re: Ollama and gguf

#50

Post by ignoramous »

Ycros wrote: post #49
rjzzleep wrote: post #45

Earlier quoted context omitted.

Georgi gave a response to some of the issues ollama has in the attached thread[1] > Looking at ollama's modifications in ggml, they have too much branching in their MXFP4 kernels and the attention sinks implementation is really inefficient. Along with other inefficiencies, I expect the performance is going to be quite bad in ollama. ollama responded to that > Ollama has worked to correctly implement MXFP4, and for la…

ollama has always had a weird attitude towards upstream, and then they wonder why many in the community don't like them

> they wonder why many in the community don't like them

Do they? They probably care more about their "partners".

As GP said:

  By reimplementing this layer, Ollama gets to enjoy a kind of LTS status that their partners rely on
Post reply on HN