Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ollama and gguf

github.com

Post reply on HN
Print view View original

31–40 of 97 posts

Re: Ollama and gguf

#31

Post by ekianjo »

scosman wrote: post #29
indigodaddy wrote: post #2

ggerganov explains the issue: https://github.com/ollama/ollama/issues/11714#issuecomment-3...

This is the comment people should read. GG is amazing. Ollama forked to get it working for day 1 compatibility. They need to get their system back in line with mainline because of that choice. That's kinda how open source works. The uproar over this (mostly on reddit and x) seems unwarranted. New models regularly have compatibility issues for much longer than this.

GG clearly mentioned they did not contribute anything to upstream.

Re: Ollama and gguf

#32

Post by [deleted] »

jychang wrote: post #22
hodgehog11 wrote: post #13

Earlier quoted context omitted.

I got to speak with some of the leads at Ollama and asked more or less this same question. The reason they abandoned llama.cpp is because it does not align with their goals. llama.cpp is designed to rapidly adopt research-level optimisations and features, but the downside is that reported speeds change all the time (sometimes faster, sometimes slower) and things break really often. You can't hope to establish contrac…

That's a dumb answer from them. What's wrong with using an older well-tested build of llama.cpp, instead of reinventing the wheel? Like every linux distro ever who's ever ran into this issue? Red Hat doesn't ship the latest build of the linux kernel to production. And Red Hat didn't reinvent the linux kernel for shits and giggles.

[deleted]

Re: Ollama and gguf

#33

Post by MarkSweep »

tarruda wrote: post #27
hodgehog11 wrote: post #23

Earlier quoted context omitted.

The Linux kernel does not break userspace. > What's wrong with using an older well-tested build of llama.cpp, instead of reinventing the wheel? Yeah, they tried this, this was the old setup as I understand it. But every time they needed support for a new model and had to update llama.cpp, an old model would break and one of their partners would go ape on them. They said it happened more than once, but one particular…

> every time they needed support for a new model and had to update llama.cpp, an old model would break and one of their partners would go ape on them. They said it happened more than once, but one particular case (wish I could remember what it was) was so bad they felt they had no choice but to reimplement. It's the lowest risk strategy. A much lower risk strategy would be using multiple versions of llama-server to k…

The Ollama distribution size is already pretty big (at least on Windows) due to all the GPU support libraries and whatnot. Having to multiple that by the number of llama.cpp versions supported would not be great.

Re: Ollama and gguf

#34

Post by ekianjo »

flakiness wrote: post #14
12345hn6789 wrote: post #11

Just days ago ollama devs claimed[0] that ollama no longer relies on ggml / llama.cpp. here is their pull request(+165,966 −47,980) to reimplement (copy) llama.cpp code in their repository. https://github.com/ollama/ollama/pull/11823 [0] https://news.ycombinator.com/item?id=44802414#44805396

not against overall sentiment here, but quote the counterpoint from the linked HN comment to be fair: > Ollama does not use llama.cpp anymore; we do still keep it and occasionally update it to remain compatible for older models for when we used it. The linked PR is doing "occasionally update it" I guess? Note that "vendored" in the PR title often means to take a snapshot to pin a specific version.

gpt-oss is not an "older model"

Re: Ollama and gguf

#35

Post by refulgentis »

hodgehog11 wrote: post #13
tarruda wrote: post #8

I recently discovered that ollama no longer uses llama.cpp as a library, and instead they link to the low level library (ggml) which requires them to reinvent a lot of wheel for absolutely no benefit (if there's some benefit I'm missing, please let me know). Even using llama.cpp as a library seems like an overkill for most use cases. Ollama could make its life much easier by spawning llama-server as a subprocess list…

I got to speak with some of the leads at Ollama and asked more or less this same question. The reason they abandoned llama.cpp is because it does not align with their goals. llama.cpp is designed to rapidly adopt research-level optimisations and features, but the downside is that reported speeds change all the time (sometimes faster, sometimes slower) and things break really often. You can't hope to establish contrac…

This is a good handwave-y answer for them but truth is they've always been allergic to ever mentioning llama.cpp, even when legally required, they made a political decision instead of an engineering one, and now justify it to themselves and you by handwaving about it somehow being less stable than the core of it, which they still depend on.

A lot of things happened to get to the point they're getting called out aggressively in public on their own repo by nice people, and I hope people don't misread a weak excuse made in conversation as solid rationale, based on innuendo. llama.cpp has been just fine for me, running on CI on every platform you can think of, for 2 years.

EDIT: I can't reply, but, see anoncareer2012's reply.

Re: Ollama and gguf

#36

Post by hodgehog11 »

refulgentis wrote: post #35
hodgehog11 wrote: post #13

Earlier quoted context omitted.

I got to speak with some of the leads at Ollama and asked more or less this same question. The reason they abandoned llama.cpp is because it does not align with their goals. llama.cpp is designed to rapidly adopt research-level optimisations and features, but the downside is that reported speeds change all the time (sometimes faster, sometimes slower) and things break really often. You can't hope to establish contrac…

This is a good handwave-y answer for them but truth is they've always been allergic to ever mentioning llama.cpp, even when legally required, they made a political decision instead of an engineering one, and now justify it to themselves and you by handwaving about it somehow being less stable than the core of it, which they still depend on. A lot of things happened to get to the point they're getting called out aggre…

It's clear you have a better handle on the situation than I do, so it's a shame you weren't the one to talk to them face-to-face.

> llama.cpp has been just fine for me.

Of course, so you really shouldn't use Ollama then.

Ollama isn't a hobby project anymore, they were the only ones at the table with OpenAI many months before the release of GPT-OSS. I honestly don't think they care one bit about the community drama at this point. We don't have to like it, but I guess now they get to shape the narrative. That's their stance, and likely the stance of their industry partners too. I'm just the messenger.

Re: Ollama and gguf

#37

Post by cwt137 »

tarruda wrote: post #8

I recently discovered that ollama no longer uses llama.cpp as a library, and instead they link to the low level library (ggml) which requires them to reinvent a lot of wheel for absolutely no benefit (if there's some benefit I'm missing, please let me know). Even using llama.cpp as a library seems like an overkill for most use cases. Ollama could make its life much easier by spawning llama-server as a subprocess list…

It is not true that Ollama doesn't use llama.cpp anymore. They built their own library, which is the default, but also really far from being feature complete. If a model is not supported by their library, they fall back to llama.cpp. For example, there is a group of people trying to get the new IBM models working with Ollama [1]. Their quick/short term solution is to bump the version of llama.cpp included with Ollama to a newer version that has support. And then at a later time, add support in Ollama's library.

1) https://github.com/ollama/ollama/issues/10557

Re: Ollama and gguf

#38

Post by wubrr »

tarruda wrote: post #19
wubrr wrote: post #16

Earlier quoted context omitted.

> Does ollama support strict structured output or strict tool calls adhering to a json schema? As far as I understand this is generally not possible at the model level. Best you can do is wrap the call in a (non-llm) json schema validator, and emit an error json in case the llm output does not match the schema, which is what some APIs do for you, but not very complicated to do yourself. Someone correct me if I'm wron…

The inference engine (llama.CPP) has full control over the possible tokens during inference. It can "force" the llm to output only valid tokens so that it produces valid json

Ahh, I stand corrected, very cool!

Re: Ollama and gguf

#39

Post by wubrr »

kristjansson wrote: post #25
tarruda wrote: post #19

Earlier quoted context omitted.

The inference engine (llama.CPP) has full control over the possible tokens during inference. It can "force" the llm to output only valid tokens so that it produces valid json

and in fact leverages that control to constrain outputs to those matching user-specified BNFs https://github.com/ggml-org/llama.cpp/tree/master/grammars

Very cool!

Re: Ollama and gguf

#40

Post by wubrr »

mangoman wrote: post #17
wubrr wrote: post #16

Earlier quoted context omitted.

> Does ollama support strict structured output or strict tool calls adhering to a json schema? As far as I understand this is generally not possible at the model level. Best you can do is wrap the call in a (non-llm) json schema validator, and emit an error json in case the llm output does not match the schema, which is what some APIs do for you, but not very complicated to do yourself. Someone correct me if I'm wron…

no that's incorrect - llama.cpp has support for providing a context free grammar while sampling and only samples tokens that would conform to the grammar, rather than sampling tokens that would violate the grammar

Very interesting, thank you!
Post reply on HN