Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ollama and gguf

github.com

Post reply on HN
Print view View original

21–30 of 97 posts

Re: Ollama and gguf

#21

Post by [deleted] »

hodgehog11 wrote: post #13
tarruda wrote: post #8

I recently discovered that ollama no longer uses llama.cpp as a library, and instead they link to the low level library (ggml) which requires them to reinvent a lot of wheel for absolutely no benefit (if there's some benefit I'm missing, please let me know). Even using llama.cpp as a library seems like an overkill for most use cases. Ollama could make its life much easier by spawning llama-server as a subprocess list…

I got to speak with some of the leads at Ollama and asked more or less this same question. The reason they abandoned llama.cpp is because it does not align with their goals. llama.cpp is designed to rapidly adopt research-level optimisations and features, but the downside is that reported speeds change all the time (sometimes faster, sometimes slower) and things break really often. You can't hope to establish contrac…

[deleted]

Re: Ollama and gguf

#22

Post by jychang »

hodgehog11 wrote: post #13
tarruda wrote: post #8

I recently discovered that ollama no longer uses llama.cpp as a library, and instead they link to the low level library (ggml) which requires them to reinvent a lot of wheel for absolutely no benefit (if there's some benefit I'm missing, please let me know). Even using llama.cpp as a library seems like an overkill for most use cases. Ollama could make its life much easier by spawning llama-server as a subprocess list…

I got to speak with some of the leads at Ollama and asked more or less this same question. The reason they abandoned llama.cpp is because it does not align with their goals. llama.cpp is designed to rapidly adopt research-level optimisations and features, but the downside is that reported speeds change all the time (sometimes faster, sometimes slower) and things break really often. You can't hope to establish contrac…

That's a dumb answer from them.

What's wrong with using an older well-tested build of llama.cpp, instead of reinventing the wheel? Like every linux distro ever who's ever ran into this issue?

Red Hat doesn't ship the latest build of the linux kernel to production. And Red Hat didn't reinvent the linux kernel for shits and giggles.

Re: Ollama and gguf

#23

Post by hodgehog11 »

jychang wrote: post #22
hodgehog11 wrote: post #13

Earlier quoted context omitted.

I got to speak with some of the leads at Ollama and asked more or less this same question. The reason they abandoned llama.cpp is because it does not align with their goals. llama.cpp is designed to rapidly adopt research-level optimisations and features, but the downside is that reported speeds change all the time (sometimes faster, sometimes slower) and things break really often. You can't hope to establish contrac…

That's a dumb answer from them. What's wrong with using an older well-tested build of llama.cpp, instead of reinventing the wheel? Like every linux distro ever who's ever ran into this issue? Red Hat doesn't ship the latest build of the linux kernel to production. And Red Hat didn't reinvent the linux kernel for shits and giggles.

The Linux kernel does not break userspace.

> What's wrong with using an older well-tested build of llama.cpp, instead of reinventing the wheel?

Yeah, they tried this, this was the old setup as I understand it. But every time they needed support for a new model and had to update llama.cpp, an old model would break and one of their partners would go ape on them. They said it happened more than once, but one particular case (wish I could remember what it was) was so bad they felt they had no choice but to reimplement. It's the lowest risk strategy.

Re: Ollama and gguf

#24

Post by hodgehog11 »

A4ET8a8uTh0_v2 wrote: post #20
hodgehog11 wrote: post #13

Earlier quoted context omitted.

I got to speak with some of the leads at Ollama and asked more or less this same question. The reason they abandoned llama.cpp is because it does not align with their goals. llama.cpp is designed to rapidly adopt research-level optimisations and features, but the downside is that reported speeds change all the time (sometimes faster, sometimes slower) and things break really often. You can't hope to establish contrac…

Thank you. This is genuinely a valid reason even from a simple consistency perspective. (edit: I think -- after I read some of the links -- I understand why Ollama comes across as less of a hero. Still, I am giving them some benefit of the doubt since they made local models very accessible to plebs like me; and maybe I can graduate to no ollama )

I think this is the thing: if you can use llama.cpp, you probably shouldn't use Ollama. It's designed for the beginner.

Re: Ollama and gguf

#25

Post by kristjansson »

tarruda wrote: post #19
wubrr wrote: post #16

Earlier quoted context omitted.

> Does ollama support strict structured output or strict tool calls adhering to a json schema? As far as I understand this is generally not possible at the model level. Best you can do is wrap the call in a (non-llm) json schema validator, and emit an error json in case the llm output does not match the schema, which is what some APIs do for you, but not very complicated to do yourself. Someone correct me if I'm wron…

The inference engine (llama.CPP) has full control over the possible tokens during inference. It can "force" the llm to output only valid tokens so that it produces valid json

and in fact leverages that control to constrain outputs to those matching user-specified BNFs

https://github.com/ggml-org/llama.cpp/tree/master/grammars

Re: Ollama and gguf

#26

Post by [deleted] »

wubrr wrote: post #16
tarruda wrote: post #8

I recently discovered that ollama no longer uses llama.cpp as a library, and instead they link to the low level library (ggml) which requires them to reinvent a lot of wheel for absolutely no benefit (if there's some benefit I'm missing, please let me know). Even using llama.cpp as a library seems like an overkill for most use cases. Ollama could make its life much easier by spawning llama-server as a subprocess list…

> Does ollama support strict structured output or strict tool calls adhering to a json schema? As far as I understand this is generally not possible at the model level. Best you can do is wrap the call in a (non-llm) json schema validator, and emit an error json in case the llm output does not match the schema, which is what some APIs do for you, but not very complicated to do yourself. Someone correct me if I'm wron…

[deleted]

Re: Ollama and gguf

#27

Post by tarruda »

hodgehog11 wrote: post #23
jychang wrote: post #22

Earlier quoted context omitted.

That's a dumb answer from them. What's wrong with using an older well-tested build of llama.cpp, instead of reinventing the wheel? Like every linux distro ever who's ever ran into this issue? Red Hat doesn't ship the latest build of the linux kernel to production. And Red Hat didn't reinvent the linux kernel for shits and giggles.

The Linux kernel does not break userspace. > What's wrong with using an older well-tested build of llama.cpp, instead of reinventing the wheel? Yeah, they tried this, this was the old setup as I understand it. But every time they needed support for a new model and had to update llama.cpp, an old model would break and one of their partners would go ape on them. They said it happened more than once, but one particular…

> every time they needed support for a new model and had to update llama.cpp, an old model would break and one of their partners would go ape on them. They said it happened more than once, but one particular case (wish I could remember what it was) was so bad they felt they had no choice but to reimplement. It's the lowest risk strategy.

A much lower risk strategy would be using multiple versions of llama-server to keep supporting old models that would break on newer llama.cpp versions.

Re: Ollama and gguf

#28

Post by ozim »

hodgehog11 wrote: post #13
tarruda wrote: post #8

I recently discovered that ollama no longer uses llama.cpp as a library, and instead they link to the low level library (ggml) which requires them to reinvent a lot of wheel for absolutely no benefit (if there's some benefit I'm missing, please let me know). Even using llama.cpp as a library seems like an overkill for most use cases. Ollama could make its life much easier by spawning llama-server as a subprocess list…

I got to speak with some of the leads at Ollama and asked more or less this same question. The reason they abandoned llama.cpp is because it does not align with their goals. llama.cpp is designed to rapidly adopt research-level optimisations and features, but the downside is that reported speeds change all the time (sometimes faster, sometimes slower) and things break really often. You can't hope to establish contrac…

Feels like BS I guess wrapping 2 or even more versions should not be that much of a problem.

There was drama that ollama doesn’t credit llama.cpp and most likely crediting it was „not aligning with their goals”.

Re: Ollama and gguf

#29

Post by scosman »

indigodaddy wrote: post #2

ggerganov explains the issue: https://github.com/ollama/ollama/issues/11714#issuecomment-3...

This is the comment people should read. GG is amazing.

Ollama forked to get it working for day 1 compatibility. They need to get their system back in line with mainline because of that choice. That's kinda how open source works.

The uproar over this (mostly on reddit and x) seems unwarranted. New models regularly have compatibility issues for much longer than this.

Post reply on HN