Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ollama and gguf

github.com

Post reply on HN
Print view View original

61–70 of 97 posts

Re: Ollama and gguf

#61

Post by brabel »

llmthrowaway wrote: post #7

Confusing title - thought this was about Ollama finally supporting sharded GGUF (ie. the Huggingface default for large gguf over 48gb). https://github.com/ollama/ollama/issues/5245 Sadly it is not and the issue still remains open after over a year meaning ollama cannot run the latest SOTA open source models unless they covert them to their proprietary format which they do not consistently do. No surprise I guess give…

I want to add an inference engine to my product. I was hoping to use ollama because it really helps, I think, make sure you have a model with the right metadata that you can count on working (I've seen that with llama.cpp, it's easy to get the metadata wrong and start getting rubbish from the LLM because the "stop_token" was wrong or something). I'd thought ollama was a proponent of the GGUF, which I really like as it standardizes metadata?!

What would be the best way to use llama.cpp and models that use GGUF these days? ramallama is a good alternative (I guess it is, but it's not completely clear from your message)? Or just use llama.cpp directly, in which case how to ensure I don't get rubbish (like the model asking and answering questions by itself without ever stopping)??

Re: Ollama and gguf

#62

Post by diimdeep »

ollama is certain in the future rent seeking wrapper by docker fame.

classic Docker Hub playbook: spread the habit for free → capture workflows → charge for scale.

moat isn't in inference speed — it’s in controlling the distribution & default UX, once they own that, can start rent-gate it.

Re: Ollama and gguf

#63

Post by elaus »

buyucu wrote: post #56

ollama is a lost cause. they are going through a very aggressive phase of enshittification right now.

What is the currently favored alternative for simply running 1-2 models locally, exposed via an API? One big advantage of Ollama seems to be that they provide fully configured models, so I don't have to fiddle with stop words, etc.

Re: Ollama and gguf

#64

Post by buyucu »

elaus wrote: post #63
buyucu wrote: post #56

ollama is a lost cause. they are going through a very aggressive phase of enshittification right now.

What is the currently favored alternative for simply running 1-2 models locally, exposed via an API? One big advantage of Ollama seems to be that they provide fully configured models, so I don't have to fiddle with stop words, etc.

I just use llama.cpp server. It works really well. Some people recommend llama-swap or kobold but I never tried them.

Re: Ollama and gguf

#65

Post by tarruda »

gcr wrote: post #46
wubrr wrote: post #16

Earlier quoted context omitted.

> Does ollama support strict structured output or strict tool calls adhering to a json schema? As far as I understand this is generally not possible at the model level. Best you can do is wrap the call in a (non-llm) json schema validator, and emit an error json in case the llm output does not match the schema, which is what some APIs do for you, but not very complicated to do yourself. Someone correct me if I'm wron…

This is misinformation. Ollama’s supported structured outputs that conform to a given JSON-schema for months. Here’s a post about this from last year: https://ollama.com/blog/structured-outputs This is absolutely possible to do at the model level via logit shaping. Llama-cpp’s functionality for this is called GBNF. It’s tightly integrated into the token sampling infrastructure, and is what ollama builds upon for thei…

> It’s tightly integrated into the token sampling infrastructure, and is what ollama builds upon for their json schema functionality.

Do you mean the functionality of generating ebnf grammar and from a json schema use it for sampling is part of ggml, and all they have to do is use it?

I assumed that this was part of llama.cpp, and another feature they have to re-implement and maintain.

Re: Ollama and gguf

#66

Post by mijoharas »

hodgehog11 wrote: post #13
tarruda wrote: post #8

I recently discovered that ollama no longer uses llama.cpp as a library, and instead they link to the low level library (ggml) which requires them to reinvent a lot of wheel for absolutely no benefit (if there's some benefit I'm missing, please let me know). Even using llama.cpp as a library seems like an overkill for most use cases. Ollama could make its life much easier by spawning llama-server as a subprocess list…

I got to speak with some of the leads at Ollama and asked more or less this same question. The reason they abandoned llama.cpp is because it does not align with their goals. llama.cpp is designed to rapidly adopt research-level optimisations and features, but the downside is that reported speeds change all the time (sometimes faster, sometimes slower) and things break really often. You can't hope to establish contrac…

> llama.cpp is designed to rapidly adopt research-level optimisations and features, but the downside is that reported speeds change all the time

Ironic that (according to the article) ollama rushed to implement GPT-OSS support, and thus broke the rest of the gguf quants (iiuc correctly).

Re: Ollama and gguf

#68

Post by arcanemachiner »

elaus wrote: post #63
buyucu wrote: post #56

ollama is a lost cause. they are going through a very aggressive phase of enshittification right now.

What is the currently favored alternative for simply running 1-2 models locally, exposed via an API? One big advantage of Ollama seems to be that they provide fully configured models, so I don't have to fiddle with stop words, etc.

llama-swap if you need more than 1 model. It wraps llama.cpp, and has a Docker container version that is pretty easy to work with.

Re: Ollama and gguf

#69

Post by buyucu »

clarionbell wrote: post #67

Why is anyone still using this? You can spin up llama.cpp server and have more optimized runtime. And if you insist on containers you can go for ramallama https://ramalama.ai/

I think people just don't know any better. I also used Ollama way longer than I should have. I didn't know that Ollama was just llama.cpp with a thin wrapper. My quality of life improved a lot after I discovered llama.cpp.

Re: Ollama and gguf

#70

Post by otabdeveloper4 »

hodgehog11 wrote: post #13
tarruda wrote: post #8

I recently discovered that ollama no longer uses llama.cpp as a library, and instead they link to the low level library (ggml) which requires them to reinvent a lot of wheel for absolutely no benefit (if there's some benefit I'm missing, please let me know). Even using llama.cpp as a library seems like an overkill for most use cases. Ollama could make its life much easier by spawning llama-server as a subprocess list…

I got to speak with some of the leads at Ollama and asked more or less this same question. The reason they abandoned llama.cpp is because it does not align with their goals. llama.cpp is designed to rapidly adopt research-level optimisations and features, but the downside is that reported speeds change all the time (sometimes faster, sometimes slower) and things break really often. You can't hope to establish contrac…

> it does not align with their goals

Ollama is a scam trying to E-E-E the rising hype wave of local LLMs while the getting is still good.

Sorry, but somebody has to voice the elephant in the room here.

Post reply on HN