Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ollama and gguf

github.com

Post reply on HN
Print view View original

81–90 of 97 posts

Re: Ollama and gguf

#81

Post by dpkirchner »

otabdeveloper4 wrote: post #70
hodgehog11 wrote: post #13

Earlier quoted context omitted.

I got to speak with some of the leads at Ollama and asked more or less this same question. The reason they abandoned llama.cpp is because it does not align with their goals. llama.cpp is designed to rapidly adopt research-level optimisations and features, but the downside is that reported speeds change all the time (sometimes faster, sometimes slower) and things break really often. You can't hope to establish contrac…

> it does not align with their goals Ollama is a scam trying to E-E-E the rising hype wave of local LLMs while the getting is still good. Sorry, but somebody has to voice the elephant in the room here.

It'd be easy enough for ollama alternatives -- they just need to make a CLI front end that lets you run a model with reasonable efficiency without passing any flags. That's really ollama's value, as far as I can tell.

Re: Ollama and gguf

#82

Post by llmthrowaway »

brabel wrote: post #61
llmthrowaway wrote: post #7

Confusing title - thought this was about Ollama finally supporting sharded GGUF (ie. the Huggingface default for large gguf over 48gb). https://github.com/ollama/ollama/issues/5245 Sadly it is not and the issue still remains open after over a year meaning ollama cannot run the latest SOTA open source models unless they covert them to their proprietary format which they do not consistently do. No surprise I guess give…

I want to add an inference engine to my product. I was hoping to use ollama because it really helps, I think, make sure you have a model with the right metadata that you can count on working (I've seen that with llama.cpp, it's easy to get the metadata wrong and start getting rubbish from the LLM because the "stop_token" was wrong or something). I'd thought ollama was a proponent of the GGUF, which I really like as i…

Meant to say llama-swap instead of llama-server. llama-swap adds a gui and dynamic model switching on top of llama-server. Somewhat tricky to set up as it relies on a .yaml file that is poorly documented for using with docker but something like:

  "GLM4-Air":
    env:
      - "CUDA_VISIBLE_DEVICES=1"
    proxy: "http://127.0.0.1:9999"
    cmd: >
      /app/llama-server
      --cache-type-k q8_0 --cache-type-v q8_0
      --flash-attn
      --ctx-size 32684
      --jinja
      -ngl 20
      --model /modelfiles/GLM-4.5-Air-UD-Q4_K_XL-00001-of-00002.gguf
      --port 9999
When run via docker this gets you a similar setup to ollama. The yaml file also needs TTL set if you want it to unload models after an idle period.

Ollama native models in their marketplace have these params supposedly set correctly to save you having to do this config but in practice this is hit or miss and often these change from day 0 of the release.

Re: Ollama and gguf

#83

Post by dpkirchner »

brabel wrote: post #75
dd3boh wrote: post #74

Earlier quoted context omitted.

By the way, you can download models straight from hugging face with llama.cpp. It might be a few characters longer than the command you would run on ollama, but still.

Then you need to also provide appropriate metadata and format messages correctly according to the format. Which I believe llama.cpp doesn’t do by default, or it can do it? I had trouble formatting messages correctly using llama.cpp due to possibly mismatch in metadata, which ollama seems to handle, but would love to know if this is wrong.

Plus a huggingface token to access models that require you to beg for approval. Ollama hosted models don't require that (which may not be legit but most users don't care).

Re: Ollama and gguf

#84

Post by otabdeveloper4 »

dpkirchner wrote: post #81
otabdeveloper4 wrote: post #70

Earlier quoted context omitted.

> it does not align with their goals Ollama is a scam trying to E-E-E the rising hype wave of local LLMs while the getting is still good. Sorry, but somebody has to voice the elephant in the room here.

It'd be easy enough for ollama alternatives -- they just need to make a CLI front end that lets you run a model with reasonable efficiency without passing any flags. That's really ollama's value, as far as I can tell.

Ollama itself doesn't pass that test. (Broken context settings, non-standard formats and crazy model names.)

Re: Ollama and gguf

#85

Post by loudmax »

dd3boh wrote: post #74
jjice wrote: post #73

Earlier quoted context omitted.

Ollama is a more out of the box solution. I also prefer llama.cpp for the more FOSS aspects, but Ollama is a simpler install, model download (this is the biggest convenience IMO), and execution. For those reasons, that's why I believe it's still fairly popular as a solution.

By the way, you can download models straight from hugging face with llama.cpp. It might be a few characters longer than the command you would run on ollama, but still.

You can, but you have to know where to look, and you have to have some idea of what you're doing. The benefit of Ollama is that the barrier to entry is really low, as long as you have the right hardware.

To me, one of the benefits of running a model locally is learning how all this stuff works, so Ollama never had any appeal. But most people just want stuff to work without putting in the effort to understand how it all fits together. Ollama meets that demand.

Re: Ollama and gguf

#86

Post by buyucu »

hodgehog11 wrote: post #36
refulgentis wrote: post #35

Earlier quoted context omitted.

This is a good handwave-y answer for them but truth is they've always been allergic to ever mentioning llama.cpp, even when legally required, they made a political decision instead of an engineering one, and now justify it to themselves and you by handwaving about it somehow being less stable than the core of it, which they still depend on. A lot of things happened to get to the point they're getting called out aggre…

It's clear you have a better handle on the situation than I do, so it's a shame you weren't the one to talk to them face-to-face. > llama.cpp has been just fine for me. Of course, so you really shouldn't use Ollama then. Ollama isn't a hobby project anymore, they were the only ones at the table with OpenAI many months before the release of GPT-OSS. I honestly don't think they care one bit about the community drama at…

Ollama isn't anything anymore. Ollama had a value proposition a year ago, they don't have one now.

And also, Ollama is claiming not to use llama.cpp, despite continuing to use llama.cpp.

Re: Ollama and gguf

#87

Post by refulgentis »

hodgehog11 wrote: post #55
anoncareer0212 wrote: post #41

Earlier quoted context omitted.

> ...they were the only ones at the table with OpenAI many months before the release of GPT-OSS In the spirit of TFA: This isn't true, at all. I don't know where the idea comes from. You've been repeating this claim frequently. You were corrected on this 2 hours ago . llama.cpp had early access to it just as well. It's bizarre for several reasons: 1. It is a fantasy that engineering involves seats at tables and bands…

Wow, I wasn't expecting this. These are fair critiques, as I am only somewhat informed about what is clearly a very sensitive topic. For transparency, I attended ICML2025 where Ollama had set up a booth and had a casual conversation with the representatives there (one of whom turned out to lead the Ollama project) before they went to their 2nd birthday celebration. I'm repeating what I can remember from the conversat…

You should revisit this in a year, I think you'll understand how you came off a bit better. TL;DR: horrible idea to show up gossiping to cast aspersions, then disclaim responsibility because it's by proxy and you didn't understand, on a professional specialist site.

Your rationale for your choices doesn't really matter, you made the choice to cast aspersions, repeatedly, on multiple stories in multiple comments.

Handwaving about how the truth will come out through discussion, even when you repeatedly cast aspersions you disclaim understanding of, while also expressing surprise you got a reply, while also always making up more stuff in service of justifying the initial aspersions is indicative of how coherent your approach seems to the rest of us.

In general, I understand why you feel patronized. Between the general incoherence, this thread, and the previous thread where you're applying not-even-wrong, in the Pauli sense, concepts like LTS and Linux kernel dev to this situation, the only real choice that lets anyone assume the best of you is you're 15-25 and don't have a great grasp of tech yet.

Otherwise, you're just some guy gossiping, getting smarmy about it, with enough of an IQ to explain back to yourself why you didn't mean to do it. 0 idea why someone older and technical would do all that on behalf of software they don't use and don't understand.

Re: Ollama and gguf

#88

Post by gcr »

tarruda wrote: post #65
gcr wrote: post #46

Earlier quoted context omitted.

This is misinformation. Ollama’s supported structured outputs that conform to a given JSON-schema for months. Here’s a post about this from last year: https://ollama.com/blog/structured-outputs This is absolutely possible to do at the model level via logit shaping. Llama-cpp’s functionality for this is called GBNF. It’s tightly integrated into the token sampling infrastructure, and is what ollama builds upon for thei…

> It’s tightly integrated into the token sampling infrastructure, and is what ollama builds upon for their json schema functionality. Do you mean the functionality of generating ebnf grammar and from a json schema use it for sampling is part of ggml, and all they have to do is use it? I assumed that this was part of llama.cpp, and another feature they have to re-implement and maintain.

The whole point of GBNF is to serve as part of the API that lets downstream applications control token sampling in a high-level way without having to drop to raw logit distributions or pull model-specific tricks.

Ollama has a hardcoded GBNF grammar to force generic json output for example, the code is here: https://github.com/ollama/ollama/blob/da09488fbfc437c55a94bc...

Ollama can also turn user-passed json schema into a more tightly specified GBNF grammar, the code is here and is a bit harder to understand: https://github.com/ollama/ollama/blob/da09488fbfc437c55a94bc...

This thread was about doing structured generation in a model-agnostic way without wrapping try/except around json.parse(), and GBNF is _the_ way to do that.

Re: Ollama and gguf

#89

Post by dpkirchner »

otabdeveloper4 wrote: post #84
dpkirchner wrote: post #81

Earlier quoted context omitted.

It'd be easy enough for ollama alternatives -- they just need to make a CLI front end that lets you run a model with reasonable efficiency without passing any flags. That's really ollama's value, as far as I can tell.

Ollama itself doesn't pass that test. (Broken context settings, non-standard formats and crazy model names.)

I haven't experienced this personally, but I have stuck with pretty mainstream models like llama, gemma, deepseek, etc.

Re: Ollama and gguf

#90

Post by A4ET8a8uTh0_v2 »

otabdeveloper4 wrote: post #71
hodgehog11 wrote: post #24

Earlier quoted context omitted.

I think this is the thing: if you can use llama.cpp, you probably shouldn't use Ollama. It's designed for the beginner.

You shouldn't use Ollama as a beginner either. It comes with crazy begginer-hostile defaults out of the box.

Hmm? I would argue against that line of argumentation. It is ridiculously easy to start out of box and working. Once the user starts moving against obvious restrictions resulting from the trade-offs in defaults, they can move on to something more custom. Woulnd't that be the definition of beginner friendly?

I am biased since I effectively started with Ollama as my main local llm so take this response for what it is.

Still, you got me curious. Which defaults you consider hostile ( not disagreeing; this is pure curiosity )?

Post reply on HN