Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What's in a GGUF, besides the weights – and what's still missing?

nobodywho.ooo

Post reply on HN
Print view View original

21–30 of 65 posts

Re: What's in a GGUF, besides the weights – and what's still missing?

#21

Post by stavros »

aktuel wrote: post #20
amelius wrote: post #17

> user Hi there! model Hi there, how can I help you today Good lord, they managed to invent a format that is even less readable than XML.

It is not supposed to be readable by humans. You rarely have to look at it. It is designed to not get confused with the actual content, where the content can be any random text from the internet. For that, you have to use a format that is not used anywhere else.

Are these markers actual text? Or does the model "see" one token per marker?

Re: What's in a GGUF, besides the weights – and what's still missing?

#22

Post by bashbjorn »

stavros wrote: post #21
aktuel wrote: post #20

Earlier quoted context omitted.

It is not supposed to be readable by humans. You rarely have to look at it. It is designed to not get confused with the actual content, where the content can be any random text from the internet. For that, you have to use a format that is not used anywhere else.

Are these markers actual text? Or does the model "see" one token per marker?

The model sees one token per marker - but the overlap with ingested actual text is still relevant, because the tokenizer will ingest regular text, where it will turn "" into the same token.

For this reason, it can be tricky to work on the runtime for a model with the same model. This really feels like an accidental problem, but I'm not sure if it's really solvable without abandoning the text representations altogether (and the jinja abstraction along with it).

Re: What's in a GGUF, besides the weights – and what's still missing?

#24

Post by Philpax »

theapadayo wrote: post #18

IMO the biggest thing still missing is an actual way to define the model architecture outside of being hard coded into the current build. It doesn't need to be a 1:1 performance parity with the fully supported models. Having proper, vendor validated support for day 1 is what is the difference between people thinking a model is amazing vs horrible. See recent Gemma vs Qwen releases. Not sure what the solution is, othe…

Yeah, I intentionally left space for the computation graph to be included in the GGUF spec in the hopes that this would be picked up by someone. I would have loved to have it in the first version, but I was prioritising getting the MVP spec out and implemented.

I'd still love to see this, but it would need a cheerleader very familiar with the current state of the GGML IR.

Re: What's in a GGUF, besides the weights – and what's still missing?

#25

Post by Philpax »

Sharlin wrote: post #16

> The really neat thing about GGUF is that it's just one file. Compare this to a typical safetensors repo on huggingface, where there's a pile of necessary JSON files scattered around [...] Funny, to me AI models have "always" been single files, as that's what has been the norm in the local image gen business. Safetensors files allow stuffing all kinds of stuff inside them too, no GGUF needed for that. Though given t…

Single-file deployments were an intentional design goal on my part. While most image models were/are single-file, LLM safetensors (at least at the time) were not, and I wanted to ensure that we enforced that at a structural level. I also didn't want to mandate a JSON reader for executors (e.g. llama.cpp), which the ST approach would have required. The bigger issue at the time, if I recall, was that ST couldn't support the new-and-upcoming quants that GGML had, and having our own file format offered us flexibility that ST couldn't.

Re: What's in a GGUF, besides the weights – and what's still missing?

#26

Post by Philpax »

I regret that the projection models ended up separate, and I too would have preferred for them to be in a single file. I'm not entirely sure why that ended up happening, but it very much runs counter to the single-file ethos I had in mind when I designed GGUF.

Hoping that someone will shepherd the cause of merging the two; I think I'm too out of the loop to do it this time around :-)

Re: What's in a GGUF, besides the weights – and what's still missing?

#27

Post by badsectoracula »

stavros wrote: post #21
aktuel wrote: post #20

Earlier quoted context omitted.

It is not supposed to be readable by humans. You rarely have to look at it. It is designed to not get confused with the actual content, where the content can be any random text from the internet. For that, you have to use a format that is not used anywhere else.

Are these markers actual text? Or does the model "see" one token per marker?

AFAIK[0] they are (usually) so-called "special" tokens - e.g is token id 105 for the vocabulary Gemma4 uses. When you are tokenizing text you can either tokenize the "" as a single token (105) or as a series of other tokens (236820, 236909, 887 and 236813 for the "" tokens) with the idea being that the model will treat "105" as the actual separator but can also use "" as part of the content.

Though using text-based templates make this a bit tricky regardless. AFAIK llama.cpp tries to avoid this confusion by having their Jinja2 implementation use a custom string type that contains metadata about where characters "come from" so that it can distinguish between special tokens (which would be part of the Jinja2 template) and content (which would be either generated text or text given in by the user) - i.e. even if a string is "" the metadata would be used to tell if it is meant to be tokenized as a special token or as a series of non-special tokens.

[0] i might be wrong, this is based on my understanding by messing around with the llama.cpp code, but i never implemented an LLM inference or training engine

Re: What's in a GGUF, besides the weights – and what's still missing?

#28

Post by lifis »

bashbjorn wrote: post #22
stavros wrote: post #21

Earlier quoted context omitted.

Are these markers actual text? Or does the model "see" one token per marker?

The model sees one token per marker - but the overlap with ingested actual text is still relevant, because the tokenizer will ingest regular text, where it will turn " " into the same token. For this reason, it can be tricky to work on the runtime for a model with the same model. This really feels like an accidental problem, but I'm not sure if it's really solvable without abandoning the text representations altogeth…

Surely one can just escape the input, no? Seems astonishing if someone isn't doing that

Re: What's in a GGUF, besides the weights – and what's still missing?

#29

Post by maxbond »

lifis wrote: post #28
bashbjorn wrote: post #22

Earlier quoted context omitted.

The model sees one token per marker - but the overlap with ingested actual text is still relevant, because the tokenizer will ingest regular text, where it will turn " " into the same token. For this reason, it can be tricky to work on the runtime for a model with the same model. This really feels like an accidental problem, but I'm not sure if it's really solvable without abandoning the text representations altogeth…

Surely one can just escape the input, no? Seems astonishing if someone isn't doing that

The escape algorithm here is very simple, you remove special tokens from the runtime tokenizer's vocabulary so that it's forced to encode them as multiple non-special tokens. (That doesn't actually mean the LLM won't treat them as special tokens though, so this isn't sufficient on it's own.)
Post reply on HN