Earlier quoted context omitted.
whoops, my bad. Just a typo in the markdown. Fixed :)
What're you using to render this blog? Any chance there could be an RSS feed?
There is an RSS feed now: https://nobodywho.ooo/feed.xml
What's in a GGUF, besides the weights – and what's still missing?
51–60 of 65 posts
Re: What's in a GGUF, besides the weights – and what's still missing?
#52> user Hi there! model Hi there, how can I help you today Good lord, they managed to invent a format that is even less readable than XML.
You're right. It does seem like a suboptimal format in terms of memory usage efficiency
The tokens all have int IDs, this is just how they're rendered.
Re: What's in a GGUF, besides the weights – and what's still missing?
#53Earlier quoted context omitted.
It is not supposed to be readable by humans. You rarely have to look at it. It is designed to not get confused with the actual content, where the content can be any random text from the internet. For that, you have to use a format that is not used anywhere else.
Are these markers actual text? Or does the model "see" one token per marker?
> Or does the model "see" one token per marker?
All models ever see are just tokens, even when you pass images or what not.
In this case, is likely Token ID 1, is Token ID 2 and so on, these common "markers" are all just tokens in, tokens out.
Re: What's in a GGUF, besides the weights – and what's still missing?
#54I regret that the projection models ended up separate, and I too would have preferred for them to be in a single file. I'm not entirely sure why that ended up happening, but it very much runs counter to the single-file ethos I had in mind when I designed GGUF. Hoping that someone will shepherd the cause of merging the two; I think I'm too out of the loop to do it this time around :-)
[deleted]
Re: What's in a GGUF, besides the weights – and what's still missing?
#55> The really neat thing about GGUF is that it's just one file. Compare this to a typical safetensors repo on huggingface, where there's a pile of necessary JSON files scattered around [...] Funny, to me AI models have "always" been single files, as that's what has been the norm in the local image gen business. Safetensors files allow stuffing all kinds of stuff inside them too, no GGUF needed for that. Though given t…
> to me AI models have "always" been single files, as that's what has been the norm in the local image gen business
That doesn't even make sense in the "local image gen business", you don't use a single weights file, you need a bunch of encoders/decoders and what not to actually be able to run the architecture with the weights.
Maybe the tooling you use hides those things from you, but they're still there under the surface.
Re: What's in a GGUF, besides the weights – and what's still missing?
#56Earlier quoted context omitted.
The escape algorithm here is very simple, you remove special tokens from the runtime tokenizer's vocabulary so that it's forced to encode them as multiple non-special tokens. (That doesn't actually mean the LLM won't treat them as special tokens though, so this isn't sufficient on it's own.)
Cool technique, but I'm not sure I'd call it simple. Doing this means that you can't just tokenize the string output of the chat template as one big string. You might need to tokenize things separately, and combine them after.
If you want the token sequence, you ought to avoid discarding it when you produce the string output. This is because, even ignoring special tokens, different token sequences map to the same strings.
From a space perspective, this is actually better because tokenization tends to compress text quite well. For example, common tokens in English text take up ~4 characters on average (expands to 32 bits), but only take up a fraction of that to store (15-18 bits/token depending on vocabulary size)
In fact it appears that designing the tokens as a text compression encoding is a decent approach, since it's roughly what some LLMs do. For example, early GPT tokenizers followed byte pair encoding to create the vocabulary, which is a text compression algorithm from the 90s.
Re: What's in a GGUF, besides the weights – and what's still missing?
#57Earlier quoted context omitted.
The escape algorithm here is very simple, you remove special tokens from the runtime tokenizer's vocabulary so that it's forced to encode them as multiple non-special tokens. (That doesn't actually mean the LLM won't treat them as special tokens though, so this isn't sufficient on it's own.)
Cool technique, but I'm not sure I'd call it simple. Doing this means that you can't just tokenize the string output of the chat template as one big string. You might need to tokenize things separately, and combine them after.
Good catch. We'd have to integrate with jinja2 (or similar) and tokenzize as we format the context, so that we know which spans are instructions and which spans are data. Which makes it more complex but still very achievable.
Re: What's in a GGUF, besides the weights – and what's still missing?
#58Re: What's in a GGUF, besides the weights – and what's still missing?
#59Fun lore, GGUFs were once called GGJTs until I caught the "JT" (Justine Tunney) stealing the memory map code from a user who did 99% of the work in a draft PR (slaren) and lying about it, and misrepresenting or not understanding how memory map worked. She wanted her initials in the file format for bragging rights because it was claimed that it caused 90% memory reduction (actually it was just lazy loading into memory…
Have you ever read my side of the story? https://justine.lol/dox/4chan.txt
Re: What's in a GGUF, besides the weights – and what's still missing?
#60I regret that the projection models ended up separate, and I too would have preferred for them to be in a single file. I'm not entirely sure why that ended up happening, but it very much runs counter to the single-file ethos I had in mind when I designed GGUF. Hoping that someone will shepherd the cause of merging the two; I think I'm too out of the loop to do it this time around :-)
Currently not many people would finetune mmproj, so mmproj is reusable. The mmproj for Qwen 3.6 27B can be reused on all its finetunes. While the MTP model usually needs to be finetuned with the main model to get the best performance, which is being studied in Heretic.