31–40 of 65 posts
GGML & GGUF have been extremely important to the open-source ML/AI space. Projects like llama.cpp, whisper.cpp, and stable-diffusion.cpp tend to just work perfectly, across a whole bunch of different platforms and hardware backends.
GGML & GGUF have been extremely important to the open-source ML/AI space. Projects like llama.cpp, whisper.cpp, and stable-diffusion.cpp tend to just work perfectly, across a whole bunch of different platforms and hardware backends.
while llama.cpp is an meta creation, and meta as I loathe them with a passion, I do admit it's the easiest out of the others. Compile this, give it brain - run. And you get a webui and api.
I regret that the projection models ended up separate, and I too would have preferred for them to be in a single file. I'm not entirely sure why that ended up happening, but it very much runs counter to the single-file ethos I had in mind when I designed GGUF. Hoping that someone will shepherd the cause of merging the two; I think I'm too out of the loop to do it this time around :-)
Which I'm happy for. So given that decision, I don't think it's unreasonable to think that they might be open to including Mmproj files in the GGUF.
Only issue I can think of is, which one? BF16, F16? Etc
I regret that the projection models ended up separate, and I too would have preferred for them to be in a single file. I'm not entirely sure why that ended up happening, but it very much runs counter to the single-file ethos I had in mind when I designed GGUF. Hoping that someone will shepherd the cause of merging the two; I think I'm too out of the loop to do it this time around :-)
Well considering right now MTP support is being developed, there was a conversation in that that seemed to throw around the idea of separating the MTP model out of the main GGUF, like with Mmproj. This was rejected. Which I'm happy for. So given that decision, I don't think it's unreasonable to think that they might be open to including Mmproj files in the GGUF. Only issue I can think of is, which one? BF16, F16? Etc
Earlier quoted context omitted.
while llama.cpp is an meta creation, and meta as I loathe them with a passion, I do admit it's the easiest out of the others. Compile this, give it brain - run. And you get a webui and api.
llama.cpp doesn't really have much to do with Meta other than it was originally developed for the first Llama model released by Meta. The creator doesn't and didn't work for Meta when it was written.
I regret that the projection models ended up separate, and I too would have preferred for them to be in a single file. I'm not entirely sure why that ended up happening, but it very much runs counter to the single-file ethos I had in mind when I designed GGUF. Hoping that someone will shepherd the cause of merging the two; I think I'm too out of the loop to do it this time around :-)
Well considering right now MTP support is being developed, there was a conversation in that that seemed to throw around the idea of separating the MTP model out of the main GGUF, like with Mmproj. This was rejected. Which I'm happy for. So given that decision, I don't think it's unreasonable to think that they might be open to including Mmproj files in the GGUF. Only issue I can think of is, which one? BF16, F16? Etc
Earlier quoted context omitted.
The model sees one token per marker - but the overlap with ingested actual text is still relevant, because the tokenizer will ingest regular text, where it will turn " " into the same token. For this reason, it can be tricky to work on the runtime for a model with the same model. This really feels like an accidental problem, but I'm not sure if it's really solvable without abandoning the text representations altogeth…
Surely one can just escape the input, no? Seems astonishing if someone isn't doing that
Obviously the prefix-with-backslash convention won't do it. The escaping system could be something like inserting a character on the second position in the text repr, and reversing that on output too if it matches an escaped known special token.
Changing the vocab on the fly requires tokenizing things separately, breaking the chat template.
Anecdotally, even claude code has an anneurism sometimes when listing special tokens. Idk exactly what claude's token is, but I'm fairly sure I've seen it stop generation when it tried to generate it before.
I should also say that I've (clearly) not thought about this deeply. There should be a simpler way to do it.
Earlier quoted context omitted.
Surely one can just escape the input, no? Seems astonishing if someone isn't doing that
The escape algorithm here is very simple, you remove special tokens from the runtime tokenizer's vocabulary so that it's forced to encode them as multiple non-special tokens. (That doesn't actually mean the LLM won't treat them as special tokens though, so this isn't sufficient on it's own.)
Doing this means that you can't just tokenize the string output of the chat template as one big string. You might need to tokenize things separately, and combine them after.
Earlier quoted context omitted.
whoops, my bad. Just a typo in the markdown. Fixed :)
What're you using to render this blog? Any chance there could be an RSS feed?
No RSS feed currently, but it's a good idea to add one!