Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What's in a GGUF, besides the weights – and what's still missing?

nobodywho.ooo

Post reply on HN
Print view View original

41–50 of 65 posts

Re: What's in a GGUF, besides the weights – and what's still missing?

#41

Post by monocasa »

I mean, one if the big issues I've had is that it doesn't really store the compute graph. It only stores a string of the foundational architecture, along with parameter metadata to allow you to rebuild the compute graph.

That means that every foundational model architecture requires new code in whatever is consuming the gguf to support that model.

Re: What's in a GGUF, besides the weights – and what's still missing?

#42

Post by halyconWays »

Fun lore, GGUFs were once called GGJTs until I caught the "JT" (Justine Tunney) stealing the memory map code from a user who did 99% of the work in a draft PR (slaren) and lying about it, and misrepresenting or not understanding how memory map worked. She wanted her initials in the file format for bragging rights because it was claimed that it caused 90% memory reduction (actually it was just lazy loading into memory). Gerganov was quite angry when he found out what happened. Jart (JT) was then banned from the llama.cpp repo but managed to get back in a year or so later.

Re: What's in a GGUF, besides the weights – and what's still missing?

#43

Post by macNchz »

ge96 wrote: post #13
mixtureoftakes wrote: post #12

Earlier quoted context omitted.

7b mistral is quite outdated. On a 12gb 4070 you can run qwen 3.5 9b q4km or qwen 3.6 35b, the latter will be a lot smarter but also a lot slower due to ram offload. Try both in lm studio, they really are surprisingly capable

I have 80gb of ram but it's slow capped by i9 CPU or specific asus mobo sucks I think only 2400mhz despite being ddr4 Tried all the stuff bios, volting

Gemma 4 26B-A4B might be interesting to try on your machine. The latest optimizations make MoE models work pretty nicely on setups like that with a decent GPU and lots of slowish RAM. I have a 16gb GPU and 64gb of 3200mhz DDR4 and get 15-20 tokens/sec out of that model with zero finagling or tweaking. I’ve been very impressed by it, even having run just about every other open weight model that would fit on my machine over the last few years.

Re: What's in a GGUF, besides the weights – and what's still missing?

#44

Post by ge96 »

macNchz wrote: post #43
ge96 wrote: post #13

Earlier quoted context omitted.

I have 80gb of ram but it's slow capped by i9 CPU or specific asus mobo sucks I think only 2400mhz despite being ddr4 Tried all the stuff bios, volting

Gemma 4 26B-A4B might be interesting to try on your machine. The latest optimizations make MoE models work pretty nicely on setups like that with a decent GPU and lots of slowish RAM. I have a 16gb GPU and 64gb of 3200mhz DDR4 and get 15-20 tokens/sec out of that model with zero finagling or tweaking. I’ve been very impressed by it, even having run just about every other open weight model that would fit on my machine…

that seems slow? 15-20, was expecting 50-60 like mistral although I have not measured that yet on my setup

I've been asking other people but what do you use it for?

Re: What's in a GGUF, besides the weights – and what's still missing?

#45

Post by paradox460 »

ganelonhb wrote: post #3
ge96 wrote: post #2

Nice, I recently pulled down TheBloke 7B mistral to try out I have a 4070.

I have a 2070 and can confirm it works amazingly fast. I love TheBloke I wish he still made stuff

A lot of the same spirit lives on in TheDrunmer

They're mostly aimed at role play and sillytavern, but they're still generally good models, with lots of quants available

Re: What's in a GGUF, besides the weights – and what's still missing?

#48

Post by aiscoming »

macNchz wrote: post #43
ge96 wrote: post #13

Earlier quoted context omitted.

I have 80gb of ram but it's slow capped by i9 CPU or specific asus mobo sucks I think only 2400mhz despite being ddr4 Tried all the stuff bios, volting

Gemma 4 26B-A4B might be interesting to try on your machine. The latest optimizations make MoE models work pretty nicely on setups like that with a decent GPU and lots of slowish RAM. I have a 16gb GPU and 64gb of 3200mhz DDR4 and get 15-20 tokens/sec out of that model with zero finagling or tweaking. I’ve been very impressed by it, even having run just about every other open weight model that would fit on my machine…

[dead]

Re: What's in a GGUF, besides the weights – and what's still missing?

#50

Post by rexthonyy »

amelius wrote: post #17

> user Hi there! model Hi there, how can I help you today Good lord, they managed to invent a format that is even less readable than XML.

You're right. It does seem like a suboptimal format in terms of memory usage efficiency
Post reply on HN