71–74 of 74 posts
This isn't just a good explainer of quantization, it's a good overview of LLMs in general.
Earlier quoted context omitted.
Newer quantization approaches are even better, 4-bits gets you no meaningful loss relative to FP16: https://github.com/z-lab/paroquant Hopefully Microsoft keeps pushing BitNet too, so only "1.58" bits are needed. I think fractional representations are only relevant for training at this point, and bf16 is sufficient, no need for fp4 and such.
Learned rotations for INT4 are cool! Seems similar to SpinQuant? https://arxiv.org/abs/2405.16406 In my personal opinion I don’t think the 1.58 bit work is going to make it into the mainstream. Not sure why you think fractional representations are only useful for training? Being able to natively compute in lower precisions can be a huge performance boost at inference time.
Indeed, but much better! More accurate, less time and space overhead, beats AWQ on almost every bench. I hope it becomes the standard.
> In my personal opinion I don’t think the 1.58 bit work is going to make it into the mainstream.
I hope you're wrong! I'm more optimistic. Definitely a bit more work to be done, but still very promising.
> Being able to natively compute in lower precisions can be a huge performance boost at inference time.
ParoQuant is barely worse than FP16. Any less precise fractional representation is going to be worse than just using that IMO.
Earlier quoted context omitted.
I've been watching the drizzle of LLM papers come through, and I think we're going to hit a 1T param MoE on consumer hardware before this year is out. It'll still be behind the bigco models, but it'll be a force multiplier. Ideally, we'd get these models to run on a CPU. MS BitNet is one way to do this. You can already run ternary LLMs on consumer CPUs with a decent tps.
Though what is consumer hardware right now? Can we still classify 5090s as consumer hardware given how expensive they are? They're £3k at the moment, and it looks like it's only going to get worse unless the AI bubble pops.
Earlier quoted context omitted.
14B even at Q4 isn't realistic for coding on a single 12GB RTX 3060. Token speed is too slow. After all they are dense models. You aren't getting a good MoE model under 30B. You can do OCR, STT, TTS really well and for LLMs, good use cases are classification, summarization and extraction with <10B models.
Dual 3060s run 24B Q6 and 32B Q4 at ~15 tok/sec. That's fast enough to be usable. Add a third one and you can run Qwen 3.5 27B Q6 with 128k ctx. For less than the price of a 3090.
> 3x RTX 3060 less tgab the price of a 3090
Interesting, here it is around the same. 200-250€ for a used 12GB 3060 and 600-800 for a used 3090€.