5-10% accuracy is like the difference between a usable model, and unusable model.
[flagged]
Quantization from the Ground Up
61–70 of 74 posts
Re: Quantization from the Ground Up
#62The hardware situation is way better than you think, and quantization is a huge part of why. Take Qwen 3.5 27B, which is a solid coding model. At FP16 it needs 54GB of VRAM. Nobody's running that on consumer hardware. At Q4_K_M quantization, it needs 16GB. A used RTX 3090 has 24GB and goes for about $900. That model runs locally with room for context. For 14B coding models at Q4, you're looking at about 10GB. A used…
14B even at Q4 isn't realistic for coding on a single 12GB RTX 3060. Token speed is too slow. After all they are dense models. You aren't getting a good MoE model under 30B. You can do OCR, STT, TTS really well and for LLMs, good use cases are classification, summarization and extraction with <10B models.
Dual 3060s run 24B Q6 and 32B Q4 at ~15 tok/sec. That's fast enough to be usable.
Add a third one and you can run Qwen 3.5 27B Q6 with 128k ctx. For less than the price of a 3090.
Re: Quantization from the Ground Up
#63[dead]
Re: Quantization from the Ground Up
#645-10% accuracy is like the difference between a usable model, and unusable model.
[flagged]
Re: Quantization from the Ground Up
#65[dead]
Re: Quantization from the Ground Up
#66Earlier quoted context omitted.
U are better off just buying their coding plan. Running LLM makes no sense whatsoever
Remaining dependent on proprietary frontier models that you can only access via an API makes no sense whatsoever. My hope is that the future is open weight models running on local hardware.
Eventually, yes. ParoQuant is hopefully the future here, 4-bit weights with no real degradation from FP16:
Re: Quantization from the Ground Up
#67Earlier quoted context omitted.
Thank you! I was really surprised how robust models are to losing information. It seems wrong that they can be compressed so much and still function at all, never mind function quite closely to the original size. Think we're only going to keep seeing more progress in this area on the research side, too.
You can even train in 4 & 8 bits with newer microscaled formats! From https://arxiv.org/pdf/2310.10537 to gpt-oss being trained (partially) natively in MXFP4 - https://huggingface.co/blog/RakshitAralimatti/learn-ai-with-... To Nemotron 3 Super, which had 25T of nvfp4 native pretraining! https://docs.nvidia.com/nemotron/0.1.0/nemotron/super3/pretr...
Newer quantization approaches are even better, 4-bits gets you no meaningful loss relative to FP16: https://github.com/z-lab/paroquant
Hopefully Microsoft keeps pushing BitNet too, so only "1.58" bits are needed.
I think fractional representations are only relevant for training at this point, and bf16 is sufficient, no need for fp4 and such.
Re: Quantization from the Ground Up
#68This isn't just a good explainer of quantization, it's a good overview of LLMs in general.
Re: Quantization from the Ground Up
#69Earlier quoted context omitted.
You can even train in 4 & 8 bits with newer microscaled formats! From https://arxiv.org/pdf/2310.10537 to gpt-oss being trained (partially) natively in MXFP4 - https://huggingface.co/blog/RakshitAralimatti/learn-ai-with-... To Nemotron 3 Super, which had 25T of nvfp4 native pretraining! https://docs.nvidia.com/nemotron/0.1.0/nemotron/super3/pretr...
Newer quantization approaches are even better, 4-bits gets you no meaningful loss relative to FP16: https://github.com/z-lab/paroquant Hopefully Microsoft keeps pushing BitNet too, so only "1.58" bits are needed. I think fractional representations are only relevant for training at this point, and bf16 is sufficient, no need for fp4 and such.
Learned rotations for INT4 are cool! Seems similar to SpinQuant? https://arxiv.org/abs/2405.16406
In my personal opinion I don’t think the 1.58 bit work is going to make it into the mainstream.
Not sure why you think fractional representations are only useful for training? Being able to natively compute in lower precisions can be a huge performance boost at inference time.
Re: Quantization from the Ground Up
#70I was a little confused by this part: "This is what's happening to the parameters of models when they're quantized down to sizes that are possible to run on your laptop. Instead of floats, small integers are what get stored and loaded into memory. When the time comes to use the quantized values, to generate an answer to a question for example, the values are dequantized on the fly. You might think this sounds slower,…
Hardware support will vary widely, as will speed on these smaller FP formats, sometimes intentionally nerfed in consumer cards.
Lots of devices with embedded "AI accelerators" will also only do things like INT8, and for some reason INT8 is generally worse than the same size FP8 (maybe that could be fixed with smarter quantization).