Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Quantization from the Ground Up

ngrok.com

Post reply on HN
Print view View original

11–20 of 74 posts

Re: Quantization from the Ground Up

#11

Post by buildbot »

samwho wrote: post #5
armcat wrote: post #2

This is beautifully written and visualised, well done! The KL divergence comparisons between original and different quantisation levels is on-point. I'm not sure people realize how powerful quantisation methods are and what they've done for democratising local AI. And there are some great players out there like Unsloth and Pruna.

Thank you! I was really surprised how robust models are to losing information. It seems wrong that they can be compressed so much and still function at all, never mind function quite closely to the original size. Think we're only going to keep seeing more progress in this area on the research side, too.

You can even train in 4 & 8 bits with newer microscaled formats! From https://arxiv.org/pdf/2310.10537 to gpt-oss being trained (partially) natively in MXFP4 - https://huggingface.co/blog/RakshitAralimatti/learn-ai-with-...

To Nemotron 3 Super, which had 25T of nvfp4 native pretraining! https://docs.nvidia.com/nemotron/0.1.0/nemotron/super3/pretr...

Re: Quantization from the Ground Up

#12

Post by fcpk »

buildbot wrote: post #10
fcpk wrote: post #9

something I have been wondering about is doing regressive layer specific quantization based on large test sets. ie reduce very specifically layers that don't improve general quality.

This is a thing! For example, https://arxiv.org/abs/2511.06516

that's brilliant, I wonder why we haven't seen much use of it to do very heavy quantization

Re: Quantization from the Ground Up

#13

Post by polotics »

aarondf wrote: post #7

My word... samwho is doing some of the best technical explainers on the internet right now.

Leading to my question: Ok keeping a zero and a minus-zero does make sense for some limits calculations... But when all you have is 4 bits, is this not quite wasteful? Would using the bits for eg. a 2.5 not improve the model?

Re: Quantization from the Ground Up

#14

Post by polotics »

polotics wrote: post #13
aarondf wrote: post #7

My word... samwho is doing some of the best technical explainers on the internet right now.

Leading to my question: Ok keeping a zero and a minus-zero does make sense for some limits calculations... But when all you have is 4 bits, is this not quite wasteful? Would using the bits for eg. a 2.5 not improve the model?

Oh well that's a rabbit hole: NVIDIA Blackwell has this, also GGUFs sidestep this with Qi_j / Qi_K... Great article, spikes curiosity!

Re: Quantization from the Ground Up

#18

Post by gavinray »

I read the entire thing top-to-bottom, as a visual learner this is superb.

One nitpick -- in the "asymmetric quantification" code, shouldn't "zero" be called "midpoint" or similar? Or is "zero" an accepted mathematics term in this domain?

Re: Quantization from the Ground Up

#19

Post by qcautomation »

mrsilencedogood wrote: post #8

Quantization is important for me because it's the only way out I can see for a future of programming that doesn't involve going through a giant bigco who can run, as the article says, a machine with 2TB of memory. And not just memory, but my understanding is that for the model to be performant, it has to be VRAM to boot. This comes as the latest concern of mine in a long line around "how software gets written" remain…

[dead]

Re: Quantization from the Ground Up

#20

Post by samwho »

gavinray wrote: post #18

I read the entire thing top-to-bottom, as a visual learner this is superb. One nitpick -- in the "asymmetric quantification" code, shouldn't "zero" be called "midpoint" or similar? Or is "zero" an accepted mathematics term in this domain?

“Zero point” is how I saw it referred to in the literature, so that’s what I went with. I personally prefer to think of it as an offset, but I try to stick with terms folks are likely to see in the wild.
Post reply on HN