Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

QuIP#: 2-bit Quantization for LLMs

cornell-relaxml.github.io

Post reply on HN
Print view View original

41–50 of 62 posts

Re: QuIP#: 2-bit Quantization for LLMs

#41

Post by shahbazac »

Can someone answer CS 101 questions about this please.

I know there are other methods related to matrix factorization, but I’m asking specifically about quantization.

Does quantization literally mean the weight matrix floats are being represented using fewer bits than the 64 bit standard?

Second, if fewer bits are being used, are CPUs able to do math directly on fewer bits? Aren’t CPU registers still 64 bit? Are these floats converted back to 64 bit for math, or is there some clever packing technique where a 64 bit float actually represents many numbers (sort of a hackey simd instruction)? Or do modern CPUs have the hardware to do math on fewer bits?

Re: QuIP#: 2-bit Quantization for LLMs

#43

Post by garrisonhh »

shahbazac wrote: post #41

Can someone answer CS 101 questions about this please. I know there are other methods related to matrix factorization, but I’m asking specifically about quantization. Does quantization literally mean the weight matrix floats are being represented using fewer bits than the 64 bit standard? Second, if fewer bits are being used, are CPUs able to do math directly on fewer bits? Aren’t CPU registers still 64 bit? Are thes…

Yes but no. The actual values represented by the quantized bits don't use a representation akin to IEEE floating point, but they are able to act like floating point values due to mathematical transformations during propagation. The floating point values a quantized value corresponds to are chosen using some kind of precomputation depending on the quantization method

Re: QuIP#: 2-bit Quantization for LLMs

#44

Post by pyinstallwoes »

wizzard0 wrote: post #31
pyinstallwoes wrote: post #30

Since a pixel can have more states than binary, could you have more space and compute by leveraging RGBA-space for data/compute than binary? Maybe a stupid question.

A pixel takes more than 1 bit to store, too

on a video card? I'm not sure that is true wrt to the geometry and charge.

Re: QuIP#: 2-bit Quantization for LLMs

#45

Post by p1esk »

shahbazac wrote: post #41

Can someone answer CS 101 questions about this please. I know there are other methods related to matrix factorization, but I’m asking specifically about quantization. Does quantization literally mean the weight matrix floats are being represented using fewer bits than the 64 bit standard? Second, if fewer bits are being used, are CPUs able to do math directly on fewer bits? Aren’t CPU registers still 64 bit? Are thes…

This is for GPUs, not CPUs. GPUs do have lower precision ALUs to do math on fewer bits. Though not 2 bits - I believe there’s support for 1, 4 and 8 bit computation in modern Nvidia cards.

But even without such support there’s a benefit of model size compression so that bigger models can fit in GPU memory, eliminating costly CPU/GPU data transfers.

Re: QuIP#: 2-bit Quantization for LLMs

#46

Post by saberience »

I’m a layperson when it comes to this topic but does this mean every value in the network is a value from 00 to 11? I.e: 00, 01, 10, and 11?

I struggle to understand how a network with only two bits of precision could ever generate text or numbers or anything really.

Is my intuition wrong here? If so, can someone give an example of what it means to quantize the network down to 2 bits only.

Re: QuIP#: 2-bit Quantization for LLMs

#47

Post by thulle »

skavi wrote: post #2

can anyone comment on running the 2b quantized llama 70b on consumer cards like the 4090?

Better output than the smaller llamas in my limited testing, but it's surprisingly slow:

Output generated in 101.74 seconds (0.98 tokens/s, 100 tokens, context 82, seed 532878022)

Output generated in 515.46 seconds (0.99 tokens/s, 511 tokens, context 27, seed 660997525)

Checking nvidia-smi it stalls at ~130W (out of ~470 W max) power usage, ~25% GPU usage and ~10% memory bandwidth usage. There's fairly much traffic on the pci-bus though, and the python process is stable at 100% usage of one core. GPU possibly limited by some thing handled in python? Pausing the GPU-accelerated video-decoding of a twitch stream it get a surprisingly large boost:

Output generated in 380.42 seconds (1.34 tokens/s, 511 tokens, context 26, seed 648992918)

Re: QuIP#: 2-bit Quantization for LLMs

#48

Post by p1esk »

saberience wrote: post #46

I’m a layperson when it comes to this topic but does this mean every value in the network is a value from 00 to 11? I.e: 00, 01, 10, and 11? I struggle to understand how a network with only two bits of precision could ever generate text or numbers or anything really. Is my intuition wrong here? If so, can someone give an example of what it means to quantize the network down to 2 bits only.

It’s better to think about mapping groups of weights to an entry in a large codebook, and then using clever methods to do weight lookups.

2 bits of precision per weight is perfectly fine as long as you have enough weights. The information encoded by a neural network is measured in total number of bits, so you can compress it by either reducing number of weights or reducing number of bits per weight.

Re: QuIP#: 2-bit Quantization for LLMs

#49

Post by MrNeon »

tarruda wrote: post #23

If this quantization method works with smaller models, it would enable running up to 33B models with only 12GB VRAM. Especially important for democratizing access to Mistral MoE new model.

IIRC quantizing small models causes a higher relative drop in the metrics.

Re: QuIP#: 2-bit Quantization for LLMs

#50

Post by pyinstallwoes »

pyinstallwoes wrote: post #30

Since a pixel can have more states than binary, could you have more space and compute by leveraging RGBA-space for data/compute than binary? Maybe a stupid question.

Found this https://news.ycombinator.com/item?id=27409371

Kinda related. Especially the comments.

Post reply on HN