Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

QuIP#: 2-bit Quantization for LLMs

cornell-relaxml.github.io

Post reply on HN
Print view View original

21–30 of 62 posts

Re: QuIP#: 2-bit Quantization for LLMs

#21

Post by acosmism »

0xDEADFED5 wrote: post #18
lhl wrote: post #12

Earlier quoted context omitted.

It's been on my list to do a proper shootout of all the various new quant formats floating around (my list here: https://llm-tracker.info/books/llms/page/quantization-overvi... ) but a lot of them don't have very good production code yet (eg, a few months ago, when I tried OmniQuant, some of the important bits of code wasn't even included and had to be gotten directly from the authors: https://llm-tracker.info/books/…

i humbly request you to add mlc-llm to your quant test when/if you get around to doing it

more specifically on a non-cuda gpu - mali on orangepi via opencl

Re: QuIP#: 2-bit Quantization for LLMs

#22

Post by orbital-decay »

skykooler wrote: post #9

I wonder whether quantization to 1-bit would be functional?

Functional? Sure, it will work. Useful? Unlikely. It would be surprising to see quantization of pretrained models to surpass the existing research around binary NNs (trained from scratch as 1-bit while using full precision for the most critical parts)

Re: QuIP#: 2-bit Quantization for LLMs

#24

Post by philipswood »

epcoa wrote: post #17
taneq wrote: post #5

Earlier quoted context omitted.

I don’t know too many humans who could answer that half as well.

"For example, the addition of two numbers, 2+2, can be seen as a repeated application of the multiplication operation," I bet most humans ever born did not know what the word "multiplication" meant. Not sure how your metric is useful.

Sometimes, to achieve some perspective, I also zoom out and declare normal over the estimated +-120,000,000,000 people who have ever lived.

Re: QuIP#: 2-bit Quantization for LLMs

#25

Post by skavi »

beacon294 wrote: post #13
skavi wrote: post #8

Earlier quoted context omitted.

to clarify, is that with https://huggingface.co/relaxml/Llama-2-70b-chat-E8P-2Bit

It is Llama-2-70b-chat. I quantized it to 2q_k using `quantize` with llama.cpp.

So your experience isn’t representative of the work presented by this post? Or does llama.cpp use the same technique for quantization?

Re: QuIP#: 2-bit Quantization for LLMs

#26

Post by brucethemoose2 »

lhl wrote: post #12
skavi wrote: post #2

can anyone comment on running the 2b quantized llama 70b on consumer cards like the 4090?

It's been on my list to do a proper shootout of all the various new quant formats floating around (my list here: https://llm-tracker.info/books/llms/page/quantization-overvi... ) but a lot of them don't have very good production code yet (eg, a few months ago, when I tried OmniQuant, some of the important bits of code wasn't even included and had to be gotten directly from the authors: https://llm-tracker.info/books/…

> https://llm-tracker.info/books/llms/page/quantization-overvi...

This is a very cool resource, thanks!

Gems like this, even in areas I follow pretty closely, are why I keep coming back to HN.

Re: QuIP#: 2-bit Quantization for LLMs

#27

Post by samus »

SeanAnderson wrote: post #15

Just to make sure I'm understanding this correctly. This paper signals that the authors have found a way to run Llama 2 70B, but with 1/8th the VRAM requirements as compared to the original model, right? And the output is on-par with the original along some metrics (ArcE/PiQA), within 25% on others (Wiki/C4), and the trajectory of their progress hints that there's even more ground to gain in the future?

Your last paragraph is the key. Without their improvements, the quality tradeoff would have been hard to stomach.

Re: QuIP#: 2-bit Quantization for LLMs

#29

Post by Philpax »

beacon294 wrote: post #3
skavi wrote: post #2

can anyone comment on running the 2b quantized llama 70b on consumer cards like the 4090?

I have quantized Llama-2-chat-70b to 2q_k with llama.cpp then run on a 4090 for 50/83 layers. I was able to get 1.3 tokens per second. It answers this question, which smaller models have trouble with: [INST]Hello. The functions addition, multiplication, and exponentiation can be considered each as a higher order of the previous. For example, 2+2+2 = 2*3 which is significant because there are three twos, which is the…

To be clear, this is not using QuIP#, and is thus not particularly relevant to the discussion at hand.
Post reply on HN