Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

QuIP#: 2-bit Quantization for LLMs

cornell-relaxml.github.io

Post reply on HN
Print view View original

11–20 of 62 posts

Re: QuIP#: 2-bit Quantization for LLMs

#11

Post by skykooler »

beacon294 wrote: post #3
skavi wrote: post #2

can anyone comment on running the 2b quantized llama 70b on consumer cards like the 4090?

I have quantized Llama-2-chat-70b to 2q_k with llama.cpp then run on a 4090 for 50/83 layers. I was able to get 1.3 tokens per second. It answers this question, which smaller models have trouble with: [INST]Hello. The functions addition, multiplication, and exponentiation can be considered each as a higher order of the previous. For example, 2+2+2 = 2*3 which is significant because there are three twos, which is the…

Note that it got all the math wrong. 2+2 is a repetition of the succession operator, not multiplication; 2+2 equals 4, not 6; multiplication of two numbers is repeated addition, not exponentiation; 2*2 equals 4, not 8; and tetration(2,2) is 4, not 65536. This is actually the invariant in that n-ation(2,2) equals 4 for all natural numbers n.

Re: QuIP#: 2-bit Quantization for LLMs

#12

Post by lhl »

skavi wrote: post #2

can anyone comment on running the 2b quantized llama 70b on consumer cards like the 4090?

It's been on my list to do a proper shootout of all the various new quant formats floating around (my list here: https://llm-tracker.info/books/llms/page/quantization-overvi...) but a lot of them don't have very good production code yet (eg, a few months ago, when I tried OmniQuant, some of the important bits of code wasn't even included and had to be gotten directly from the authors: https://llm-tracker.info/books/llms/page/omniquant).

If you're looking for the best widely deployed quant format atm, it's probably ExLlamaV2's EXL2 - it supports arbitrary bpw w/ a calibration file, and also 8-bit kvcache support. I haven't tested EXL2 much at lower bpws though.

Note, both llama.cpp and AirLLM allow layer offloading to system memory (or in AirLLM's case, even to disk?!).

r/LocalLlama probably is the best place to search for if you're looking for people's experiences w/ quants. I know some people have been testing, like: https://www.reddit.com/r/LocalLLaMA/comments/17klaa5/tested_...

Re: QuIP#: 2-bit Quantization for LLMs

#13

Post by beacon294 »

skavi wrote: post #8
beacon294 wrote: post #3

Earlier quoted context omitted.

I have quantized Llama-2-chat-70b to 2q_k with llama.cpp then run on a 4090 for 50/83 layers. I was able to get 1.3 tokens per second. It answers this question, which smaller models have trouble with: [INST]Hello. The functions addition, multiplication, and exponentiation can be considered each as a higher order of the previous. For example, 2+2+2 = 2*3 which is significant because there are three twos, which is the…

to clarify, is that with https://huggingface.co/relaxml/Llama-2-70b-chat-E8P-2Bit

It is Llama-2-70b-chat. I quantized it to 2q_k using `quantize` with llama.cpp.

Re: QuIP#: 2-bit Quantization for LLMs

#14

Post by gumby »

skykooler wrote: post #9

I wonder whether quantization to 1-bit would be functional?

There was a paper from the Allen institute from around 2017 successfully using 1 bit quantization but I can’t find it right now. We started using it where I was working at the time but I’m no longer there so I don’t know how it all turned out.

Re: QuIP#: 2-bit Quantization for LLMs

#15

Post by SeanAnderson »

Just to make sure I'm understanding this correctly.

This paper signals that the authors have found a way to run Llama 2 70B, but with 1/8th the VRAM requirements as compared to the original model, right?

And the output is on-par with the original along some metrics (ArcE/PiQA), within 25% on others (Wiki/C4), and the trajectory of their progress hints that there's even more ground to gain in the future?

Re: QuIP#: 2-bit Quantization for LLMs

#17

Post by epcoa »

taneq wrote: post #5
beacon294 wrote: post #4

Earlier quoted context omitted.

The answer seems kind of low quality, but it got further than others. It could probably self correct.

I don’t know too many humans who could answer that half as well.

"For example, the addition of two numbers, 2+2, can be seen as a repeated application of the multiplication operation,"

I bet most humans ever born did not know what the word "multiplication" meant. Not sure how your metric is useful.

Re: QuIP#: 2-bit Quantization for LLMs

#18

Post by 0xDEADFED5 »

lhl wrote: post #12
skavi wrote: post #2

can anyone comment on running the 2b quantized llama 70b on consumer cards like the 4090?

It's been on my list to do a proper shootout of all the various new quant formats floating around (my list here: https://llm-tracker.info/books/llms/page/quantization-overvi... ) but a lot of them don't have very good production code yet (eg, a few months ago, when I tried OmniQuant, some of the important bits of code wasn't even included and had to be gotten directly from the authors: https://llm-tracker.info/books/…

i humbly request you to add mlc-llm to your quant test when/if you get around to doing it

Re: QuIP#: 2-bit Quantization for LLMs

#19

Post by lhl »

0xDEADFED5 wrote: post #18
lhl wrote: post #12

Earlier quoted context omitted.

It's been on my list to do a proper shootout of all the various new quant formats floating around (my list here: https://llm-tracker.info/books/llms/page/quantization-overvi... ) but a lot of them don't have very good production code yet (eg, a few months ago, when I tried OmniQuant, some of the important bits of code wasn't even included and had to be gotten directly from the authors: https://llm-tracker.info/books/…

i humbly request you to add mlc-llm to your quant test when/if you get around to doing it

Sure, I think their quant format is pretty basic, something similar to bnb q4 - my plan will to be scripting a framework for testing, so should do that as well since the omniquant implementation is in mlc-llm anyways.

Re: QuIP#: 2-bit Quantization for LLMs

#20

Post by acosmism »

0xDEADFED5 wrote: post #18
lhl wrote: post #12

Earlier quoted context omitted.

It's been on my list to do a proper shootout of all the various new quant formats floating around (my list here: https://llm-tracker.info/books/llms/page/quantization-overvi... ) but a lot of them don't have very good production code yet (eg, a few months ago, when I tried OmniQuant, some of the important bits of code wasn't even included and had to be gotten directly from the authors: https://llm-tracker.info/books/…

i humbly request you to add mlc-llm to your quant test when/if you get around to doing it

i was trying to get this to work with mlc-llm. i'd appreciate any pointers
Post reply on HN