Live data from Hacker News

QuIP#: 2-bit Quantization for LLMs

cornell-relaxml.github.io

51–60 of 62 posts

Re: QuIP#: 2-bit Quantization for LLMs

#51
post #25

Earlier quoted context omitted.

It is Llama-2-70b-chat. I quantized it to 2q_k using `quantize` with llama.cpp.

So your experience isn’t representative of the work presented by this post? Or does llama.cpp use the same technique for quantization?

I don't know, hopefully it will help set general expectations.

Re: QuIP#: 2-bit Quantization for LLMs

#52

I wonder whether quantization to 1-bit would be functional?

Those networks are named as discrete neural networks. There are already research on those, mainly for (homomorphic) encryption purposes (because it’s much easier to homomorphically encrypt NNs of just 0’s and 1’s than normal NNs).

Re: QuIP#: 2-bit Quantization for LLMs

#53

I wonder whether quantization to 1-bit would be functional?

I predict that even less-than-1-bit viable quantization methods will be found eventually. Of course, the bits-per-weight figure should be interpreted as something average: e.g., group-quantizing 32 weights into 24 bits would be 0.75 bits per weight.

Re: QuIP#: 2-bit Quantization for LLMs

#54
post #28

I wonder whether quantization to 1-bit would be functional?

If we get 1-bit quantization, wouldn't it be basically a bunch of nested if's and else's?

No. The catch is that what's quantized here is not individual weights but groups of them.

Re: QuIP#: 2-bit Quantization for LLMs

#56
post #17

Earlier quoted context omitted.

"For example, the addition of two numbers, 2+2, can be seen as a repeated application of the multiplication operation," I bet most humans ever born did not know what the word "multiplication" meant. Not sure how your metric is useful.

Sometimes, to achieve some perspective, I also zoom out and declare normal over the estimated +-120,000,000,000 people who have ever lived.

I think the point went over your head (which just further proves mine).

Hint: Perhaps read the original output more carefully: "For example, the addition of two numbers, 2+2, can be seen as a repeated application of the multiplication operation,". Or are you counted in the majority of that 120,000,000,000?

Re: QuIP#: 2-bit Quantization for LLMs

#57
post #56

Earlier quoted context omitted.

Sometimes, to achieve some perspective, I also zoom out and declare normal over the estimated +-120,000,000,000 people who have ever lived.

I think the point went over your head (which just further proves mine). Hint: Perhaps read the original output more carefully: "For example, the addition of two numbers, 2+2, can be seen as a repeated application of the multiplication operation,". Or are you counted in the majority of that 120,000,000,000?

Paris

in the

the spring

Re: QuIP#: 2-bit Quantization for LLMs

#60
post #58

Earlier quoted context omitted.

yes. common bit depths to represent pixels range up to 64-bits.

But the physical space?

3840 × 2160 Total number of pixels = 8,294,400

log2(8,294,400) ≈ 23 (each additional bit doubles the number of unique addresses that can be represented)

so 24 bits are the minimum required to locate a point in 2-d space at the resolution of "4k". when you add in color, 60 bits per pixel.

19 bits for 640x480.

Post reply on HN