QuIP#: 2-bit Quantization for LLMs
cornell-relaxml.github.io
QuIP#: 2-bit Quantization for LLMs
1–10 of 62 posts
Re: QuIP#: 2-bit Quantization for LLMs
#2can anyone comment on running the 2b quantized llama 70b on consumer cards like the 4090?
Re: QuIP#: 2-bit Quantization for LLMs
#3can anyone comment on running the 2b quantized llama 70b on consumer cards like the 4090?
I have quantized Llama-2-chat-70b to 2q_k with llama.cpp then run on a 4090 for 50/83 layers. I was able to get 1.3 tokens per second. It answers this question, which smaller models have trouble with:
[INST]Hello. The functions addition, multiplication, and exponentiation can be considered each as a higher order of the previous. For example, 2+2+2 = 2*3 which is significant because there are three twos, which is the multiplication factor. Likewise 2*2*2=2^3, again, the power number represents the count of the numbers being multiplied. What is this concept and what is the next higher order function?[/INST] The concept you are referring to is called the "iterative" or "recursive" nature of arithmetic operations. Each operation can be seen as a repeated application of the previous operation, with the multiplication factor representing the number of times the operation is applied.
For example, the addition of two numbers, 2+2, can be seen as a repeated application of the multiplication operation, where the multiplicand is 2 and the multiplier is 3 (i.e., 2*3 = 6, which is the result of the addition). Similarly, the multiplication of two numbers, 2*2, can be seen as a repeated application of the exponentiation operation, where the base is 2 and the power is 3 (i.e., 2^3 = 8, which is the result of the multiplication).
The next higher order function would be the "tetration" operation, which is the iterative application of the exponential function. In other words, tetration(n,x) = x^(x^(x^(...(x^n)...))), where n is the number of times the exponential function is applied.
For example, tetration(2,2) = 2^(2^(2^(2))), which can be simplified to 2^16, or 65536.
It's worth noting that this concept can be extended even further to higher levels of abstraction, such as "hyper-operations" and "super -operations", but these are more advanced and less commonly used concepts in mathematics.*
Re: QuIP#: 2-bit Quantization for LLMs
#4can anyone comment on running the 2b quantized llama 70b on consumer cards like the 4090?
I have quantized Llama-2-chat-70b to 2q_k with llama.cpp then run on a 4090 for 50/83 layers. I was able to get 1.3 tokens per second. It answers this question, which smaller models have trouble with: [INST]Hello. The functions addition, multiplication, and exponentiation can be considered each as a higher order of the previous. For example, 2+2+2 = 2*3 which is significant because there are three twos, which is the…
The answer seems kind of low quality, but it got further than others. It could probably self correct.
Re: QuIP#: 2-bit Quantization for LLMs
#5Earlier quoted context omitted.
I have quantized Llama-2-chat-70b to 2q_k with llama.cpp then run on a 4090 for 50/83 layers. I was able to get 1.3 tokens per second. It answers this question, which smaller models have trouble with: [INST]Hello. The functions addition, multiplication, and exponentiation can be considered each as a higher order of the previous. For example, 2+2+2 = 2*3 which is significant because there are three twos, which is the…
The answer seems kind of low quality, but it got further than others. It could probably self correct.
I don’t know too many humans who could answer that half as well.
Re: QuIP#: 2-bit Quantization for LLMs
#6Re: QuIP#: 2-bit Quantization for LLMs
#7One of the best papers I've read in a long time. This could be huge.
Re: QuIP#: 2-bit Quantization for LLMs
#8can anyone comment on running the 2b quantized llama 70b on consumer cards like the 4090?
I have quantized Llama-2-chat-70b to 2q_k with llama.cpp then run on a 4090 for 50/83 layers. I was able to get 1.3 tokens per second. It answers this question, which smaller models have trouble with: [INST]Hello. The functions addition, multiplication, and exponentiation can be considered each as a higher order of the previous. For example, 2+2+2 = 2*3 which is significant because there are three twos, which is the…
to clarify, is that with https://huggingface.co/relaxml/Llama-2-70b-chat-E8P-2Bit
Re: QuIP#: 2-bit Quantization for LLMs
#9I wonder whether quantization to 1-bit would be functional?
Re: QuIP#: 2-bit Quantization for LLMs
#10I wonder whether quantization to 1-bit would be functional?
[deleted]