Since a pixel can have more states than binary, could you have more space and compute by leveraging RGBA-space for data/compute than binary? Maybe a stupid question.
A pixel takes more than 1 bit to store, too
QuIP#: 2-bit Quantization for LLMs
31–40 of 62 posts
Re: QuIP#: 2-bit Quantization for LLMs
#32Just to make sure I'm understanding this correctly. This paper signals that the authors have found a way to run Llama 2 70B, but with 1/8th the VRAM requirements as compared to the original model, right? And the output is on-par with the original along some metrics (ArcE/PiQA), within 25% on others (Wiki/C4), and the trajectory of their progress hints that there's even more ground to gain in the future?
Your last paragraph is the key. Without their improvements, the quality tradeoff would have been hard to stomach.
When considering quality, you should really compare not to the original model, but to smaller models which use the same vram/compute
Re: QuIP#: 2-bit Quantization for LLMs
#33Earlier quoted context omitted.
Your last paragraph is the key. Without their improvements, the quality tradeoff would have been hard to stomach.
When considering quality, you should really compare not to the original model, but to smaller models which use the same vram/compute
But people use large model for better metrics, this defeats the purpose.
Re: QuIP#: 2-bit Quantization for LLMs
#34For quantization, you should always verify directly on your own intended tasks, not trusting the quantization will preserve accuracy on a boarder spectrums of tasks, because surprises are not that infrequent.
Re: QuIP#: 2-bit Quantization for LLMs
#35Does LM Studio support it?
By the way, what’s your favorite easy-to-use LLM front end?
Re: QuIP#: 2-bit Quantization for LLMs
#36Re: QuIP#: 2-bit Quantization for LLMs
#37Does LM Studio support it? By the way, what’s your favorite easy-to-use LLM front end?
LM Studio, sort of. Unfortunately my Macbook is incapable of running all but the smallest models. It would be great if LM Studio can connect to a remote model running on a GPU server that I can rent.
Re: QuIP#: 2-bit Quantization for LLMs
#38Does LM Studio support it? By the way, what’s your favorite easy-to-use LLM front end?
LM Studio, sort of. Unfortunately my Macbook is incapable of running all but the smallest models. It would be great if LM Studio can connect to a remote model running on a GPU server that I can rent.
You could set up something similar using gradio quite easily:
Re: QuIP#: 2-bit Quantization for LLMs
#39How does this 2-bit quantization method compare to HQQ which was posted yesterday?
Re: QuIP#: 2-bit Quantization for LLMs
#40[dead]