Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

QLoRA: Efficient Finetuning of Quantized LLMs

arxiv.org

Post reply on HN
Print view View original

31–40 of 116 posts

Re: QLoRA: Efficient Finetuning of Quantized LLMs

#31

Post by brucethemoose2 »

andy_xor_andrew wrote: post #15

Hold on. I need someone to explain something to me. The colab notebook shows an example of loading the vanilla, unquantized model "decapoda-research/llama-7b-hf", using the flag "load_in_4bit" to load it as 4bits. When... when did this become possible? My understanding, from playing with these models daily for the past few months, is that quantization of LLaMA-based models is done via this: https://github.com/qwopqwo…

Thats just one method...

- bitsandbytes was always used for on the fly 8 bit quant, just like its being used for 4-bit now. - llama.cpp (and derivatives) quantize ahead of time, but its not resource intense. - mlc llm (vulkan/metal llm inference via tvm) do require lots of ram for their quantization

Re: QLoRA: Efficient Finetuning of Quantized LLMs

#32

Post by sanxiyn »

sp332 wrote: post #24
orost wrote: post #20

Earlier quoted context omitted.

Quantization isn't (and wasn't) expensive, it's mostly just data shuffling. A good PC will do a 7B model in half a minute, up to a few minutes for a larger model. Quantized models being made available for download is more for the benefit of less technical users who may not be comfortable with the command-line tools, or for people with slow or metered connections who'd much rather download 15GB of data than download 6…

The question is whether this step is actually doing the GPTQ optimized quantization, or simple truncation.

This work introduces a new quantization scheme, NF4, for 4-bit NormalFloat, based on previous work on quantile quantization, so it's not a simple truncation, but it's also not a GPTQ-like optimization method. Figure 3 of the paper shows accuracy improvement of NF4 over FP4.

Re: QLoRA: Efficient Finetuning of Quantized LLMs

#33

Post by sanxiyn »

api wrote: post #17
sva_ wrote: post #11

Earlier quoted context omitted.

To be fair, Altman has been fairly outspoken about not limiting open source models. In how far that was just for streetcred, I cannot say however.

If progress really continues like this any regulation that does not limit open models would be a pointless exercise. It’s only a matter of time before people really crack distributed training algorithms that can be run in a less organized swarm configuration. At that point open trainers could actually train near the frontier of what is possible. Most of the data is open.

> It's only a matter of time before people really crack distributed training algorithms that can be run in a less organized swarm configuration.

Why do you think this? This continually failed, and seems extremely unlikely to me. Barring surprising breakthrough, there is inherent communication complexity, and physical limit to communication bandwidth.

Re: QLoRA: Efficient Finetuning of Quantized LLMs

#34

Post by sanxiyn »

gumballindie wrote: post #29
CuriouslyC wrote: post #16

Earlier quoted context omitted.

Yes, in a sense. It won't work well for distribution of training on a single subject, but you distribute training based on subject and then combine LoRAs.

> and then combine LoRAs How do you combine them? > but you distribute training based on subject Perhaps this could work like a set of hashed and trusted data sets split but subject and topic? Each node downloads one at random and trains against that single subset of a topic or something.

You can, like, just do weighted average. It works.

Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Models (this is Tim Dettmers's previous work!)

> These ELMs can be added and removed to update data coverage, ensembled to generalize to new domains, or averaged to collapse back to a single LM for efficient inference.

https://arxiv.org/abs/2208.03306

Re: QLoRA: Efficient Finetuning of Quantized LLMs

#35

Post by gumballindie »

sanxiyn wrote: post #34
gumballindie wrote: post #29

Earlier quoted context omitted.

> and then combine LoRAs How do you combine them? > but you distribute training based on subject Perhaps this could work like a set of hashed and trusted data sets split but subject and topic? Each node downloads one at random and trains against that single subset of a topic or something.

You can, like, just do weighted average. It works. Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Models (this is Tim Dettmers's previous work!) > These ELMs can be added and removed to update data coverage, ensembled to generalize to new domains, or averaged to collapse back to a single LM for efficient inference. https://arxiv.org/abs/2208.03306

So that means a massively distributed model training network with cryptocurrency like incentives is incoming? Where and how to begin? This could free up companies such as openai and potentially lead to the first agi.

(mentioning crypto because that will motivate switch hordes of miners that already have the gpu power available)

Re: QLoRA: Efficient Finetuning of Quantized LLMs

#36

Post by sanxiyn »

gumballindie wrote: post #35
sanxiyn wrote: post #34

Earlier quoted context omitted.

You can, like, just do weighted average. It works. Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Models (this is Tim Dettmers's previous work!) > These ELMs can be added and removed to update data coverage, ensembled to generalize to new domains, or averaged to collapse back to a single LM for efficient inference. https://arxiv.org/abs/2208.03306

So that means a massively distributed model training network with cryptocurrency like incentives is incoming? Where and how to begin? This could free up companies such as openai and potentially lead to the first agi. (mentioning crypto because that will motivate switch hordes of miners that already have the gpu power available)

BTM is very promising, but it is unclear how much it scales, let alone "massively". The paper scaled it to 64 domains and it worked, but you probably want more than 64 nodes.

Since domain is actually important to its performance, you can't randomly split to 64 pieces, see Table 4 of the paper, "Domain expert ensemble outperforms random split ensemble". Performance difference is large.

So if you want to begin, I would start by researching how to scale domain split.

Re: QLoRA: Efficient Finetuning of Quantized LLMs

#37

Post by make3 »

andy_xor_andrew wrote: post #15

Hold on. I need someone to explain something to me. The colab notebook shows an example of loading the vanilla, unquantized model "decapoda-research/llama-7b-hf", using the flag "load_in_4bit" to load it as 4bits. When... when did this become possible? My understanding, from playing with these models daily for the past few months, is that quantization of LLaMA-based models is done via this: https://github.com/qwopqwo…

There are a very large number of quantization schemes in existence, definitely not just one, & they all have potentially very different ideas and schemes.

LLM8 was introduced before https://arxiv.org/abs/2208.07339 of the same first and last author as QLora & is still what can be used in Huggingface Transformers with the `load_in_8bits` parameter.

The idea was just to quantize all weights to 8 bits except a few outliers, which are kept in original precision. This scheme kept the computations extremely accurate, and was really fast to do.

I haven't read the new paper, but I assume they came up with a more advanced fast distributional setup.

Re: QLoRA: Efficient Finetuning of Quantized LLMs

#38

Post by [deleted] »

sanxiyn wrote: post #32
sp332 wrote: post #24

Earlier quoted context omitted.

The question is whether this step is actually doing the GPTQ optimized quantization, or simple truncation.

This work introduces a new quantization scheme, NF4, for 4-bit NormalFloat, based on previous work on quantile quantization, so it's not a simple truncation, but it's also not a GPTQ-like optimization method. Figure 3 of the paper shows accuracy improvement of NF4 over FP4.

[deleted]

Re: QLoRA: Efficient Finetuning of Quantized LLMs

#39

Post by ukuina »

CSSer wrote: post #3
simonw wrote: post #2

I'm very impressed at the quality of Guanaco 33B, the model that accompanies this paper. You can try it out here: https://huggingface.co/spaces/uwnlp/guanaco-playground-tgi I tried "You are a sentient cheesecake that teaches people SQL, with cheesecake analogies to illustrate different points. Teach me to use count and group by" and got a good result from it: https://twitter.com/simonw/status/1661460336334241794/phot…

My prompt: You are a sentient cow with a PHD in mathematics. You can speak English, but you randomly insert Cow-like “Moo” sounds into parts of your dialogue. Explain to me why 2+2=4. Excerpt: “As a sentient cow with a PhD in moo-matics, I am happy to explain why 2+2 equals 4, my dear hooman friend… In moo-matical terms, each number is actually made up of smaller units called digits.” I approve.

Which is heavier: a pound of feathers, or a great british pound?

> Both weights are equal.

Re: QLoRA: Efficient Finetuning of Quantized LLMs

#40

Post by CuriouslyC »

gumballindie wrote: post #35
sanxiyn wrote: post #34

Earlier quoted context omitted.

You can, like, just do weighted average. It works. Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Models (this is Tim Dettmers's previous work!) > These ELMs can be added and removed to update data coverage, ensembled to generalize to new domains, or averaged to collapse back to a single LM for efficient inference. https://arxiv.org/abs/2208.03306

So that means a massively distributed model training network with cryptocurrency like incentives is incoming? Where and how to begin? This could free up companies such as openai and potentially lead to the first agi. (mentioning crypto because that will motivate switch hordes of miners that already have the gpu power available)

We already have exactly that for stable diffusion with Civitai.com. People have published a variety of LoRAs for different subjects just as you describe. The local LLM community is very much following the lead of the stable diffusion community in terms of how it's organizing, so I expect that we'll see a proliferation of domain LoRAs being published on an aggregator for LLM stuff before too long.
Post reply on HN