Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DyLoRA: Parameter Efficient Tuning of Pre-Trained Models

arxiv.org

Post reply on HN
Print view View original

21–30 of 37 posts

Re: DyLoRA: Parameter Efficient Tuning of Pre-Trained Models

#21

Post by Majromax »

whimsicalism wrote: post #4
turnsout wrote: post #2

So this can tune a model 7X faster than LoRA, which was already a massive speed boost? Curious to see what this will do to the LLaMA-derivative community in particular.

7x faster compared to grid-search LoRA for best rank. I am not convinced that the "best rank" is not just the highest possible with your compute budget, personally.

Highest posssible in which combination, though? If you’re fine tuning a model with N layers, then you could apply LoRA to any or all of them. Maybe it’s better to concentrate effort unevenly, in which case a uniform increase of adaptation rank (to compute budget) could still be subpar.

Re: DyLoRA: Parameter Efficient Tuning of Pre-Trained Models

#22

Post by FrostKiwi »

fancyfredbot wrote: post #5

When fine tuning an LLM you can use the LORA technique to make the fine tuning faster. LORA involves fine tuning a subset of parameters (really it's a low rank approximation of the weight matrix determined by picking the n largest eigenvalues in the SVD decomposition). The size of the subset is determined by the rank. The smaller the rank the faster the fine tuning. However if you make the rank too small then quality…

Fascinating progress.

Would you say the following understanding is correct?:

- You can fine-tune a model, regardless of whether it has been quantized (as in the 4-bit versions of models made to fit in consumer grade RAM sizes) or not.

- You can fine-tune any model on any hardware, provided it fits into RAM. That means, that the 30B llama-derived models in their 4-bit quantized version and 19.5GB of VRAM requirement can be fine-tuned on consumer grade GPUs with 24gb of VRAM. (Like the RTX 3090 and 4090)

Re: DyLoRA: Parameter Efficient Tuning of Pre-Trained Models

#23

Post by whimsicalism »

Majromax wrote: post #21
whimsicalism wrote: post #4

Earlier quoted context omitted.

7x faster compared to grid-search LoRA for best rank. I am not convinced that the "best rank" is not just the highest possible with your compute budget, personally.

Highest posssible in which combination, though? If you’re fine tuning a model with N layers, then you could apply LoRA to any or all of them. Maybe it’s better to concentrate effort unevenly, in which case a uniform increase of adaptation rank (to compute budget) could still be subpar.

Right but the way that this paper proposes determining the best rank is by training a LoRA with the full rank.

Re: DyLoRA: Parameter Efficient Tuning of Pre-Trained Models

#24

Post by whimsicalism »

FrostKiwi wrote: post #22
fancyfredbot wrote: post #5

When fine tuning an LLM you can use the LORA technique to make the fine tuning faster. LORA involves fine tuning a subset of parameters (really it's a low rank approximation of the weight matrix determined by picking the n largest eigenvalues in the SVD decomposition). The size of the subset is determined by the rank. The smaller the rank the faster the fine tuning. However if you make the rank too small then quality…

Fascinating progress. Would you say the following understanding is correct?: - You can fine-tune a model, regardless of whether it has been quantized (as in the 4-bit versions of models made to fit in consumer grade RAM sizes) or not. - You can fine-tune any model on any hardware, provided it fits into RAM. That means, that the 30B llama-derived models in their 4-bit quantized version and 19.5GB of VRAM requirement c…

Yes to the first.

To the second, I'm not sure that the RAM requirements are the same to train because you have to preserve the state which takes extra memory.

Re: DyLoRA: Parameter Efficient Tuning of Pre-Trained Models

#25

Post by brucethemoose2 »

kernelsanderz wrote: post #17
lxe wrote: post #6

Kudos for the authors for providing the code https://github.com/huawei-noah/KD-NLP/tree/main/DyLoRA and the roberta example. Considering the current state of the OSS LLM community, I'm guessing someone is already porting it to Llama and gpt-style models.

Adding this to the huggingface peft library would be amazing. That's the main library that people using LoRA are currently using. https://github.com/huggingface/peft/issues/289

The stable diffusion community has, unfortunately, largely ignored peft because the training/inference scripts largely ignored diffusers.

Re: DyLoRA: Parameter Efficient Tuning of Pre-Trained Models

#26

Post by alchemist1e9 »

whimsicalism wrote: post #24
FrostKiwi wrote: post #22

Earlier quoted context omitted.

Fascinating progress. Would you say the following understanding is correct?: - You can fine-tune a model, regardless of whether it has been quantized (as in the 4-bit versions of models made to fit in consumer grade RAM sizes) or not. - You can fine-tune any model on any hardware, provided it fits into RAM. That means, that the 30B llama-derived models in their 4-bit quantized version and 19.5GB of VRAM requirement c…

Yes to the first. To the second, I'm not sure that the RAM requirements are the same to train because you have to preserve the state which takes extra memory.

Is it possible for many people to simultaneously fine tune models on different data and then combine the new models into something improved?

Re: DyLoRA: Parameter Efficient Tuning of Pre-Trained Models

#27

Post by yorwba »

alchemist1e9 wrote: post #26
whimsicalism wrote: post #24

Earlier quoted context omitted.

Yes to the first. To the second, I'm not sure that the RAM requirements are the same to train because you have to preserve the state which takes extra memory.

Is it possible for many people to simultaneously fine tune models on different data and then combine the new models into something improved?

Yes, but the naïve way to combine rank k adaptations created by n different people would be to concatenate them to a rank nk adaptation, which wouldn't be as lightweight and easy to share, so you'd likely be better off mushing them into the baseline model.

Re: DyLoRA: Parameter Efficient Tuning of Pre-Trained Models

#28

Post by Ldorigo »

whimsicalism wrote: post #8
sitkack wrote: post #7

Earlier quoted context omitted.

What is the fastest way to show that?

Fastest way to show what? That you should train with the maximum sized LoRA you can? Because the only upsides to having a smaller LoRA are in the training time, and if you are already able to train a DynLoRA with max rank 8, then you should just train a LoRA with that rank.

Why is the only advantage at training time? I might misunderstand something but with this method you can train once, and then deploy models that use arbitrary rank (according to end-users compute requirements) and expect to have a model that performs best for that specific rank.

Re: DyLoRA: Parameter Efficient Tuning of Pre-Trained Models

#29

Post by fancyfredbot »

alchemist1e9 wrote: post #26
whimsicalism wrote: post #24

Earlier quoted context omitted.

Yes to the first. To the second, I'm not sure that the RAM requirements are the same to train because you have to preserve the state which takes extra memory.

Is it possible for many people to simultaneously fine tune models on different data and then combine the new models into something improved?

One approach is to have the model learn to select between several separately fine tuned adapters by learning which adapter works best in a given context. So at any given time it's only really using one adapter but can switch to another. In this case one adapter can't really improve another but the overall impact might be a model which is improved in a variety of different contexts.

Re: DyLoRA: Parameter Efficient Tuning of Pre-Trained Models

#30

Post by alchemist1e9 »

yorwba wrote: post #27
alchemist1e9 wrote: post #26

Earlier quoted context omitted.

Is it possible for many people to simultaneously fine tune models on different data and then combine the new models into something improved?

Yes, but the naïve way to combine rank k adaptations created by n different people would be to concatenate them to a rank nk adaptation, which wouldn't be as lightweight and easy to share, so you'd likely be better off mushing them into the baseline model.

Can they mathematically be “mushed” and then create an improved model?

I have yet to understand the difference between fine tuning and training and therefore yet to understand if a distributed decentralized eventually consistent training approach is a possibility or simply not realistic.

Post reply on HN