I really wish people would be more careful about choosing names for these things. LoRa has been a popular wireless protocol for like 10 years.
Yes, but this is LoRA, clearly not LoRa.
LoRA Learns Less and Forgets Less
11–20 of 65 posts
Re: LoRA Learns Less and Forgets Less
#12I really wish people would be more careful about choosing names for these things. LoRa has been a popular wireless protocol for like 10 years.
Yes, but this is LoRA, clearly not LoRa.
PP has a point though. I entered "LoRA" on Google, DuckDuckGo, Startpage and Bing, and all returned results in all first pages were about the communication protocol (1). They could have inferred my interests from previous searches, but I never used Bing in the last year or so, so it seems to me someone didn't care about name clashes.
(1) well, except Google which -surprise- returned about mid page an ad of a local quite expensive chandeliers brand called "LORA".
Re: LoRA Learns Less and Forgets Less
#13The findings are that the best fine-tune performance comes from fine-tuning all weights, followed my MLPs, followed by attention heads, using LoRA. Authors assert that the performance difference is based on the target module of the NN. Isn’t an equally valid argument that MLPs tend to constitute a greater number of weights in transformer networks than attention heads, and the performance difference can be traced to a…
as a follow up curiosity, has anyone tried using LoRA on the entire model for pretraining to compare regular training model performance to LoRA?
Do you mean leaving most of the model in its initial, randomised state and only training a LoRA?
Re: LoRA Learns Less and Forgets Less
#14What can we learn about Low Rank Acronyms today?
Re: LoRA Learns Less and Forgets Less
#15The findings are that the best fine-tune performance comes from fine-tuning all weights, followed my MLPs, followed by attention heads, using LoRA. Authors assert that the performance difference is based on the target module of the NN. Isn’t an equally valid argument that MLPs tend to constitute a greater number of weights in transformer networks than attention heads, and the performance difference can be traced to a…
as a follow up curiosity, has anyone tried using LoRA on the entire model for pretraining to compare regular training model performance to LoRA?
Yes, I’ve tested this out. It does train, but the scaling doesn’t seem to pan out. It’ll perform slightly better than the number of trainable parameters, but never improves as you scale, so for now there’s no benefit.
Re: LoRA Learns Less and Forgets Less
#16Earlier quoted context omitted.
as a follow up curiosity, has anyone tried using LoRA on the entire model for pretraining to compare regular training model performance to LoRA?
Do you mean leaving most of the model in its initial, randomised state and only training a LoRA?
I’ve tested specifically this (on my personal time) :) It will train but I found the loss is proportional to the number of trainable parameters. So roughly to hit the performance of a standard 70m param model, you need to train ~70m lora params anyway.
Re: LoRA Learns Less and Forgets Less
#17Earlier quoted context omitted.
as a follow up curiosity, has anyone tried using LoRA on the entire model for pretraining to compare regular training model performance to LoRA?
This paper [1] does atempt that and reports similar performance compared to conventional pre-training. However, they do start off by doing a normal full-rank training and claim that it is needed to 'warm start' the training process. [1] https://arxiv.org/abs/2307.05695
Oh yes this paper! The main issue is the scaling of the A and B LoRA matrices. Some papers show scaling the B matrix with larger learning rates (LoRA+) could be beneficial. DoRA for eg learns an auto scaling vector of numbers which tries to alleviate these issues.
Galore might be more equivalent to full pretraining with the gradients being low rank.
Re: LoRA Learns Less and Forgets Less
#18Earlier quoted context omitted.
Do you mean leaving most of the model in its initial, randomised state and only training a LoRA?
I’ve tested specifically this (on my personal time) :) It will train but I found the loss is proportional to the number of trainable parameters. So roughly to hit the performance of a standard 70m param model, you need to train ~70m lora params anyway.
It's worse than that, because lora requires two matrices per layer. At full rank, you have an additional NxN parameters to learn versus full finetuning, where N is min(input_features, output_features).
For example, tuning a layer of 128 in x 256 out is 32k params. Learning a full-rank lora for that layer would be two matrices of 128x128 and 128x256 = 48k params.
Re: LoRA Learns Less and Forgets Less
#19The findings are that the best fine-tune performance comes from fine-tuning all weights, followed my MLPs, followed by attention heads, using LoRA. Authors assert that the performance difference is based on the target module of the NN. Isn’t an equally valid argument that MLPs tend to constitute a greater number of weights in transformer networks than attention heads, and the performance difference can be traced to a…
I think the QLoRA paper https://arxiv.org/pdf/2305.14314 paper also showed LoRA on all MLP + Attention layers > all MLP layers > just Attention layers.
Other papers show finetuning a select few layers can also work well.
Re: LoRA Learns Less and Forgets Less
#20This is Low-rank adaptation. Not to be confused with Lake of the Ozarks Recreation Area.
Apparently constructed in 1929. You think those wireless people would have been more careful when they reappropriated the name.