This study is great and addresses a question I've had about LoRA for a while.
In a continual learning paper from last year, I found LoRA was extremely effective for faster fine-tuning and not forgetting the original dataset:
LoRA Learns Less and Forgets Less
21–30 of 65 posts
Re: LoRA Learns Less and Forgets Less
#22[dead]
Re: LoRA Learns Less and Forgets Less
#23I really wish people would be more careful about choosing names for these things. LoRa has been a popular wireless protocol for like 10 years.
Yes, but this is LoRA, clearly not LoRa.
I think you mean LoRa®, a registered trademark of Semtech® Corporation, for use only with permission and within specific guidelines. https://www.semtech.com/uploads/company/FAQ-for-Use-of-LoRa-...
Re: LoRA Learns Less and Forgets Less
#24I really wish people would be more careful about choosing names for these things. LoRa has been a popular wireless protocol for like 10 years.
I addressed the same in a previous 'lora' post on HN. For me the name is already reserved for the radio telecommunication meaning. Nothing going to change that.
Re: LoRA Learns Less and Forgets Less
#25The findings are that the best fine-tune performance comes from fine-tuning all weights, followed my MLPs, followed by attention heads, using LoRA. Authors assert that the performance difference is based on the target module of the NN. Isn’t an equally valid argument that MLPs tend to constitute a greater number of weights in transformer networks than attention heads, and the performance difference can be traced to a…
as a follow up curiosity, has anyone tried using LoRA on the entire model for pretraining to compare regular training model performance to LoRA?
i would be shocked if this worked well
Re: LoRA Learns Less and Forgets Less
#26I really wish people would be more careful about choosing names for these things. LoRa has been a popular wireless protocol for like 10 years.
Seriously, that was a terrible name for the wireless system since it's been used by the Loyola Online Records Access system for half a decade or more before the radio company shamelessly copied the name.
Re: LoRA Learns Less and Forgets Less
#27I really wish people would be more careful about choosing names for these things. LoRa has been a popular wireless protocol for like 10 years.
Not sure about "popular"
99% of ML engineers wouldn't know what it is.
Re: LoRA Learns Less and Forgets Less
#28Earlier quoted context omitted.
I’ve tested specifically this (on my personal time) :) It will train but I found the loss is proportional to the number of trainable parameters. So roughly to hit the performance of a standard 70m param model, you need to train ~70m lora params anyway.
It's worse than that, because lora requires two matrices per layer. At full rank, you have an additional NxN parameters to learn versus full finetuning, where N is min(input_features, output_features). For example, tuning a layer of 128 in x 256 out is 32k params. Learning a full-rank lora for that layer would be two matrices of 128x128 and 128x256 = 48k params.
Yeah, exactly. Though the 48k param lora might be as good as a 48k param layer of higher rank, I haven't looked into that case really.
Re: LoRA Learns Less and Forgets Less
#29I really wish people would be more careful about choosing names for these things. LoRa has been a popular wireless protocol for like 10 years.
Not sure about "popular" 99% of ML engineers wouldn't know what it is.
I'm not an ML engineer and the only reason I know that the wireless protocol exists is because in every HN article, there's a comment repeating the same complaint
Re: LoRA Learns Less and Forgets Less
#30I really wish people would be more careful about choosing names for these things. LoRa has been a popular wireless protocol for like 10 years.
Not sure about "popular" 99% of ML engineers wouldn't know what it is.
99% of the engineers who are still working in ML (Machine Language) would.
A much smaller percent among those who write in ML (the functional programming language) probably, though.