LoRA Learns Less and Forgets Less
arxiv.org
LoRA Learns Less and Forgets Less
1–10 of 65 posts
Re: LoRA Learns Less and Forgets Less
#2This is about Low-rank adaptation. Not to be confused with LoRa the long range proprietary radio communication technique, which hopefully doesn't learn at all.
Re: LoRA Learns Less and Forgets Less
#3This is "Low-Rank Adaptation", "a widely-used parameter-efficient finetuning method for large language models."
Not to be confused with LoRa ("long range") [1], an Internet of Things radio technology.
Re: LoRA Learns Less and Forgets Less
#4This is "Low-Rank Adaptation", "a widely-used parameter-efficient finetuning method for large language models." Not to be confused with LoRa ("long range") [1], an Internet of Things radio technology. [1] https://en.wikipedia.org/wiki/LoRa
Isn’t this fairly obvious after a two second glance at the abstract
Re: LoRA Learns Less and Forgets Less
#5This is Low-rank adaptation. Not to be confused with Lake of the Ozarks Recreation Area.
Re: LoRA Learns Less and Forgets Less
#6The findings are that the best fine-tune performance comes from fine-tuning all weights, followed my MLPs, followed by attention heads, using LoRA. Authors assert that the performance difference is based on the target module of the NN.
Isn’t an equally valid argument that MLPs tend to constitute a greater number of weights in transformer networks than attention heads, and the performance difference can be traced to a greater number of weights having freedom to change? I’d be curious to know if randomly choosing a subset of matrices to train, regardless of where they are in the network, would provide analogous performance to LoRA on a specific module with comparable learnable weights.
Re: LoRA Learns Less and Forgets Less
#7The findings are that the best fine-tune performance comes from fine-tuning all weights, followed my MLPs, followed by attention heads, using LoRA. Authors assert that the performance difference is based on the target module of the NN. Isn’t an equally valid argument that MLPs tend to constitute a greater number of weights in transformer networks than attention heads, and the performance difference can be traced to a…
as a follow up curiosity, has anyone tried using LoRA on the entire model for pretraining to compare regular training model performance to LoRA?
Re: LoRA Learns Less and Forgets Less
#8This is about Low-rank adaptation. Not to be confused with LoRa the long range proprietary radio communication technique, which hopefully doesn't learn at all.
"Why the hell is LoRa learning" was indeed my first thought...
Re: LoRA Learns Less and Forgets Less
#9The findings are that the best fine-tune performance comes from fine-tuning all weights, followed my MLPs, followed by attention heads, using LoRA. Authors assert that the performance difference is based on the target module of the NN. Isn’t an equally valid argument that MLPs tend to constitute a greater number of weights in transformer networks than attention heads, and the performance difference can be traced to a…
as a follow up curiosity, has anyone tried using LoRA on the entire model for pretraining to compare regular training model performance to LoRA?
This paper [1] does atempt that and reports similar performance compared to conventional pre-training. However, they do start off by doing a normal full-rank training and claim that it is needed to 'warm start' the training process.
Re: LoRA Learns Less and Forgets Less
#10I really wish people would be more careful about choosing names for these things.
LoRa has been a popular wireless protocol for like 10 years.