DyLoRA: Parameter Efficient Tuning of Pre-Trained Models
1–10 of 37 posts
Re: DyLoRA: Parameter Efficient Tuning of Pre-Trained Models
#2So this can tune a model 7X faster than LoRA, which was already a massive speed boost? Curious to see what this will do to the LLaMA-derivative community in particular.
Re: DyLoRA: Parameter Efficient Tuning of Pre-Trained Models
#3I'm unsure of the value of dynamically reducing the rank of the LoRA matrix at inference time given that probably most of the parameter count comes from the original weights rather than the LoRA diff.
But nonetheless, training time improvements look interesting.
e: Oh I see, the training time improvement is compared to a grid search over the LoRA rank. Not for a single run.
I am not convinced that you shouldn't just train on the highest possible rank that you can with your compute budget. If you can train a DynLoRA with rank 8, why not just train a LoRA with that rank?
Re: DyLoRA: Parameter Efficient Tuning of Pre-Trained Models
#4So this can tune a model 7X faster than LoRA, which was already a massive speed boost? Curious to see what this will do to the LLaMA-derivative community in particular.
7x faster compared to grid-search LoRA for best rank.
I am not convinced that the "best rank" is not just the highest possible with your compute budget, personally.
Re: DyLoRA: Parameter Efficient Tuning of Pre-Trained Models
#5When fine tuning an LLM you can use the LORA technique to make the fine tuning faster. LORA involves fine tuning a subset of parameters (really it's a low rank approximation of the weight matrix determined by picking the n largest eigenvalues in the SVD decomposition). The size of the subset is determined by the rank. The smaller the rank the faster the fine tuning. However if you make the rank too small then quality will suffer. So you want to pick the optimal rank. This paper describes a technique which can be used to find the optimal rank more easily.
Re: DyLoRA: Parameter Efficient Tuning of Pre-Trained Models
#6Kudos for the authors for providing the code https://github.com/huawei-noah/KD-NLP/tree/main/DyLoRA and the roberta example. Considering the current state of the OSS LLM community, I'm guessing someone is already porting it to Llama and gpt-style models.
Re: DyLoRA: Parameter Efficient Tuning of Pre-Trained Models
#7So this can tune a model 7X faster than LoRA, which was already a massive speed boost? Curious to see what this will do to the LLaMA-derivative community in particular.
7x faster compared to grid-search LoRA for best rank. I am not convinced that the "best rank" is not just the highest possible with your compute budget, personally.
What is the fastest way to show that?
Re: DyLoRA: Parameter Efficient Tuning of Pre-Trained Models
#8Earlier quoted context omitted.
7x faster compared to grid-search LoRA for best rank. I am not convinced that the "best rank" is not just the highest possible with your compute budget, personally.
What is the fastest way to show that?
Fastest way to show what? That you should train with the maximum sized LoRA you can? Because the only upsides to having a smaller LoRA are in the training time, and if you are already able to train a DynLoRA with max rank 8, then you should just train a LoRA with that rank.
Re: DyLoRA: Parameter Efficient Tuning of Pre-Trained Models
#9I'm unsure of the value of dynamically reducing the rank of the LoRA matrix at inference time given that probably most of the parameter count comes from the original weights rather than the LoRA diff. But nonetheless, training time improvements look interesting. e: Oh I see, the training time improvement is compared to a grid search over the LoRA rank. Not for a single run. I am not convinced that you shouldn't just…
Yea, this is interesting but I can't see the immidiate value (not that there isn't).
Maybe if the "optimal rank" of LORA applies to any adaptation and you interested in training multiple adaptations for different use cases?
Re: DyLoRA: Parameter Efficient Tuning of Pre-Trained Models
#10Earlier quoted context omitted.
What is the fastest way to show that?
Fastest way to show what? That you should train with the maximum sized LoRA you can? Because the only upsides to having a smaller LoRA are in the training time, and if you are already able to train a DynLoRA with max rank 8, then you should just train a LoRA with that rank.
You get diminishing returns as you increase the rank, so with a fixed training budget it's not clear whether you get the best return from increasing rank vs increasing something else. If you start off by training DynLORA with max rank 8 you can see returns diminish fast beyond rank 5. Then you can use rank 5 for the rest of your training. You wouldn't know that with LoRA. I think this is the idea behind the paper. If you are just going to use your entire budget training a DyLoRA with max rank 8 then you're right there's no advantage over LoRA with rank 8. You'd have to use the ability to assess multiple ranks in order to see some benefit.