[dead]
LoRA from scratch: implementation for LLM finetuning
61–70 of 87 posts
Re: LoRA from scratch: implementation for LLM finetuning
#62Nice article, I'm not in this field, however, my understanding of the original paper was that the LoRA was applied only on the last dense layer, and not to all independently (maybe I misread it originally). Digging a bit in why the implementation is like this in the link, I found that in QLoRA they used this and it seems to have some interesting effects, maybe adding a note on the QLoRA decision would be nice :) I'm…
Like most things in ML, the answer of which layers to use come down to empirical evidence more than theory. In a typical Lora training pipeline, you freeze the contents of the base model and just adjust the Lora layers. The more layers you convert to lora layers the more degrees of freedom you have for the optimization. There are some finetuning regimens that only recommend finetuning the last layer since this is the…
Yeah, but if I remember correctly the paper, LoRA followed the logic that only the last layers on a llm changed drastically during finetuning, and the layers above remained almost unchanged, so it made sense to alterate only the last ones, breaking this by adding a LoRA at each linear layer doesn't seem to follow the logic of why LoRA was created and why it works.
Re: LoRA from scratch: implementation for LLM finetuning
#63[dead]
nice, looks very cool and useful! I'll definitely try it!
Re: LoRA from scratch: implementation for LLM finetuning
#64Earlier quoted context omitted.
Like most things in ML, the answer of which layers to use come down to empirical evidence more than theory. In a typical Lora training pipeline, you freeze the contents of the base model and just adjust the Lora layers. The more layers you convert to lora layers the more degrees of freedom you have for the optimization. There are some finetuning regimens that only recommend finetuning the last layer since this is the…
Yeah, but if I remember correctly the paper, LoRA followed the logic that only the last layers on a llm changed drastically during finetuning, and the layers above remained almost unchanged, so it made sense to alterate only the last ones, breaking this by adding a LoRA at each linear layer doesn't seem to follow the logic of why LoRA was created and why it works.
Well, Lora works just because it's a low rank approximation of full updates - much in the same way that SVD works, and regular gradient updating works. It delivers good results by both acting as a regularizer and by allowing larger models to be updated with smaller memory footprints.
My point is that the original Lora paper choosing the last layer is one choice. And it is likely the most common one because of its higher symbolic nature typically being all that's needed for good performance on downstream tasks.
Depending on the size of your finetuning job I've personally seen updating more layers (or updating some only on a certain learning rate schedule) to be more effective. Lora is just the mathematical technique of updating, it doesn't really have a hypothesis on the ideal training regimen.
Re: LoRA from scratch: implementation for LLM finetuning
#65LoRA != LoRa. I keep on getting confused and hate that they chose to reuse an existing acronym
Likewise. My day job is machine learning and I still, or maybe consequently, do a double-take every time I see the acronym with minimal context (like on the HN front page, where either usage would be normal).
And my day job involves a lot of LoRa. I always do a double take on these. I'm grateful that at least the caps is now being done differently.
Re: LoRA from scratch: implementation for LLM finetuning
#66LoRA != LoRa. I keep on getting confused and hate that they chose to reuse an existing acronym
I hate the trend of software guys naming things after hardware related stuff
Re: LoRA from scratch: implementation for LLM finetuning
#67LoRA != LoRa. I keep on getting confused and hate that they chose to reuse an existing acronym
That's what happens when people specialize and don't pay attention to what's going on outside their bubble.
Re: LoRA from scratch: implementation for LLM finetuning
#68LoRA != LoRa. I keep on getting confused and hate that they chose to reuse an existing acronym
It's unfortunate that those two so far unrelated technologies have the same acronym.
LoRa the radio tech was first, so as far as I'm concerned it's the canonical definition of the acronym. But I'm biased, I'm an embedded firmware dev
Re: LoRA from scratch: implementation for LLM finetuning
#69HN friends, What are the most popular libraries for fine-tuning? (Not from scratch)