LoRA from scratch: implementation for LLM finetuning
1–10 of 87 posts
Re: LoRA from scratch: implementation for LLM finetuning
#2I've been keeping track of the techniques through Maxime Labonne's LLMs 101: https://github.com/mlabonne/llm-course#4-supervised-fine-tun...
Re: LoRA from scratch: implementation for LLM finetuning
#3Not to be confused with LoRa ("long range"), a radio communication protocol. At first I thought this could be about using LLMs to find optimal protocol parameters, but alas.
Re: LoRA from scratch: implementation for LLM finetuning
#4Not to be confused with LoRa ("long range"), a radio communication protocol. At first I thought this could be about using LLMs to find optimal protocol parameters, but alas.
I had the exact same confusion
Re: LoRA from scratch: implementation for LLM finetuning
#5Not to be confused with LoRa ("long range"), a radio communication protocol. At first I thought this could be about using LLMs to find optimal protocol parameters, but alas.
It's the first thing that comes to my mind too, but this is mentioned in every thread (and there are far more of them for LoRA than LoRa atm), and in this case there's unlikely to be much confusion because it starts by spelling out the acronym: 'LoRA, which stands for Low Rank Adaptation, [...]'.
Re: LoRA from scratch: implementation for LLM finetuning
#6Not to be confused with LoRa ("long range"), a radio communication protocol. At first I thought this could be about using LLMs to find optimal protocol parameters, but alas.
Hah, yeah that's LoRA as in Low-Rank Adaptation :P
Re: LoRA from scratch: implementation for LLM finetuning
#7"From scratch" seems to be a matter of opinion. "Pure pytorch" maybe, except it uses HF transformers. So it's LoRA on top of common frameworks...
Re: LoRA from scratch: implementation for LLM finetuning
#8"From scratch" seems to be a matter of opinion. "Pure pytorch" maybe, except it uses HF transformers. So it's LoRA on top of common frameworks...
Yeah, the LoRA part is from scratch. The LLM backbone in this example is not, this is to provide a concrete example. But you could apply the exact same LoRA from scratch code to a pure PyTorch model if you wanted to:
E.g.
class MultilayerPerceptron(nn.Module):
def __init__(self, num_features, num_hidden_1, num_hidden_2, num_classes):
super().__init__()
self.layers = nn.Sequential(
nn.Linear(num_features, num_hidden_1),
nn.ReLU(),
nn.Linear(num_hidden_1, num_hidden_2),
nn.ReLU(),
nn.Linear(num_hidden_2, num_classes)
)
def forward(self, x):
x = self.layers(x)
return x
model = MultilayerPerceptron(
num_features=num_features,
num_hidden_1=num_hidden_1,
num_hidden_2=num_hidden_2,
num_classes=num_classes
)
model.layers[0] = LinearWithLoRA(model.layers[0], rank=4, alpha=1)
model.layers[2] = LinearWithLoRA(model.layers[2], rank=4, alpha=1)
model.layers[4] = LinearWithLoRA(model.layers[4], rank=4, alpha=1)Re: LoRA from scratch: implementation for LLM finetuning
#9"From scratch" seems to be a matter of opinion. "Pure pytorch" maybe, except it uses HF transformers. So it's LoRA on top of common frameworks...
This apple pie recipe claims to be from scratch, but they cooked it in an off the shelf oven. So it's from scratch on top of the universe...
Re: LoRA from scratch: implementation for LLM finetuning
#10[flagged]