I thought this was going to be some neat software defined radio stuff. Still quite interesting though.
LoRA from scratch: implementation for LLM finetuning
71–80 of 87 posts
Re: LoRA from scratch: implementation for LLM finetuning
#72I thought this was going to be some neat software defined radio stuff. Still quite interesting though.
it's all about whether the 'A' is capitalized or not.
LoRa - radio
LoRA - machine learning
Re: LoRA from scratch: implementation for LLM finetuning
#73It's still strange to me to work in a field of computer science where we say things like "we're not exactly sure how these numbers (hyper parameters) affect the result, so just try a bunch of different values and see which one works best."
I feel like it's the difference between something that has been engineered and something that has been discovered. I feel like most of our industry up until now has been engineered. LLMs were discovered.
LLMs were very much engineered... the exact results they yield are hard to determine since they're large statistical models, but I don't think that categorizes the LLMs themselves as a 'discovery' (like say Penicilin)
Re: LoRA from scratch: implementation for LLM finetuning
#74It's still not too clear to me when we should fine tune versus RAG. In the past, I used to believe that finetuning is mostly for model behavioral change, but recently it seems that certain companies are also using fine-tuning for knowledge addition. What are the main use cases for fine tuning?
These are autoregressive models. When you have a new type of sequence where future elements are able to be predicted from previous parts of the sequence, but in a new kind of way than the models have seen before, it would make sense to finetune.
Admittedly, that's a pretty vague descriptor for how to decide what to do for a given data scenario, but it might be good enough as a rough heuristic. Now, whether knowledge addition falls under that, might be a question of taste (without experiments).
Re: LoRA from scratch: implementation for LLM finetuning
#75It's still strange to me to work in a field of computer science where we say things like "we're not exactly sure how these numbers (hyper parameters) affect the result, so just try a bunch of different values and see which one works best."
I mean, it’s kind of in the name isn’t it? Computer science. Science is empirical, often poorly understood and even the best theories don’t fully explain all observations, especially when a field gets new tools to observe phenomena. It takes a while for a good theory to come along and make sense of everything in science and that seems like more or less exactly where we are today.
Re: LoRA from scratch: implementation for LLM finetuning
#76Earlier quoted context omitted.
Yeah, but if I remember correctly the paper, LoRA followed the logic that only the last layers on a llm changed drastically during finetuning, and the layers above remained almost unchanged, so it made sense to alterate only the last ones, breaking this by adding a LoRA at each linear layer doesn't seem to follow the logic of why LoRA was created and why it works.
Well, Lora works just because it's a low rank approximation of full updates - much in the same way that SVD works, and regular gradient updating works. It delivers good results by both acting as a regularizer and by allowing larger models to be updated with smaller memory footprints. My point is that the original Lora paper choosing the last layer is one choice. And it is likely the most common one because of its hig…
Thanks, I'll meditate on that and re read the paper with this view in mind.
The last sentence makes sense to me, if the finetuning job changes significatively more the weights of other layers than just the last one, it is kinda normal to to use Lora on them. I had the impression that it was rarely the case, but I must be mistaken. I'll think about applications where this is the case.
Re: LoRA from scratch: implementation for LLM finetuning
#77I prefer the not from scratch, but from configuration approach by Axolotl. Aolotl supports fine-tuning mistral, llama-2, with lots of the latest techniques - sample packing, flash attention, xformers. I concentrate on collecting and curating the fine-tuning data, do "data-centric" fine-tuning - not learning LoRA from scratch.
this is also what our (Lightning AI) lit-gpt library does. https://github.com/Lightning-AI/lit-gpt
Thanks, hadn't seen this.
Re: LoRA from scratch: implementation for LLM finetuning
#78"From scratch" seems to be a matter of opinion. "Pure pytorch" maybe, except it uses HF transformers. So it's LoRA on top of common frameworks...
If anyone is interested in a more 'pure' or 'scratch' implementation, check out https://github.com/michaelnny/QLoRA-LLM. (author here) It also supports 4-bit quantized LoRA, using only PyTorch and bitsandbytes, without any other tools.
Re: LoRA from scratch: implementation for LLM finetuning
#79It's cheap and sleazy to steal a name from another project to ride it's fame.
Re: LoRA from scratch: implementation for LLM finetuning
#80Earlier quoted context omitted.
I feel like it's the difference between something that has been engineered and something that has been discovered. I feel like most of our industry up until now has been engineered. LLMs were discovered.
LLMs were very much engineered... the exact results they yield are hard to determine since they're large statistical models, but I don't think that categorizes the LLMs themselves as a 'discovery' (like say Penicilin)
There’s an argument that all maths are discovered instead of invented or engineered. LLM hardware certainly is hard engineering but the numbers you put in it aren’t, once you have them; if you stumbled upon them by chance or they were revealed to you in your sleep it’d work just as well. (‘ollama run mixtral’ is good enough for a dream to me!)