LoRA != LoRa. I keep on getting confused and hate that they chose to reuse an existing acronym
It's unfortunate that those two so far unrelated technologies have the same acronym.
LoRA from scratch: implementation for LLM finetuning
31–40 of 87 posts
Re: LoRA from scratch: implementation for LLM finetuning
#32It's still not too clear to me when we should fine tune versus RAG. In the past, I used to believe that finetuning is mostly for model behavioral change, but recently it seems that certain companies are also using fine-tuning for knowledge addition. What are the main use cases for fine tuning?
From what I gather, fine-tuning is unreasonably effective [0] because in-context learning really depends on how powerful the underlying model is and just how you do RAG (process queries, retrieve embeddings, rank outcomes, etc [1]). Per this paper I read, fine-tuning may add new domain knowledge (but as another commenter pointed out, knowledge is better represented from data of the pre-training stage) or boost specific knowledge; while RAG is limited to boosting only; nevertheless, both techniques turn out to be similarly capable with different trade-offs [2].
--
[0] Fast.ai: Can Models learn from one sample, https://www.fast.ai/posts/2023-09-04-learning-jumps/ / https://archive.is/eJMPR
[1] LlamaIndex: Advanced RAG, https://blog.llamaindex.ai/a-cheat-sheet-and-some-recipes-fo... / https://archive.is/qtBXX
[2] Microsoft: RAG vs Fine-tuning: Pipelines, Tradeoffs, and a Case Study, https://arxiv.org/html/2401.08406v2#S6 / https://archive.is/UQ8Sa#S6
Re: LoRA from scratch: implementation for LLM finetuning
#33It's still strange to me to work in a field of computer science where we say things like "we're not exactly sure how these numbers (hyper parameters) affect the result, so just try a bunch of different values and see which one works best."
I feel like it's the difference between something that has been engineered and something that has been discovered. I feel like most of our industry up until now has been engineered. LLMs were discovered.
I understand your distinction, I think, but I would say it is more engineering than ever. It's like the early days of the steam engine or firearms development. It's not a hard science, not formal analysis, it's engineering: tinkering, testing, experimenting, iterating.
Re: LoRA from scratch: implementation for LLM finetuning
#34It's still strange to me to work in a field of computer science where we say things like "we're not exactly sure how these numbers (hyper parameters) affect the result, so just try a bunch of different values and see which one works best."
I feel like it's the difference between something that has been engineered and something that has been discovered. I feel like most of our industry up until now has been engineered. LLMs were discovered.
and finally, this justifies the "science" in Computer Science.
Re: LoRA from scratch: implementation for LLM finetuning
#35It's still strange to me to work in a field of computer science where we say things like "we're not exactly sure how these numbers (hyper parameters) affect the result, so just try a bunch of different values and see which one works best."
it's a new paradigm
Re: LoRA from scratch: implementation for LLM finetuning
#36It's still not too clear to me when we should fine tune versus RAG. In the past, I used to believe that finetuning is mostly for model behavioral change, but recently it seems that certain companies are also using fine-tuning for knowledge addition. What are the main use cases for fine tuning?
Fine tuning is better than RAG when the additional data isn't concise, or requires context. This is because too much context (or "unfocused" context) can dilute prompt following behavior, and RAG doesn't help the model with higher order token associations so you have to get lucky and pull what you need from the augmentation material, at which point it's not much better than a fancy search engine. Of course this is mostly an issue when you're dealing with a specialized corpus with its own micro-dialect that isn't well represented in public data sets, such as with government/big corporation internal documents.
Re: LoRA from scratch: implementation for LLM finetuning
#37What's the performance penalty of LoRA?
Re: LoRA from scratch: implementation for LLM finetuning
#38LoRA != LoRa. I keep on getting confused and hate that they chose to reuse an existing acronym
Likewise. My day job is machine learning and I still, or maybe consequently, do a double-take every time I see the acronym with minimal context (like on the HN front page, where either usage would be normal).
Re: LoRA from scratch: implementation for LLM finetuning
#39LoRA != LoRa. I keep on getting confused and hate that they chose to reuse an existing acronym
Wait, what is the meaning other than "Low-Rank Adaptation"? It's hard to google the difference.
Re: LoRA from scratch: implementation for LLM finetuning
#40What's the performance penalty of LoRA?
During training, it's more efficient than full finetuning because you only update a fraction of the parameters via backprop.
During inference, it can ...
1) ... be theoretically a tad slower if you add the LoRA values dynamically during the forward pass (however, this is also an advantage if you want to keep a separate small weight set per customer, for example; you run only one large base model and can apply the different LoRA weights per customer on the fly)
2) ... have the exact same performance as the base model if you merge the LoRA weights back with the base model.