RAG sucks. Microsoft is the force behind it because they don't allow training on their chatgpt models. Fine-tuning even Lora on the open source models is nearly always better than these other approaches
RAG is the ONLY way to make sure you models are keeping true to facts and source material. Fine-tuning a model before using RAG helps with shaping the style of the summary, and gravitating towards more important facts presented.
It still doesn't give guarantees, it just makes hallucinations much less likely.
One of the main problems with LLMs today is they are gigantic, and this is because they are shipping the entire compressed memry of the training data with them.
Future LLMs are likely to have much smaller size and have outside long term memory/training knowledge as well as work memory (a'la RAG approach).
RAG sucks. Microsoft is the force behind it because they don't allow training on their chatgpt models. Fine-tuning even Lora on the open source models is nearly always better than these other approaches
How does RAG not meet your expectations? When I use chatgpt I provide source material for my questions and typically get much higher quality responses.
One of the main problems with LLMs today is they are gigantic, and this is because they are shipping the entire compressed memry of the training data with them. Future LLMs are likely to have much smaller size and have outside long term memory/training knowledge as well as work memory (a'la RAG approach).
Isn't this just ignoring all the lessons we learned from pretraining large models? Seems like a different flavor of the bitter lesson.
I agree with summary. When I first wanted to tackle a hard problem I thought to reach for fine-tuning with lots of input and output pairs, but it wasn't needed. Past few shot and RAG, you can overcome context window limits if you find ways to break a single request into many, each with specific context and then roll them up somehow. This can help get past context window limits. Claude 2 has a large context window, bu…
How much do you make use of Claude's large context window, and has it helped? Or did you basically revert back to small context size + multiple steps?
Not OP but, I finally got access to GPT-4 32k and I thought that for a thing as simple as a summary it would be better than the smaller context-window model. But then I realized that now I have (practically) infinite context in input but what the LLM can output is always between 1-1.5k tokens. That is, I think, because of the samples in its training, not because it can't output a more lengthy output.
In short, I think that multiple steps are better, for example in summarization.
I tried finetuning the 13b LLaMa model to insert the knowledge from my own documents, but my experiments weren't successful. My conclusion is that you need billions of tokens to make a LLM reason based on your own dataset. And even if they did acquire these reasoning skills, they probably wouldn't beat GPT-4. And we are not even getting into the costs of self-hosting these LLM's. So why bother? Just use an API from t…
Did you try a Retrieval Augmented Generation (RAG) approach? If so, what made you give it up in favor of fine-tuning?
This is the approach I am using right now. My intuition for trying to finetune was that for complex questions it would be better if the model could naturally deal with those intrincacies instead of reading documents with concepts that are connected but not very explicitly so. There is also the problem with context window limit; sometimes I have to truncate the relevant documents, limiting the capacity of the model to offer a good answer.
But I am very impressed with GPT-4's abilities in finding the correct answer just by reading those documents, so I think the only problem still enduring is the context window size.
Fine-tuning is such a dangerous phrased bc it sounds perfect.
“We’ll just fine-tune based on our (we think valuable + special) data”
Fine-tuning doesn’t enhance the model w/ new “knowledge” but a new narrowly-defined task
One other “cost” to consider is fine-tuning a 3rd-party model means if that foundation model changes or goes away that effort/cost needs to be repeated
I love how clearly this article is written. The author uses a table with the columns being "Initial Motivation for Fine-tuning" and "Why a Base LLM is Sufficient" - exactly what you need to learn why "You probably don’t need to fine-tune LLMs", which precisely the title. Using text to convey something that can be expressed as a table or a chart is just as bad as trying to do math without math notation. Stellar work!