What are some os / hardware specific challenges you guys faced?
How we got fine-tuning Mistral-7B to not suck
11–20 of 50 posts
Re: How we got fine-tuning Mistral-7B to not suck
#12I always thought that fine tuning is more like getting a style rather than memorizing information word to word or at least the facts. What are the next steps to ensure that it doesn't start pulling info from the base knowledge and reference the docs instead? How long does it usually take to train? 10-15 minutes on what doc size?
Fine tuning is just more training -- so it's definitely possible to teach the model facts this way too. In practice we've found that it's a bit of a balancing act to teach the model the new knowledge without destroying existing knowledge, but it's just a matter of tuning the parameters carefully. We're also researching whether we can fine-tune a brand new expert in a MoE model like Mixtral, I've also seen work on fin…
Have you tried generating two sets of qapairs, one with bad answers, and using DPO?
Re: How we got fine-tuning Mistral-7B to not suck
#13For helix, I notice that GitHub is listed as a data source, but there's nothing in the docs about this. I'd really love to see what a model trained on my commonly used git repos (which generally are newer than The Stack etc), and in particular their commit history. Ideally these would make it easier for code completion to have the historical context as well as the current code to play with in determining what to write next.
I often wonder how you'd go about organizing training data for a full historic github repo in a way that makes sense for training (or RAG)? The vast majority of the data is previous changes to the repo. I think this would generally mean that it would outweigh the current information and cause problems (i.e. old method names before refactoring etc.)
Also, perhaps being able to expand that out to doing the same thing for a bunch of consumers of the library that I'm maintaining would be neat.
Sprinkle in the PR and Issue history, docs website, API docs, and discord history and I think you'd have a helluva model.
Re: How we got fine-tuning Mistral-7B to not suck
#14Unsloth’s colab notebooks for fine-tuning Mistral-7B are super easy to use and run fine in just about any colab instance: https://github.com/unslothai/unsloth It’s my default now for experimenting and basic training. If I want to get into the weeds, I use axolotl, but 9/10, it’s not really necessary.
How much improvement do you get by finetuning?
Re: How we got fine-tuning Mistral-7B to not suck
#15If you look at the source [1] you can see how they solved their what are the doctors going to do problem. It is literally included in one of the prompts now :-)
Users tend to ask broad, vague questions of the document in order to test that the system is working. We want those queries to work well. For example, a user would ask "what are the doctors going to do?" of a document that is about a junior doctors' strike. Take this into account when generating the questions - in particular, refer to noun phrases by less specific descriptions, so for example instead of "junior doctors", say "doctors" in your questions.
[1]: https://github.com/helixml/helix/blob/main/api/pkg/dataprep/...
Re: How we got fine-tuning Mistral-7B to not suck
#16Glad to see that more people outside the big ai labs are figuring out how to do fine tuning. Some open source LLM authors also seem to have figured it out.
I think many users get put off it because just pushing a button doesn’t work and the whole thing seems like a black box that you don’t know how to fix when it breaks.
It turns out that finetuning can be debugged, but the methods aren’t well documented (yet), eg by generating q/a, oversampling them, etc
When you get it to work it’s powerful - new abilities emerge beyond memorization.
Just like how llama2/claude2/gpt4 learned reasoning by memorizing sentences from Reddit posts :P
Also, I don’t get the comparison of rag vs finetuning in articles like this - why not do both. RAG is easy to setup - it’s push button. Just do it on all models (including finetuned models).
Re: How we got fine-tuning Mistral-7B to not suck
#17Interesting article but, IMHO, completely impractical. Teaching the model about specific content is totally what you should not do. What you should do is to teach the model how to effectively retrieve the information even if it is unsuccessful on the first try.
Re: How we got fine-tuning Mistral-7B to not suck
#18I always thought that fine tuning is more like getting a style rather than memorizing information word to word or at least the facts. What are the next steps to ensure that it doesn't start pulling info from the base knowledge and reference the docs instead? How long does it usually take to train? 10-15 minutes on what doc size?
Your sentiment is correct, but it's more of a spectrum. Fine tuning can learn facts (otherwise how would the foundation models learn facts?). But it needs those facts in the training dataset. If you have an infinite amount of facts, then you can memorise all of them. The challenge arises when it becomes hard to generate that training data. If you just have the raw text and pop that in the context (i.e. RAG), then the…
> If you have an infinite amount of facts, then you can memorise all of them
pigeon-hole?
Re: How we got fine-tuning Mistral-7B to not suck
#19Interesting article but, IMHO, completely impractical. Teaching the model about specific content is totally what you should not do. What you should do is to teach the model how to effectively retrieve the information even if it is unsuccessful on the first try.
I really don’t get this sentiment - why not do both?
Retrieval allows looking up facts - eg in a Google search
Finetuning allows reasoning using new knowledge.
Humans do both.
Re: How we got fine-tuning Mistral-7B to not suck
#20Interesting article but, IMHO, completely impractical. Teaching the model about specific content is totally what you should not do. What you should do is to teach the model how to effectively retrieve the information even if it is unsuccessful on the first try.
I really don’t get this sentiment - why not do both? Retrieval allows looking up facts - eg in a Google search Finetuning allows reasoning using new knowledge. Humans do both.
I think fine tuning makes sense when you need some domain specific knowledge to properly read, analyze, and interpret the information you're passing to it. But its not an information store itself.
The most valuable skill an LLM can have is good reasoning skills and a broad enough knowledge base to understand. From there you can pass it the important bits it needs.