Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How we got fine-tuning Mistral-7B to not suck

helixml.substack.com

Post reply on HN
Print view View original

21–30 of 50 posts

Re: How we got fine-tuning Mistral-7B to not suck

#21

Post by mendeza »

aCoreyJ wrote: post #7
lewq wrote: post #4

Earlier quoted context omitted.

Fine tuning is just more training -- so it's definitely possible to teach the model facts this way too. In practice we've found that it's a bit of a balancing act to teach the model the new knowledge without destroying existing knowledge, but it's just a matter of tuning the parameters carefully. We're also researching whether we can fine-tune a brand new expert in a MoE model like Mixtral, I've also seen work on fin…

What is the advantage over using Retrieval Augmented Generation ?

RAG adds context to the users question to reduce hallucination. https://docs.llamaindex.ai/en/stable/getting_started/concept...

Re: How we got fine-tuning Mistral-7B to not suck

#23

Post by gdiamos »

swalsh wrote: post #20
gdiamos wrote: post #19

Earlier quoted context omitted.

I really don’t get this sentiment - why not do both? Retrieval allows looking up facts - eg in a Google search Finetuning allows reasoning using new knowledge. Humans do both.

I think fine tuning makes sense when you need some domain specific knowledge to properly read, analyze, and interpret the information you're passing to it. But its not an information store itself. The most valuable skill an LLM can have is good reasoning skills and a broad enough knowledge base to understand. From there you can pass it the important bits it needs.

I think we are saying the same thing.

The key ingredients are:

reasoning(skills) + knowledge + important bits/facts

The best systems have all of these

Re: How we got fine-tuning Mistral-7B to not suck

#24

Post by josh-sematic »

cuuupid wrote: post #10

Not in love with axolotl but appreciate the advantages. This is an interesting approach, but you can also finetune easily on providers who wrap axolotl like Replicate [1], Modal [2], or if you want to run the infra, LLM Engine [3]. My only gripe with Helix would be that it's smaller than the above and my org would be peeved about data security. The ability to self host is cool, but too much can go wrong too quickly w…

I’ll add https://airtrain.ai to that list of push-button fine tunes using axolotl. Disclaimer: I’m the engineer who built it.

Re: How we got fine-tuning Mistral-7B to not suck

#25

Post by gdiamos »

gpderetta wrote: post #18
drphilwinder wrote: post #3

Earlier quoted context omitted.

Your sentiment is correct, but it's more of a spectrum. Fine tuning can learn facts (otherwise how would the foundation models learn facts?). But it needs those facts in the training dataset. If you have an infinite amount of facts, then you can memorise all of them. The challenge arises when it becomes hard to generate that training data. If you just have the raw text and pop that in the context (i.e. RAG), then the…

> If you have an infinite amount of facts, then you can memorise all of them pigeon-hole?

Not literally infinite, but Llama2 scale models can handle about 10 trillion tokens.

Re: How we got fine-tuning Mistral-7B to not suck

#26

Post by nl »

I've done fine tuning too, but the reasons they mention in "Why not just use RAG?" aren't very good.

People way understimate what RAG can do, even if in general people don't talk about the right things. For example LlamaIndex spends a lot of time talking about various extractors which is the easy part. The hard thing is deciding what you are actually searching for given a chat context.

RAG is a horrible hack (and the more you understand about the more it seems so!) but it does work.

I (and I'm sure everyone else) is experimenting with surgery on an LLM so it takes a vector representation of the docs directly alongside a text input so you don't have to do the lossy doc vector -> text -> LLM context -> vector thing. Not sure why no one has shipped this yet though!

Re: How we got fine-tuning Mistral-7B to not suck

#27

Post by bugglebeetle »

3abiton wrote: post #14
bugglebeetle wrote: post #6

Unsloth’s colab notebooks for fine-tuning Mistral-7B are super easy to use and run fine in just about any colab instance: https://github.com/unslothai/unsloth It’s my default now for experimenting and basic training. If I want to get into the weeds, I use axolotl, but 9/10, it’s not really necessary.

How much improvement do you get by finetuning?

Quite a bit, as I’m generally using it for highly-specific text extraction and conversion tasks.

Re: How we got fine-tuning Mistral-7B to not suck

#28

Post by formercoder »

nl wrote: post #26

I've done fine tuning too, but the reasons they mention in "Why not just use RAG?" aren't very good. People way understimate what RAG can do, even if in general people don't talk about the right things. For example LlamaIndex spends a lot of time talking about various extractors which is the easy part. The hard thing is deciding what you are actually searching for given a chat context. RAG is a horrible hack (and the…

Why is RAG a horrible hack? LLMs can draw from only 2 sources of data: their parametric knowledge and the prompt. The prompt seems like a pretty good place to put new information they need to reason with.

Re: How we got fine-tuning Mistral-7B to not suck

#29

Post by mrfox321 »

formercoder wrote: post #28
nl wrote: post #26

I've done fine tuning too, but the reasons they mention in "Why not just use RAG?" aren't very good. People way understimate what RAG can do, even if in general people don't talk about the right things. For example LlamaIndex spends a lot of time talking about various extractors which is the easy part. The hard thing is deciding what you are actually searching for given a chat context. RAG is a horrible hack (and the…

Why is RAG a horrible hack? LLMs can draw from only 2 sources of data: their parametric knowledge and the prompt. The prompt seems like a pretty good place to put new information they need to reason with.

I guess op may be envisioning an end-to-end solution that can train a model in the context of an external document store.

I.e. One day we want to be able to backprop through the database.

Search systems face equivalent problems. The hierarchy of ML retrieval systems are separately optimized (trained). Maybe this helps regularize things, but, given enough compute / complexity, it is theoretically possible to differentiate through more of the stack.

Post reply on HN