Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How we got fine-tuning Mistral-7B to not suck

helixml.substack.com

Post reply on HN
Print view View original

41–50 of 50 posts

Re: How we got fine-tuning Mistral-7B to not suck

#41

Post by kkzz99 »

benpacker wrote: post #33
nl wrote: post #26

I've done fine tuning too, but the reasons they mention in "Why not just use RAG?" aren't very good. People way understimate what RAG can do, even if in general people don't talk about the right things. For example LlamaIndex spends a lot of time talking about various extractors which is the easy part. The hard thing is deciding what you are actually searching for given a chat context. RAG is a horrible hack (and the…

Is that lossy? Isn't the vector representation of the text (and the ANN index itself) lossy, and the source text itself the source of truth?

You use the vector embedding only for retrieval of the actual text, which then gets injected into the prompt.

Re: How we got fine-tuning Mistral-7B to not suck

#42

Post by binocarlos »

gdiamos wrote: post #16

Glad to see that more people outside the big ai labs are figuring out how to do fine tuning. Some open source LLM authors also seem to have figured it out. I think many users get put off it because just pushing a button doesn’t work and the whole thing seems like a black box that you don’t know how to fix when it breaks. It turns out that finetuning can be debugged, but the methods aren’t well documented (yet), eg by…

Thanks for the feedback - I agree that fine tuning a) has potential and b) is not easy :-)

> Also, I don’t get the comparison of rag vs finetuning in articles like this - why not do both

It's interesting you say this because we are very close to adding RAG support to Helix sessions and it will be "both at the same time" not an "either or" setup. You can choose to do either or but we are interested in seeing if doing both at the same time yields better results than either or - watch this space!

disclaimer: I work on Helix

Re: How we got fine-tuning Mistral-7B to not suck

#43

Post by binocarlos »

_pdp_ wrote: post #17

Interesting article but, IMHO, completely impractical. Teaching the model about specific content is totally what you should not do. What you should do is to teach the model how to effectively retrieve the information even if it is unsuccessful on the first try.

We are finding that fine tuning is very good at setting the style and tone of responses. A potential use case we are thinking about is what if your star sales person leaves the company? Could you fine tune an LLM on their conversations with customers and then do inference where it would write text in the style of your star sales person.

We are also adding function calling so the model would know to reach out to an external API to fetch some data before generating a response.

disclaimer: I work on Helix

Re: How we got fine-tuning Mistral-7B to not suck

#44

Post by binocarlos »

nicolezhu wrote: post #11

What are some os / hardware specific challenges you guys faced?

Great question! scheduling workloads onto GPUs in a way where VRAM is being utilised efficiently was quite the challenge.

What we found was the IO latency for loading model weights into VRAM will kill responsiveness if you don't "re-use" sessions (i.e. where the model weights remain loaded and you run multiple inference sessions over the same loaded weights).

Obviously projects like https://github.com/vllm-project/vllm exist but we needed to build out a scheduler that can run a fleet of GPUs for a matrix of text/image vs inference/finetune sessions.

disclaimer: I work on Helix

Re: How we got fine-tuning Mistral-7B to not suck

#45

Post by binocarlos »

bugglebeetle wrote: post #6

Unsloth’s colab notebooks for fine-tuning Mistral-7B are super easy to use and run fine in just about any colab instance: https://github.com/unslothai/unsloth It’s my default now for experimenting and basic training. If I want to get into the weeds, I use axolotl, but 9/10, it’s not really necessary.

Excellent link thank you - I will make sure we check this out because as the README says:

> Finetune Mistral, Llama 2-5x faster with 70% less memory!

Could be very useful for us!

Disclaimer: I work on Helix

Re: How we got fine-tuning Mistral-7B to not suck

#46

Post by ivanvas »

binocarlos wrote: post #37
AznHisoka wrote: post #30

Does fine tuning it on a set of docs in your “knowledge base” help for generalizing it so it can answer questions pertaining to new documents that come in (with a “similar” style/structure but with different content/fscts)?

Fine tuning on your documents will really help to answer questions in the style and tone of those documents, so in that way, yes it helps. It would be possible to include some parts of the new documents in the prompt so you can answer questions about new facts in the style and tone of your old documents, which we feel is useful. We are also experimenting with adding Retrieval Augmented Generation alongside fine tunin…

Is it En only or other languages would work?

Re: How we got fine-tuning Mistral-7B to not suck

#47

Post by nl »

formercoder wrote: post #28
nl wrote: post #26

I've done fine tuning too, but the reasons they mention in "Why not just use RAG?" aren't very good. People way understimate what RAG can do, even if in general people don't talk about the right things. For example LlamaIndex spends a lot of time talking about various extractors which is the easy part. The hard thing is deciding what you are actually searching for given a chat context. RAG is a horrible hack (and the…

Why is RAG a horrible hack? LLMs can draw from only 2 sources of data: their parametric knowledge and the prompt. The prompt seems like a pretty good place to put new information they need to reason with.

RAG is a hack for lots of reasons, but the reason I'm focused on at the moment is the pipeline.

Say you are trying to do RAG in a chat-type application. You do the following:

1) Summarize the context of chat into some text that is suitable for a search (lossy).

2) Turn this into a vector embedded in a particular vector space.

3) Use this vector to query a vector database, which returns reference to documents or document fragments (which themselves have been indexed as a lossy vector).

4) Take the text of these fragments and put them in the context of the LLM as input.

5) Modify the prompt to explain what these fragments are.

6) Then the prompt is sent to the LLM, which turns it into it's own vector representation.

An obvious improvement to this is that the VectorDB and the LLM should share an internal representation, and the VectorDB should understand this. The LLM should take this vector input as a second input alongside the text context and the LLM should combine them (in the same way you can put a text and image into a multi-modal model)

Re: How we got fine-tuning Mistral-7B to not suck

#48

Post by nl »

kkzz99 wrote: post #41
benpacker wrote: post #33

Earlier quoted context omitted.

Is that lossy? Isn't the vector representation of the text (and the ANN index itself) lossy, and the source text itself the source of truth?

You use the vector embedding only for retrieval of the actual text, which then gets injected into the prompt.

The conversion of a conversation context to vector for retrieval is lossy and often very manual.

Re: How we got fine-tuning Mistral-7B to not suck

#49

Post by danielhanchen »

binocarlos wrote: post #45
bugglebeetle wrote: post #6

Unsloth’s colab notebooks for fine-tuning Mistral-7B are super easy to use and run fine in just about any colab instance: https://github.com/unslothai/unsloth It’s my default now for experimenting and basic training. If I want to get into the weeds, I use axolotl, but 9/10, it’s not really necessary.

Excellent link thank you - I will make sure we check this out because as the README says: > Finetune Mistral, Llama 2-5x faster with 70% less memory! Could be very useful for us! Disclaimer: I work on Helix

Great article on Helix - love the QA generation part :) If you have any questions on Unsloth, more than happy to answer them :) If you need help setting stuff up, I'm also here to help! :) ( I'm the engineer behind Unsloth :) )

Re: How we got fine-tuning Mistral-7B to not suck

#50

Post by danielhanchen »

bugglebeetle wrote: post #6

Unsloth’s colab notebooks for fine-tuning Mistral-7B are super easy to use and run fine in just about any colab instance: https://github.com/unslothai/unsloth It’s my default now for experimenting and basic training. If I want to get into the weeds, I use axolotl, but 9/10, it’s not really necessary.

Super thanks for sharing Unsloth! :) Highly appreciative and high praise as well :))
Post reply on HN