Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

You probably don’t need to fine-tune an LLM

tidepool.so

Post reply on HN
Print view View original

61–70 of 75 posts

Re: You probably don’t need to fine-tune an LLM

#61

Post by fnordpiglet »

gsuuon wrote: post #57

Besides the three mentioned in the article (stringent accuracy requirements, fast edge inference, involved style transfer task) are there other good reasons to fine-tune?

I fine tune smaller LLMs with fanfic to generate stories in a genre. For instance, most models have no meaningful Minecraft content. By feeding lots of fanfic into a Lora and supplementing with RAG, I get pretty good results. RAG alone is garbage as there’s too much broad context on say Herobrine spanning many many stories, but the base models know almost nothing. Few shot doesn’t help because there’s not enough semantic support in the model weights. Etc.

Re: You probably don’t need to fine-tune an LLM

#62

Post by zargon »

gk1 wrote: post #43
gdiamos wrote: post #42

How can you edit or remove guardrails without fine tuning?

How would guardrails be applied through fine-tuning? I haven't heard of that, so genuinely curious. What I'm familiar with is guardrails applied on top of the LLM. For instance through prompting, managing the available data inside the vector database that could be used for context (if using RAG), or through something like NeMo[1]. https://www.pinecone.io/learn/nemo-guardrails-intro/

The technique is called reinforcement learning from human feedback (RLHF). It's how, for example, Facebook can train Llama models to meet a certain safety score before releasing them into the wild.

Re: You probably don’t need to fine-tune an LLM

#63

Post by gsuuon »

fnordpiglet wrote: post #61
gsuuon wrote: post #57

Besides the three mentioned in the article (stringent accuracy requirements, fast edge inference, involved style transfer task) are there other good reasons to fine-tune?

I fine tune smaller LLMs with fanfic to generate stories in a genre. For instance, most models have no meaningful Minecraft content. By feeding lots of fanfic into a Lora and supplementing with RAG, I get pretty good results. RAG alone is garbage as there’s too much broad context on say Herobrine spanning many many stories, but the base models know almost nothing. Few shot doesn’t help because there’s not enough sema…

That's really interesting! I've wanted to do something similar but assumed that the token count required to ingrain a meaningful amount of information would make it very difficult. How many tokens do you typically need to feed into these base models to get consistent content out? Does generated text ever end up leaving the specialized domain? (like League of Legends characters mixed into a Minecraft story)

Re: You probably don’t need to fine-tune an LLM

#64

Post by treprinum »

binarymax wrote: post #60
treprinum wrote: post #52

Earlier quoted context omitted.

There are other methods than RAG over vector database; you can use basic TF-IDF or even full-text search to find candidate paragraphs and put them into the context.

Of course. “Retrieval” in RAG doesn’t require a special kind of retriever. As long as the relevance is tuned for the top documents to seed the prompt context, it doesn’t matter what kind of search backend you use.

Agreed though R in RAG typically means specifically vector search. Not sure how this was called before vector DBs were popularized by LLaMAIndex, probably just retrieval systems with LLMs.

Re: You probably don’t need to fine-tune an LLM

#65

Post by natsucks »

binarymax wrote: post #7
norwalkbear wrote: post #2

RAG sucks. Microsoft is the force behind it because they don't allow training on their chatgpt models. Fine-tuning even Lora on the open source models is nearly always better than these other approaches

RAG is the ONLY way to make sure you models are keeping true to facts and source material. Fine-tuning a model before using RAG helps with shaping the style of the summary, and gravitating towards more important facts presented.

not to mention staying true to their benchmarks...

Re: You probably don’t need to fine-tune an LLM

#67

Post by ilaksh »

herdcall wrote: post #15

We're using LLMs (OpenAI) to generate SQL queries to search customer data, and the current approach using chat API frequently generates queries using the wrong record/column names. I'm exploring use of fine tuning to improve accuracy on a customer/customer basis to train on their set of data, isn't that a good use case?

1. Use GPT-4

2. Clearly specify the column names as part of the prompt and that there are no other columns.

3. Occasionally you may get an error that you have to feed back to GPT-4.

Re: You probably don’t need to fine-tune an LLM

#68

Post by ilaksh »

jdmccarty wrote: post #58

Is there a method for this to be "Augmented" and not "Replacement" eg in the example from the blog post, "retriever=vectorstore.as_retriever()" which I believe would return something like "I don't know" if the content is not in the vectorstore. In humans, a person might say something like, "I'm not an expert, but X" and I think being able to default back to the underlying LLM would be useful.

I assume those frameworks have a way to specify the cut off for similarity and a default. And they are flexible in terms of the granularity that you use them, so you should be able to specify the prompt directly in theory.

But using the OpenAI API directly is not that complicated. And neither is using something like pg_vector or just a plain cosine similarity calculation from Stack Overflow.

Re: You probably don’t need to fine-tune an LLM

#69

Post by ilaksh »

I wonder if the approach may change a bit when OpenAI releases fine tuning for the chat models. I think it depends on how well it works. If they find some way to significantly decrease the amount of training data needed or someone creates a tool to easily generate lots of training examples (using OpenAI), the advice might change again.

What also matters is the size of the context window and how effectively the models can follow large amounts of instructions. So new models might change the advice again.

Re: You probably don’t need to fine-tune an LLM

#70

Post by clharman »

rafaelero wrote: post #11

I tried finetuning the 13b LLaMa model to insert the knowledge from my own documents, but my experiments weren't successful. My conclusion is that you need billions of tokens to make a LLM reason based on your own dataset. And even if they did acquire these reasoning skills, they probably wouldn't beat GPT-4. And we are not even getting into the costs of self-hosting these LLM's. So why bother? Just use an API from t…

@rafaelero I'm working on a blogpost (https://colinharman.substack.com/) to demonstrate this fact since I get a lot of tiresome questions like "why don't you just train instead of retrieving"

Do you have any scripts you could share for the training/eval process? Would love to credit you in the post

Post reply on HN