Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

You probably don’t need to fine-tune an LLM

tidepool.so

Post reply on HN
Print view View original

41–50 of 75 posts

Re: You probably don’t need to fine-tune an LLM

#41

Post by CharlesW »

rafaelero wrote: post #38
CharlesW wrote: post #26

Earlier quoted context omitted.

Did you try a Retrieval Augmented Generation (RAG) approach? If so, what made you give it up in favor of fine-tuning?

This is the approach I am using right now. My intuition for trying to finetune was that for complex questions it would be better if the model could naturally deal with those intrincacies instead of reading documents with concepts that are connected but not very explicitly so. There is also the problem with context window limit; sometimes I have to truncate the relevant documents, limiting the capacity of the model to…

> My intuition for trying to finetune was that for complex questions it would be better if the model could naturally deal with those intrincacies instead of reading documents with concepts that are connected but not very explicitly so.

That was my impression as well, which is why your comment was so interesting to me. Have you found tools/projects for the DAG approach that you'd recommend?

Re: You probably don’t need to fine-tune an LLM

#43

Post by gk1 »

gdiamos wrote: post #42

How can you edit or remove guardrails without fine tuning?

How would guardrails be applied through fine-tuning?

I haven't heard of that, so genuinely curious. What I'm familiar with is guardrails applied on top of the LLM. For instance through prompting, managing the available data inside the vector database that could be used for context (if using RAG), or through something like NeMo[1].

https://www.pinecone.io/learn/nemo-guardrails-intro/

Re: You probably don’t need to fine-tune an LLM

#45

Post by adamgordonbell »

gk1 wrote: post #25
adamgordonbell wrote: post #4

I agree with summary. When I first wanted to tackle a hard problem I thought to reach for fine-tuning with lots of input and output pairs, but it wasn't needed. Past few shot and RAG, you can overcome context window limits if you find ways to break a single request into many, each with specific context and then roll them up somehow. This can help get past context window limits. Claude 2 has a large context window, bu…

How much do you make use of Claude's large context window, and has it helped? Or did you basically revert back to small context size + multiple steps?

I am just using GPT4-8k and breaking tasks up. When I tested with Claude 1 ( 2 wasn't out yet) you could feed it lots of context, but it didn't always seem like it paid attention to it all. It would go off the rails.

For example, if I put instructions first, then a lot of context, it would forget the task, so instead put a small explanation, then context, then question, but still it would seem to sometimes loose the plot.

For feedback on prose though, I found it Claude very strong. It's nice to be able to give it half of a book, in text format and have it guide you to important parts and summarize sections. Some things, like what are the themes of this work, what's surprising finding, don't work well with vector DBs.

Re: You probably don’t need to fine-tune an LLM

#46

Post by nico »

ZunarJ5 wrote: post #29
rafaelero wrote: post #11

I tried finetuning the 13b LLaMa model to insert the knowledge from my own documents, but my experiments weren't successful. My conclusion is that you need billions of tokens to make a LLM reason based on your own dataset. And even if they did acquire these reasoning skills, they probably wouldn't beat GPT-4. And we are not even getting into the costs of self-hosting these LLM's. So why bother? Just use an API from t…

https://github.com/khoj-ai/khoj This is the easiest I found, on here too.

The readme says it’s offline and doesn’t mention any LLMs

Does it not use any LLMs, or does it have its own, or how how does it work?

Thank you

Re: You probably don’t need to fine-tune an LLM

#47

Post by adamgordonbell »

freediver wrote: post #32

One of the main problems with LLMs today is they are gigantic, and this is because they are shipping the entire compressed memry of the training data with them. Future LLMs are likely to have much smaller size and have outside long term memory/training knowledge as well as work memory (a'la RAG approach).

But in compressing all that data, they seem to build up internal representations that let them generalize on data they haven't seem. That reasoning ability, if not the actual data, is what gives them their power.

Re: You probably don’t need to fine-tune an LLM

#48

Post by gk1 »

adamgordonbell wrote: post #45
gk1 wrote: post #25

Earlier quoted context omitted.

How much do you make use of Claude's large context window, and has it helped? Or did you basically revert back to small context size + multiple steps?

I am just using GPT4-8k and breaking tasks up. When I tested with Claude 1 ( 2 wasn't out yet) you could feed it lots of context, but it didn't always seem like it paid attention to it all. It would go off the rails. For example, if I put instructions first, then a lot of context, it would forget the task, so instead put a small explanation, then context, then question, but still it would seem to sometimes loose the…

Interesting, thanks for sharing. We also found[1] that context stuffing leads (generally) to worse quality results, as did a recent study.[2] But that's for question-answering use cases and not summarization.

[1] https://www.pinecone.io/blog/why-use-retrieval-instead-of-la...

[2] https://arxiv.org/pdf/2307.03172.pdf

Re: You probably don’t need to fine-tune an LLM

#49

Post by ZunarJ5 »

nico wrote: post #46
ZunarJ5 wrote: post #29

Earlier quoted context omitted.

https://github.com/khoj-ai/khoj This is the easiest I found, on here too.

The readme says it’s offline and doesn’t mention any LLMs Does it not use any LLMs, or does it have its own, or how how does it work? Thank you

You can set it up so that it downloads Llama for offline use in the UI. You do not have to do it manually.

Re: You probably don’t need to fine-tune an LLM

#50

Post by zackproser »

Great post!

We've got some additional resources for folks looking to better understand Retrieval Augmented Generation (RAG) and even see it in action - in this example we demonstrate a potentially very dangerous hallucination (that has to do with driving) and how to fix it using RAG: https://www.pinecone.io/learn/retrieval-augmented-generation...

If you're curious to actually try out the difference between an LLM without domain-specific context and an LLM that is using RAG, you can try our live demo here: https://pinecone-vercel-starter.vercel.app/

And if you'd like to fork and make your own tweaks to the above demo ^ chatbot, in order to, for example, swap in your own company logo and extend it for your purposes, you can find our Vercel template here: https://github.com/pinecone-io/pinecone-vercel-starter

In our opinion, RAG is indeed an effective technique partly because you don't need to be a machine learning expert in order to implement it in your Generative AI applications.

Post reply on HN