Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Solving the out-of-context chunk problem for RAG

d-star.ai

Post reply on HN
Print view View original

81–90 of 93 posts

Re: Solving the out-of-context chunk problem for RAG

#81

Post by bashfulpup »

zby wrote: post #67
Satam wrote: post #2

RAG feels hacky to me. We’re coming up with these pseudo-technical solutions to help but really they should be solved at the level of the model by researchers. Until this is solved natively, the attempts will be hacky duct-taped solutions.

I guess you can imagine an LLM that contains all information there is - but it would have to be at least as big as all information there is or it would have to hallucinate. And also you Not to mention that it seems that you would also require it to learn everything immediately. I don't see any realistic way to reach that goal. To reach their potential LLMs need to know how to use external sources. Update: After some…

A CL agent is next generation AI.

When CL is properly implemented in an LLM agent format, most of these systems vanish.

Re: Solving the out-of-context chunk problem for RAG

#82

Post by bashfulpup »

CGamesPlay wrote: post #75
l72 wrote: post #25

Earlier quoted context omitted.

I've described it this way to my colleagues: RAG is a bit like having a pretty smart person take an open book test on a subject they are not an expert in. If your book has a good chapter layout and index, you probably do an ok job trying to find relevant information, quickly read it, and try to come up with an answer. But your not going to be able to test for a deep understanding of the material. This person is going…

What you said about RAG makes sense, but my understanding is that fine-tuning is actually not very good at getting deeper understanding out of LLMs. It's more useful for teaching general instructions like output format rather than teaching deep concepts like a new domain of science.

This is true if you don't know what you're doing, so it is good advice for the vast majority.

Fine tuning is just training. You can completely change the model if you want make learn anything you want.

But there are MANY challenges in doing so.

Re: Solving the out-of-context chunk problem for RAG

#83

Post by CGamesPlay »

bashfulpup wrote: post #82
CGamesPlay wrote: post #75

Earlier quoted context omitted.

What you said about RAG makes sense, but my understanding is that fine-tuning is actually not very good at getting deeper understanding out of LLMs. It's more useful for teaching general instructions like output format rather than teaching deep concepts like a new domain of science.

This is true if you don't know what you're doing, so it is good advice for the vast majority. Fine tuning is just training. You can completely change the model if you want make learn anything you want. But there are MANY challenges in doing so.

This isn't true either, because if you don't have access to the original data set, the model will overfit on your fine tuning data set and (in the extreme cases) lose its ability to even do basic reasoning.

Re: Solving the out-of-context chunk problem for RAG

#84

Post by bashfulpup »

CGamesPlay wrote: post #83
bashfulpup wrote: post #82

Earlier quoted context omitted.

This is true if you don't know what you're doing, so it is good advice for the vast majority. Fine tuning is just training. You can completely change the model if you want make learn anything you want. But there are MANY challenges in doing so.

This isn't true either, because if you don't have access to the original data set, the model will overfit on your fine tuning data set and (in the extreme cases) lose its ability to even do basic reasoning.

Again, that's why I said it is challenging.

I regularly do fine tuning on a model with fine results and little damage to the base functionality.

It is possible, but it's too complex for the majority of users. It requires a lot of work per dataset you want trained on.

Re: Solving the out-of-context chunk problem for RAG

#85

Post by ramoz »

ianbutler wrote: post #59
simonw wrote: post #45

Earlier quoted context omitted.

It doesn't have to be a vector DB - and in fact I'm seeing increasing skepticism that embedding vector DBs are the best way to implement RAG. A full-text search index using BM25 or similar may actually work a lot better for many RAG applications. I wrote up some notes on building FTS-based RAG here: https://simonwillison.net/2024/Jun/21/search-based-rag/

In 2019 I was using vector search to narrow the search space within 100s of millions of documents and then do full text search on the top 10k or so docs. That seems like a better stacking of the technologies even now

Interesting. Why did you need to “narrow” the search space using vector space? Did you build custom embeddings and feel confident about retrieval segments?

I did similar in 2019 but typically in reverse, FTS, and a dual tower model to rerank. Vector search was an additional capability but never augmented the FTS.

Re: Solving the out-of-context chunk problem for RAG

#86

Post by DebtDeflation »

CGamesPlay wrote: post #83
bashfulpup wrote: post #82

Earlier quoted context omitted.

This is true if you don't know what you're doing, so it is good advice for the vast majority. Fine tuning is just training. You can completely change the model if you want make learn anything you want. But there are MANY challenges in doing so.

This isn't true either, because if you don't have access to the original data set, the model will overfit on your fine tuning data set and (in the extreme cases) lose its ability to even do basic reasoning.

Yes. It's called "catastrophic forgetting". These models were trained on trillions of tokens and then underwent a significant RLHF process. Fine tuning them on your tiny data set (relative to the original training data) almost always results in the model performing worse at everything else. There's also the issue of updating changed information. This is easy with RAG - replace the document in the repository with a new version and it just works. Not so easy with fine tuning since you can't identify and update just the weights that were changed (there's research in this area but it's early days).

Re: Solving the out-of-context chunk problem for RAG

#88

Post by zmccormick7 »

simonw wrote: post #80
kgeist wrote: post #77

Earlier quoted context omitted.

>but because embeddings search orders by similarity score it will ALWAYS return results, really scraping the bottom of the barrel if it has to Why not have a similarity threshold? Say, if the distance is below 0.7, do not accept the search result.

It turns out picking that threshold is extremely difficult - I've tried! The value seems to differ for different searches, so picking eg 0.7 as a fixed value isn't actually as useful as you would expect.

Agreed that thresholds don't work when applied to the cosine similarity of embeddings. But I have found that the similarity score returned by high-quality rerankers, especially Cohere, are consistent and meaningful enough that using a threshold works well there.

Re: Solving the out-of-context chunk problem for RAG

#89

Post by jairuhme »

kgeist wrote: post #77
simonw wrote: post #63

Earlier quoted context omitted.

The biggest one is that it's hard to get "zero matches" from an embeddings database. You get back all results ordered by distance from the user's query, but it will really scrape the bottom of the barrel if there aren't any great matches - which can lead to bugs like this one: https://simonwillison.net/2024/Jun/6/accidental-prompt-injec... The other problem is that embeddings search can miss things that a direct keyw…

>but because embeddings search orders by similarity score it will ALWAYS return results, really scraping the bottom of the barrel if it has to Why not have a similarity threshold? Say, if the distance is below 0.7, do not accept the search result.

I'll add to what the other commenter noted, but sometimes the difference between results get very granular (i.e. .65789 vs .65788) so deciding on where that threshold should be is little trickier.

Re: Solving the out-of-context chunk problem for RAG

#90

Post by hackernoteng »

I experience worse IR performance adding title/headers to chunks. It really depends on the nature of the documents. The only successful RAG systems I see are ones specifically tuned to a single domain and document type. If your document collection is diverse in domains or formats, good luck.
Post reply on HN