Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Solving the out-of-context chunk problem for RAG

d-star.ai

Post reply on HN
Print view View original

71–80 of 93 posts

Re: Solving the out-of-context chunk problem for RAG

#71

Post by benreesman »

k__ wrote: post #41
bob1029 wrote: post #37

I've found the best approach is to start with traditional full text search. Get it to a point where manual human searches are useful - Especially for users who don't have a stake in the development of an AI solution. Then , look at building a RAG-style solution around the FTS. I never could get much beyond the basic search piece. I don't see how mixing in a black box AI model with probabilistic outcomes could add any…

I always wondered why a RAG index has to be a vector DB. If the model understands text/code and can generate text/code it should be able to talk to OpenSearch no problem.

Inner product similarity in an embedding space is often a very valuable feature in a ranker, and the effort/wow ratio at the prototype phase is good, but the idea that it’s the only pillar of an IR stack is SaaS marketing copy.

Vector DBs are cool, you want one handy (particularly for recommender tasks). I recommend FAISS as a solid baseline all these years later. If you’re on modern x86_64 then SVS is pretty shit hot.

A search engine that only uses a vector DB is a PoC.

For folks who want to go deeper on the topic, Lars basically invented the modern “news feed”, which looks a lot like a production RAG system would [1].

1. https://youtu.be/BuE3DIJGWOw

Re: Solving the out-of-context chunk problem for RAG

#72

Post by nostrebored »

derefr wrote: post #43
siquick wrote: post #11

I can’t imagine any serious RAG application is not doing this - adding a contextual title, summary, keywords, and questions to the metadata of each chunk is a pretty low effort/high return implementation.

> adding a contextual title, summary, keywords, and questions That's interesting; do you then transform the question-as-prompt before embedding it at runtime, so that it "asks for" that metadata to be in the response? Because otherwise, it would seem to me that you're just making it harder for the prompt vector and the document vectors to match. (I guess, if it's equally harder in all cases, then that might be fine.…

You can also train query awareness into the embedding model. This avoids LLMs rewriting questions poorly and lets you embed questions the way your customers actually ask them.

For an example with multimodal: https://www.marqo.ai/blog/generalized-contrastive-learning-f...

But the same approach works with text.

Re: Solving the out-of-context chunk problem for RAG

#73

Post by nostrebored »

zmccormick7 wrote: post #53
michalwarda wrote: post #32

Earlier quoted context omitted.

I guess it's because people are not using tools enough yet. In my tests giving LLM access to tools for retrieval works much better then trying to guess what the RAG would need to answer. ie. LLM decides if it has all of the necessary information to answer the question. If not, let it search for it. If it still fails than let it search more :D

Agreed. Retrieval performance is very dependent on the quality of the search queries. Letting the LLM generate the search queries is much more reliable than just embedding the user input. Also, no retrieval system is going to return everything needed on the first try, so using a multi-step agent approach to retrieving information is the only way I've found to get extremely high accuracy.

The queries you see and the resulting user interaction should be trained into the embedding model.

This is a foundational problem that requires your data. The way you search Etsy is different than the way you search Amazon. The queries these systems see are different and so are the desired results.

Trying to solve the problem with pretrained models is not currently realistic.

Re: Solving the out-of-context chunk problem for RAG

#74

Post by CGamesPlay »

An interesting paper that was recently published that talks about a different approach: Human-like Episodic Memory for Infinite Context LLMs https://arxiv.org/abs/2407.09450>

This wasn't focused on RAG, but there seems to be a lot of crossover to me. Using the LLM to make "episodes" is a similar problem to chunking, and letting the LLM decide the boundary might also yield good results.

Re: Solving the out-of-context chunk problem for RAG

#75

Post by CGamesPlay »

l72 wrote: post #25
Satam wrote: post #2

RAG feels hacky to me. We’re coming up with these pseudo-technical solutions to help but really they should be solved at the level of the model by researchers. Until this is solved natively, the attempts will be hacky duct-taped solutions.

I've described it this way to my colleagues: RAG is a bit like having a pretty smart person take an open book test on a subject they are not an expert in. If your book has a good chapter layout and index, you probably do an ok job trying to find relevant information, quickly read it, and try to come up with an answer. But your not going to be able to test for a deep understanding of the material. This person is going…

What you said about RAG makes sense, but my understanding is that fine-tuning is actually not very good at getting deeper understanding out of LLMs. It's more useful for teaching general instructions like output format rather than teaching deep concepts like a new domain of science.

Re: Solving the out-of-context chunk problem for RAG

#77

Post by kgeist »

simonw wrote: post #63
niam wrote: post #62

Earlier quoted context omitted.

What are the arguments for embedded vector DBs being suboptimal in RAG, out of curiosity?

The biggest one is that it's hard to get "zero matches" from an embeddings database. You get back all results ordered by distance from the user's query, but it will really scrape the bottom of the barrel if there aren't any great matches - which can lead to bugs like this one: https://simonwillison.net/2024/Jun/6/accidental-prompt-injec... The other problem is that embeddings search can miss things that a direct keyw…

>but because embeddings search orders by similarity score it will ALWAYS return results, really scraping the bottom of the barrel if it has to

Why not have a similarity threshold? Say, if the distance is below 0.7, do not accept the search result.

Re: Solving the out-of-context chunk problem for RAG

#78

Post by iAkashPaul »

One quick way to improve results greatly is to ask questions with 2/3 chunks & in the lookup for these chunks mention the IDs of the other chunks, qdrant allows for easy metadata addition. So just generate a synthetic question bank & then do vSearch against the same instead of hoping for the chunks to match up with user questions.

Re: Solving the out-of-context chunk problem for RAG

#79

Post by mediaman »

zby wrote: post #66
bob1029 wrote: post #37

I've found the best approach is to start with traditional full text search. Get it to a point where manual human searches are useful - Especially for users who don't have a stake in the development of an AI solution. Then , look at building a RAG-style solution around the FTS. I never could get much beyond the basic search piece. I don't see how mixing in a black box AI model with probabilistic outcomes could add any…

Traditional FTS returns the whole document - people take over from that point and locate the interesting content there. The problem with RAG is that it does not follow that procedure - it tries to find the interesting chunk in one step. Even though since ReAct we know that LLMs could follow the same procedure as humans. But we need an iterative RAG anyway: https://zzbbyy.substack.com/p/why-iterative-thinking-is-cruc.…

For my application we do a land-and-expand strategy, where we use a mix of BM25 and semantic search to find a chunk, but before showing it to the LLM we then expand to include everything on that page.

It works pretty well. It might benefit from including some material on the page prior and after, but it mostly solves the "isolated chunk" problem.

Re: Solving the out-of-context chunk problem for RAG

#80

Post by simonw »

kgeist wrote: post #77
simonw wrote: post #63

Earlier quoted context omitted.

The biggest one is that it's hard to get "zero matches" from an embeddings database. You get back all results ordered by distance from the user's query, but it will really scrape the bottom of the barrel if there aren't any great matches - which can lead to bugs like this one: https://simonwillison.net/2024/Jun/6/accidental-prompt-injec... The other problem is that embeddings search can miss things that a direct keyw…

>but because embeddings search orders by similarity score it will ALWAYS return results, really scraping the bottom of the barrel if it has to Why not have a similarity threshold? Say, if the distance is below 0.7, do not accept the search result.

It turns out picking that threshold is extremely difficult - I've tried! The value seems to differ for different searches, so picking eg 0.7 as a fixed value isn't actually as useful as you would expect.
Post reply on HN