Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Solving the out-of-context chunk problem for RAG

d-star.ai

Post reply on HN
Print view View original

61–70 of 93 posts

Re: Solving the out-of-context chunk problem for RAG

#61

Post by ec109685 »

CharlieDigital wrote: post #4

The easiest solution to this is to stuff the heading into the chunk. The heading is hierarchical navigation within the sections of the document. I found Azure Document Intelligence specifically with the Layout Model to be fantastic for this because it can identify headers. All the better if you write a parser for the output JSON to track depth and stuff multiple headers from the path into the chunk.

Would it be better to go all the way and completely rewrite the source material in a way more suitable for retrieval? To some extent these headers are a step in that direction, but you’re still at the mercy of the chunk of text being suitable to answer the question.

Instead, completely transforming the text into a dense set of denormalized “notes” that cover every concept present in the text seems like it would be easier to mine for answers to user questions.

Essentially, it would be like taking comprehensive notes from a book and handing them to a friend who didn’t take the class for a test. What would they need to be effective?

Longer term, the sequence would likely be “get question”, hand it to research assistant who has full access to source material and can run a variety of AI / retrieval strategies to customize the notes, and then hand those notes back for answers. By spending more time on the note gathering step, it will be more likely the LLM will be able to answer the question.

Re: Solving the out-of-context chunk problem for RAG

#62

Post by niam »

simonw wrote: post #45
k__ wrote: post #41

Earlier quoted context omitted.

I always wondered why a RAG index has to be a vector DB. If the model understands text/code and can generate text/code it should be able to talk to OpenSearch no problem.

It doesn't have to be a vector DB - and in fact I'm seeing increasing skepticism that embedding vector DBs are the best way to implement RAG. A full-text search index using BM25 or similar may actually work a lot better for many RAG applications. I wrote up some notes on building FTS-based RAG here: https://simonwillison.net/2024/Jun/21/search-based-rag/

What are the arguments for embedded vector DBs being suboptimal in RAG, out of curiosity?

Re: Solving the out-of-context chunk problem for RAG

#63

Post by simonw »

niam wrote: post #62
simonw wrote: post #45

Earlier quoted context omitted.

It doesn't have to be a vector DB - and in fact I'm seeing increasing skepticism that embedding vector DBs are the best way to implement RAG. A full-text search index using BM25 or similar may actually work a lot better for many RAG applications. I wrote up some notes on building FTS-based RAG here: https://simonwillison.net/2024/Jun/21/search-based-rag/

What are the arguments for embedded vector DBs being suboptimal in RAG, out of curiosity?

The biggest one is that it's hard to get "zero matches" from an embeddings database. You get back all results ordered by distance from the user's query, but it will really scrape the bottom of the barrel if there aren't any great matches - which can lead to bugs like this one: https://simonwillison.net/2024/Jun/6/accidental-prompt-injec...

The other problem is that embeddings search can miss things that a direct keyword match would have caught. If you have key terms that are specific to your corpus - product names for example - there's a risk that a vector match might not score those as highly as BM25 would have so you may miss the most relevant documents.

Finally, embeddings are much more black box and hard to debug and reason about. We have decades of experience tweaking and debugging and improving BM25-style FTS search - the whole field of "Information Retrieval". Throwing that all away in favour of weird new embedding vectors is suboptimal.

Re: Solving the out-of-context chunk problem for RAG

#64

Post by drittich »

gillesjacobs wrote: post #58
drittich wrote: post #55

Earlier quoted context omitted.

How do you weight results between vector search and bm25? Do you fall back to bm25 when vector similarity is below a threshold, or maybe you tweak the weights by hand for each data set?

The algorithm I use to get a final ranking from multiple rankings is called "reciprocal ranked fusion". I use the implementation described here: https://docs.llamaindex.ai/en/stable/examples/low_level/fusi... Which is the implementation from the original paper.

Thanks, much appreciated!

Re: Solving the out-of-context chunk problem for RAG

#65

Post by lowdest »

simonw wrote: post #23
viraptor wrote: post #21

Earlier quoted context omitted.

You don't need to update the whole model for everyone. Fine tuning exists and is even available as a service in openai. The updates are only visible in the specific models you see.

Maintaining a fine-tuned model for every one of your users - even with techniques like LoRA - sounds complicated and expensive to me!

It is, but it's also not that bad. A copy of the weights is X GB of cloud storage, which can be stored as a diff if it helps, and added compute time for loading a custom model and unloading for the next customer. It's not free, but it's an approachable cost for a premium service.

Re: Solving the out-of-context chunk problem for RAG

#66

Post by zby »

bob1029 wrote: post #37

I've found the best approach is to start with traditional full text search. Get it to a point where manual human searches are useful - Especially for users who don't have a stake in the development of an AI solution. Then , look at building a RAG-style solution around the FTS. I never could get much beyond the basic search piece. I don't see how mixing in a black box AI model with probabilistic outcomes could add any…

Traditional FTS returns the whole document - people take over from that point and locate the interesting content there. The problem with RAG is that it does not follow that procedure - it tries to find the interesting chunk in one step. Even though since ReAct we know that LLMs could follow the same procedure as humans.

But we need an iterative RAG anyway: https://zzbbyy.substack.com/p/why-iterative-thinking-is-cruc...

Re: Solving the out-of-context chunk problem for RAG

#67

Post by zby »

Satam wrote: post #2

RAG feels hacky to me. We’re coming up with these pseudo-technical solutions to help but really they should be solved at the level of the model by researchers. Until this is solved natively, the attempts will be hacky duct-taped solutions.

I guess you can imagine an LLM that contains all information there is - but it would have to be at least as big as all information there is or it would have to hallucinate. And also you Not to mention that it seems that you would also require it to learn everything immediately. I don't see any realistic way to reach that goal.

To reach their potential LLMs need to know how to use external sources.

Update: After some more thinking - if you required it to know information about itself - then this would lead to some paradox - I am sure.

Re: Solving the out-of-context chunk problem for RAG

#68

Post by Sharlin »

“An Outside Context Problem was the sort of thing most civilisations encountered just once, and which they tended to encounter rather in the same way a sentence encountered a full stop.”

https://www.goodreads.com/quotes/9605621-an-outside-context-...

(Sorry, I just had to post this quote because it was the first thing that came to my mind when I saw the title, and I've been re-reading Banks lately.)

Re: Solving the out-of-context chunk problem for RAG

#70

Post by CharlieDigital »

ec109685 wrote: post #61
CharlieDigital wrote: post #4

The easiest solution to this is to stuff the heading into the chunk. The heading is hierarchical navigation within the sections of the document. I found Azure Document Intelligence specifically with the Layout Model to be fantastic for this because it can identify headers. All the better if you write a parser for the output JSON to track depth and stuff multiple headers from the path into the chunk.

Would it be better to go all the way and completely rewrite the source material in a way more suitable for retrieval? To some extent these headers are a step in that direction, but you’re still at the mercy of the chunk of text being suitable to answer the question. Instead, completely transforming the text into a dense set of denormalized “notes” that cover every concept present in the text seems like it would be ea…

For a large corpus, this would be quite expensive in terms of time and storage space. My experience is that embeddings work pretty well around 144-160 tokens (pure trial and error) with clinical trial protocols. I am certain that this value will be different by domain and document types.

If you generate and then "stuff" more text into this, my hunch is that accuracy drops off as the token count increases and it becomes "muddy". GRAG or even normal RAG can solve this to an extent because -- as you propose -- you can generate a congruent "note" and then embed that and link them together.

I'd propose something more flexible: expand on the input query instead and basically multiplex it to the related topics and ideas instead and perform cheap embedding search using more than 1 input vector.

Post reply on HN