Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Solving the out-of-context chunk problem for RAG

d-star.ai

Post reply on HN
Print view View original

21–30 of 93 posts

Re: Solving the out-of-context chunk problem for RAG

#21

Post by viraptor »

simonw wrote: post #20
Satam wrote: post #13

Earlier quoted context omitted.

There’s probably lack of cpabalities on multiple fronts. RAG might have the right general idea but currently the retrieval seems to be too seperated from the model itself. I don’t know how our brains do it, but retrieval looks to be more integrated there. Models currently also have no way to update themselves with new info besides us putting data into their context window. They don’t learn after the initial training.…

Don't forget the importance of data privacy. Updating a model with fresh information makes that information available to ALL users of that model. This often isn't what you want - you can run RAG against a user's private email to answer just their queries, without making that email "baked in" to the model.

You don't need to update the whole model for everyone. Fine tuning exists and is even available as a service in openai. The updates are only visible in the specific models you see.

Re: Solving the out-of-context chunk problem for RAG

#22

Post by danielbln »

ac1spkrbox wrote: post #17
Satam wrote: post #2

RAG feels hacky to me. We’re coming up with these pseudo-technical solutions to help but really they should be solved at the level of the model by researchers. Until this is solved natively, the attempts will be hacky duct-taped solutions.

The set of techniques for retrieval is immature, but it's important to note that just relying on model context or few-shot prompting has many drawbacks. Perhaps the most important is that retrieval as a task should not rely on generative outputs.

It's also subject to significantly more hallucination when the knowledge is baked into the model, vs being injected into the context at runtime.

Re: Solving the out-of-context chunk problem for RAG

#23

Post by simonw »

viraptor wrote: post #21
simonw wrote: post #20

Earlier quoted context omitted.

Don't forget the importance of data privacy. Updating a model with fresh information makes that information available to ALL users of that model. This often isn't what you want - you can run RAG against a user's private email to answer just their queries, without making that email "baked in" to the model.

You don't need to update the whole model for everyone. Fine tuning exists and is even available as a service in openai. The updates are only visible in the specific models you see.

Maintaining a fine-tuned model for every one of your users - even with techniques like LoRA - sounds complicated and expensive to me!

Re: Solving the out-of-context chunk problem for RAG

#24

Post by SkyPuncher »

aster0id wrote: post #7

I'd like to see more evaluation data. There are 100s of RAG strategies, most of them only work on specific types of queries.

RAG is akin to “search engine”.

It’s such a broad term that it’s essentially useless. Nearly anyone doing anything interesting with LLMs is doing RAG.

Re: Solving the out-of-context chunk problem for RAG

#25

Post by l72 »

Satam wrote: post #2

RAG feels hacky to me. We’re coming up with these pseudo-technical solutions to help but really they should be solved at the level of the model by researchers. Until this is solved natively, the attempts will be hacky duct-taped solutions.

I've described it this way to my colleagues:

RAG is a bit like having a pretty smart person take an open book test on a subject they are not an expert in. If your book has a good chapter layout and index, you probably do an ok job trying to find relevant information, quickly read it, and try to come up with an answer. But your not going to be able to test for a deep understanding of the material. This person is going to struggle if each chapter/concept builds on the previous concept, as you can't just look up something in Chapter 10 and be able to understand it without understanding Chapter 1-9.

Fine-tuning is a bit more like having someone go off and do a phd and specialize in a specific area. They get a much deeper understanding for the problem space and can conceptualize at a different level.

Re: Solving the out-of-context chunk problem for RAG

#26

Post by simonw »

SkyPuncher wrote: post #24
aster0id wrote: post #7

I'd like to see more evaluation data. There are 100s of RAG strategies, most of them only work on specific types of queries.

RAG is akin to “search engine”. It’s such a broad term that it’s essentially useless. Nearly anyone doing anything interesting with LLMs is doing RAG.

The definition for RAG that works for me is that you perform some form of "retrieval" (could be full-text search, could be vector search, could be some combination of the two or even another technique like a regular expression search) and you then include the results of that retrieval in the context.

I think it's a useful term.

Re: Solving the out-of-context chunk problem for RAG

#27

Post by CharlieDigital »

williamcotton wrote: post #8
CharlieDigital wrote: post #4

The easiest solution to this is to stuff the heading into the chunk. The heading is hierarchical navigation within the sections of the document. I found Azure Document Intelligence specifically with the Layout Model to be fantastic for this because it can identify headers. All the better if you write a parser for the output JSON to track depth and stuff multiple headers from the path into the chunk.

Contextual chunk headers The idea here is to add in higher-level context to the chunk by prepending a chunk header. This chunk header could be as simple as just the document title, or it could use a combination of document title, a concise document summary, and the full hierarchy of section and sub-section titles. That is from the article. Is this different from your suggested approach?

No, but this is also not really a novel solution.

Re: Solving the out-of-context chunk problem for RAG

#28

Post by CharlieDigital »

lmeyerov wrote: post #12
CharlieDigital wrote: post #4

The easiest solution to this is to stuff the heading into the chunk. The heading is hierarchical navigation within the sections of the document. I found Azure Document Intelligence specifically with the Layout Model to be fantastic for this because it can identify headers. All the better if you write a parser for the output JSON to track depth and stuff multiple headers from the path into the chunk.

So subtle! The article is on doing that, which is something we are doing a lot on right now... though it seems to snatch defeat from the jaws of victory: If we think about what this is about, it is basically entity augmentation & lexical linking / citations. Ex: A patient document may be all about patient id 123. That won't be spelled out in every paragraph, but by carrying along the patient ID (semantic entity) and…

Also working with GRAG (via Neo4j) and I'm somewhat skeptical that for most cases where a natural hierarchical structure already exists that graph will significantly exceed RAG with the hierarchical structure.

A better solution I had thought about its "local RAG". I came across this while processing embeddings from chunks parsed from Azure Document Intelligence JSON. The realization is that relevant topics are often localized within a document. Even across a corpus of documents, relevant passages are localized.

Because the chunks are processed sequentially, one needs only to keep track o the sequence number of the chunk. Assume that the embedding matches with a chunk n, then it would follow that the most important context are the chunks localized at n - m and n + p. So find the top x chunks via hybrid embedding + full text match and expand outwards from each of the chunks to grab the chunks around it.

While a chunk may represent just a few sentences of a larger block of text, this strategy will grab possibly the whole section or page of text localized around the chunk with the highest match.

Re: Solving the out-of-context chunk problem for RAG

#29

Post by PaulHoule »

Satam wrote: post #13
viraptor wrote: post #9

Earlier quoted context omitted.

That's so vague I can't tell what you're suggesting. What specifically do you think needs solving at the model level? What should work differently?

There’s probably lack of cpabalities on multiple fronts. RAG might have the right general idea but currently the retrieval seems to be too seperated from the model itself. I don’t know how our brains do it, but retrieval looks to be more integrated there. Models currently also have no way to update themselves with new info besides us putting data into their context window. They don’t learn after the initial training.…

I can answer questions off the cuff based on the weights of the neural network in my head. If I really wanted to get the right answers I would do "RAG" in the sense of looking up answers on the web or at the library and summarizing them.

For instance I have a policy that I try hard not to say anything like "most people think that..." without providing links because I work at an archive of public opinion data and if it gets out that one of our people was spouting false information about our domain, even if we weren't advertising the affiliation, that would look bad.

Re: Solving the out-of-context chunk problem for RAG

#30

Post by michalwarda »

CharlieDigital wrote: post #28
lmeyerov wrote: post #12

Earlier quoted context omitted.

So subtle! The article is on doing that, which is something we are doing a lot on right now... though it seems to snatch defeat from the jaws of victory: If we think about what this is about, it is basically entity augmentation & lexical linking / citations. Ex: A patient document may be all about patient id 123. That won't be spelled out in every paragraph, but by carrying along the patient ID (semantic entity) and…

Also working with GRAG (via Neo4j) and I'm somewhat skeptical that for most cases where a natural hierarchical structure already exists that graph will significantly exceed RAG with the hierarchical structure. A better solution I had thought about its "local RAG". I came across this while processing embeddings from chunks parsed from Azure Document Intelligence JSON. The realization is that relevant topics are often…

This works until relevant information is colocated. Sometimes though, for example in financial documents, important parts reference each other through keywords etc. That's why you can always try and retrieve not only positionally related chunks but also semantically related ones.

Go for chunk n, n - m, n + p and n' where n' are closest chunks to n semantically.

Moreover you can give this traversal possibility to your LLM to use itself as a tool or w/e whenever it is missing crucial information to answer the question. Thanks to that you don't always retrieve thousands of tokens even when not needed.

Post reply on HN