Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Solving the out-of-context chunk problem for RAG

d-star.ai

Post reply on HN
Print view View original

41–50 of 93 posts

Re: Solving the out-of-context chunk problem for RAG

#41

Post by k__ »

bob1029 wrote: post #37

I've found the best approach is to start with traditional full text search. Get it to a point where manual human searches are useful - Especially for users who don't have a stake in the development of an AI solution. Then , look at building a RAG-style solution around the FTS. I never could get much beyond the basic search piece. I don't see how mixing in a black box AI model with probabilistic outcomes could add any…

I always wondered why a RAG index has to be a vector DB.

If the model understands text/code and can generate text/code it should be able to talk to OpenSearch no problem.

Re: Solving the out-of-context chunk problem for RAG

#42

Post by lmeyerov »

CharlieDigital wrote: post #35
michalwarda wrote: post #30

Earlier quoted context omitted.

This works until relevant information is colocated. Sometimes though, for example in financial documents, important parts reference each other through keywords etc. That's why you can always try and retrieve not only positionally related chunks but also semantically related ones. Go for chunk n, n - m, n + p and n' where n' are closest chunks to n semantically. Moreover you can give this traversal possibility to your…

> positionally related chunks but also semantically related ones That's why the entry point would still be an embedding search; it's just that instead of using the first 20 embedding hits, you take the first 5 and if the reference is "semantically adjacent" to the entry concept, we would expect that some of the first few chunks would capture it in most cases. I think where GRAG yields more relevancy is when the refer…

Neo4j graph rag is typically not graph rag in the AI sense / MSR Graph RAG paper sense, but KG or lexical extraction & embedding, and some retrieval time hope of the neighborhood being ok

GRAG in the direction of the MSR paper adds some important areas:

- summary indexes that can be lexical (document hierarchy) or not (topic, patient ID, etc), esp via careful entity extraction & linking

- domain-optimized summarization templates, both automated & manual

- + as mentioned, wider context around these at retrieval

- introducing a more generalized framework for handling different kinds of concept relations, summary indexing, and retrieval around these

Ex: The same patient over time & docz, and seperately, similar kinds of patients across documents

Note that I'm not actually a big fan of how the MSR paper indirects the work through KG extraction, as that exits the semantic domain, and we don't do it that way

Fundamentally, that both moves away from paltry retrieval result sets that are small/gaps/etc, and enables cleaner input to the runtime query

I agree it is a quick win if quality can be low and you have low budget/time. Like combine a few out of the box index types and do rank retrieval. But a lot of the power gets lost. We are working on infra (+ OSSing it) because that is an unfortunate and unnecessary state of affairs. Right now llamaindex/langchain and raw vector DBs feel like adhoc and unprincipled ways to build these pipelines in a software engineering and AI perspective, so from an investment side, moving away from hacks and to more semantic, composable, & scalable pipelines is important IMO.

Re: Solving the out-of-context chunk problem for RAG

#43

Post by derefr »

siquick wrote: post #11

I can’t imagine any serious RAG application is not doing this - adding a contextual title, summary, keywords, and questions to the metadata of each chunk is a pretty low effort/high return implementation.

> adding a contextual title, summary, keywords, and questions

That's interesting; do you then transform the question-as-prompt before embedding it at runtime, so that it "asks for" that metadata to be in the response? Because otherwise, it would seem to me that you're just making it harder for the prompt vector and the document vectors to match.

(I guess, if it's equally harder in all cases, then that might be fine. But if some of your documents have few tags or no title or something, they might be unfairly advantaged in a vector-distance-ranked search, because the formats of the documents more closely resemble the response format the question was expecting...)

Re: Solving the out-of-context chunk problem for RAG

#44

Post by te_chris »

k__ wrote: post #41
bob1029 wrote: post #37

I've found the best approach is to start with traditional full text search. Get it to a point where manual human searches are useful - Especially for users who don't have a stake in the development of an AI solution. Then , look at building a RAG-style solution around the FTS. I never could get much beyond the basic search piece. I don't see how mixing in a black box AI model with probabilistic outcomes could add any…

I always wondered why a RAG index has to be a vector DB. If the model understands text/code and can generate text/code it should be able to talk to OpenSearch no problem.

Honestly you clocked the secret: it doesn’t.

It makes sense for the hype, though. As we got LLM’s we also got wayyyy better embedding models, but they’re not dependencies.

Re: Solving the out-of-context chunk problem for RAG

#45

Post by simonw »

k__ wrote: post #41
bob1029 wrote: post #37

I've found the best approach is to start with traditional full text search. Get it to a point where manual human searches are useful - Especially for users who don't have a stake in the development of an AI solution. Then , look at building a RAG-style solution around the FTS. I never could get much beyond the basic search piece. I don't see how mixing in a black box AI model with probabilistic outcomes could add any…

I always wondered why a RAG index has to be a vector DB. If the model understands text/code and can generate text/code it should be able to talk to OpenSearch no problem.

It doesn't have to be a vector DB - and in fact I'm seeing increasing skepticism that embedding vector DBs are the best way to implement RAG.

A full-text search index using BM25 or similar may actually work a lot better for many RAG applications.

I wrote up some notes on building FTS-based RAG here: https://simonwillison.net/2024/Jun/21/search-based-rag/

Re: Solving the out-of-context chunk problem for RAG

#46

Post by petercooper »

bob1029 wrote: post #37

I've found the best approach is to start with traditional full text search. Get it to a point where manual human searches are useful - Especially for users who don't have a stake in the development of an AI solution. Then , look at building a RAG-style solution around the FTS. I never could get much beyond the basic search piece. I don't see how mixing in a black box AI model with probabilistic outcomes could add any…

You're right, and it's also possible to still use LLMs and vector search in such a system, but instead you use them to enrich the queries made to traditional, pre-existing knowledge bases and search systems. Arguably you could call this "generative assisted retrieval" or GAR.. sadly I didn't coin the term, there's a paper about it ;-) https://aclanthology.org/2021.acl-long.316/

Re: Solving the out-of-context chunk problem for RAG

#47

Post by derefr »

visarga wrote: post #19
siquick wrote: post #11

I can’t imagine any serious RAG application is not doing this - adding a contextual title, summary, keywords, and questions to the metadata of each chunk is a pretty low effort/high return implementation.

Text embeds don't capture inferred data, like "second letter of this text" does not embed close to "e". LLM chain of thought is required to deduce the meaning more completely.

Given current SOTA, no, they don’t.

But there’s no reason why they couldn’t — just capture the vectors of some of the earlier hidden layers during the RAG encoder’s inference run, and append these intermediate vectors to the final embedding vector of the output layer to become the vectors you throw into your vector DB. (And then do the same at runtime for embedding your query prompts.)

Probably you’d want to bias those internal-layer vectors, giving them an increasingly-high “artificial distance” coefficient for increasingly-early layers — so that a document closely matching in token space or word space or syntax-node space improves its retrieval rank a bit, but not nearly as much as if the document were a close match in concept space. (But maybe do something nonlinear instead of multiplication here — you might want near-identical token-wise or syntax-wise matches to show up despite different meanings, depending on your use-case.)

Come to think, you could probably build a pretty good source-code search RAG off of this approach.

(Also, it should hopefully be obvious here that if you fine-tuned an encoder-decoder LLM to label matches based on criteria where some of those criteria are only available in earlier layers, then you’d be training pass-through vector dimensions into the intermediate layers of the encoder — such that using such an encoder on its own for RAG embedding should produce the same effect as capturing + weighting the intermediate layers of a non-fine-tuned LLM.)

Re: Solving the out-of-context chunk problem for RAG

#48

Post by CharlieDigital »

lmeyerov wrote: post #42
CharlieDigital wrote: post #35

Earlier quoted context omitted.

> positionally related chunks but also semantically related ones That's why the entry point would still be an embedding search; it's just that instead of using the first 20 embedding hits, you take the first 5 and if the reference is "semantically adjacent" to the entry concept, we would expect that some of the first few chunks would capture it in most cases. I think where GRAG yields more relevancy is when the refer…

Neo4j graph rag is typically not graph rag in the AI sense / MSR Graph RAG paper sense, but KG or lexical extraction & embedding, and some retrieval time hope of the neighborhood being ok GRAG in the direction of the MSR paper adds some important areas: - summary indexes that can be lexical (document hierarchy) or not (topic, patient ID, etc), esp via careful entity extraction & linking - domain-optimized summarizati…

    > Neo4j graph rag is typically not graph rag
I would mildly disagree with this; Neo4j just serves as an underlying storage mechanism much like Postgres+pgvector could be the underlying storage mechanism for embedding-only RAG. How one extracts entities and connects them in the graph happens a layer above the storage layer of Neo4j (though Neo4j can also do this internally). Neo4j is not magic; the application layer and data modelling still has to define which entities and how they are connected.

But why Neo4j? Neo4j has some nice amenities for building GRAG on top of. In particular, it has packages to support community partitioning including Leiden[0] (also used by Microsoft's GraphRAG[1]) and Louvain[2] as well as several other community detection algorithms. The built-in support for node embeddings[3] as well as external AI APIs[4] make the DX -- in so far as building the underlying storage for complex retrieval -- quite good, IMO.

The approach that we are taking is that we are importing a corpus of information into Neo4j and performing ETL on the way in to create additional relationships; effectively connecting individual chunks by some related "facet". Then we plan to run community detection over it to identify communities of interest and use a combination of communities, locality, and embedding match to retrieve chunks.

I just started exploring it over the past week and I would say that if your team is going to end up doing some more complex GRAG, then Neo4j feels like it has the right tooling to be the underlying storage layer and you could even feasibly implement other parts of your workflow in there as well, but entity extraction and such feels like it belongs one layer up in the application layer. Most notably, having direct query access to the underlying graph with a graph query language (Cypher) means that you will have more control and different ways to experiment with retrieval. However; as I mentioned, I would encourage most teams to be more clever with embedding RAG before adding more infrastructure like Neo4j.

[0] https://neo4j.com/docs/graph-data-science/current/algorithms...

[1] https://microsoft.github.io/graphrag/

[2] https://neo4j.com/docs/graph-data-science/current/algorithms...

[3] https://neo4j.com/docs/graph-data-science/current/machine-le...

[4] https://neo4j.com/labs/apoc/5/ml/openai/

Re: Solving the out-of-context chunk problem for RAG

#49

Post by alexmolas »

bob1029 wrote: post #37

I've found the best approach is to start with traditional full text search. Get it to a point where manual human searches are useful - Especially for users who don't have a stake in the development of an AI solution. Then , look at building a RAG-style solution around the FTS. I never could get much beyond the basic search piece. I don't see how mixing in a black box AI model with probabilistic outcomes could add any…

But with FTS you don't solve the "out-of-context chunk problem". You'll still miss relevant chunks with FTS. You still can apply the approach proposed in the post to FTS, but instead of using similarity you could use BM25.

Re: Solving the out-of-context chunk problem for RAG

#50

Post by ankit219 »

As is typical with any RAG strategy/algorithm, the implicit thing is it works on a specific dataset. Then, it solves a very specific use case. The thing is, if you have a dataset and a use case, you can have a very custom algorithm which would work wonders in terms of output you need. There need not be anything generic.

My instinct at this point is, these algos look attractive because we are constrained to giving a user a wow moment where they upload something and get to chat with the doc/dataset within minutes. As attractive as that is, it is a distinct second priority to building a system that works 99% of the time, even if takes a day or two to set up. You get a feel of the data, have a feel of type of questions that may be asked, and create an algo that works for a specific type of dataset-usecase combo (assuming any more data you add in this system would be similar and work pretty well). There is no silver bullet that we seem to be searching for.

Post reply on HN