Knowledge Graphs in RAG: Hype vs. Ragas Analysis
aiencoder.substack.com
Knowledge Graphs in RAG: Hype vs. Ragas Analysis
1–10 of 25 posts
Re: Knowledge Graphs in RAG: Hype vs. Ragas Analysis
#2[deleted]
Re: Knowledge Graphs in RAG: Hype vs. Ragas Analysis
#3This is a nice sandbox walkthrough of the author's objective which was to test MSFT claims in the paper -- but with all due respect the buzz of graphs is because they add whole third layer in a combined approach like Reciprocal Rank Fusion (RRF). You do a BM25 search then you do a vector based nearest neighbors search and now you can add a KG search then all combined with local and global reranking etc the expectation is this produces a better final outcome. These findings aside, it still makes sense that adding KG to a hybrid search pipeline is going to be useful.
Re: Knowledge Graphs in RAG: Hype vs. Ragas Analysis
#4Knowledge graphs where created to solve the problem of making natural,free flowing text machine processable. We now have a technology that completely understands natural free flowing text and can extract meaning. Why would going back to structure help when that structure can never be as rich as just text. I get it if the kb has new information, that's not what I'm saying.
Re: Knowledge Graphs in RAG: Hype vs. Ragas Analysis
#5The Microsoft GraphRAG paper focuses on global sensemaking through hierarchical summarization, which is a fundamental aspect of their approach. The blog post analysis, however, doesn't address this core feature at all. Another issue is the corpus size, the paper focuses on sizes on the order of 1M tokens, while the reference text used in the blog post is probably shorter. On shorter text a simple LLM call could do summarization directly.
Re: Knowledge Graphs in RAG: Hype vs. Ragas Analysis
#6Knowledge graphs where created to solve the problem of making natural,free flowing text machine processable. We now have a technology that completely understands natural free flowing text and can extract meaning. Why would going back to structure help when that structure can never be as rich as just text. I get it if the kb has new information, that's not what I'm saying.
But RAG without graphs just relies on similarity search, which isn't very smart.
Re: Knowledge Graphs in RAG: Hype vs. Ragas Analysis
#7Knowledge graphs where created to solve the problem of making natural,free flowing text machine processable. We now have a technology that completely understands natural free flowing text and can extract meaning. Why would going back to structure help when that structure can never be as rich as just text. I get it if the kb has new information, that's not what I'm saying.
> Why would going back to structure help
When your corpus is large it is useful to split it up and hierarchically combine. In their place I would do both bottom-up and top-down summarization passes, so information can percolate from a leaf to the root and from the root to a different leaf. Global context can illuminate local summaries, for example think of the twist in a novel, it sheds new light on everything.
Re: Knowledge Graphs in RAG: Hype vs. Ragas Analysis
#8Looks like the test-setup confuses knowledge graphs with graph databases. The code just creates a neo4j database from a document, not a knowledge graph (basically uses neo4j as vector database). A knowledge graph would be created by a LLM as a preprocessing step (and queried similary by an LLM). This is a different approach than was tested, an approach that trades preprocessing time and domain knowledge for accuracy. Reference: https://python.langchain.com/v0.1/docs/use_cases/graph/const...
Re: Knowledge Graphs in RAG: Hype vs. Ragas Analysis
#9Knowledge graphs where created to solve the problem of making natural,free flowing text machine processable. We now have a technology that completely understands natural free flowing text and can extract meaning. Why would going back to structure help when that structure can never be as rich as just text. I get it if the kb has new information, that's not what I'm saying.
[dead]
Re: Knowledge Graphs in RAG: Hype vs. Ragas Analysis
#10This is a nice sandbox walkthrough of the author's objective which was to test MSFT claims in the paper -- but with all due respect the buzz of graphs is because they add whole third layer in a combined approach like Reciprocal Rank Fusion (RRF). You do a BM25 search then you do a vector based nearest neighbors search and now you can add a KG search then all combined with local and global reranking etc the expectatio…
[dead]