Knowledge Graphs in RAG: Hype vs. Ragas Analysis • Hacker News Board

Re: Knowledge Graphs in RAG: Hype vs. Ragas Analysis

#21

Post by yetanotherjosh » Wed, Jul 10, 2024, 4:08 PM UTC

It seems to me that the "knowledge graph" generated in this article is incredibly naive and not comparable to the process in the MS paper, which requires multiple rounds of preprocessing the source content using LLMs to extract, summarize, find relationships at multiple levels and model them in the graph store. This just splats chunks and words into a vector graph and is barely defensible as a "knowledge graph".

Please tell me I'm missing something because this is egregious. How can you expect a graph approach to improve over naive rag if you don't actually build a knowledge graph that captures high quality, higher level entity relationships?

Re: Knowledge Graphs in RAG: Hype vs. Ragas Analysis

#22

Post by mkehrt » Wed, Jul 10, 2024, 7:22 PM UTC

davedx wrote: post #14

This seems highly relevant: https://arxiv.org/abs/2406.01506 > In this paper, we study the two foundational questions in this area. First, how are categorical concepts, such as {'mammal', 'bird', 'reptile', 'fish'}, represented? Second, how are hierarchical relations between concepts encoded? For example, how is the fact that 'dog' is a kind of 'mammal' encoded? We show how to extend the linear representation hypothe…

As an GenAI skeptic, I think this is a very cool finding. My experience with AI tools is that they are complete bullshit artists. But to a large extent that's just a result of the way they are trained. If this description of how the data is structured is correct, it indicates that these programs do encode a real model about the world. Perhaps alternative ways of training these same models, or fixing the data afterwards, will result in more truthful models.

Re: Knowledge Graphs in RAG: Hype vs. Ragas Analysis

#23

Post by th0ma5 » Wed, Jul 10, 2024, 7:38 PM UTC

itkovian_ wrote: post #4

Knowledge graphs where created to solve the problem of making natural,free flowing text machine processable. We now have a technology that completely understands natural free flowing text and can extract meaning. Why would going back to structure help when that structure can never be as rich as just text. I get it if the kb has new information, that's not what I'm saying.

> We now have a technology that completely understands natural free flowing text and can extract meaning.

Actually we don't. I know it certainly feels like LLMs do this but no one would dare stake their life on their output if they know how they work. Still useful!

Re: Knowledge Graphs in RAG: Hype vs. Ragas Analysis

#24

Post by neeleshs » Wed, Jul 10, 2024, 8:36 PM UTC

CharlieDigital wrote: post #16

davedx wrote: post #14

This seems highly relevant: https://arxiv.org/abs/2406.01506 > In this paper, we study the two foundational questions in this area. First, how are categorical concepts, such as {'mammal', 'bird', 'reptile', 'fish'}, represented? Second, how are hierarchical relations between concepts encoded? For example, how is the fact that 'dog' is a kind of 'mammal' encoded? We show how to extend the linear representation hypothe…

> Basically, LLM's already partially encode information as semantic graphs internally. There's an (underutilized?) technique here to take advantage of that internal graph: have the LLM tell you the related concepts first and then perform the RAG using not just the original concept, but the expanded set of related concepts. So: concept → [related concepts] → [[.. rag-rc1],[.. rag-rc2],[.. rag-rcn]] → summarize With GP…

Even with old GPT, if the summary is decent, it works reasonably well with even no RAG. We are a data management platform and allow users to build data pipelines around a data model. This is basically a DAG. We autogenerate documentation for these pipelines, using gpt4, and feed a summarized version of the data pipeline - expressed as graphviz dot file format in the prompt. gpt4 understands this format well, and seeminlgy understands the graph itself reasonably well!

It performs poorly expressing the higher level intent of the pipeline, but tactical details are accurately documented. We are trying to push prompting itself more, before turning to RAG & finetuning

Re: Knowledge Graphs in RAG: Hype vs. Ragas Analysis

#25

Post by DrStartup » Thu, Jul 11, 2024, 10:11 AM UTC

Knowledge / property graphs provide truths that can guide the retrieval. LLMs lack a truth function, ie causality. The KPG provides this as sorta a lace across the llm vector space. A KPG can either be used as a filter or a router of sorts. I expect we’ll see kpgs colocated with vector data of the llm and a tuned router layer uses it to guide retrieval and course correct the output. Kind of like MoE.