Have you considered this approach? Worked well for us: https://news.ycombinator.com/item?id=40998497
Solving the out-of-context chunk problem for RAG
51–60 of 93 posts
Re: Solving the out-of-context chunk problem for RAG
#52As is typical with any RAG strategy/algorithm, the implicit thing is it works on a specific dataset. Then, it solves a very specific use case. The thing is, if you have a dataset and a use case, you can have a very custom algorithm which would work wonders in terms of output you need. There need not be anything generic. My instinct at this point is, these algos look attractive because we are constrained to giving a u…
100% agree with you. I've built a # of RAG systems and find that simple Q&A-style use cases actually do fine with traditional chunking approaches.
... and then you have situations where people ask complex questions with multiple logical steps, or knowledge gathering requirements, and using some sort of hierarchical RAG strategy works better.
I think a lot of solutions (including this post) abstract to building knowledge graphs of some sort... But knowledge graphs still require an ontology associated to the problem you're solving and will fail outside of those domains.
Re: Solving the out-of-context chunk problem for RAG
#53Earlier quoted context omitted.
There’s probably lack of cpabalities on multiple fronts. RAG might have the right general idea but currently the retrieval seems to be too seperated from the model itself. I don’t know how our brains do it, but retrieval looks to be more integrated there. Models currently also have no way to update themselves with new info besides us putting data into their context window. They don’t learn after the initial training.…
I guess it's because people are not using tools enough yet. In my tests giving LLM access to tools for retrieval works much better then trying to guess what the RAG would need to answer. ie. LLM decides if it has all of the necessary information to answer the question. If not, let it search for it. If it still fails than let it search more :D
Agreed. Retrieval performance is very dependent on the quality of the search queries. Letting the LLM generate the search queries is much more reliable than just embedding the user input. Also, no retrieval system is going to return everything needed on the first try, so using a multi-step agent approach to retrieving information is the only way I've found to get extremely high accuracy.
Re: Solving the out-of-context chunk problem for RAG
#54Earlier quoted context omitted.
I always wondered why a RAG index has to be a vector DB. If the model understands text/code and can generate text/code it should be able to talk to OpenSearch no problem.
It doesn't have to be a vector DB - and in fact I'm seeing increasing skepticism that embedding vector DBs are the best way to implement RAG. A full-text search index using BM25 or similar may actually work a lot better for many RAG applications. I wrote up some notes on building FTS-based RAG here: https://simonwillison.net/2024/Jun/21/search-based-rag/
I've been using SQLite FTS (which is essentially BM25) and it works so well I haven't really bothered with vector databases, or Postgres, or anything else yet. Maybe when my corpus exceeds 2GB...
Re: Solving the out-of-context chunk problem for RAG
#55I'd like to see more evaluation data. There are 100s of RAG strategies, most of them only work on specific types of queries.
Yeah exactly, existing benchmark datasets available are underutilized (eg KILT, Natural questions, etc.). But it is only natural that different QA use cases require different strategies. I built 3 production RAG systems / virtual assistant now, and 4 that didn't make it past PoC and what advanced techniques works really depends on document type, text content and genre, use case, source knowledgebase structure and met…
How do you weight results between vector search and bm25? Do you fall back to bm25 when vector similarity is below a threshold, or maybe you tweak the weights by hand for each data set?
Re: Solving the out-of-context chunk problem for RAG
#56I've found the best approach is to start with traditional full text search. Get it to a point where manual human searches are useful - Especially for users who don't have a stake in the development of an AI solution. Then , look at building a RAG-style solution around the FTS. I never could get much beyond the basic search piece. I don't see how mixing in a black box AI model with probabilistic outcomes could add any…
I always wondered why a RAG index has to be a vector DB. If the model understands text/code and can generate text/code it should be able to talk to OpenSearch no problem.
You can view RAG as a bigger word2vec. The canonical example being "king - man + woman = queen". Words, or now chunks, have geometric distribution, cluster, and relationships... on semantic levels
What is happening is that text is being embedded into a different space, and that format is an array of floats (a point in the embedding space). When we do retrieval, we embed the query and then find other points close to that query. The reason for Vector DB is (1) to optimize for this use-case, we have many specialized data stores / indexes (redis, elastic, dolt, RDBMS) (2) often to be memory based for faster retrieval. PgVector will be interesting to watch. I personally use Qdrant
Full-text search will never be able to do some of the things that are possible in the embedding space. The most capable systems will use both techniques
Re: Solving the out-of-context chunk problem for RAG
#57Earlier quoted context omitted.
What about fresh data like an extremely relevant news headline that was published 10 minutes ago? Private data that I don’t want stored offsite but am okay trusting an enterprise no log api? Providing realtime context to LLMs isn’t “hacky”, model intelligence and RAG can complement each other and make advancements in tandem
One of my favorite cases is sports chat. I'd expect ChatGPT to be able to talk about sports legends but not be able to talk about a game that happened last weekend. Copilot usually does a good job because it can look up the game on Bing and them summarize but the other day i asked it "What happened last week in the NFL" and it told me about a Buffalo Bills game from last year (did it know I was in the Bills geography…
For the current game, it seems solvable by providing it the Boxscore and the radio commentary as context, perhaps with some additional data derived from recent games and news.
I think you’d get a close approximation of speaking with someone who was watching the game with you.
Re: Solving the out-of-context chunk problem for RAG
#58Earlier quoted context omitted.
Yeah exactly, existing benchmark datasets available are underutilized (eg KILT, Natural questions, etc.). But it is only natural that different QA use cases require different strategies. I built 3 production RAG systems / virtual assistant now, and 4 that didn't make it past PoC and what advanced techniques works really depends on document type, text content and genre, use case, source knowledgebase structure and met…
How do you weight results between vector search and bm25? Do you fall back to bm25 when vector similarity is below a threshold, or maybe you tweak the weights by hand for each data set?
The algorithm I use to get a final ranking from multiple rankings is called "reciprocal ranked fusion". I use the implementation described here: https://docs.llamaindex.ai/en/stable/examples/low_level/fusi...
Which is the implementation from the original paper.
Re: Solving the out-of-context chunk problem for RAG
#59Earlier quoted context omitted.
I always wondered why a RAG index has to be a vector DB. If the model understands text/code and can generate text/code it should be able to talk to OpenSearch no problem.
It doesn't have to be a vector DB - and in fact I'm seeing increasing skepticism that embedding vector DBs are the best way to implement RAG. A full-text search index using BM25 or similar may actually work a lot better for many RAG applications. I wrote up some notes on building FTS-based RAG here: https://simonwillison.net/2024/Jun/21/search-based-rag/
In 2019 I was using vector search to narrow the search space within 100s of millions of documents and then do full text search on the top 10k or so docs.
That seems like a better stacking of the technologies even now
Re: Solving the out-of-context chunk problem for RAG
#60Earlier quoted context omitted.
Neo4j graph rag is typically not graph rag in the AI sense / MSR Graph RAG paper sense, but KG or lexical extraction & embedding, and some retrieval time hope of the neighborhood being ok GRAG in the direction of the MSR paper adds some important areas: - summary indexes that can be lexical (document hierarchy) or not (topic, patient ID, etc), esp via careful entity extraction & linking - domain-optimized summarizati…
> Neo4j graph rag is typically not graph rag I would mildly disagree with this; Neo4j just serves as an underlying storage mechanism much like Postgres+pgvector could be the underlying storage mechanism for embedding-only RAG. How one extracts entities and connects them in the graph happens a layer above the storage layer of Neo4j (though Neo4j can also do this internally). Neo4j is not magic; the application layer a…
We generally stick with using neo4j/neptune/etc for more operational OLTP graph queries, basically large-scale managed storage for small neighborhood lookups. As soon as the task becomes more compute-tier AI workloads, like LLM summary indexing of 1M tweets or 10K documents, we prefer to use GPU-based compute stacks & external APIs with more fidelity. Think pipelines combining bulk embeddings, rich enrichment & wrangling, GNNs, community detection, etc. We only dump into DBs at the end. Speedups are generally in the 2-100X territory with even cheapo GPUs, so this ends up a big deal for both development + production. Likewise, continuous update flows end up being awkward in these environments vs full compute-tier ones, even ignoring the GPU aspect.
Separately, we're still unsure about vector search inside vs outside the graph DB during retrieval, both in the graph RAG scenario and the more general intelligence work domains. I'm more optimistic there for keeping in these graph DB, especially for small cases (And agreed, it's unfortunate neo4j uses graph RAG to market a variety of mostly bad quality solutions and conflate it with graph db storage, and the MSR researchers used it for a more specific and more notable technique (in AI circles) that doesn't need a graph DB and IMO, fundamentally, not even a KG. It's especially confusing that both groups are 'winning' on the term... in different circles.