Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Solving the out-of-context chunk problem for RAG

d-star.ai

Post reply on HN
Print view View original

91–93 of 93 posts

Re: Solving the out-of-context chunk problem for RAG

#91

Post by ianbutler »

ramoz wrote: post #85
ianbutler wrote: post #59

Earlier quoted context omitted.

In 2019 I was using vector search to narrow the search space within 100s of millions of documents and then do full text search on the top 10k or so docs. That seems like a better stacking of the technologies even now

Interesting. Why did you need to “narrow” the search space using vector space? Did you build custom embeddings and feel confident about retrieval segments? I did similar in 2019 but typically in reverse, FTS, and a dual tower model to rerank. Vector search was an additional capability but never augmented the FTS.

It was in consideration of how slow our FTS at the time was over large amount of documents and the window we wanted to keep response times in and you're correct, we had custom embeddings and we had a reasonably high confidence.

So vector search would reduce the space to like 10k documents and then we'd take the document ids and FTS acted as the final authority on the ranking.

Re: Solving the out-of-context chunk problem for RAG

#92

Post by k__ »

verdverm wrote: post #56
k__ wrote: post #41

Earlier quoted context omitted.

I always wondered why a RAG index has to be a vector DB. If the model understands text/code and can generate text/code it should be able to talk to OpenSearch no problem.

You can view RAG as a bigger word2vec. The canonical example being "king - man + woman = queen". Words, or now chunks, have geometric distribution, cluster, and relationships... on semantic levels What is happening is that text is being embedded into a different space, and that format is an array of floats (a point in the embedding space). When we do retrieval, we embed the query and then find other points close to t…

"When we do retrieval, we embed the query and then find other points close to that query."

To me that just sounds like OpenSearch with extra steps.

How is this different/better than a search engine?

Re: Solving the out-of-context chunk problem for RAG

#93

Post by kgeist »

zmccormick7 wrote: post #88
simonw wrote: post #80

Earlier quoted context omitted.

It turns out picking that threshold is extremely difficult - I've tried! The value seems to differ for different searches, so picking eg 0.7 as a fixed value isn't actually as useful as you would expect.

Agreed that thresholds don't work when applied to the cosine similarity of embeddings. But I have found that the similarity score returned by high-quality rerankers, especially Cohere, are consistent and meaningful enough that using a threshold works well there.

I use similarity threshold (to remove absolutely irrelevant results) and then use a reranker to get Top N.
Post reply on HN