Better RAG Results with Reciprocal Rank Fusion and Hybrid Search • Hacker News Board

Re: Better RAG Results with Reciprocal Rank Fusion and Hybrid Search

#51

Post by softwaredoug » Fri, May 31, 2024, 3:21 PM UTC

eskibars wrote: post #46

Having worked building out a RAG SaaS platform for the past year and having worked on the vendor side of several keyword-based search systems in the past 10 years, I can say it's absolutely necessary to have some kind of hybrid search for most use cases I've seen. The problem is that most people don't have experience optimizing even 1 of the retrieval systems (vector or keyword), so a lot of users that try to DIY bui…

People dramatically underestimate the complexity of even reasonably relevant search systems.

One reason is unlike other data products - it’s an active, conscious action of users. If ads or recommendations are wrong nobody gets mad. But screw up search and it’s like the shop sales person taking you to the wrong aisle. It’s actively frustrating.

So basically every useful search system is disliked to some degree because it will get some things wrong some of the time.

Re: Better RAG Results with Reciprocal Rank Fusion and Hybrid Search

#52

Post by s-macke » Fri, May 31, 2024, 3:54 PM UTC

spencerchubb wrote: post #50

s-macke wrote: post #45

Earlier quoted context omitted.

I find these discussions funny. For decades we had search engines based on the query terms (keywords). Then there were lots of discussions and some implementations to put a semantic search on top of it to improve the keyword search. A hybrid search. Google Search did exactly that already in 2015 [1]. Now we start from pure semantic search and put keyword search on top of it to improve the semantic search and call it…

Except now the semantic capabilities are so much stronger. The transformer allows the model to get meaning from words that are far apart from each other

You are talking about English, right? And only for searches without any special technical terms or abbreviations?

Also my use case includes more than 20 languages. To find usable embeddings for all languages is next to impossible. However, there are keyword plugins for most languages in Solr or ElasticSearch.

Btw. In my benchmarks the result look something like this in English (MAP=mean average precision):

BM25(keyword search) -> MAP=45%

Embedding (Ada-002) -> MAP=49%

Hybrid (BM25 + Embedding) -> MAP=57%

Hybrid (Embedding + BM25) -> MAP=57%

And that's before you use synonym dictionaries for keyword searches.

Re: Better RAG Results with Reciprocal Rank Fusion and Hybrid Search

#53

Post by marcyb5st » Fri, May 31, 2024, 3:57 PM UTC

As soon as the indexed documents contain lingo of any kind you need hybrid search IMHO.

Additionally, if you can add conditional fuzzy matching into the mix so fat fingering something still yields a workable result is even better for UX (something along the lines of "the results from the tf-idf search are garbage, let's redo the search with fuzzy matching this time).

Re: Better RAG Results with Reciprocal Rank Fusion and Hybrid Search

#54

Post by wadefletch » Fri, May 31, 2024, 5:19 PM UTC

Mkengine wrote: post #16

throwaway115 wrote: post #11

I've implemented a very similar RAG hybrid solution, and it has improved LLM responses enormously. There are other things you can do too that have huge improvements, like destructuring your data and placing it into a graph structure, with queryable edge relationships. I think we're just scratching the surface.

This is really interesting, do you have other recommendations for improvements (gladly with sources I you have any)? I have to build a RAG solution for my job and right now I am collecting information to determine the best way to go ahead.

I'm exploring tooling for building these graphs and would love to pick your brain about your use case, if you're willing. No pressure! wade at tractorbeam dot ai

Re: Better RAG Results with Reciprocal Rank Fusion and Hybrid Search

#55

Post by cpursley » Fri, May 31, 2024, 5:55 PM UTC

pamelafox wrote: post #17

If you're looking for an example of RRF + Hybrid Search with PostgreSQL, I've put together a FastAPI app here that uses RAG with those options: https://github.com/Azure-Samples/rag-postgres-openai-python/ Here's the RRF+Hybrid part: https://github.com/Azure-Samples/rag-postgres-openai-python/... That's largely based off a sample from the pgvector repo, with a few tweaks. Agreed that Hybrid is the way to go, it's what…

This is awesome, thank you.

Re: Better RAG Results with Reciprocal Rank Fusion and Hybrid Search

#56

Post by spencerchubb » Fri, May 31, 2024, 8:08 PM UTC

s-macke wrote: post #52

spencerchubb wrote: post #50

Earlier quoted context omitted.

Except now the semantic capabilities are so much stronger. The transformer allows the model to get meaning from words that are far apart from each other

You are talking about English, right? And only for searches without any special technical terms or abbreviations? Also my use case includes more than 20 languages. To find usable embeddings for all languages is next to impossible. However, there are keyword plugins for most languages in Solr or ElasticSearch. Btw. In my benchmarks the result look something like this in English (MAP=mean average precision): BM25(keywo…

I'm curious, in your benchmark, what's the difference between BM25+Embedding and Embedding+BM25? And what do you use to make the embedding

If you make the embedding with an LLM, it should work for any language the LLM is trained on.

Re: Better RAG Results with Reciprocal Rank Fusion and Hybrid Search

#57

Post by cpursley » Mon, Jun 03, 2024, 5:52 PM UTC

cpursley wrote: post #55

pamelafox wrote: post #17

If you're looking for an example of RRF + Hybrid Search with PostgreSQL, I've put together a FastAPI app here that uses RAG with those options: https://github.com/Azure-Samples/rag-postgres-openai-python/ Here's the RRF+Hybrid part: https://github.com/Azure-Samples/rag-postgres-openai-python/... That's largely based off a sample from the pgvector repo, with a few tweaks. Agreed that Hybrid is the way to go, it's what…

This is awesome, thank you.

First take at hybrid search with Postgres pg_vector based on this: https://gist.github.com/cpursley/dae0a0be442f27e6af79d6bfc2b...

Re: Better RAG Results with Reciprocal Rank Fusion and Hybrid Search

#58

Post by s-macke » Tue, Jun 04, 2024, 8:14 AM UTC

spencerchubb wrote: post #56

s-macke wrote: post #52

Earlier quoted context omitted.

You are talking about English, right? And only for searches without any special technical terms or abbreviations? Also my use case includes more than 20 languages. To find usable embeddings for all languages is next to impossible. However, there are keyword plugins for most languages in Solr or ElasticSearch. Btw. In my benchmarks the result look something like this in English (MAP=mean average precision): BM25(keywo…

I'm curious, in your benchmark, what's the difference between BM25+Embedding and Embedding+BM25? And what do you use to make the embedding If you make the embedding with an LLM, it should work for any language the LLM is trained on.

BM25+Embedding and Embedding+BM25 is exactly the same and shows the commutative relation whether you start from keyword search or semantic search.

For my tests, I used Ada-002. As data I used small news articles and no chunking and no preprocessing. The query for the articles is embedded directly.

Of course, improvements can be done for both approaches. That should just exemplify, what you might expect with hybrid search.

Re: Better RAG Results with Reciprocal Rank Fusion and Hybrid Search

#59

Post by thefourthchime » Fri, Jun 07, 2024, 3:00 AM UTC

siquick wrote: post #32

thefourthchime wrote: post #28

I also found pure RAG with vector search to not work. I was creating a bot that could find answers to questions about things by looking at Slack discussions. At first, I downloaded entire channels, loaded them into a vector DB, and did RAG. The results sucked. Vector searches don't understand things very well, and in this world, specific keywords and error messages are very searchable. Instead, I take the user's quer…

When you’re creating your embedding you can store keywords from the content (using an LLM) in the metadata of each chunk which would positively increase the relevancy of results turned from the retrieval. LlamaIndex does this out of the box.

That's interesting! I didn't know that