Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

RAG is more than just embedding search

jxnl.github.io

Post reply on HN
Print view View original

21–30 of 62 posts

Re: RAG is more than just embedding search

#21

Post by lukev »

throwaway4aday wrote: post #9
pplonski86 wrote: post #5

Earlier quoted context omitted.

Before learning about RAG I thought that it is recurrent LLM agent that traverse over documents. After some study I must say that VectorDBs are boring.

It can be as simple or as complicated as you want. The article starts off by saying the naive approach of just embedding the query and looking for similar documents is a bad approach and what you actually want to embed and compare is something similar to the expected result. They don't go into detail on this but using their example of "what is the capital of France" you would conceivably transform that into "list of…

> none of this process is needed to answer the actual question which the LLM would know from its training data.

I think this isn't true; even if the model has the answer stored implicitly in its weights, it has no way of "citing it's source" or demonstrating that the answer is correct.

Re: RAG is more than just embedding search

#22

Post by simonw »

"Query-Document Mismatch: This model assumes that query embedding and the content embedding are similar in the embedding space, which is not always true based on the text you're trying to search over."

There are embeddings models that take this into account, which are pretty fascinating.

I've been exploring https://huggingface.co/intfloat/e5-large-v2 which lets you calculate two different types of embeddings in the same space. Example from their README:

    passage: As a general guideline, the CDC's average requirement of protein for women ages 19 to 70 is 46 grams per day
    query: how much protein should a female eat
You can then build your embedding database out of "passage: " embeddings, then run "query: " embeddings against it to try and find passages that can answer the question.

I've had pretty great initial results trying that out against paragraphs from my blog: https://til.simonwillison.net/llms/embed-paragraphs#user-con...

This won't help address other challenges mentioned in that post, like "what problems did we fix last week?" - but it's still a useful starting point.

Re: RAG is more than just embedding search

#23

Post by [deleted] »

simonw wrote: post #22

"Query-Document Mismatch: This model assumes that query embedding and the content embedding are similar in the embedding space, which is not always true based on the text you're trying to search over." There are embeddings models that take this into account, which are pretty fascinating. I've been exploring https://huggingface.co/intfloat/e5-large-v2 which lets you calculate two different types of embeddings in the s…

[deleted]

Re: RAG is more than just embedding search

#24

Post by natsucks »

binarymax wrote: post #20
natsucks wrote: post #14

Earlier quoted context omitted.

So in your opinion what are some examples of highly effective RAG systems/implementations?

Any good search you used before all this LLM stuff started happening is a perfect candidate for RAG. How do you know if a search was good? If you weren't pulling your hair out and actually got decent results for your queries (search is a thankless job like that - everyone expects it to work and complains when it doesnt). The reason good search is best for RAG is because the prompt is seeded by the top results for the…

Interesting. Do you think that points to the current limitations of RAG or a mismatch in what a user truly wants from search?

Re: RAG is more than just embedding search

#25

Post by tunesmith »

The pattern of "one request yields multiple kinds of responses" is challenging. You're basically looking at either having the client ask for the results and get them back, and then send the results to backend to get back the summary, OR, you're setting up some sort of sockets/server-sent-events thing where the frontend request establishes a connection and subscribes, while the backend sends back different sorts of "response events" as they become available.

Re: RAG is more than just embedding search

#26

Post by throwaway4aday »

lukev wrote: post #21
throwaway4aday wrote: post #9

Earlier quoted context omitted.

It can be as simple or as complicated as you want. The article starts off by saying the naive approach of just embedding the query and looking for similar documents is a bad approach and what you actually want to embed and compare is something similar to the expected result. They don't go into detail on this but using their example of "what is the capital of France" you would conceivably transform that into "list of…

> none of this process is needed to answer the actual question which the LLM would know from its training data. I think this isn't true; even if the model has the answer stored implicitly in its weights, it has no way of "citing it's source" or demonstrating that the answer is correct.

if your model can't predict the completion of "the capital of France is _" then it's going to really suck for other completions

Re: RAG is more than just embedding search

#27

Post by lukev »

throwaway4aday wrote: post #26
lukev wrote: post #21

Earlier quoted context omitted.

> none of this process is needed to answer the actual question which the LLM would know from its training data. I think this isn't true; even if the model has the answer stored implicitly in its weights, it has no way of "citing it's source" or demonstrating that the answer is correct.

if your model can't predict the completion of "the capital of France is _" then it's going to really suck for other completions

This is a great example of something GPT-4 gets confidently wrong, today. I just ran this query:

Prompt: "The year is 894 AD. The capital of France is: Response: "In 894 AD, the capital of France was Paris."

This is incorrect. According to Wikipedia, "In the 10th century Paris was a provincial cathedral city of little political or economic significance..."

The problem is that there's no good way to tell from this interaction whether it's true or false, because the mechanism that GPT-4 uses to return an answer is the same whether it's correct or incorrect.

Unless you already know the answer, the only way to be confident that a LLM is answering correctly is to use RAG to find a citation.

Re: RAG is more than just embedding search

#28

Post by jxnlco »

tunesmith wrote: post #25

The pattern of "one request yields multiple kinds of responses" is challenging. You're basically looking at either having the client ask for the results and get them back, and then send the results to backend to get back the summary, OR, you're setting up some sort of sockets/server-sent-events thing where the frontend request establishes a connection and subscribes, while the backend sends back different sorts of "r…

why not just asyncio.gather?

Re: RAG is more than just embedding search

#29

Post by [deleted] »

binarymax wrote: post #7

I agree with the premise of the article, but I’m not sure about the proposed solution. Search relevance tuning is a thing. Learn how to use a search engine and combine multiple features into ranking signals with relevance judgement data. I recommend the books “Relevant Search” and “AI Powered Search” (the latter of which I’m a contributing author). You’ll find that having a well tuned retriever is the backbone for mo…

[deleted]

Re: RAG is more than just embedding search

#30

Post by bthomas »

simonw wrote: post #22

"Query-Document Mismatch: This model assumes that query embedding and the content embedding are similar in the embedding space, which is not always true based on the text you're trying to search over." There are embeddings models that take this into account, which are pretty fascinating. I've been exploring https://huggingface.co/intfloat/e5-large-v2 which lets you calculate two different types of embeddings in the s…

Neat! Do you happen to have the analogous similarity queries with a default embedding? Curious to see them side by side.

(I know I can reproduce myself and I appreciate all the code you posted there - thought I'd ask first!)

Post reply on HN