RAG is more than just embedding search
jxnl.github.io
RAG is more than just embedding search
1–10 of 62 posts
Re: RAG is more than just embedding search
#2So instead of asking google or wikipedia, you ask a natural language tokenizer to break your query into several possible queries, then feed that to an LLM in order to get an essay that might answer your question.
Do I have that basically correct?
edit, 43 minutes later: the first three responders say yes. So, it's a way of increasing the verbosity and reducing the reliability of responses to search queries. Yay! Who would not want such a thing?
(me. And probably you.)
Re: RAG is more than just embedding search
#3So instead of asking google or wikipedia, you ask a natural language tokenizer to break your query into several possible queries, then feed that to an LLM in order to get an essay that might answer your question. Do I have that basically correct? edit, 43 minutes later: the first three responders say yes. So, it's a way of increasing the verbosity and reducing the reliability of responses to search queries. Yay! Who…
Not quite, the advantage is that you can give it any documents you want to search even ones that aren't available to google or wikipedia. But I think otherwise that is essentially what is proposed here. The nice part is that since you know which documents got looked up when forumlating the answer you can also provide those as part of the output to the user so they can then go check the source data to confirm what was stated by the LLM.
Re: RAG is more than just embedding search
#4So instead of asking google or wikipedia, you ask a natural language tokenizer to break your query into several possible queries, then feed that to an LLM in order to get an essay that might answer your question. Do I have that basically correct? edit, 43 minutes later: the first three responders say yes. So, it's a way of increasing the verbosity and reducing the reliability of responses to search queries. Yay! Who…
Yes, a bit, though an important feature here is it's still searching the underlying data sources (e.g. Google, Wikipedia, or others) and then using a LLM to summarize the results.
The "natural language tokenizer" itself is often an LLM (they do a pretty good job of this).
A further extension this article doesn't talk about is to have a LLM with a different prompt analyze the answer before returning to the user, and do more queries if it doesn't believe the question has been well answered (imagine clicking "next page" of google search results under the hood).
The potential complexity of this scales all the way up to a full "research assistant" LLM "agent" that calls itself recursively.
Re: RAG is more than just embedding search
#5So instead of asking google or wikipedia, you ask a natural language tokenizer to break your query into several possible queries, then feed that to an LLM in order to get an essay that might answer your question. Do I have that basically correct? edit, 43 minutes later: the first three responders say yes. So, it's a way of increasing the verbosity and reducing the reliability of responses to search queries. Yay! Who…
Yes, a bit, though an important feature here is it's still searching the underlying data sources (e.g. Google, Wikipedia, or others) and then using a LLM to summarize the results. The "natural language tokenizer" itself is often an LLM (they do a pretty good job of this). A further extension this article doesn't talk about is to have a LLM with a different prompt analyze the answer before returning to the user, and d…
Before learning about RAG I thought that it is recurrent LLM agent that traverse over documents. After some study I must say that VectorDBs are boring.
Re: RAG is more than just embedding search
#6So instead of asking google or wikipedia, you ask a natural language tokenizer to break your query into several possible queries, then feed that to an LLM in order to get an essay that might answer your question. Do I have that basically correct? edit, 43 minutes later: the first three responders say yes. So, it's a way of increasing the verbosity and reducing the reliability of responses to search queries. Yay! Who…
At its most basic perhaps. But the LLM has an enormous semantic corpus embedded in its model that augments the retrieved document. The retrieved document in a way cements the context better to help prevent wandering into hallucinations. So the LLM would indeed be able to summarize the retrieved document, but also synthesize it with other “knowledge” embedded in its model.
But the more important thing is you can interrogate the LLM to ask it the specific questions you have based on what it has said and your goals. Contrast this to an information retrieval based methods where you read the article hoping your questions are answered, and when they aren’t you are stuck digging through less and less relevant results or refining a search string hoping to find the right incantation that tweaks the index in the right way, sifting through documents that may contain the kernel of information somewhere if it wasn’t SEO’ed out of existence. This is a really unnatural way of discovering information - the natural way, say with a teacher, is to be told background, ask questions, and iterate to understanding. This is how chat based LLMs work.
However with RAG you can ground them more concretely, as their model is a massive mishmash of everything that may or may not embed the information sought, but it’s also mixed in with everything else trained. You can bring in factual information into context that may not have even been trained. However the facts are a small aspect of knowledge - the overall semantics in the total corpus supports the facts in adjacent areas.
Re: RAG is more than just embedding search
#7I agree with the premise of the article, but I’m not sure about the proposed solution.
Search relevance tuning is a thing. Learn how to use a search engine and combine multiple features into ranking signals with relevance judgement data.
I recommend the books “Relevant Search” and “AI Powered Search” (the latter of which I’m a contributing author).
You’ll find that having a well tuned retriever is the backbone for most complex text AI. Learn the best practices from people who have been in the field for years, instead of trying to reinvent the wheel.
Re: RAG is more than just embedding search
#8So instead of asking google or wikipedia, you ask a natural language tokenizer to break your query into several possible queries, then feed that to an LLM in order to get an essay that might answer your question. Do I have that basically correct? edit, 43 minutes later: the first three responders say yes. So, it's a way of increasing the verbosity and reducing the reliability of responses to search queries. Yay! Who…
At its most basic perhaps. But the LLM has an enormous semantic corpus embedded in its model that augments the retrieved document. The retrieved document in a way cements the context better to help prevent wandering into hallucinations. So the LLM would indeed be able to summarize the retrieved document, but also synthesize it with other “knowledge” embedded in its model. But the more important thing is you can inter…
You could also introduce a classifier step that takes the result of the query and asks the LLM if the results truly are relevant or not before passing them on to the summarization step. You can even add more steps (with possibly diminishing returns) such as taking the more relevant results and crafting a new query that is a very condensed summary, embedding it and then finding more results that are semantically similar to it.
Re: RAG is more than just embedding search
#9Earlier quoted context omitted.
Yes, a bit, though an important feature here is it's still searching the underlying data sources (e.g. Google, Wikipedia, or others) and then using a LLM to summarize the results. The "natural language tokenizer" itself is often an LLM (they do a pretty good job of this). A further extension this article doesn't talk about is to have a LLM with a different prompt analyze the answer before returning to the user, and d…
Before learning about RAG I thought that it is recurrent LLM agent that traverse over documents. After some study I must say that VectorDBs are boring.
It can be as simple or as complicated as you want. The article starts off by saying the naive approach of just embedding the query and looking for similar documents is a bad approach and what you actually want to embed and compare is something similar to the expected result. They don't go into detail on this but using their example of "what is the capital of France" you would conceivably transform that into "list of European capital cities" or "list of cities in France" using an LLM, embed that, find the similar documents, feed those documents into an LLM along with the query and some system instructions about how to format the response and then return that. Keep in mind this is an absurdly simplified example query and none of this process is needed to answer the actual question which the LLM would know from its training data but you would want this process in place to ensure accurate results for more complex or specialized queries.
Re: RAG is more than just embedding search
#10I agree with the premise of the article, but I’m not sure about the proposed solution. Search relevance tuning is a thing. Learn how to use a search engine and combine multiple features into ranking signals with relevance judgement data. I recommend the books “Relevant Search” and “AI Powered Search” (the latter of which I’m a contributing author). You’ll find that having a well tuned retriever is the backbone for mo…
Agree with your sentiment, though the article explicitly mentions precision/recall, suggesting at least some level of tuning. Query understanding via structured attributes is SOTA and used at top companies. Rewriting the query as a method is weird, and yeah I'm not so convinced.
One reoccuring problem - the hacker ethos doesn't scale with AI products. "Mess around until it works" is ok to prototype. This is effectively using the dev's intuition on the 10 examples they look at as the offline eval function.
But many (most?) new-wave AI products don't have consistent offline metrics they optimize for. I think this quickly stops working when you've absorbed the obvious gains.