Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

RAG is more than just embedding search

jxnl.github.io

Post reply on HN
Print view View original

11–20 of 62 posts

Re: RAG is more than just embedding search

#11

Post by sroussey »

binarymax wrote: post #7

I agree with the premise of the article, but I’m not sure about the proposed solution. Search relevance tuning is a thing. Learn how to use a search engine and combine multiple features into ranking signals with relevance judgement data. I recommend the books “Relevant Search” and “AI Powered Search” (the latter of which I’m a contributing author). You’ll find that having a well tuned retriever is the backbone for mo…

I went to buy it, but apparently I already have an account, so I did a password reset, and then it wants my previous password to activate the account, and well, I can’t buy it.

Re: RAG is more than just embedding search

#12

Post by fnordpiglet »

throwaway4aday wrote: post #8
fnordpiglet wrote: post #6

Earlier quoted context omitted.

At its most basic perhaps. But the LLM has an enormous semantic corpus embedded in its model that augments the retrieved document. The retrieved document in a way cements the context better to help prevent wandering into hallucinations. So the LLM would indeed be able to summarize the retrieved document, but also synthesize it with other “knowledge” embedded in its model. But the more important thing is you can inter…

You could also introduce a classifier step that takes the result of the query and asks the LLM if the results truly are relevant or not before passing them on to the summarization step. You can even add more steps (with possibly diminishing returns) such as taking the more relevant results and crafting a new query that is a very condensed summary, embedding it and then finding more results that are semantically simil…

Yep. But the idea that a RAG backed LLM is merely an efficient summarizer is missing the real power, which is it can summarize then be interrogated iteratively to refine in a semantic sense the actual questions you have, or explore adjacent spaces. It’s not just a search engine that can summarize, it’s a search engine that you can interrogate in natural language and it responds directly to your questions, as opposed to throwing a bunch of documents at you that have a probability of being related to your query.

Re: RAG is more than just embedding search

#14

Post by natsucks »

binarymax wrote: post #7

I agree with the premise of the article, but I’m not sure about the proposed solution. Search relevance tuning is a thing. Learn how to use a search engine and combine multiple features into ranking signals with relevance judgement data. I recommend the books “Relevant Search” and “AI Powered Search” (the latter of which I’m a contributing author). You’ll find that having a well tuned retriever is the backbone for mo…

So in your opinion what are some examples of highly effective RAG systems/implementations?

Re: RAG is more than just embedding search

#15

Post by natsucks »

majorbadass wrote: post #10
binarymax wrote: post #7

I agree with the premise of the article, but I’m not sure about the proposed solution. Search relevance tuning is a thing. Learn how to use a search engine and combine multiple features into ranking signals with relevance judgement data. I recommend the books “Relevant Search” and “AI Powered Search” (the latter of which I’m a contributing author). You’ll find that having a well tuned retriever is the backbone for mo…

Agree with your sentiment, though the article explicitly mentions precision/recall, suggesting at least some level of tuning. Query understanding via structured attributes is SOTA and used at top companies. Rewriting the query as a method is weird, and yeah I'm not so convinced. One reoccuring problem - the hacker ethos doesn't scale with AI products. "Mess around until it works" is ok to prototype. This is effective…

Do you know of a good example demonstrating RAG with query understanding via structured attributes?

Re: RAG is more than just embedding search

#16

Post by softwaredoug »

binarymax wrote: post #7

I agree with the premise of the article, but I’m not sure about the proposed solution. Search relevance tuning is a thing. Learn how to use a search engine and combine multiple features into ranking signals with relevance judgement data. I recommend the books “Relevant Search” and “AI Powered Search” (the latter of which I’m a contributing author). You’ll find that having a well tuned retriever is the backbone for mo…

I actually wonder why people dump gobs of user input to the vector db, or try to tokenize it into something smart, instead of being smarter and asking for queries to be generated. Such as:

--

Given a Jira issue database, I want to give you additional context to answer a question about a project called FooBar. The Jira project id is FOOBAR. Please generate JQL that you would like to use to answer this question

My question is: what are the major areas of technical debt in project FOOBAR?

--

Given a search engine for the wiki for project foobar, generate queries that help you answer this question:

What's the current status of project foobar?

---

Or somesuch...

(and hi Max, thanks for plugging our book :-p )

Re: RAG is more than just embedding search

#18

Post by binarymax »

softwaredoug wrote: post #16
binarymax wrote: post #7

I agree with the premise of the article, but I’m not sure about the proposed solution. Search relevance tuning is a thing. Learn how to use a search engine and combine multiple features into ranking signals with relevance judgement data. I recommend the books “Relevant Search” and “AI Powered Search” (the latter of which I’m a contributing author). You’ll find that having a well tuned retriever is the backbone for mo…

I actually wonder why people dump gobs of user input to the vector db, or try to tokenize it into something smart, instead of being smarter and asking for queries to be generated. Such as: -- Given a Jira issue database, I want to give you additional context to answer a question about a project called FooBar. The Jira project id is FOOBAR. Please generate JQL that you would like to use to answer this question My ques…

:waves: Hi Doug! (he's co-author of Relevant Search and contributing author of AI Powered Search too)

That's definitely a thing. But alarms go off in my head when I think about query latency and cost. Can't imagine running 1k qps while sending every single one to GPT or LLama - thats the stuff of production nightmares for me!

If you've got less demand and have a couple queries a second, then maybe it's OK - but you're probably adding a good second on top of your query latency.

Re: RAG is more than just embedding search

#19

Post by binarymax »

sroussey wrote: post #11
binarymax wrote: post #7

I agree with the premise of the article, but I’m not sure about the proposed solution. Search relevance tuning is a thing. Learn how to use a search engine and combine multiple features into ranking signals with relevance judgement data. I recommend the books “Relevant Search” and “AI Powered Search” (the latter of which I’m a contributing author). You’ll find that having a well tuned retriever is the backbone for mo…

I went to buy it, but apparently I already have an account, so I did a password reset, and then it wants my previous password to activate the account, and well, I can’t buy it.

Hi! Send me an email (it's in my profile) and maybe we can figure it out for you!

Re: RAG is more than just embedding search

#20

Post by binarymax »

natsucks wrote: post #14
binarymax wrote: post #7

I agree with the premise of the article, but I’m not sure about the proposed solution. Search relevance tuning is a thing. Learn how to use a search engine and combine multiple features into ranking signals with relevance judgement data. I recommend the books “Relevant Search” and “AI Powered Search” (the latter of which I’m a contributing author). You’ll find that having a well tuned retriever is the backbone for mo…

So in your opinion what are some examples of highly effective RAG systems/implementations?

Any good search you used before all this LLM stuff started happening is a perfect candidate for RAG. How do you know if a search was good? If you weren't pulling your hair out and actually got decent results for your queries (search is a thankless job like that - everyone expects it to work and complains when it doesnt).

The reason good search is best for RAG is because the prompt is seeded by the top results for the query. The only thing RAG does is summarize things for you and gives you answers instead of a list of documents.

And now I gotta confess something, after making RAG systems for clients and having to use them with all the web search engines these days - I kinda miss the list of documents, and find myself just skipping the summary at the top half the time and going back to reading the 10 blue links.

Post reply on HN