Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

RAG is more than just embedding search

jxnl.github.io

Post reply on HN
Print view View original

41–50 of 62 posts

Re: RAG is more than just embedding search

#41

Post by fudged71 »

rolisz wrote: post #38
darkteflon wrote: post #37

Earlier quoted context omitted.

Could you elaborate on “asymmetric embeddings”? That’s the first time I’ve heard that term used in this context.

It's embeddings that are generated differently for queries and for documents. The idea of that queries are usually short, while documents are longer, so if you embed them in the same way, the most relevant docs will be far from the query. Instruct from HKU is an example of such asymmetric embeddings

I've heard the concept of applying a linear transformation to an embedding model output, is that a similar idea?

Re: RAG is more than just embedding search

#42

Post by rolisz »

fudged71 wrote: post #41
rolisz wrote: post #38

Earlier quoted context omitted.

It's embeddings that are generated differently for queries and for documents. The idea of that queries are usually short, while documents are longer, so if you embed them in the same way, the most relevant docs will be far from the query. Instruct from HKU is an example of such asymmetric embeddings

I've heard the concept of applying a linear transformation to an embedding model output, is that a similar idea?

That's a sort of adapting an embedding to a certain task/domain. But I wouldn't call it asymmetric embeddings.

Re: RAG is more than just embedding search

#44

Post by nobodyminus »

> Query-Document Mismatch: This model assumes that query embedding and the content embedding are similar in the embedding space, which is not always true based on the text you're trying to search over. Only using queries that are semantically similar to the content is a huge limitation!

It seems like fine tuning for joint embeddings between your queries and content is a far more elegant way to solve this problem.

Re: RAG is more than just embedding search

#45

Post by jxnlco »

cuuupid wrote: post #36

I think these have common solutions that don't require building more systems, running more queries and jacking up GPU bills, which is the direction we should be moving in at this point. e.g. asymmetric embeddings, instruct-based embeddings, and retrieval-rerank all address parts of the problems the author is presenting, all while keeping things generally light on infra.

> asymmetric embeddings, instruct-based embeddings, and retrieval-rerank

how would you handle a relative time range? what if you're provided a search client that does not support embeddings

(maybe say, google calendar api)

Re: RAG is more than just embedding search

#46

Post by levkk »

natsucks wrote: post #14
binarymax wrote: post #7

I agree with the premise of the article, but I’m not sure about the proposed solution. Search relevance tuning is a thing. Learn how to use a search engine and combine multiple features into ranking signals with relevance judgement data. I recommend the books “Relevant Search” and “AI Powered Search” (the latter of which I’m a contributing author). You’ll find that having a well tuned retriever is the backbone for mo…

So in your opinion what are some examples of highly effective RAG systems/implementations?

How about the one that works directly in your database: https://postgresml.org

Re: RAG is more than just embedding search

#47

Post by suresk »

softwaredoug wrote: post #16
binarymax wrote: post #7

I agree with the premise of the article, but I’m not sure about the proposed solution. Search relevance tuning is a thing. Learn how to use a search engine and combine multiple features into ranking signals with relevance judgement data. I recommend the books “Relevant Search” and “AI Powered Search” (the latter of which I’m a contributing author). You’ll find that having a well tuned retriever is the backbone for mo…

I actually wonder why people dump gobs of user input to the vector db, or try to tokenize it into something smart, instead of being smarter and asking for queries to be generated. Such as: -- Given a Jira issue database, I want to give you additional context to answer a question about a project called FooBar. The Jira project id is FOOBAR. Please generate JQL that you would like to use to answer this question My ques…

GAR - Generation-Augmented Retrieval?

I've actually had some success with getting ChatGPT to create Redshift queries based on user text and then I can run them and render results, which has some interesting use-cases.

Max calls out the biggest problem with using something like ChatGPT in a search flow - it is way too slow. I've talked to a lot of people wondering if we can just shove a catalog at ChatGPT and have it magically do a really good job of search, and token limits + latency are two pretty hard stops there (plus I think it would be generally a worse experience in many cases).

What I'm trying to look at now is how LLMs can be used to make documents better suited for search by pulling out useful metadata, summarizing related content, etc. Things that can be done at index time instead of search time, so the latency requirements are less of an issue.

Re: RAG is more than just embedding search

#48

Post by tallytarik »

darkteflon wrote: post #39
binarymax wrote: post #7

I agree with the premise of the article, but I’m not sure about the proposed solution. Search relevance tuning is a thing. Learn how to use a search engine and combine multiple features into ranking signals with relevance judgement data. I recommend the books “Relevant Search” and “AI Powered Search” (the latter of which I’m a contributing author). You’ll find that having a well tuned retriever is the backbone for mo…

To someone not familiar with the space, search seems like an incredibly complex and difficult space to get right. In your view, is it reasonable for the average developer prepared to read both of those books to expect to come out the other side and construct something ready for production? Thanks!

IMO: yes.

I've used both of these in my current role to make substantial improvements to our Solr search engine.

They include a good range of techniques between "quick wins you could implement and test in an hour" and "complex machine learning pipelines based on millions of data points".

AI Powered Search was probably the more interesting and useful but it's also a bit of a misnomer. Half of the techniques aren't related to AI (which is fine) and the half that are, are rapidly out of date. Semantic/vector search is now miles ahead of what the book talks about, with dense vector support in Solr/Elastic/Opensearch; sparse models; hybrid search/RRF... but I digress :)

If you're interested in how to improve the magic black box that is search, they're worthwhile reads.

Just remember that expectations are everything. There are no two books, or twenty books, that'll turn your out-of-the-box Solr instance into Google or Bing quality. But you can end up with a magic black box that serves much better results, which is nice!

Re: RAG is more than just embedding search

#49

Post by computerex »

darkteflon wrote: post #37
cuuupid wrote: post #36

I think these have common solutions that don't require building more systems, running more queries and jacking up GPU bills, which is the direction we should be moving in at this point. e.g. asymmetric embeddings, instruct-based embeddings, and retrieval-rerank all address parts of the problems the author is presenting, all while keeping things generally light on infra.

Could you elaborate on “asymmetric embeddings”? That’s the first time I’ve heard that term used in this context.

It's actually not asymmetric embeddings, but rather asymmetric searches. This is referring to the case where the query and documents being search against are not interchangeable. If the query and documents are interchangeable, then that's symmetric search.

It has implications for retrieval. There are embeddings models that are optimized for the symmetric search use case, and then there are models optimized for asymmetric search. You have to use the appropriate model for the task. Furthermore, you can use a LLM to transform your query into the same class as the docs being retrieved, to turn asymmetric search into symmetric search.

Re: RAG is more than just embedding search

#50

Post by WinLychee »

darkteflon wrote: post #39
binarymax wrote: post #7

I agree with the premise of the article, but I’m not sure about the proposed solution. Search relevance tuning is a thing. Learn how to use a search engine and combine multiple features into ranking signals with relevance judgement data. I recommend the books “Relevant Search” and “AI Powered Search” (the latter of which I’m a contributing author). You’ll find that having a well tuned retriever is the backbone for mo…

To someone not familiar with the space, search seems like an incredibly complex and difficult space to get right. In your view, is it reasonable for the average developer prepared to read both of those books to expect to come out the other side and construct something ready for production? Thanks!

It's a problem with a long tail, and it very much depends on what objective you're optimizing for. In search at least, you aim for "good" and "better", but will never achieve "perfect". It's a pretty interesting space at the meeting point of software and data science. You probably don't necessarily need to read full books before diving in, but play around with "learning to rank" https://xgboost.readthedocs.io/en/latest/tutorials/learning_... and maybe check out https://www.microsoft.com/en-us/research/uploads/prod/2017/0... . Also https://www.tensorflow.org/recommenders/examples/basic_retri... .
Post reply on HN