Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

RAG is more than just embedding search

jxnl.github.io

Post reply on HN
Print view View original

31–40 of 62 posts

Re: RAG is more than just embedding search

#31

Post by simonw »

bthomas wrote: post #30
simonw wrote: post #22

"Query-Document Mismatch: This model assumes that query embedding and the content embedding are similar in the embedding space, which is not always true based on the text you're trying to search over." There are embeddings models that take this into account, which are pretty fascinating. I've been exploring https://huggingface.co/intfloat/e5-large-v2 which lets you calculate two different types of embeddings in the s…

Neat! Do you happen to have the analogous similarity queries with a default embedding? Curious to see them side by side. (I know I can reproduce myself and I appreciate all the code you posted there - thought I'd ask first!)

No, I haven't been disciplined enough to have good examples for that yet.

One of my goals right now is to put together a solid RAG system based on top of LLM and Datasette that makes it really easy to compare different embedding models, chunking strategies and prompts to figure out what works best - but that's still just an idea in my head at the moment.

Re: RAG is more than just embedding search

#33

Post by ivalm »

natsucks wrote: post #15
majorbadass wrote: post #10

Earlier quoted context omitted.

Agree with your sentiment, though the article explicitly mentions precision/recall, suggesting at least some level of tuning. Query understanding via structured attributes is SOTA and used at top companies. Rewriting the query as a method is weird, and yeah I'm not so convinced. One reoccuring problem - the hacker ethos doesn't scale with AI products. "Mess around until it works" is ok to prototype. This is effective…

Do you know of a good example demonstrating RAG with query understanding via structured attributes?

A bit of a plug but

https://auxhealth.io/try

Does it’s generations with RAG with a mix of structured attributes + semantic retrieval.

Re: RAG is more than just embedding search

#34

Post by binarymax »

natsucks wrote: post #24
binarymax wrote: post #20

Earlier quoted context omitted.

Any good search you used before all this LLM stuff started happening is a perfect candidate for RAG. How do you know if a search was good? If you weren't pulling your hair out and actually got decent results for your queries (search is a thankless job like that - everyone expects it to work and complains when it doesnt). The reason good search is best for RAG is because the prompt is seeded by the top results for the…

Interesting. Do you think that points to the current limitations of RAG or a mismatch in what a user truly wants from search?

I think it works well when it's not a blob of text. One issue is that most of them are really long-winded. For example, if the answer can be nouns, just give me the list of nouns instead of a full sentence or paragraph.

Take for example this search: https://search.brave.com/search?q=what+are+the+captain+ameri...

Why the paragraph? Just give me a bulleted list! It's hard to read and kinda annoying.

Another issue for me is trust. Web search is oft polluted with web spam (this is not new). Mentally, one can see a URL and skip a site that doesn't have strong authority. So now in RAG, I either need to trust the answer, or I need to look at the embedded citation and find the document and then see if it's trustworthy. This adds friction.

This is also not unique to web search. Private search can also have poor relevance - do I know the LLM is being given the best context? Or is it getting bad context and hallucinating? I need to look at the results to be sure anyway.

I think when used in appropriate ways it can be good. But the experience of "summarize these 10 results for me" might not be the best for every query.

Re: RAG is more than just embedding search

#35

Post by ramoz »

binarymax wrote: post #7

I agree with the premise of the article, but I’m not sure about the proposed solution. Search relevance tuning is a thing. Learn how to use a search engine and combine multiple features into ranking signals with relevance judgement data. I recommend the books “Relevant Search” and “AI Powered Search” (the latter of which I’m a contributing author). You’ll find that having a well tuned retriever is the backbone for mo…

it also seems costly to deploy such a robust search backend (eg Elastic cluster, vector db, reranking ensemble, LLM for complex parsing... these are not cheap technologies)

Re: RAG is more than just embedding search

#36

Post by cuuupid »

I think these have common solutions that don't require building more systems, running more queries and jacking up GPU bills, which is the direction we should be moving in at this point.

e.g. asymmetric embeddings, instruct-based embeddings, and retrieval-rerank all address parts of the problems the author is presenting, all while keeping things generally light on infra.

Re: RAG is more than just embedding search

#37

Post by darkteflon »

cuuupid wrote: post #36

I think these have common solutions that don't require building more systems, running more queries and jacking up GPU bills, which is the direction we should be moving in at this point. e.g. asymmetric embeddings, instruct-based embeddings, and retrieval-rerank all address parts of the problems the author is presenting, all while keeping things generally light on infra.

Could you elaborate on “asymmetric embeddings”? That’s the first time I’ve heard that term used in this context.

Re: RAG is more than just embedding search

#38

Post by rolisz »

darkteflon wrote: post #37
cuuupid wrote: post #36

I think these have common solutions that don't require building more systems, running more queries and jacking up GPU bills, which is the direction we should be moving in at this point. e.g. asymmetric embeddings, instruct-based embeddings, and retrieval-rerank all address parts of the problems the author is presenting, all while keeping things generally light on infra.

Could you elaborate on “asymmetric embeddings”? That’s the first time I’ve heard that term used in this context.

It's embeddings that are generated differently for queries and for documents. The idea of that queries are usually short, while documents are longer, so if you embed them in the same way, the most relevant docs will be far from the query. Instruct from HKU is an example of such asymmetric embeddings

Re: RAG is more than just embedding search

#39

Post by darkteflon »

binarymax wrote: post #7

I agree with the premise of the article, but I’m not sure about the proposed solution. Search relevance tuning is a thing. Learn how to use a search engine and combine multiple features into ranking signals with relevance judgement data. I recommend the books “Relevant Search” and “AI Powered Search” (the latter of which I’m a contributing author). You’ll find that having a well tuned retriever is the backbone for mo…

To someone not familiar with the space, search seems like an incredibly complex and difficult space to get right. In your view, is it reasonable for the average developer prepared to read both of those books to expect to come out the other side and construct something ready for production? Thanks!

Re: RAG is more than just embedding search

#40

Post by DebtDeflation »

binarymax wrote: post #34
natsucks wrote: post #24

Earlier quoted context omitted.

Interesting. Do you think that points to the current limitations of RAG or a mismatch in what a user truly wants from search?

I think it works well when it's not a blob of text. One issue is that most of them are really long-winded. For example, if the answer can be nouns, just give me the list of nouns instead of a full sentence or paragraph. Take for example this search: https://search.brave.com/search?q=what+are+the+captain+ameri... Why the paragraph? Just give me a bulleted list! It's hard to read and kinda annoying. Another issue for m…

You're referring to what in the NLP subfield of Question Answering Systems would be known as a "factoid question". Historically, things like knowledge graphs and RDF triple stores would be used for answering these types of questions. I'm still not sold on the idea that an LLM is the answer to all QA/Chat problems and this is one example.
Post reply on HN