Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Build a search engine, not a vector DB

blog.elicit.com

Post reply on HN
Print view View original

21–30 of 84 posts

Re: Build a search engine, not a vector DB

#21

Post by danielbln »

deckar01 wrote: post #9

Instead of embedding the user prompt, I let the LLM invert it into keywords and search the embedding of that. It very much does feel like a magic bullet.

Using the LLM to mutate the user query is the way to go. A common practice for example to take the chat history of a chat, and rephrase a follow up question that might not have a lot of information density (e.g. follow up question is "and then what?" which is useless for search, but the LLM turns it into "after a contract cancellation, what steps have to be taken afterwards" or something similar, which provides a lot more meat to search with.

Using the LLM to mutate the input so it can be used better for search is a path that works very well (ignoring added latency and cost).

Re: Build a search engine, not a vector DB

#22

Post by m-i-l »

CGamesPlay wrote: post #14
kristiandupont wrote: post #11

Earlier quoted context omitted.

"Search the embedding"? Could you elaborate on this, it sounds interesting!

I think OP means to filter the user input through an LLM with “convert this question into a keyword list” and then calculating the embedding of the LLM’s output (instead of calculating the embedding of the user input directly). The “search the embedding” is the normal vector DB part.

"Query expansion"[0] has been an information retrieval technique for a while, but using LLMs to help with query expansion is fairly new and promising, e.g. "Query Expansion by Prompting Large Language Models"[1], and "Query2doc: Query Expansion with Large Language Models"[2].

[0] https://en.wikipedia.org/wiki/Query_expansion

[1] https://arxiv.org/abs/2305.03653

[2] https://arxiv.org/abs/2303.07678

Re: Build a search engine, not a vector DB

#23

Post by m-i-l »

jankovicsandras wrote: post #7

I agree too. My impression is that almost all RAG tutorials _only_ talk about vector DBs, when these are not strictly required for Retrieval Augmented Generation. I'm guessing vector DBs are useful when you have massive amounts of documents on diverse topics. Some gotchas I experienced (but I might be using the wrong embedding/vector DB: spaCy/FAISS): - Short user questions might result a low signal query vector, e.…

> "Has somebody experience with Apache Lucene / Solr or Elasticsearch?"

I've been working on a RAG with Solr, and quickly hit some of the issues you describe when dealing with real-world messy data and user input, e.g. using all-MiniLM-L6-v2 and cosine similarity, "Can you summarize Immanuel Kant's biography?" matched a chunk containing just the word "Biography" rather than one which started "Immanuel Kant, born in 1724...", and "How high is Ben Nevis?" matched a chunk of text about someone called Benjamin rather than a chunk about mountains containing the words "Ben Nevis" and its height[0]. Switching embedding model has helped, but still not convinced that vector search alone is the silver bullet some claim it is. Still lots more to try though, e.g. hybrid search[1], query expansion[2], knowledge graphs etc.

[0] https://www.michael-lewis.com/posts/vector-search-and-retrie...

[1] https://sease.io/2023/12/hybrid-search-with-apache-solr.html

[2] https://news.ycombinator.com/item?id=38706913

Re: Build a search engine, not a vector DB

#24

Post by poulpy123 »

deckar01 wrote: post #9

Instead of embedding the user prompt, I let the LLM invert it into keywords and search the embedding of that. It very much does feel like a magic bullet.

I'm sure it would be possible to fine tune a LLM like mistral to search a database or a document

Re: Build a search engine, not a vector DB

#25

Post by Grimblewald »

bambax wrote: post #17

Many, many big companies don't see any value in search. They simply use the defaults, and when those defaults are abysmal (like in the case of Confluence for example), well... they just suffer through it in silence. I have so far mostly failed in trying to explain 1/ why search matters and 2/ that not all "search" functionality are equal and that building good search is an art form.

it may still not work, but try explaining using flashy analogies. For example, the internet without search algorithms is not the economic powerhouse we know it as today, and the quality of search made companies like google the giants they are. All this is because of the enormous economic impact good search has, say a user must make just 5 searches a day, but this turns into 20 because of poor search results, resulting in re-querying in an attempt to turn up the right result, multiply that wasted time by all employees and at face value you're costing yourself an enormous amount of money as a company, not to mention the compounding loss due to workflow interruption. With a graph or two you should be able to convince most of the fact good search = massive productivity gain.

Re: Build a search engine, not a vector DB

#26

Post by hobofan »

m-i-l wrote: post #23
jankovicsandras wrote: post #7

I agree too. My impression is that almost all RAG tutorials _only_ talk about vector DBs, when these are not strictly required for Retrieval Augmented Generation. I'm guessing vector DBs are useful when you have massive amounts of documents on diverse topics. Some gotchas I experienced (but I might be using the wrong embedding/vector DB: spaCy/FAISS): - Short user questions might result a low signal query vector, e.…

> "Has somebody experience with Apache Lucene / Solr or Elasticsearch?" I've been working on a RAG with Solr, and quickly hit some of the issues you describe when dealing with real-world messy data and user input, e.g. using all-MiniLM-L6-v2 and cosine similarity, "Can you summarize Immanuel Kant's biography?" matched a chunk containing just the word "Biography" rather than one which started "Immanuel Kant, born in 1…

If you know that your search queries will be actual questions (like in the example you listed), you can possibly use the HyDE[0] to create a hypothetical answer which will usually have an embedding that's closer to the RAG chunks you are looking for.

It has the downside that an LLM (rather than just a embedding model) is used in the query path, but it has helped me multiple times in the past to strongly reduce problems with RAG like the ones you outlined, where it likes to latch onto individual words.

[0]: https://arxiv.org/abs/2212.10496

Re: Build a search engine, not a vector DB

#27

Post by hobofan »

mnd999 wrote: post #19
jankovicsandras wrote: post #7

I agree too. My impression is that almost all RAG tutorials _only_ talk about vector DBs, when these are not strictly required for Retrieval Augmented Generation. I'm guessing vector DBs are useful when you have massive amounts of documents on diverse topics. Some gotchas I experienced (but I might be using the wrong embedding/vector DB: spaCy/FAISS): - Short user questions might result a low signal query vector, e.…

Neo4j are mixing vector embeddings with knowledge graphs - https://neo4j.com/generativeai/

It's generally good etiquette around here to disclose your affiliation if you post comments that advertise the products of your employer.

Re: Build a search engine, not a vector DB

#28

Post by mnd999 »

hobofan wrote: post #27
mnd999 wrote: post #19

Earlier quoted context omitted.

Neo4j are mixing vector embeddings with knowledge graphs - https://neo4j.com/generativeai/

It's generally good etiquette around here to disclose your affiliation if you post comments that advertise the products of your employer.

My views are my own and as such I do not disclose my employment or otherwise on here.

I did think twice about posting it, as I don't usually but it's relevant and i might be helpful so why not? If you don't like it, thanks for the downvote.

Re: Build a search engine, not a vector DB

#29

Post by marginalia_nu »

bambax wrote: post #17

Many, many big companies don't see any value in search. They simply use the defaults, and when those defaults are abysmal (like in the case of Confluence for example), well... they just suffer through it in silence. I have so far mostly failed in trying to explain 1/ why search matters and 2/ that not all "search" functionality are equal and that building good search is an art form.

> I have so far mostly failed in trying to explain 1/ why search matters and 2/ that not all "search" functionality are equal and that building good search is an art form.

Yeah, it takes an absurd amount of tuning to make search work well. Given how poorly the average search field works in almost anything, it's fair to say this crucial step isn't happening.

I suspect a lot of organizations just don't have workflows that would tolerate someone spending a month tweaking search algorithm parameters. It doesn't look enough like work.

Re: Build a search engine, not a vector DB

#30

Post by Tomte »

bambax wrote: post #17

Many, many big companies don't see any value in search. They simply use the defaults, and when those defaults are abysmal (like in the case of Confluence for example), well... they just suffer through it in silence. I have so far mostly failed in trying to explain 1/ why search matters and 2/ that not all "search" functionality are equal and that building good search is an art form.

"we have no stemming support in Confluence" goes far beyond unfortunate defaults.
Post reply on HN