Live data from Hacker News

Choosing vector database: a side-by-side comparison

benchmark.vectorview.ai

101–110 of 125 posts

Re: Choosing vector database: a side-by-side comparison

#101

As others have correctly pointed out, to make a vector search or recommendation application requires a lot more than similarity alone. We have seen the HNSW become commoditised and the real value lies elsewhere. Just because a database has vector functionality doesn’t mean it will actually service anything beyond “hello world” type semantic search applications. IMHO these have questionable value, much like the simple…

> IMHO these have questionable value

In general I find they're incredible good for being able to rapidly build out search engines for things that would it would normally be difficult to do with plain text.

The most obvious example is code search where you can describe the function's behavior and get a match. But you could also make a searchable list of recipes that would allow a user to search something like "a hearty beef dish for a cold fall night". Or searching support tickets where full text might not match, "all the cases where users had trouble signing on".

Interestingly Q & A is ultimately a (imho fairly boring) implementation of this pattern.

The really nice part is that you can implement working demos of this projects in just a few lines of code once you have the vector db set up. Once you start thinking in terms of semantic search over text matching, you realize you can build old-Google style search engines for basically any text available to you.

One thing that is a bit odd about the space is, from what I've experienced and heard, is that setup and performance on most of this products is not all that great. Given that you can implement the demo version of a vector db in a few lines of numpy, you would hope that investing in a full vector db product we get you an easily scalable solution.

Re: Choosing vector database: a side-by-side comparison

#102
This is interesting because it does not mention Vector database powered by Apache Cassandra or the hosted serverless version DataStax Astra. Here is write up we did on 5 hard problems in Vector database and how we solved them. https://thenewstack.io/5-hard-problems-in-vector-search-and-...

In full transparency: I work for DataStatx and lead engineering for Vector database.

Re: Choosing vector database: a side-by-side comparison

#103
post #28

Let me half hijack to ask a related question: I'm building a RAG for my personal use: Say I have a lot of notes on various topics I've compiled over the years. They're scattered over a lot of text files (and org nodes). I want to be able to ask questions in a natural language and have the system query my notes and give me an answer. The approach I'm going for is to store those notes in a vector DB. When I ask my quer…

RAG in this case is essentially the same as a recommender system so you can approach it with the same metrics you would there.

You'll need to build a data set with known correct answers but then it's basically, NDCG (Normalized Discounted Cumulative Gain) is a good place to start, MRR (Mean Reciprocal Rank) and MAP (Mean Absolute Precision) are other options. You could also just look at the accuracy of getting your result in the top K results for various thresholds for k (which can be interpreted as the "probability of getting your result in 'k' results).

Re: Choosing vector database: a side-by-side comparison

#104
post #14

I'd love to know how vector databases compare in their ability to do hybrid queries, vector similarity filtered by metadata values. For example, find the 100 items with the closest cosine similarity where genre = jazz and publication date between 1990 and 2000. Can the vector index operate on a subset of records? Or when searching for 100 closest matches does the database have to find 1000 matches and then apply the…

You can totally do this in Cassandra. See https://docs.datastax.com/en/astra-serverless/docs/vector-se...

Re: Choosing vector database: a side-by-side comparison

#105
post #28

Let me half hijack to ask a related question: I'm building a RAG for my personal use: Say I have a lot of notes on various topics I've compiled over the years. They're scattered over a lot of text files (and org nodes). I want to be able to ask questions in a natural language and have the system query my notes and give me an answer. The approach I'm going for is to store those notes in a vector DB. When I ask my quer…

I just did a talk with Jerry from LlamaIndex earlier this week. https://www.youtube.com/watch?v=eLXivBehPGo

Included here is a bit of the old tried and true: NDCG/MRR/Precision @k - what you really want for measuring your information retrieval systems.

But we also talk through a bit of the "new", how to use Evals to generate the building blocks for those metrics above. You will want both hand labels and the automated Evals in the end to evaluate your system.

Re: Choosing vector database: a side-by-side comparison

#106

Earlier quoted context omitted.

The thing is if you need a vector _database_ there is no reason why it can't be a pg extensions. And if you project is only small scale there is probably some HNSW pg extension library you could use. But what is most times needed instead of a vector database is a efficient fast responsive vectore approximate KNN search system with fast attribute filtering which overlaps with a fast an efficient text search system (e.…

Why is text search so related to vector search by your opinion?

because any production use case I'm aware of sooner or later uses both searches and combined the results

e.g. vector search is fundamentally terrible at finding keywords, but keywords search is fundamentally terrible at finding equal things which use slightly different words

Re: Choosing vector database: a side-by-side comparison

#107
post #28

Let me half hijack to ask a related question: I'm building a RAG for my personal use: Say I have a lot of notes on various topics I've compiled over the years. They're scattered over a lot of text files (and org nodes). I want to be able to ask questions in a natural language and have the system query my notes and give me an answer. The approach I'm going for is to store those notes in a vector DB. When I ask my quer…

Langchain previously had a nice blogpost about how they build their RAG chatbot, maybe there are some helpful hints in there: https://blog.langchain.dev/building-chat-langchain-2/

Re: Choosing vector database: a side-by-side comparison

#108

I'll add txtai to the list: https://github.com/neuml/txtai txtai is an all-in-one embeddings database for semantic search, LLM orchestration and language model workflows. Embeddings databases are a union of vector indexes (sparse and dense), graph networks and relational databases. This enables vector search with SQL, topic modeling and retrieval augmented generation. txtai adopts a local-first approach. A production…

I know David and have used txtai extensively. txtai is awesome and if you care at all about open source, you should give it a shot!

Thank you!

Re: Choosing vector database: a side-by-side comparison

#109

Latency from embedding models is still going to be the bottleneck for performance however fast the DB is going to be. Plus adding all the overhead of synthesising answers and summaries from a LLM is going to weigh you down.

Embeddings can be precomputed. Imagine a related videos section a video sharing site. Each video's embedding is relatively static.

If you are building a search engine or a QA bot, the embedding of the query still needs to be calculated. The results do depend on the quality of the model, and if you are using a large on it does take time.

Re: Choosing vector database: a side-by-side comparison

#110
post #66

Earlier quoted context omitted.

They have beef with ES since they took the software, made a bunch of cash on it, then never contributed back. ES called them out and it started a feud. I'd go on ES over Amazon-built software any day. I worked on RDS and I've used RDS at several companies, it's a mess. Longer story: One day one of our table went missing on Aurora, we couldn't figure out why, it was in the schema, etc. Devops panicked and restarted th…

>Longer story: One day one of our table went missing on Aurora, we couldn't figure out why, it was in the schema, etc. Devops panicked and restarted the instance, and then another table was missing. We ended up creating 10 empty tables and restarted it until it hit one of those. Are there any report this? How come this is the first time I heard of this? How can companies trust this kind of managed DB services?

We worked with dedicated support on this, but I don't think they had enough knowledge to dig deep into it and just gave up. There is a huge backlog of critical issues at most AWS services. It looks great from the outside in, but the sausage making process is extremely messy.
Post reply on HN