Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Vector database built for scalable similarity search

milvus.io

Post reply on HN
Print view View original

61–70 of 95 posts

Re: Vector database built for scalable similarity search

#61

Post by qwick23 »

killthebuddha wrote: post #41
qwick23 wrote: post #39

Earlier quoted context omitted.

What do you think about Weaviate or Qdrant? There is a nice benchmarks overview with all the major players https://qdrant.tech/benchmarks/

I haven't really looked into either. So far I've tested pinecone, Redis, chroma, elasticsearch, and pgvector. I'm not really considering performance at all, just looking for something dead-simple to deploy and use. At the moment it looks like pgvector on supabase is the winner.

Yeah, if you don't need performance, then just take a simple ANN library. No need for a database at all. In terms of databases, Qdrant and Pinecone are the simplest I've tried so far. But Pinecone isn't open-source, not an option for on-prem. PGvector is too much imho, do not want to have all the other Sequel stuff if I just need an NN search.

Re: Vector database built for scalable similarity search

#62

Post by elbartus »

qwick23 wrote: post #39
killthebuddha wrote: post #37

Earlier quoted context omitted.

I'm currently evaluating different vector stores and passed on Redis today after spending about a half day looking into it. Here's my reasoning 1. The Node.js client is designed to be just a thin wrapper around Redis commands. The client's docs basically just point you straight at the Redis docs. 2. The `@redis/search` API is slightly different than the FS.SEARCH Redis command's api. The difference is not documented…

What do you think about Weaviate or Qdrant? There is a nice benchmarks overview with all the major players https://qdrant.tech/benchmarks/

Qdrant v1.1 was released recently and its quantization feature is just fantastic . See: https://github.com/qdrant/qdrant

Re: Vector database built for scalable similarity search

#63

Post by leobg »

elbigbad wrote: post #42

I haven’t tried many alternatives, but I needed a fast, self-hosted vector similarity search that had the ability to cull results based on a second criterion. Milvus has worked really well for me. It does take a ton of memory though such that I can’t run on small VMs, and runs a large number of supporting services.

hnswlib? Local. Single file. Allows pre-filtering.

Re: Vector database built for scalable similarity search

#65

Post by hcentelles »

ChocoluvH wrote: post #43
billythemaniam wrote: post #27

Earlier quoted context omitted.

ES has support for vector search now too. Really you want both in use cases where the user expects the the top results to contain the search keywords, but also wants results that are synonyms or conceptually similar. TF/IDF and BM25 help with first part and vectors help with the second. Theoretically only vectors should be needed, but that isn't my experience in practice.

Totally agree. The thing is that ElasticSearch does not meet our requirements in vector searching. I am currently running with Milvus + ElasticSearch, works perfect. The latest Milvus version is super fast and scalable (>50M vectors). Haven't tried Zilliz Cloud. Have to find out what the cost is. I am old school. IMO ElasticSearch is only good for keyword search and these so called "vector databases" products are onl…

Could you please elaborate on how you utilize both of them together, and for which specific use case? I'm attempting to gain a better understanding of the hybrid approach.

Re: Vector database built for scalable similarity search

#66

Post by andre-z »

adobrawy wrote: post #40
qwick23 wrote: post #36

Earlier quoted context omitted.

With Postgres, you can do almost everything, also a full-text search, but you still have Elasticsearch, Mejlisearch, etc when you need performance and advanced features. The multitool approach is suboptimal in most cases.

In small teams, the infrastructure is often not able to be fully utilized, so performance is not an issue. However, feature richness allows this team to deliver higher-level feature faster. Think early stage startup (one or two engineers) or hairdressers-like business (they use a ready-made framework that targets a popular database and limits its feature to have a wide range of users). As a result, you can have a lot…

For small startups is better just to utilize a managed solution like Pinecone or Qdrant and do not take about infra at all.

Re: Vector database built for scalable similarity search

#67

Post by billythemaniam »

ChocoluvH wrote: post #43
billythemaniam wrote: post #27

Earlier quoted context omitted.

ES has support for vector search now too. Really you want both in use cases where the user expects the the top results to contain the search keywords, but also wants results that are synonyms or conceptually similar. TF/IDF and BM25 help with first part and vectors help with the second. Theoretically only vectors should be needed, but that isn't my experience in practice.

Totally agree. The thing is that ElasticSearch does not meet our requirements in vector searching. I am currently running with Milvus + ElasticSearch, works perfect. The latest Milvus version is super fast and scalable (>50M vectors). Haven't tried Zilliz Cloud. Have to find out what the cost is. I am old school. IMO ElasticSearch is only good for keyword search and these so called "vector databases" products are onl…

If ES doesn't work for you, I recommend Vespa. https://github.com/vespa-engine/vespa

Others have made other suggestions, but Vespa has two unique features. First it is battle tested at a large scale, second it supports combining the keyword and vector scores in several ways. The latter is something that other hybrid systems don't do very well in my experience.

Re: Vector database built for scalable similarity search

#70

Post by anonymousDan »

ttt3ts wrote: post #52

Learn faiss first. Mullvus is very complicated in comparison and faiss will work in many cases. I mean look at their system diagram https://milvus.io/static/0bc2e74d0a1b20bbfb91bdbd03f77e5e/bb... . Not fun...

Looks pretty standard to me for a high availability cloud setup with disaggregated storage?
Post reply on HN