Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Vector database built for scalable similarity search

milvus.io

Post reply on HN
Print view View original

41–50 of 95 posts

Re: Vector database built for scalable similarity search

#41

Post by killthebuddha »

qwick23 wrote: post #39
killthebuddha wrote: post #37

Earlier quoted context omitted.

I'm currently evaluating different vector stores and passed on Redis today after spending about a half day looking into it. Here's my reasoning 1. The Node.js client is designed to be just a thin wrapper around Redis commands. The client's docs basically just point you straight at the Redis docs. 2. The `@redis/search` API is slightly different than the FS.SEARCH Redis command's api. The difference is not documented…

What do you think about Weaviate or Qdrant? There is a nice benchmarks overview with all the major players https://qdrant.tech/benchmarks/

I haven't really looked into either. So far I've tested pinecone, Redis, chroma, elasticsearch, and pgvector. I'm not really considering performance at all, just looking for something dead-simple to deploy and use. At the moment it looks like pgvector on supabase is the winner.

Re: Vector database built for scalable similarity search

#42

Post by elbigbad »

I haven’t tried many alternatives, but I needed a fast, self-hosted vector similarity search that had the ability to cull results based on a second criterion. Milvus has worked really well for me. It does take a ton of memory though such that I can’t run on small VMs, and runs a large number of supporting services.

Re: Vector database built for scalable similarity search

#43

Post by ChocoluvH »

billythemaniam wrote: post #27
simplify wrote: post #19

I only heard about vector databases along with the recent advents of AI. Assuming they've been around for a while, what were the benefits of using them over "normal" search engines (e.g. ElasticSearch)?

ES has support for vector search now too. Really you want both in use cases where the user expects the the top results to contain the search keywords, but also wants results that are synonyms or conceptually similar. TF/IDF and BM25 help with first part and vectors help with the second. Theoretically only vectors should be needed, but that isn't my experience in practice.

Totally agree. The thing is that ElasticSearch does not meet our requirements in vector searching.

I am currently running with Milvus + ElasticSearch, works perfect. The latest Milvus version is super fast and scalable (>50M vectors). Haven't tried Zilliz Cloud. Have to find out what the cost is.

I am old school. IMO ElasticSearch is only good for keyword search and these so called "vector databases" products are only good for vector search.

Re: Vector database built for scalable similarity search

#45

Post by jamesblonde »

mikeortman wrote: post #34
gavi wrote: post #26

How does this compare to pinecone?

Pinecone is closed-source AND hosted-infra only... which is a non-starter for many companies.

Pinecone offers a free managed tier, which was quite nice until it lost my data last month. They did eventually recover it a few days later, to be fair to them.

Re: Vector database built for scalable similarity search

#46

Post by jamesblonde »

simplify wrote: post #19

I only heard about vector databases along with the recent advents of AI. Assuming they've been around for a while, what were the benefits of using them over "normal" search engines (e.g. ElasticSearch)?

OpenSearch (elasticsearch open-source fork by AWS) has supported similarity search for embeddings for a couple of years now with its k-NN plugin. It supports 2 engines - FAISS and HNSW - and has post-result filtering support and replication support. A good option, imo.

Re: Vector database built for scalable similarity search

#47

Post by jeadie »

ChocoluvH wrote: post #43
billythemaniam wrote: post #27

Earlier quoted context omitted.

ES has support for vector search now too. Really you want both in use cases where the user expects the the top results to contain the search keywords, but also wants results that are synonyms or conceptually similar. TF/IDF and BM25 help with first part and vectors help with the second. Theoretically only vectors should be needed, but that isn't my experience in practice.

Totally agree. The thing is that ElasticSearch does not meet our requirements in vector searching. I am currently running with Milvus + ElasticSearch, works perfect. The latest Milvus version is super fast and scalable (>50M vectors). Haven't tried Zilliz Cloud. Have to find out what the cost is. I am old school. IMO ElasticSearch is only good for keyword search and these so called "vector databases" products are onl…

Didn't even realise Milvus was so lacking. https://github.com/marqo-ai/marqo also has a hybrid approach. It's just a more complete/end-to-end platform than pinecone, so it really just depends on what you're building

Re: Vector database built for scalable similarity search

#48

Post by Seattle3503 »

dandiep wrote: post #25

I really don't want another database. I just want to have a solution built in for Postgres, and more specifically, RDS, which we use. I know there will be some extra difficulty that I will have to manage (e.g. reindexing to a new model that is outputting different embeddings), but I really don't want another piece of infrastructure. If anyone from AWS/Google/Azure is listening, please add pgvector [1] into your manag…

I'm with you there. It seems like an extension to existing DBs would be better. I would like something like this for a file based DB like sqlite.

Re: Vector database built for scalable similarity search

#49

Post by bottlepalm »

killthebuddha wrote: post #38
dandiep wrote: post #25

I really don't want another database. I just want to have a solution built in for Postgres, and more specifically, RDS, which we use. I know there will be some extra difficulty that I will have to manage (e.g. reindexing to a new model that is outputting different embeddings), but I really don't want another piece of infrastructure. If anyone from AWS/Google/Azure is listening, please add pgvector [1] into your manag…

FWIW supabase recently added support: https://supabase.com/docs/guides/database/extensions/pgvecto...

Very cool, thanks.

Re: Vector database built for scalable similarity search

#50

Post by simonw »

Seattle3503 wrote: post #48
dandiep wrote: post #25

I really don't want another database. I just want to have a solution built in for Postgres, and more specifically, RDS, which we use. I know there will be some extra difficulty that I will have to manage (e.g. reindexing to a new model that is outputting different embeddings), but I really don't want another piece of infrastructure. If anyone from AWS/Google/Azure is listening, please add pgvector [1] into your manag…

I'm with you there. It seems like an extension to existing DBs would be better. I would like something like this for a file based DB like sqlite.

sqlite-vss is an extension that adds vector search to SQLite: https://observablehq.com/@asg017/introducing-sqlite-vss
Post reply on HN