Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Vector database built for scalable similarity search

milvus.io

Post reply on HN
Print view View original

81–90 of 95 posts

Re: Vector database built for scalable similarity search

#81

Post by ChocoluvH »

jeadie wrote: post #47
ChocoluvH wrote: post #43

Earlier quoted context omitted.

Totally agree. The thing is that ElasticSearch does not meet our requirements in vector searching. I am currently running with Milvus + ElasticSearch, works perfect. The latest Milvus version is super fast and scalable (>50M vectors). Haven't tried Zilliz Cloud. Have to find out what the cost is. I am old school. IMO ElasticSearch is only good for keyword search and these so called "vector databases" products are onl…

Didn't even realise Milvus was so lacking. https://github.com/marqo-ai/marqo also has a hybrid approach. It's just a more complete/end-to-end platform than pinecone, so it really just depends on what you're building

I personally like Milvus very much.

My point is I only trust stuff that focuses their own business. Especially for small startups.

Re: Vector database built for scalable similarity search

#82

Post by ChocoluvH »

gk1 wrote: post #30
billythemaniam wrote: post #27

Earlier quoted context omitted.

ES has support for vector search now too. Really you want both in use cases where the user expects the the top results to contain the search keywords, but also wants results that are synonyms or conceptually similar. TF/IDF and BM25 help with first part and vectors help with the second. Theoretically only vectors should be needed, but that isn't my experience in practice.

Which is why Pinecone supports hybrid search, which has shown to provide better results for out-of-domain use cases than either semantic search or keyword search alone: https://www.pinecone.io/learn/hybrid-search-intro/

IMO vector databases should not mess with ElasticSearch.

The real focus should be to improve the recall of vector search. Pity that nobody is doing real AI research here. Money wasted in marketing and branding.

Re: Vector database built for scalable similarity search

#83

Post by ChocoluvH »

hcentelles wrote: post #65
ChocoluvH wrote: post #43

Earlier quoted context omitted.

Totally agree. The thing is that ElasticSearch does not meet our requirements in vector searching. I am currently running with Milvus + ElasticSearch, works perfect. The latest Milvus version is super fast and scalable (>50M vectors). Haven't tried Zilliz Cloud. Have to find out what the cost is. I am old school. IMO ElasticSearch is only good for keyword search and these so called "vector databases" products are onl…

Could you please elaborate on how you utilize both of them together, and for which specific use case? I'm attempting to gain a better understanding of the hybrid approach.

Certainly!

The thing is to make ElasticSearch scores "comparable" to Milvus scores. Lots of ways to do this, but there's no single good solution. For example you could calculate BM25 score offline, or use TF-IDF score to do some kind of filtering. Again there's no single perfect answer. You'd have to do a lot of experiment according to your own use case and your own data to get the best results.

Also a lot of tuning needs to be done during all phases: 1) query pre-processing 2) query tokenizing 3) retrieval 4) ranking and reranking

I personally would not trust any universal "hybird-search" solutions. All toy demos.

It usually takes 5-10 good engineers to build a decent search engine/system for any real use case. It also requires a lot of turning, tricks, hand-written rules to make things work.

Re: Vector database built for scalable similarity search

#84

Post by ChocoluvH »

andre-z wrote: post #66
adobrawy wrote: post #40

Earlier quoted context omitted.

In small teams, the infrastructure is often not able to be fully utilized, so performance is not an issue. However, feature richness allows this team to deliver higher-level feature faster. Think early stage startup (one or two engineers) or hairdressers-like business (they use a ready-made framework that targets a popular database and limits its feature to have a wide range of users). As a result, you can have a lot…

For small startups is better just to utilize a managed solution like Pinecone or Qdrant and do not take about infra at all.

Open source software nowadays are very easy to use.

If your guy couldn't get a single open source software straight, you had the wrong guy :(

I can only see managed service useful when I had 100X traffic and when strong SLA is required.

Re: Vector database built for scalable similarity search

#85

Post by kacperlukawski »

fzliu wrote: post #14
m1117 wrote: post #5

Ah I see, they released Milvus 2.0 which is similar to Pinecone. What are the differences?

Milvus is completely open source ( https://github.com/milvus-io/milvus ) and supports a variety of index types ( https://milvus.io/docs/overview.md#Index-types ) and support various consistency levels, scalar/metadata filtering, and time travel. We started working on Milvus back in 2018, with 2.0 being released in January 2022 ( https://github.com/milvus-io/milvus/releases/tag/v2.0.0 ). For those interested, here's a…

I'm affiliated with Qdrant and was quite surprised to see us listed in your comparisons with some false statements. If you claim to be the only database with billion-scale vector support, it would be great to make your benchmarks public, as we did: https://qdrant.tech/benchmarks/

Btw, Milvus is described in your comparisons as "a fully open source and independent project", while Weaviate and Qdrant, in contrary, are "maintained by a single commercial company offering a cloud version". Why then the suggested way in the Milvus Quick Start on github is to use Zilliz Cloud?

Re: Vector database built for scalable similarity search

#86

Post by redskyluan »

kacperlukawski wrote: post #85
fzliu wrote: post #14

Earlier quoted context omitted.

Milvus is completely open source ( https://github.com/milvus-io/milvus ) and supports a variety of index types ( https://milvus.io/docs/overview.md#Index-types ) and support various consistency levels, scalar/metadata filtering, and time travel. We started working on Milvus back in 2018, with 2.0 being released in January 2022 ( https://github.com/milvus-io/milvus/releases/tag/v2.0.0 ). For those interested, here's a…

I'm affiliated with Qdrant and was quite surprised to see us listed in your comparisons with some false statements. If you claim to be the only database with billion-scale vector support, it would be great to make your benchmarks public, as we did: https://qdrant.tech/benchmarks/ Btw, Milvus is described in your comparisons as "a fully open source and independent project", while Weaviate and Qdrant, in contrary, are…

would you please update the milvus score in your benchmark with the latest milvus, thanks

Re: Vector database built for scalable similarity search

#87

Post by simplify »

killthebuddha wrote: post #37
tommoor wrote: post #28

It feels like there are an influx of "vector databases" right now, I haven't had a strong answer out of anyone on why you'd be better off using these over Redis which offers vector storage with similarity search in a battle-tested OSS solution.

I'm currently evaluating different vector stores and passed on Redis today after spending about a half day looking into it. Here's my reasoning 1. The Node.js client is designed to be just a thin wrapper around Redis commands. The client's docs basically just point you straight at the Redis docs. 2. The `@redis/search` API is slightly different than the FS.SEARCH Redis command's api. The difference is not documented…

Not trying to say you're wrong in your decision, but in the context of the post you're replying to, aren't these points arguments against the node client rather than redis itself?

Re: Vector database built for scalable similarity search

#88

Post by ChocoluvH »

totalhack wrote: post #64

Tried Milvus first and never really got it off the ground. Ended up with Qdrant for its simplicity.

Don't start with Milvus clustered version, not unless you have like 100million vectors.

Try Milvus standalone instead, much simpler. I also just found their python version (https://github.com/milvus-io/embd-milvus), which is quite neat.

Re: Vector database built for scalable similarity search

#89

Post by winddude »

ttt3ts wrote: post #55
tommoor wrote: post #28

It feels like there are an influx of "vector databases" right now, I haven't had a strong answer out of anyone on why you'd be better off using these over Redis which offers vector storage with similarity search in a battle-tested OSS solution.

Redis uses too much memory and supports only two NN algorithms FLAT (very slow) and HNSW if you start indexing millions of large vectors you will quickly understand the problem. That being said, many of these DBs are overcomplicated for most use cases. Redis HNSW will work for many use cases.

it uses a lot of memory because it's an in memory db, that's by design.

Re: Vector database built for scalable similarity search

#90

Post by asmyers1793 »

jamesblonde wrote: post #45
mikeortman wrote: post #34

Earlier quoted context omitted.

Pinecone is closed-source AND hosted-infra only... which is a non-starter for many companies.

Pinecone offers a free managed tier, which was quite nice until it lost my data last month. They did eventually recover it a few days later, to be fair to them.

Check out trychroma.com
Post reply on HN