Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Vector database built for scalable similarity search

milvus.io

Post reply on HN
Print view View original

91–95 of 95 posts

Re: Vector database built for scalable similarity search

#91

Post by spartee »

ttt3ts wrote: post #55
tommoor wrote: post #28

It feels like there are an influx of "vector databases" right now, I haven't had a strong answer out of anyone on why you'd be better off using these over Redis which offers vector storage with similarity search in a battle-tested OSS solution.

Redis uses too much memory and supports only two NN algorithms FLAT (very slow) and HNSW if you start indexing millions of large vectors you will quickly understand the problem. That being said, many of these DBs are overcomplicated for most use cases. Redis HNSW will work for many use cases.

Enterprise has tiered memory. We have customers in the 50-100m range. benchmarked more than that as well. Single query latency will usually be better than most of the field. Also, were working on other variants of our index that will scale to 500m +

More to come. https://github.com/redisventures to keep up with our team

Re: Vector database built for scalable similarity search

#92

Post by spartee »

tommoor wrote: post #28

It feels like there are an influx of "vector databases" right now, I haven't had a strong answer out of anyone on why you'd be better off using these over Redis which offers vector storage with similarity search in a battle-tested OSS solution.

We've seen a huge uptick in users of Redis VSS. This thread is great feedback for us so thank you.

Re: Vector database built for scalable similarity search

#93

Post by leibale1998 »

killthebuddha wrote: post #37
tommoor wrote: post #28

It feels like there are an influx of "vector databases" right now, I haven't had a strong answer out of anyone on why you'd be better off using these over Redis which offers vector storage with similarity search in a battle-tested OSS solution.

I'm currently evaluating different vector stores and passed on Redis today after spending about a half day looking into it. Here's my reasoning 1. The Node.js client is designed to be just a thin wrapper around Redis commands. The client's docs basically just point you straight at the Redis docs. 2. The `@redis/search` API is slightly different than the FS.SEARCH Redis command's api. The difference is not documented…

1. node-redis or ioredis? If you are using a "modern IDE" (VS Code/WebStorm/any IDE with support for TS language server) you should get autocomplete for all the commands (in both of them).

2. The `@redis/search` package extends `@redis/client` with support for all RediSearch commands (you should get autocomplete for all RediSearch commands as well). If you want to get the "whole in one" package (support for redis vanilla + all redis modules) you can use the `redis` package instead.

3. Can you please share the command you are trying to run?

BTW, these examples might help:

1. https://github.com/redis/node-redis/blob/master/examples/sea...

2. https://github.com/redis/node-redis/blob/master/examples/sea...

3. Checkout `redis-om` https://github.com/redis/redis-om-node/tree/main

Re: Vector database built for scalable similarity search

#94

Post by ywmario »

vp834 wrote: post #79

I am not sure why Milvus is so popular. There are other pretty good options like Weaviate and Vespa - both are open source. One important limitation of Milvus that I cannot ignore is that it does not support pre-filtering like Vespa and a few others do. Pre-filtering is critically important for use cases in e-commerce etc. where filters based on structured metadata are applied e.g. brand, category etc.

I'm very sure milvus supports it. Milvus calls it hybrid search though: https://milvus.io/docs/hybridsearch.md#Conduct-a-hybrid-vect...

Re: Vector database built for scalable similarity search

#95

Post by ywmario »

ttt3ts wrote: post #54
jhgg wrote: post #20

Earlier quoted context omitted.

I was mulling over the idea of building keyword image search (say, with CLIP based embeddings). However I'm not really sure the cost, or whether or not this is the best solution. Do you have any case studies about large deployments of this software, and what the upper limits of scale might be?

Don't start with mullivus if you're learning. Too much yak shaving. Try https://github.com/criteo/autofaiss . Also, TBH, it is a lot cheaper to run a simple faiss index.

Milvus standalone is pretty neat. I made it successfully run on the first try.
Post reply on HN