Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Which vector database should I use? A comparison cheatsheet

navidre.medium.com

Post reply on HN
Print view View original

71–80 of 91 posts

Re: Which vector database should I use? A comparison cheatsheet

#71

Post by huac »

vinni2 wrote: post #3
navidres wrote: post #2

I have prepared this comparison table to help me choose a vector database. I am sharing it here, hoping it may assist you in your projects as well. Main comparison points: cost at scale, compliance, and queries per second (QPS).

Wondering why you didn’t include Elasticsearch [0] in your comparison. Also having some benchmark to compare performance would help. [0] https://www.elastic.co/guide/en/elasticsearch/reference/curr...

You should test yourself on your own use case (eg vector dimension, prefiltering, throughput, target latency). In my testing, using identical HNSW configs between OS and a purpose built vector DB, I saw 10x+ better performance with the vector DB, despite much smaller CPU usage for vector DB and even including internet latency for the vector DB (but not OS).

This may not matter if you are not doing high throughput / have tight latency requirements, but in my case, it did. Of course you should weigh that versus the convenience of preexisting ES/OS clusters and so on. You can also use ES/OS together with a separate vector DB. (these tradeoffs are, of course, what make a static benchmarking post like this one so hard to think about).

Re: Which vector database should I use? A comparison cheatsheet

#72

Post by huac »

I would suggest that anyone trying a real comparison of vector DB's consider the following

    - necessary functions / use cases (eg prefiltering, dense search)
    - embeddings version management
    - anticipated embedding size (the article only considers glove-100 on ANN-benchmarks, which is quite different from openai-ada-002 1536 - both in terms of their output distribution and the vector size)
    - required precision / recall
    - required ingestion speed 
    - required ingestion throughput / time to ingest periodic updates
    - required query speed (percentiles, not average!)
    - required query throughput
    - required RBAC, data privacy, active-active, etc
...and so much more. ANN-benchmarks is a good start for thinking about this but remember that actual throughput is quite different from whatever you see in the algorithms benchmarking!

Re: Which vector database should I use? A comparison cheatsheet

#73

Post by iamspr »

shon wrote: post #36

* Shameless plug and a free ticket: Etienne Dilocker, The Co-founder/CTO of Weaviate and Ram Sriharsha, the VP of R&D at Pinecone are both presenting at The AI Conference. Lots of other smart people are presenting including Nazneen from Hugging Face, Harrison from Langchain, Jerry from Llamaindex, Ben the co-founder of Anthropic and many more. A hackathon is happening in the evening at the event as well. If you can't…

Thanks Shon! Could grab one earlier. Appreciate the opportunity and looking forward to it!

Re: Which vector database should I use? A comparison cheatsheet

#74

Post by sebastiennight »

huac wrote: post #72

I would suggest that anyone trying a real comparison of vector DB's consider the following - necessary functions / use cases (eg prefiltering, dense search) - embeddings version management - anticipated embedding size (the article only considers glove-100 on ANN-benchmarks, which is quite different from openai-ada-002 1536 - both in terms of their output distribution and the vector size) - required precision / recall…

Wouldn't "ease of putting into production" also factor in?

For many use cases, being able to put a proof of concept out of the door in hours vs days vs weeks is the top selection criterion if everything else is "good enough".

Re: Which vector database should I use? A comparison cheatsheet

#75

Post by vinni2 »

readyplayeremma wrote: post #38
vinni2 wrote: post #3

Earlier quoted context omitted.

Wondering why you didn’t include Elasticsearch [0] in your comparison. Also having some benchmark to compare performance would help. [0] https://www.elastic.co/guide/en/elasticsearch/reference/curr...

I can also add one more data-point in favor of Elastic / OpenSearch. They benefit from a long history of providing search-specific features. Including the ability to write custom re-ranking functions to combine the benefits of traditional TF/IDF style search with the modern benefit of vector search techniques. And you can easily use OpenSearch with state of the art open embedding models like SGPT that use 2048-dimens…

Does OpenSearch support Kibana? Because I haven’t found a good Kibana replacement yet.

Re: Which vector database should I use? A comparison cheatsheet

#76

Post by shon »

neilsharma wrote: post #57
shon wrote: post #36

* Shameless plug and a free ticket: Etienne Dilocker, The Co-founder/CTO of Weaviate and Ram Sriharsha, the VP of R&D at Pinecone are both presenting at The AI Conference. Lots of other smart people are presenting including Nazneen from Hugging Face, Harrison from Langchain, Jerry from Llamaindex, Ben the co-founder of Anthropic and many more. A hackathon is happening in the evening at the event as well. If you can't…

Thanks Shon! Got one a bit earlier while they were still there -- really excited for this :)

Awesome, looking forward to seeing you there!

Re: Which vector database should I use? A comparison cheatsheet

#77

Post by shon »

shon wrote: post #36

* Shameless plug and a free ticket: Etienne Dilocker, The Co-founder/CTO of Weaviate and Ram Sriharsha, the VP of R&D at Pinecone are both presenting at The AI Conference. Lots of other smart people are presenting including Nazneen from Hugging Face, Harrison from Langchain, Jerry from Llamaindex, Ben the co-founder of Anthropic and many more. A hackathon is happening in the evening at the event as well. If you can't…

downvoted to hell lol

Re: Which vector database should I use? A comparison cheatsheet

#78

Post by sv123 »

spullara wrote: post #58

Pure vector databases are a dead end. Almost every search engine (Vespa, Elastic, etc) and every database (Postgres, SQLite, Redis, etc) already has a solution for searching vectors in addition to everything else you need to query or search. If any of these vector databases become anything they will have to also implement either a full search engine or a full database.

MS desperately needs to get on this train with SQL. Maintaining and keeping a second system in sync to do vector search is painful. I've never been more jealous of people using Postgres.
Post reply on HN