Didn't Milvus (vector db, wrapper around FAISS) come before Pinecone?
Vector search just got up to 10x faster and vertically scalable
21–30 of 54 posts
Re: Vector search just got up to 10x faster and vertically scalable
#22Earlier quoted context omitted.
Not that easy, the founder was a director at AWS. This is just devops/obfuscation on top of an open source library: FAISS
Pinecone doesn’t use Faiss, nor ScaNN. We love Faiss and even teach people to use it[1]. There happens to be a sizable population of engineers who need more than what Faiss provides (like live index updates and metadata filtering, for example), and can’t be bothered or aren’t being paid to customize and manage open-source libraries all day. [1] https://www.pinecone.io/learn/faiss/
So you guys developed and implemented state of the art neural network vector search from scratch? in a year? and something better than libraries with tens of contributors over years of research?
Re: Vector search just got up to 10x faster and vertically scalable
#23I don't understand the emphasis here on vertical scaling. Move a database to a bigger machine = more storage and faster querying. Not exactly rocket science. Horizontal scaling is the real challenge here, and the complexity of vector indexes makes it especially challenging. Milvus and Vertex AI both have horizontal scaling ANN search and the ability to do parallel indexing as well. I appreciate the post but this does…
Bigger machine doesn't automatically mean higher performance. The code needs to scale with the increased number of cores, has share-nothing or share-very-little approach to avoid contention, and uses efficient data structure to utilize the increased memory.
Re: Vector search just got up to 10x faster and vertically scalable
#24Didn't Milvus (vector db, wrapper around FAISS) come before Pinecone?
Just to clarify, Milvus is much more than a wrapper around FAISS. Our vector search component called Knowhere (https://github.com/milvus-io/knowhere) utilizes FAISS and Annoy and will soon include ScaNN, DiskANN, and in-house vector indexes as well. Milvus uses Knowhere as the compute engine, and implements a variety of database functions such as horizontal scaling, caching, replication, failover, and object storage on top of Knowhere. If you're interested, I recommend checking out our architecture page (https://milvus.io/docs/architecture_overview.md).
[EDIT]: Forgot to mention - Milvus development began in 2018 was open sourced in 2019.
Re: Vector search just got up to 10x faster and vertically scalable
#25Earlier quoted context omitted.
If you really think that's enough to build a real product, go for it. Even open-source companies (Elastic, Mongo, Scylla) have to build tons of infra around their core codebase in order to make it an actual cloud product.
Not that easy, the founder was a director at AWS. This is just devops/obfuscation on top of an open source library: FAISS
I actually built a similar solution supporting similar operations (including filtering by meta-data) using open-source libraries. Took me about 2 weeks net.
I can see a clientele for such database (people who want a turnkey solution), but honestly it looks like an attempt to use a dev-ops solution to address deeper issues with problem formulation: e.g.
1. Is there really a need to search all items in the database? can subsampling make simple similarity comparison feasible?
2. Do the embeddings really need to have that many dimensions? Can we reduce their dimensionality and fit them in RAM?
3. Is embedding accurate enough compared to pairwise comparison? Can we formulate the problem to make the latter feasible?
I also could not find any explanation of the underlying algorithms, especially around meta-data filtering, which is not solved by FAISS as well as their accuracy. (happy to hear otherwise)
Re: Vector search just got up to 10x faster and vertically scalable
#26Earlier quoted context omitted.
Which open source libraries is pinecone wrapping?
I’m not sure where the other commenter gets their confidence, but Pinecone is not wrapping any open source vector-search library. We offer three index types (in-memory, in-memory graph-based, hybrid memory + disk), and all are proprietary. We do have articles about Faiss and HNSW and all sorts of other vector-search and NLP topics, so it’s possible that’s where the confusion comes from.
so, how does your proprietary solution compare against FAISS, eg with 10M dense vectors of 1024 dimensions?
Re: Vector search just got up to 10x faster and vertically scalable
#27a custom datatype or fdw in postgresql seems interesting to me.
Re: Vector search just got up to 10x faster and vertically scalable
#28Earlier quoted context omitted.
Pinecone doesn’t use Faiss, nor ScaNN. We love Faiss and even teach people to use it[1]. There happens to be a sizable population of engineers who need more than what Faiss provides (like live index updates and metadata filtering, for example), and can’t be bothered or aren’t being paid to customize and manage open-source libraries all day. [1] https://www.pinecone.io/learn/faiss/
So you guys developed and implemented state of the art neural network vector search from scratch? in a year? and something better than libraries with tens of contributors over years of research?
Most vector search research teams are a lot smaller than you suggest, and haven't been around that long (e.g. the FAISS paper was published in 2017).
From public info, you can see they have at least one researcher working there. It's believable to me that they could have some new innovations, especially since the product space they're focusing on is different from other teams working on vector search. State-of-the-art for a specific set of constraints is still state-of-the-art.
However, considering how much of their edu-marketing content is posted to HN, it would be great if they could share more details about the internals of their index with the community. One of the great things about vector search is how many techniques are open sourced or documented in papers :).
Disclaimer: I work on vector search at a different company
Re: Vector search just got up to 10x faster and vertically scalable
#29Earlier quoted context omitted.
I’m not sure where the other commenter gets their confidence, but Pinecone is not wrapping any open source vector-search library. We offer three index types (in-memory, in-memory graph-based, hybrid memory + disk), and all are proprietary. We do have articles about Faiss and HNSW and all sorts of other vector-search and NLP topics, so it’s possible that’s where the confusion comes from.
so, how does your proprietary solution compare against FAISS, eg with 10M dense vectors of 1024 dimensions?
They used to publish some benchmarks on their site, but seem to have removed them. You can find them on archive.org[1]. I guess it is understandable, since vector search performance is pretty unpredictable, and depends on a lot of factors. If their target market is people who want vector search without needing to read a bunch of papers first, benchmarks might be more confusing than they are helpful.
edit: While I do think it's understandable, it's not great for transparency. Even if they don't want to open-source their index, I would admire it if they were willing to give ann-benchmarks[2] an API key to publish some independent results.
Disclaimer: I work on vector search at a different company
[1] https://web.archive.org/web/20210227105542/https://www.pinec... [2] https://github.com/erikbern/ann-benchmarks
Re: Vector search just got up to 10x faster and vertically scalable
#30I don't understand the emphasis here on vertical scaling. Move a database to a bigger machine = more storage and faster querying. Not exactly rocket science. Horizontal scaling is the real challenge here, and the complexity of vector indexes makes it especially challenging. Milvus and Vertex AI both have horizontal scaling ANN search and the ability to do parallel indexing as well. I appreciate the post but this does…
Bigger machine doesn't automatically mean higher performance. The code needs to scale with the increased number of cores, has share-nothing or share-very-little approach to avoid contention, and uses efficient data structure to utilize the increased memory.
Why would search queries have contention with each other? Surely it's in the domain of embarrassingly parallel.