Data Skeptic podcast is currently doing a series on K-means applications (similar to KNN). Would recommend it for anyone exploring the topic:
https://dataskeptic.com/episodes/k-means
K-Nearest Neighbors
21–30 of 38 posts
Re: K-Nearest Neighbors
#22Would HN benefit from some kind of "self promotion" or "marketing" tag?
I think it’s fine as long as the content is interesting. It gets flagged to death pretty quickly if it is not.
Re: K-Nearest Neighbors
#23I wonder how their advertised "vector database" works. kNN combined with embeddings from pre-trained deep learning models can be very useful for information retrieval, (e.g. searching for duplicate/similar images or text). In the past I have used a k-d tree [1] for this, which allows O(log n) searches in the vector space. It seems they are offering a k-d-tree-as-a-service. [1] https://en.wikipedia.org/wiki/K-d_tree
Pinecone stores and searches through dense vector embeddings using a proprietary ANN index. It also has live index updates and metadata filtering, which you’d expect from any database but is surprisingly hard to find or do with vector indexes.
As you said, common use cases include deduplication and image search, and especially semantic search (text).
Re: K-Nearest Neighbors
#24I wonder how their advertised "vector database" works. kNN combined with embeddings from pre-trained deep learning models can be very useful for information retrieval, (e.g. searching for duplicate/similar images or text). In the past I have used a k-d tree [1] for this, which allows O(log n) searches in the vector space. It seems they are offering a k-d-tree-as-a-service. [1] https://en.wikipedia.org/wiki/K-d_tree
Pinecone stores and searches through dense vector embeddings using a proprietary ANN index. It also has live index updates and metadata filtering, which you’d expect from any database but is surprisingly hard to find or do with vector indexes. As you said, common use cases include deduplication and image search, and especially semantic search (text).
Do you happen to know other implementations that allow for live updates and metadata filtering like Pinecone?
Re: K-Nearest Neighbors
#25I wonder how their advertised "vector database" works. kNN combined with embeddings from pre-trained deep learning models can be very useful for information retrieval, (e.g. searching for duplicate/similar images or text). In the past I have used a k-d tree [1] for this, which allows O(log n) searches in the vector space. It seems they are offering a k-d-tree-as-a-service. [1] https://en.wikipedia.org/wiki/K-d_tree
> kNN combined with embeddings from pre-trained deep learning models can be very useful for information retrieval Indeed! We've been able to build simple reverse image search apps and other solutions using the power of embeddings from pre-trained ML models: https://gist.github.com/fzliu/c9380a7f9ba411adeff0b727cdba15... . One quick note: k-d trees are great for indexing low-dimensional data, but for high-dimensional…
For image retrieval, have you tried using a model trained with contrastive learning (e.g. SimCLR)? This could produce better embeddings for retrieval since the model is trained to explicitly minimize euclidean distance between similar pairs.
Thanks for the reference! Nice outline of various ANN approaches.
Re: K-Nearest Neighbors
#26The infamous kNN algorithm
Re: K-Nearest Neighbors
#27Once you find the N nearest neighbors, why consider only the 0th order (locally constant) model, which is what KNN assumes? Instead you can use a larger N and fit a local linear regression. I have tried this on synthetic data sets, and it works pretty well. This is common for 1D regression but not regression with multiple predictors.
Once you do that you have something that starts looking like a Gaussian process but is a lot cheaper. I believe it could have a lot of value (for people who have too much data for Gaussian process but otherwise could do with their good properties and are in a space with an easily defined distance).
Re: K-Nearest Neighbors
#28Why would somebody use pinecone instead of a high quality open source library like FAISS？
Re: K-Nearest Neighbors
#29Earlier quoted context omitted.
> kNN combined with embeddings from pre-trained deep learning models can be very useful for information retrieval Indeed! We've been able to build simple reverse image search apps and other solutions using the power of embeddings from pre-trained ML models: https://gist.github.com/fzliu/c9380a7f9ba411adeff0b727cdba15... . One quick note: k-d trees are great for indexing low-dimensional data, but for high-dimensional…
For image retrieval, have you tried using a model trained with contrastive learning (e.g. SimCLR)? This could produce better embeddings for retrieval since the model is trained to explicitly minimize euclidean distance between similar pairs. Thanks for the reference! Nice outline of various ANN approaches.
I haven't tried SimCLR, but I did try face embedding models trained with contrastive and triplet loss. For applications where precision is the key metric, I do agree that these loss functions are much better overall.
If discovery or recall is what you're after, a generic image classification model trained with binary cross-entropy might be better. For example, performing reverse image search on a photo of a German Shepherd should always return images of GSheps in the first N pages, but showing other dog breeds in later pages and possibly even cats after that would be a desirable feature for many search/retrieval solutions. An embedding model trained with contrastive loss might have this behavior to a certain extent, but a model based on BCE should be better.
Re: K-Nearest Neighbors
#30Why would somebody use pinecone instead of a high quality open source library like FAISS？
* CRUD: Faiss indexes are static, if you want vector search over data that users can edit freely, it's not easy to use FAISS for this
* Non-vector filtering: Often you need to do a search based on some structured query + rank the results by vector similarity. FAISS can only do vector similarity.