Show HN: Embeddinghub: A vector database built for Machine Learning embeddings • Hacker News Board

Re: Show HN: Embeddinghub: A vector database built for Machine Learning embeddings

#31

Post by kevin948 » Thu, Sep 16, 2021, 9:12 PM UTC

This is really great! It speaks very much to my use-case (building user embeddings and serving them both to analysts + other ML models).

I was wondering if there was a reasonable way to store raw data next to the embeddings such that: 1. Analysts can run queries to filter down to a space they understand (the raw data). 2. Nearest neighbors can be run on top of their selection on the embedding space.

Our main use case is segmentation, so giving analysts access to the raw feature space is very important.

Re: Show HN: Embeddinghub: A vector database built for Machine Learning embeddings

#32

Post by cyrusthegreat » Thu, Sep 16, 2021, 10:14 PM UTC

kevin948 wrote: post #31

This is really great! It speaks very much to my use-case (building user embeddings and serving them both to analysts + other ML models). I was wondering if there was a reasonable way to store raw data next to the embeddings such that: 1. Analysts can run queries to filter down to a space they understand (the raw data). 2. Nearest neighbors can be run on top of their selection on the embedding space. Our main use case…

This is in the works! We'd love you feedback on the API and to learn a bit more about your use-case so we build the right thing, mind joining our slack? https://join.slack.com/t/featureform-community/shared_invite...

Re: Show HN: Embeddinghub: A vector database built for Machine Learning embeddings

#33

Post by cyrusthegreat » Thu, Sep 16, 2021, 10:17 PM UTC

barefeg wrote: post #28

Where can I find documentation on versioning? My first use case would be to versión different embeddings and use it more like a storage backend than to search for KNN. Would it be possible to not create the NN graph and just use it for versioned storage? We currently use opendistro and it nicely allows doing pre and post filtering based on other document fields (other than the embedding). Therefore I think this could…

Hey! We're actually polishing up a PR that'll add documentation and finalize the versioning API, it should be merged in this weekend. Would you be up for a quick chat with someone on our team? It would be interesting to get your feedback and see what else we're missing to be a drop-in replacement to opendistro, join our slack if so. We'll dm you :) https://join.slack.com/t/featureform-community/shared_invite...

Re: Show HN: Embeddinghub: A vector database built for Machine Learning embeddings

#34

Post by elephantum » Fri, Sep 17, 2021, 7:56 AM UTC

Nice, are there any benchmarks?

Would be interesting to see how it compares to Postgres or LevelDB for read/write of exact values

And how it compares to Faiss/Annoy for KNN

Re: Show HN: Embeddinghub: A vector database built for Machine Learning embeddings

#35

Post by ymroddi » Mon, Sep 20, 2021, 3:07 AM UTC

cyrusthegreat wrote: post #17

planetsprite wrote: post #15

What makes this different from something like gensim? They have vector search for doc2vec embeddings.

Gensim is great for generating certain types of embeddings, but not for operationalizing them. It doesn’t do approximate nearest neighbor lookup which is a deal breaker for most models that use embeddings at scale. It also do not manage versioning so you end up having to hack a workflow around it to manage embedding. Finally, it’s not really data infrastructure like this is, so you end up doing hacky things like copy…

gensim actually allows you to use both annoy and nmslib with gensim generated vectors as part of the api.

https://radimrehurek.com/gensim/similarities/nmslib.html

https://radimrehurek.com/gensim/similarities/annoy.html