Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Embeddinghub: A vector database built for Machine Learning embeddings

github.com

Post reply on HN
Print view View original

11–20 of 35 posts

Re: Show HN: Embeddinghub: A vector database built for Machine Learning embeddings

#11

Post by cyrusthegreat »

sathergate wrote: post #8
cyrusthegreat wrote: post #5

Earlier quoted context omitted.

We use HNSW internally via HNSWLIB, it's the same algorithm that Facebook uses to power their embedding search.

thanks! how did you make the decision to use hnsw over faiss and other search algorithms?

Faiss actually also uses HNSW internally, HNSWLIB is just a lighter weight implementation which allowed us to iterate faster. In the future we will switch it back out for FAISS to take advantage of its full array of functionality.

Re: Show HN: Embeddinghub: A vector database built for Machine Learning embeddings

#12

Post by cyrusthegreat »

nelsondev wrote: post #9

Cool! Nice work! Do you have any performance numbers you could share? Specifically around nearest neighbor computation latency, a regular get embedding latency, read/write rate achieved on a machine?

Not yet, this is very much an early release to get it in people's hands and to get feedback on the API and the functionality. We've purposely held off optimizing too much until we feel more confident that this is useful and our API approach makes sense for people. That said, Simba who's one of the main devs actually comes from a performance tuning background at Google. Also, it's built on HNSWLIB and RocksDB, and is being used in real world workloads today.

Re: Show HN: Embeddinghub: A vector database built for Machine Learning embeddings

#13

Post by cyrusthegreat »

localhost wrote: post #7
cyrusthegreat wrote: post #2

Hi everyone! Over the years, I've found myself building hacky solutions to serve and manage my embeddings. I’m excited to share Embeddinghub, an open-source vector database for ML embeddings. It is built with four goals in mind: Store embeddings durably and with high availability Allow for approximate nearest neighbor operations Enable other operations like partitioning, sub-indices, and averaging Manage versioning,…

Curious about how your solution is different / better than nmslib which I've tried in the past?

We actually use HNSWLIB by NMSLIB on the backend. NMSLIB is solving the approximate nearest neighbor problem, not the storage problem. It’s not a database, it’s an index. We handle everything needed to turn their index into a full fledged database with a data science workflow around it (versioning, monitoring, etc.)

Re: Show HN: Embeddinghub: A vector database built for Machine Learning embeddings

#14

Post by cyrusthegreat »

deploy wrote: post #10

This looks awesome - psyched to try! Embeddings are a bitch, nice to see some new tools for managing them :)

Thanks for the kind words! We'd love to get your feedback as we iterate. Please join our slack community: https://join.slack.com/t/featureform-community/shared_invite...

Re: Show HN: Embeddinghub: A vector database built for Machine Learning embeddings

#16

Post by tourist_on_road »

Great work! Looks like you are using HNSWLIB. From what I understand HNSW graph based approach can be memory intensive compared PQ code based approach. FAISS has support for both HNSW and PQ codes. Any plans on extending your work to support PQ code based index in future?

Re: Show HN: Embeddinghub: A vector database built for Machine Learning embeddings

#17

Post by cyrusthegreat »

planetsprite wrote: post #15

What makes this different from something like gensim? They have vector search for doc2vec embeddings.

Gensim is great for generating certain types of embeddings, but not for operationalizing them. It doesn’t do approximate nearest neighbor lookup which is a deal breaker for most models that use embeddings at scale. It also do not manage versioning so you end up having to hack a workflow around it to manage embedding. Finally, it’s not really data infrastructure like this is, so you end up doing hacky things like copying all your embeddings to every docker file. With regards to serving embeddings, gensim is just a library that supports in-memory brute force nearest neighbour look ups.

Re: Show HN: Embeddinghub: A vector database built for Machine Learning embeddings

#19

Post by ypcx »

cyrusthegreat wrote: post #2

Hi everyone! Over the years, I've found myself building hacky solutions to serve and manage my embeddings. I’m excited to share Embeddinghub, an open-source vector database for ML embeddings. It is built with four goals in mind: Store embeddings durably and with high availability Allow for approximate nearest neighbor operations Enable other operations like partitioning, sub-indices, and averaging Manage versioning,…

In the "Definitive Guide to Embeddings", in the figure "An illustration of One Hot Encoding", the "One Hot Encoding" table doesn't make any sense whatsoever. Am I wrong?

Re: Show HN: Embeddinghub: A vector database built for Machine Learning embeddings

#20

Post by make3 »

ypcx wrote: post #19
cyrusthegreat wrote: post #2

Hi everyone! Over the years, I've found myself building hacky solutions to serve and manage my embeddings. I’m excited to share Embeddinghub, an open-source vector database for ML embeddings. It is built with four goals in mind: Store embeddings durably and with high availability Allow for approximate nearest neighbor operations Enable other operations like partitioning, sub-indices, and averaging Manage versioning,…

In the "Definitive Guide to Embeddings", in the figure "An illustration of One Hot Encoding", the "One Hot Encoding" table doesn't make any sense whatsoever. Am I wrong?

no you're right ahahah wth are these
Post reply on HN