Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Introduction to vector similarity search (2022)

zilliz.com

Post reply on HN
Print view View original

11–20 of 57 posts

Re: Introduction to vector similarity search (2022)

#11

Post by fzliu »

behnamoh wrote: post #2

Can anyone please suggest a good stack for the following: - calculating text embeddings using open-source/local methods (not OpenAI) - storing them in a vector database. I'm confused by the myriad of options like Chromadb, Pinecone, etc. - running vector similarity search using open-source/local methods. Also, how granular should the text chunks be? Too short and we'll end up with a huge database, too long and we'll…

If you're just starting out, I'd use sentence-transformers for calculating embeddings. You'll want a bi-encoder model since they produce embeddings. As the author of the blog, I'm partial towards Milvus (https://github.com/milvus-io/milvus) due to its enterprise and scale, but FAISS is a great option too if you're just looking for something more local and contained.

Milvus will perform vector similarity search for you - all you need to do is give it a query vector.

Re: Introduction to vector similarity search (2022)

#12

Post by fzliu »

Xenoamorphous wrote: post #7
behnamoh wrote: post #2

Can anyone please suggest a good stack for the following: - calculating text embeddings using open-source/local methods (not OpenAI) - storing them in a vector database. I'm confused by the myriad of options like Chromadb, Pinecone, etc. - running vector similarity search using open-source/local methods. Also, how granular should the text chunks be? Too short and we'll end up with a huge database, too long and we'll…

This is what we use: BERT sentence transformers to generate the embeddings (we used Universal Sentence Encoder before that and it was good too), and ElasticSearch for storage, which has a dense vector data type. It also has a cosineSimilarity function to run searches.

I would not use Elastic for vector search due to its architectural limitations and poor performance when conducting vector search. https://zilliz.com/benchmark

Re: Introduction to vector similarity search (2022)

#13

Post by anon7725 »

NhanH wrote: post #4
mattnewton wrote: post #3

Earlier quoted context omitted.

Calculating the embeddings is probably going to be an application-specific thing. Either your application has reasonable pre-trained encoders or you train one off a mountain of matching pairs of data. Once you have the embeddings in some space, for PoC I’ve mostly seen people shove them into faiss, which handles most of the rest very well for small/medium datasets: https://github.com/facebookresearch/faiss

Could you please point to some materials to understand the data needed to train the embedding model for a specific domain?

You don’t need to train anything if you just need embeddings. The data is text. You apply the pretrained model to your text and it returns the embedding. You save it in a vector database if you’re fancy, or a big numpy array if you’re like me. Then run your similarity search (cosine, Euclidean, etc).

Re: Introduction to vector similarity search (2022)

#14

Post by jeffchuber »

chaxor wrote: post #9
jeffchuber wrote: post #6

Earlier quoted context omitted.

Really just chroma + openai is all you need. Chroma makes this easy https://docs.trychroma.com/embeddings . All you have to do is chunk... I'd start at a paragraph and experiment.

This is not always good advice. Many people require to not use off premise models, due to data ownership issues. I would therefore suggest a better default for this, such as BERT+Qdrant. It would be wonderful if there were a simpler (single file, SQLite or DuckDB like) database for vectors than the complex (and in some cases, unfortunately cloud-based) ones available now.

ah sorry, i should read OP better - chroma's default embedding model is sentence transformers - and we have many other integrated - https://github.com/chroma-core/chroma/blob/main/chromadb/uti...

> It would be wonderful if there were a simpler (single file, SQLite or DuckDB like) database for vectors than the complex (and in some cases, unfortunately cloud-based) ones available now.

This is literally chroma!

Re: Introduction to vector similarity search (2022)

#15

Post by ClassyJacket »

computerex wrote: post #5
behnamoh wrote: post #2

Can anyone please suggest a good stack for the following: - calculating text embeddings using open-source/local methods (not OpenAI) - storing them in a vector database. I'm confused by the myriad of options like Chromadb, Pinecone, etc. - running vector similarity search using open-source/local methods. Also, how granular should the text chunks be? Too short and we'll end up with a huge database, too long and we'll…

To calculate embeddings for free, use this very popular model: https://huggingface.co/sentence-transformers/all-MiniLM-L6-v... For storing the vectors and doing the vector search: https://github.com/pgvector/pgvector `ankane/pgvector` docker image is a drop in replacement for the postgres image, so you can fire this up with docker very quickly. It's a normal postgres db with a vector datatype. It can index the vector…

>This is a sentence-transformers model: It maps sentences & paragraphs to a 384 dimensional dense vector space

Interesting. Can someone explain to me why 384 specifically? How did they arrive at that number?

Re: Introduction to vector similarity search (2022)

#16

Post by fzliu »

chaxor wrote: post #9
jeffchuber wrote: post #6

Earlier quoted context omitted.

Really just chroma + openai is all you need. Chroma makes this easy https://docs.trychroma.com/embeddings . All you have to do is chunk... I'd start at a paragraph and experiment.

This is not always good advice. Many people require to not use off premise models, due to data ownership issues. I would therefore suggest a better default for this, such as BERT+Qdrant. It would be wonderful if there were a simpler (single file, SQLite or DuckDB like) database for vectors than the complex (and in some cases, unfortunately cloud-based) ones available now.

You might enjoy milvus-lite: https://zilliz.com/blog/exploring-magic-vector-databases-jup...

   pip install milvus

Re: Introduction to vector similarity search (2022)

#17

Post by Xenoamorphous »

fzliu wrote: post #12
Xenoamorphous wrote: post #7

Earlier quoted context omitted.

This is what we use: BERT sentence transformers to generate the embeddings (we used Universal Sentence Encoder before that and it was good too), and ElasticSearch for storage, which has a dense vector data type. It also has a cosineSimilarity function to run searches.

I would not use Elastic for vector search due to its architectural limitations and poor performance when conducting vector search. https://zilliz.com/benchmark

I should’ve said that we were already using it for actual search where embeddings similarity is just one component of the overall score. For pure vector stuff a dedicated solution will be faster.

Re: Introduction to vector similarity search (2022)

#18

Post by computerex »

ClassyJacket wrote: post #15
computerex wrote: post #5

Earlier quoted context omitted.

To calculate embeddings for free, use this very popular model: https://huggingface.co/sentence-transformers/all-MiniLM-L6-v... For storing the vectors and doing the vector search: https://github.com/pgvector/pgvector `ankane/pgvector` docker image is a drop in replacement for the postgres image, so you can fire this up with docker very quickly. It's a normal postgres db with a vector datatype. It can index the vector…

>This is a sentence-transformers model: It maps sentences & paragraphs to a 384 dimensional dense vector space Interesting. Can someone explain to me why 384 specifically? How did they arrive at that number?

It has to do with the architecture of the network used to create the embeddings. The embeddings is actually the output of the final layer of the model. The dimensionality is a function of the number of parameters in that layer.

Different models/architectures will produce different dimension embeddings.

Re: Introduction to vector similarity search (2022)

#19

Post by YossarianFrPrez »

behnamoh wrote: post #2

Can anyone please suggest a good stack for the following: - calculating text embeddings using open-source/local methods (not OpenAI) - storing them in a vector database. I'm confused by the myriad of options like Chromadb, Pinecone, etc. - running vector similarity search using open-source/local methods. Also, how granular should the text chunks be? Too short and we'll end up with a huge database, too long and we'll…

Depending on your use case (particularly if it is research-oriented), "scipy.spatial.distance.cdist" and "scipy.spatial.distance.pdist" are your friends. If you are doing something in production, the PG extension seems like a good bet.

One way to potentially answer your question about text-chunk-granularity is to take a random sample of 500 pieces of chunked text and look at several "most similar pairs." Do this for a few different chunk-lengths and you'll see how much information is lost...

Re: Introduction to vector similarity search (2022)

#20

Post by owen-elliott »

behnamoh wrote: post #2

Can anyone please suggest a good stack for the following: - calculating text embeddings using open-source/local methods (not OpenAI) - storing them in a vector database. I'm confused by the myriad of options like Chromadb, Pinecone, etc. - running vector similarity search using open-source/local methods. Also, how granular should the text chunks be? Too short and we'll end up with a huge database, too long and we'll…

You could use Marqo, it is a vector search engine that includes the text chunking, inference for calculating embeddings, vector storage, and vector search. You can pick from a heap of open-source models or bring your own fine-tuned ones. It all runs locally in docker https://github.com/marqo-ai/marqo
Post reply on HN