Introduction to vector similarity search (2022)
1–10 of 57 posts
Re: Introduction to vector similarity search (2022)
#2Can anyone please suggest a good stack for the following:
- calculating text embeddings using open-source/local methods (not OpenAI)
- storing them in a vector database. I'm confused by the myriad of options like Chromadb, Pinecone, etc.
- running vector similarity search using open-source/local methods.
Also, how granular should the text chunks be? Too short and we'll end up with a huge database, too long and we'll probably miss some relevant information in some chunks.
Has anyone been able to achieve reliable results from these? Preferably w/o using Langchain.
Re: Introduction to vector similarity search (2022)
#3Can anyone please suggest a good stack for the following: - calculating text embeddings using open-source/local methods (not OpenAI) - storing them in a vector database. I'm confused by the myriad of options like Chromadb, Pinecone, etc. - running vector similarity search using open-source/local methods. Also, how granular should the text chunks be? Too short and we'll end up with a huge database, too long and we'll…
Calculating the embeddings is probably going to be an application-specific thing. Either your application has reasonable pre-trained encoders or you train one off a mountain of matching pairs of data.
Once you have the embeddings in some space, for PoC I’ve mostly seen people shove them into faiss, which handles most of the rest very well for small/medium datasets: https://github.com/facebookresearch/faiss
Re: Introduction to vector similarity search (2022)
#4Can anyone please suggest a good stack for the following: - calculating text embeddings using open-source/local methods (not OpenAI) - storing them in a vector database. I'm confused by the myriad of options like Chromadb, Pinecone, etc. - running vector similarity search using open-source/local methods. Also, how granular should the text chunks be? Too short and we'll end up with a huge database, too long and we'll…
Calculating the embeddings is probably going to be an application-specific thing. Either your application has reasonable pre-trained encoders or you train one off a mountain of matching pairs of data. Once you have the embeddings in some space, for PoC I’ve mostly seen people shove them into faiss, which handles most of the rest very well for small/medium datasets: https://github.com/facebookresearch/faiss
Could you please point to some materials to understand the data needed to train the embedding model for a specific domain?
Re: Introduction to vector similarity search (2022)
#5Can anyone please suggest a good stack for the following: - calculating text embeddings using open-source/local methods (not OpenAI) - storing them in a vector database. I'm confused by the myriad of options like Chromadb, Pinecone, etc. - running vector similarity search using open-source/local methods. Also, how granular should the text chunks be? Too short and we'll end up with a huge database, too long and we'll…
To calculate embeddings for free, use this very popular model:
https://huggingface.co/sentence-transformers/all-MiniLM-L6-v...
For storing the vectors and doing the vector search: https://github.com/pgvector/pgvector
`ankane/pgvector` docker image is a drop in replacement for the postgres image, so you can fire this up with docker very quickly.
It's a normal postgres db with a vector datatype. It can index the vectors and allows efficient retrieval. Both AWS RDS and Google Cloud now support this in their managed Postgres offerings, so postgres+pgvector is a viable managed production vectordb solution.
> Also, how granular should the text chunks be?
That depends on the use case, the size of your corpus, the context of the model you are using, how much money you are willing to spend.
> Has anyone been able to achieve reliable results from these? Preferably w/o using Langchain.
Definitely. We use postgres+pgvector with php.
Re: Introduction to vector similarity search (2022)
#6Can anyone please suggest a good stack for the following: - calculating text embeddings using open-source/local methods (not OpenAI) - storing them in a vector database. I'm confused by the myriad of options like Chromadb, Pinecone, etc. - running vector similarity search using open-source/local methods. Also, how granular should the text chunks be? Too short and we'll end up with a huge database, too long and we'll…
Really just chroma + openai is all you need. Chroma makes this easy https://docs.trychroma.com/embeddings.
All you have to do is chunk... I'd start at a paragraph and experiment.
Re: Introduction to vector similarity search (2022)
#7Can anyone please suggest a good stack for the following: - calculating text embeddings using open-source/local methods (not OpenAI) - storing them in a vector database. I'm confused by the myriad of options like Chromadb, Pinecone, etc. - running vector similarity search using open-source/local methods. Also, how granular should the text chunks be? Too short and we'll end up with a huge database, too long and we'll…
This is what we use: BERT sentence transformers to generate the embeddings (we used Universal Sentence Encoder before that and it was good too), and ElasticSearch for storage, which has a dense vector data type. It also has a cosineSimilarity function to run searches.
Re: Introduction to vector similarity search (2022)
#8I’m confused by the choice of word2vec for the embeddings, for “simplicity”. Using a transformer based model like Universal Sentence Encoder or SBERT is just as easy and the results will be considerably better.
Re: Introduction to vector similarity search (2022)
#9Can anyone please suggest a good stack for the following: - calculating text embeddings using open-source/local methods (not OpenAI) - storing them in a vector database. I'm confused by the myriad of options like Chromadb, Pinecone, etc. - running vector similarity search using open-source/local methods. Also, how granular should the text chunks be? Too short and we'll end up with a huge database, too long and we'll…
Really just chroma + openai is all you need. Chroma makes this easy https://docs.trychroma.com/embeddings . All you have to do is chunk... I'd start at a paragraph and experiment.
This is not always good advice. Many people require to not use off premise models, due to data ownership issues.
I would therefore suggest a better default for this, such as BERT+Qdrant.
It would be wonderful if there were a simpler (single file, SQLite or DuckDB like) database for vectors than the complex (and in some cases, unfortunately cloud-based) ones available now.
Re: Introduction to vector similarity search (2022)
#10I’m confused by the choice of word2vec for the embeddings, for “simplicity”. Using a transformer based model like Universal Sentence Encoder or SBERT is just as easy and the results will be considerably better.
word2vec is seminal and outputs static embeddings - fairly easy for beginners to understand. The embeddings for each token change relative to context in attention-based MLMs and I figured it might be confusing for an introductory blog.