31–40 of 57 posts
Can anyone please suggest a good stack for the following: - calculating text embeddings using open-source/local methods (not OpenAI) - storing them in a vector database. I'm confused by the myriad of options like Chromadb, Pinecone, etc. - running vector similarity search using open-source/local methods. Also, how granular should the text chunks be? Too short and we'll end up with a huge database, too long and we'll…
How would I decide on which granularity level I create my vectors? For example, if I create word level embeddings with word2vec and store one vector per word I probably will only get good results for keyword based queries, right? If I create more fancy vectors, for example on the sentence level, it is not really clear to me how that is supposed to work. Why is the vector embedding of a query, which typically has the…
Earlier quoted context omitted.
You’re correct. There are lots of ways to project a string of embeddings back onto a single embedding - lots of times however these simpler methods do something like pooling the sum if the word embeddings by averaging and hoping that the answer embedding has a small angle with that projection (cosine similarity). You could try adding a way to focus on individual tokens, pairwise interactions, and more sophisticated w…
oh boy, this looks like an alchemy to me. on one hand one cant deny success of llm's on another we are shifting the responsibiliteis to non-deterministic fuzzy duck-taped functions.
Another thing that is not clear to me: Is there query directly fed into e.g. SBERT or should I ask an LLM to transform the query into something more suitable, like turning the question into a proposition?
Asked more abstractly: In a vector space like SBERT's, can I expect questions and answers about the same topic to lie near each other? Especially will the correct answers lie near their question?
Can anyone please suggest a good stack for the following: - calculating text embeddings using open-source/local methods (not OpenAI) - storing them in a vector database. I'm confused by the myriad of options like Chromadb, Pinecone, etc. - running vector similarity search using open-source/local methods. Also, how granular should the text chunks be? Too short and we'll end up with a huge database, too long and we'll…
Can anyone please suggest a good stack for the following: - calculating text embeddings using open-source/local methods (not OpenAI) - storing them in a vector database. I'm confused by the myriad of options like Chromadb, Pinecone, etc. - running vector similarity search using open-source/local methods. Also, how granular should the text chunks be? Too short and we'll end up with a huge database, too long and we'll…
Haystack allows you to pre-process your documents into smaller chunks, calculate embeddings and index them into a document store. You can wrap all of that in a modular pipeline if you want.
Next, you can query your documents using a retrieval pipeline.
Regarding document store selection: Replacing your document store is easy, so I would start with the most simple one, probably an InMemoryDocumentStore. When you want to move from experimentation to production, you‘ll want to tailor your selection to your use case. Here‘s a few things that I‘ve observed.
You don’t want to manage anything and are fine with SaaS -> Pinecone
You have a very large dataset (500M+ vectors) and you want something that you can run locally -> maybe Qdrant
You have meta data that you want to incorporate into your retrieval or you want to do hybrid search -> Opensearch/Elasticsearch
Regarding model selection:
We‘ve seen https://huggingface.co/sentence-transformers/multi-qa-distil... work well for a good semantic search baseline with fast indexing times. If you feel like the performance is lacking, you could look at the E5 models. What also works fairly well for us is a multi-step retrieval process where we retrieve ~100 documents with BM25 first and then use a cross-encoder to rank these by semantic relevance. Very fast indexing times are a benefit and you also don’t need a beefy vector db to store your documents. Latency at query time will be slightly higher though and you might need a GPU machine to run your query pipeline.
Retrieval in Haystack: https://docs.haystack.deepset.ai/docs/retriever
Cross-Encoder approach: https://docs.haystack.deepset.ai/docs/ranker
Blog Post with an end-to-end retrieval example: https://haystack.deepset.ai/blog/how-to-build-a-semantic-sea...
Can anyone please suggest a good stack for the following: - calculating text embeddings using open-source/local methods (not OpenAI) - storing them in a vector database. I'm confused by the myriad of options like Chromadb, Pinecone, etc. - running vector similarity search using open-source/local methods. Also, how granular should the text chunks be? Too short and we'll end up with a huge database, too long and we'll…
Really just chroma + openai is all you need. Chroma makes this easy https://docs.trychroma.com/embeddings . All you have to do is chunk... I'd start at a paragraph and experiment.
Can anyone please suggest a good stack for the following: - calculating text embeddings using open-source/local methods (not OpenAI) - storing them in a vector database. I'm confused by the myriad of options like Chromadb, Pinecone, etc. - running vector similarity search using open-source/local methods. Also, how granular should the text chunks be? Too short and we'll end up with a huge database, too long and we'll…
To calculate embeddings for free, use this very popular model: https://huggingface.co/sentence-transformers/all-MiniLM-L6-v... For storing the vectors and doing the vector search: https://github.com/pgvector/pgvector `ankane/pgvector` docker image is a drop in replacement for the postgres image, so you can fire this up with docker very quickly. It's a normal postgres db with a vector datatype. It can index the vector…
Can anyone please suggest a good stack for the following: - calculating text embeddings using open-source/local methods (not OpenAI) - storing them in a vector database. I'm confused by the myriad of options like Chromadb, Pinecone, etc. - running vector similarity search using open-source/local methods. Also, how granular should the text chunks be? Too short and we'll end up with a huge database, too long and we'll…
How do vector databases respond to changes in embeddings? Does one need to reindex all documents in the DB when that happens (i.e. retraining a model, switching to a model with different embeddings etc.)?