41–50 of 57 posts
Can anyone please suggest a good stack for the following: - calculating text embeddings using open-source/local methods (not OpenAI) - storing them in a vector database. I'm confused by the myriad of options like Chromadb, Pinecone, etc. - running vector similarity search using open-source/local methods. Also, how granular should the text chunks be? Too short and we'll end up with a huge database, too long and we'll…
Earlier quoted context omitted.
To calculate embeddings for free, use this very popular model: https://huggingface.co/sentence-transformers/all-MiniLM-L6-v... For storing the vectors and doing the vector search: https://github.com/pgvector/pgvector `ankane/pgvector` docker image is a drop in replacement for the postgres image, so you can fire this up with docker very quickly. It's a normal postgres db with a vector datatype. It can index the vector…
>This is a sentence-transformers model: It maps sentences & paragraphs to a 384 dimensional dense vector space Interesting. Can someone explain to me why 384 specifically? How did they arrive at that number?
Can anyone please suggest a good stack for the following: - calculating text embeddings using open-source/local methods (not OpenAI) - storing them in a vector database. I'm confused by the myriad of options like Chromadb, Pinecone, etc. - running vector similarity search using open-source/local methods. Also, how granular should the text chunks be? Too short and we'll end up with a huge database, too long and we'll…
https://jillesvangurp.github.io/kt-search/manual/KnnSearch.h...
You index into an index with a dense vector field
And when you query, you generate embeddings for your query and run a knn vector similarity search.
It uses some embeddings I generated with openai. You could use something like easybert or one of the many OSS embeddings models instead. Basically you need some code that converts your text/images/whatever into lists of numbers using such a model.
So:
1) use some magical tool that given a thing returns embeddings. You use this to extract embeddings at index time from your content and at query time for your queries.
2) put your embeddings along with your things in an Elasticsearch index (or vector db of your choice, Opensearch works similar to Elasticsearch for this)
3) when querying, create embeddings for your queries and find the nearest match.
I built this tutorial as a quick POC to figure out how easy it is with my own library. I'm not an expert. Mission accomplished and it only took me a few hours. The results are not impressive as this model is probably not very appropriate for the demo content. But it vaguely works. There are a bunch of people that are smarter than me that suggest that most oss models struggle to outperform bm25, which is just doing simple text searches.
Btw. the embeddings are the hard part. The rest is just plumbing. And of course world + dog just glosses over that. There's an interesting article that I came across recently that goes a bit more in depth on this: https://blog.metarank.ai/from-zero-to-semantic-search-embedd...
Earlier quoted context omitted.
This is not always good advice. Many people require to not use off premise models, due to data ownership issues. I would therefore suggest a better default for this, such as BERT+Qdrant. It would be wonderful if there were a simpler (single file, SQLite or DuckDB like) database for vectors than the complex (and in some cases, unfortunately cloud-based) ones available now.
ah sorry, i should read OP better - chroma's default embedding model is sentence transformers - and we have many other integrated - https://github.com/chroma-core/chroma/blob/main/chromadb/uti... > It would be wonderful if there were a simpler (single file, SQLite or DuckDB like) database for vectors than the complex (and in some cases, unfortunately cloud-based) ones available now. This is literally chroma!
Can anyone please suggest a good stack for the following: - calculating text embeddings using open-source/local methods (not OpenAI) - storing them in a vector database. I'm confused by the myriad of options like Chromadb, Pinecone, etc. - running vector similarity search using open-source/local methods. Also, how granular should the text chunks be? Too short and we'll end up with a huge database, too long and we'll…
Can anyone please suggest a good stack for the following: - calculating text embeddings using open-source/local methods (not OpenAI) - storing them in a vector database. I'm confused by the myriad of options like Chromadb, Pinecone, etc. - running vector similarity search using open-source/local methods. Also, how granular should the text chunks be? Too short and we'll end up with a huge database, too long and we'll…
Also, here's a benchmark that allows you to easily test their performance differences through a user-friendly interface. This includes both cloud solutions and open-source options. If you prefer to view pre-tested results, there are standard ones available as well. Check it out here: VectorDBBench. https://github.com/zilliztech/VectorDBBench
Can anyone please suggest a good stack for the following: - calculating text embeddings using open-source/local methods (not OpenAI) - storing them in a vector database. I'm confused by the myriad of options like Chromadb, Pinecone, etc. - running vector similarity search using open-source/local methods. Also, how granular should the text chunks be? Too short and we'll end up with a huge database, too long and we'll…
To calculate embeddings for free, use this very popular model: https://huggingface.co/sentence-transformers/all-MiniLM-L6-v... For storing the vectors and doing the vector search: https://github.com/pgvector/pgvector `ankane/pgvector` docker image is a drop in replacement for the postgres image, so you can fire this up with docker very quickly. It's a normal postgres db with a vector datatype. It can index the vector…
I think I’d need to fine tune the model to see better results with some domain-specific terms, but I couldn’t find much information about how to actually do that - what sort of input data you need, how much of it, etc.
Would be interested to hear if anyone had more to share about fine tuning these models for semantic search.
It seems to me that in a semantic embedding, "big", "huge", "enormous", and "gargantuan" should roughly point in the same direction but have different magnitudes. For instance, I might assume that the nearest neighbor to "big * 10" is "huge" or "enormous". But if embeddings must be normalized, I either can't tell the difference between these four terms, or they must point in different directions. I can't even talk about "big * 10" at all. I mean, perhaps there are "emphasis" dimensions roughly corresponding to words like "very" and "extremely", but it still seems like "big" and "enormous" would be hard to distinguish without scalars. It seems to me that the only significant difference between "big" and "huge" is in fact their magnitude, and forcing "huge" to point away from "big" so that it can be non-zero in the "very" dimension must cause it to be less "big-like" in some other small way, right?
I'm surprised that throwing away all the power of scalars is worth it. I'm not a professional in the field, so maybe there's good reason for this that I just haven't read. Can any professional comment on why vectors of magnitude != 1 are not helpful in embeddings?
Can anyone please suggest a good stack for the following: - calculating text embeddings using open-source/local methods (not OpenAI) - storing them in a vector database. I'm confused by the myriad of options like Chromadb, Pinecone, etc. - running vector similarity search using open-source/local methods. Also, how granular should the text chunks be? Too short and we'll end up with a huge database, too long and we'll…
Video: https://www.youtube.com/watch?v=hGRNcftpqAk Slides: https://presentations.clickhouse.com/meetup74/ai/
70% of the way through this article they drop "Since we have unit norm vectors ..." and later "Always remember to normalize your embeddings." I found this strange and surprising. It seems to me that in a semantic embedding, "big", "huge", "enormous", and "gargantuan" should roughly point in the same direction but have different magnitudes. For instance, I might assume that the nearest neighbor to "big * 10" is "huge"…
* The problem with large vectors is that they have large dot products with every other vector, which would imply that they are more similar to everything which doesn't make sense.
* Adding the requirement that "length==1" doesn't matter much in high-dimensional spaces, since that only removes one degree of freedom. Don't try to use too much 3D intuition here.
* It might be intuitive to think that "large" should have implications for the size of the vectors, but that really only applies to a couple of examples. We want vectors to represent thousands of unrelated concepts, so this one case is really not that relevant or important.
* In reality what ends up happening is partially the "very" dimension you're suggesting, but also just a "largeness" dimension. Individual dimensions can still have a scale!