The Theoretical Limitations of Embedding-Based Retrieval
1–10 of 42 posts
Re: The Theoretical Limitations of Embedding-Based Retrieval
#2Their idea is that capacity of even 4096-wide vectors limits their performance.
Sparse models like BM25 have a huge dimension and thus don’t suffer from this limit, but they don’t capture semantics and can’t follow instructions.
It seems like the holy grail is a sparse semantic model. I wonder how splade would do?
Re: The Theoretical Limitations of Embedding-Based Retrieval
#3we used multi-vector models at Morphik, and I can confirm the real-world effectiveness, especially when compared with dense-vector retrieval.
Re: The Theoretical Limitations of Embedding-Based Retrieval
#4Their idea is that capacity of even 4096-wide vectors limits their performance. Sparse models like BM25 have a huge dimension and thus don’t suffer from this limit, but they don’t capture semantics and can’t follow instructions. It seems like the holy grail is a sparse semantic model. I wonder how splade would do?
We already have "sparse" embeddings. Google's Matryoshka embedding schema can scale embeddings from ~150 dimensions to >3k, and it's the same embedding with layers of representational meaning. Imagine decomposing an embedding along principle components, then streaming the embedding vectors in order of their eigenvalue, kind of the idea.
Re: The Theoretical Limitations of Embedding-Based Retrieval
#5Their idea is that capacity of even 4096-wide vectors limits their performance. Sparse models like BM25 have a huge dimension and thus don’t suffer from this limit, but they don’t capture semantics and can’t follow instructions. It seems like the holy grail is a sparse semantic model. I wonder how splade would do?
Wouldn't holy grail then be parallel channels for candidate generation;
euclidean embedding
hyperbolic embedding
sparse BM25 / SPLADE lexical search
optional multi-vector signatures
↓ merge & deduplicate candidates
Re: The Theoretical Limitations of Embedding-Based Retrieval
#6we used multi-vector models at Morphik, and I can confirm the real-world effectiveness, especially when compared with dense-vector retrieval.
Curious is that colbert-like ones?
Re: The Theoretical Limitations of Embedding-Based Retrieval
#7Their idea is that capacity of even 4096-wide vectors limits their performance. Sparse models like BM25 have a huge dimension and thus don’t suffer from this limit, but they don’t capture semantics and can’t follow instructions. It seems like the holy grail is a sparse semantic model. I wonder how splade would do?
We already have "sparse" embeddings. Google's Matryoshka embedding schema can scale embeddings from ~150 dimensions to >3k, and it's the same embedding with layers of representational meaning. Imagine decomposing an embedding along principle components, then streaming the embedding vectors in order of their eigenvalue, kind of the idea.
Matryoshka embeddings are not sparse. And SPLADE can scale to tens or hundreds of thousands of dimensions.
Re: The Theoretical Limitations of Embedding-Based Retrieval
#8Earlier quoted context omitted.
We already have "sparse" embeddings. Google's Matryoshka embedding schema can scale embeddings from ~150 dimensions to >3k, and it's the same embedding with layers of representational meaning. Imagine decomposing an embedding along principle components, then streaming the embedding vectors in order of their eigenvalue, kind of the idea.
Matryoshka embeddings are not sparse. And SPLADE can scale to tens or hundreds of thousands of dimensions.
If you consider the actual latent space the full higher dimensional representation, and you take the first principle component, the other vectors are zero. Pretty sparse. No it's not a linked list sparse matrix. Don't be a pedant.
Re: The Theoretical Limitations of Embedding-Based Retrieval
#9In the theoretical section, they extrapolate assuming a polynomial from 40 to thousands of dimensions. Why do they trust a polynomial fit to extrapolate two orders of magnitude? Why do we even think it's polynomial instead of exponential in the first place? Most things like this increase exponentially with dimension.
In fact, I think we can do it in d=2k dimensions, if we're willing to have arbitrarily precise query vectors.
Embed our points as (sin(theta), cos(theta), sin(2 x theta), cos(2 x theta)..., sin(k x theta), cos(k x theta)), with theta uniformly spaced around the circle, and we should be able to select any k of them.
Using a few more dimensions we can then ease the precision requirements on the query.
Re: The Theoretical Limitations of Embedding-Based Retrieval
#10Their idea is that capacity of even 4096-wide vectors limits their performance. Sparse models like BM25 have a huge dimension and thus don’t suffer from this limit, but they don’t capture semantics and can’t follow instructions. It seems like the holy grail is a sparse semantic model. I wonder how splade would do?
We already have "sparse" embeddings. Google's Matryoshka embedding schema can scale embeddings from ~150 dimensions to >3k, and it's the same embedding with layers of representational meaning. Imagine decomposing an embedding along principle components, then streaming the embedding vectors in order of their eigenvalue, kind of the idea.
Doesn't PCA compress the embeddings in this case, ie reduce the accuracy? It's similar to quantization.