Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Theoretical Limitations of Embedding-Based Retrieval

arxiv.org

Post reply on HN
Print view View original

41–42 of 42 posts

Re: The Theoretical Limitations of Embedding-Based Retrieval

#41

Post by faxipay349 »

jxmorris12 wrote: post #7
CuriouslyC wrote: post #4

Earlier quoted context omitted.

We already have "sparse" embeddings. Google's Matryoshka embedding schema can scale embeddings from ~150 dimensions to >3k, and it's the same embedding with layers of representational meaning. Imagine decomposing an embedding along principle components, then streaming the embedding vectors in order of their eigenvalue, kind of the idea.

Matryoshka embeddings are not sparse. And SPLADE can scale to tens or hundreds of thousands of dimensions.

Yeah, the standard SPLADE model trained from BERT typically already has a vocabulary/vector size of 30,552. If the SPLADE model is based on a multilingual version of BERT, such as mBERT or XLM-R, the vocabulary size could inherently expand to approximately 100,000, as does the vector size.

Re: The Theoretical Limitations of Embedding-Based Retrieval

#42

Post by lunarmony »

yorwba wrote: post #14
lunarmony wrote: post #13

Researchers have discussed limitations of vector-based retrieval from a rank perspective in various forms for a few years. It's further been shown that better alternative exists; some low-rank approaches can theoretically approximate arbitrary high-rank distribution while permitting MIPS-level efficient inference (see e.g., Retrieval with Learned Similarities, https://arxiv.org/abs/2407.15462 ). Such solutions are al…

I don't think Mixture of Logits from the paper you link circumvents the theoretical limitations pointed out here, since their dataset size mostly stays well below the limit. In the end they still rely on Maximum Inner Product Search, just with several lookups for smaller partitions of the full embedding, and the largest dataset is Books , where this paper suggests you'd need more than 512 embedding dimensions, and Mo…

Mixture of Logits was actually already deployed on 100M scale+ datasets at Meta and at LinkedIn (https://arxiv.org/abs/2306.04039 https://arxiv.org/abs/2407.13218 etc.). The crucial departure from traditional embedding/multi-embedding approaches is in learning a query-/item- dependent gating function, which enables MoL to become a universal high-rank approximator (assuming we care about recall@1) even when the input embeddings are low rank.
Post reply on HN