PCA is good, but you could also try playing around with Sparse (robust) PCA. The sparsification loses orthogonality, but does not necessarily lose information, it can yield a different rotation and cleaner vectors. Now whether that matters in the context of LLMs/Embeddings - I cannot tell.
Honey, I shrunk the embeddings: Matryoshka vs. PCA
11–20 of 20 posts
Re: Honey, I shrunk the embeddings: Matryoshka vs. PCA
#12Good benchmarks! > You can push this further by combining quantization with truncation or PCA. The resulting vectors can be dramatically smaller while still preserving a surprising amount of retrieval quality. Counterintuitively - quantisation can also be combined with a random rotation step before the quantisation. A random rotation spreads information across more dimensions, allowing more aggressive quantisation wi…
in my experience it's very similar to what you typically need to do in order to get the best results in other fields (like audio), where being really particular/deliberate in how you manicure and format the input can pay orders-of-magnitude dividends
sometimes adding noise over a frequency range is better than removing it entirely, the opposite tends to be true for text and especially code, where you'll want lists of language and framework keywords, and then a pass on top of that to scan the codebase itself for its own slang. You can then 'double dip' and use these lists to weight the results after the fact
Re: Honey, I shrunk the embeddings: Matryoshka vs. PCA
#13Thanks for doing this benchmarking Dylan. I wanted to teach people PCA in my original article, but had no idea it would stack up this well against Matroyshka! Feels like a “just use logistic regression” moment :)
It does, tbh I was also surprised by the results!
Re: Honey, I shrunk the embeddings: Matryoshka vs. PCA
#14The original MRL paper (https://arxiv.org/pdf/2205.13147) reported an SVD baseline, which showed comparable performance (Table 1, top-1 accuracy) at d>=256, but much degraded performance at lower dims (d \in {8,32,64}). (Though Table 2, nearest-neighbor accuracy, doesn't show degradation until d In your conclusion, you report that PCA won on most dimensions. Did you investigate why you found that PCA outperforms MRL when the original paper found that their SVD baseline did not?
Re: Honey, I shrunk the embeddings: Matryoshka vs. PCA
#15The original MRL paper ( https://arxiv.org/pdf/2205.13147 ) reported an SVD baseline, which showed comparable performance (Table 1, top-1 accuracy) at d>=256, but much degraded performance at lower dims (d \in {8,32,64}). (Though Table 2, nearest-neighbor accuracy, doesn't show degradation until d In your conclusion, you report that PCA won on most dimensions. Did you investigate why you found that PCA outperforms MR…
I also found that interesting but our evaluation methodology is quite different, so I didn’t go too deep into it
Re: Honey, I shrunk the embeddings: Matryoshka vs. PCA
#16>a new problem emerged: how do you store and search all that information efficiently?
This is no way a new problem. The idea that embeddings or even search didn't exist until LLMs is simply false.
Re: Honey, I shrunk the embeddings: Matryoshka vs. PCA
#17>a new problem emerged: how do you store and search all that information efficiently? This is no way a new problem. The idea that embeddings or even search didn't exist until LLMs is simply false.
That's not a fair read of what I said
Re: Honey, I shrunk the embeddings: Matryoshka vs. PCA
#18>a new problem emerged: how do you store and search all that information efficiently? This is no way a new problem. The idea that embeddings or even search didn't exist until LLMs is simply false.
That's not a fair read of what I said
Well to me it's written in a way that suggests that users had no need for searching through their documents to find information. User's have had this need before vector search was used for ranking, so starting the story of how to search user's documents with vector databases is ignoring history.
Re: Honey, I shrunk the embeddings: Matryoshka vs. PCA
#19Earlier quoted context omitted.
That's not a fair read of what I said
Well to me it's written in a way that suggests that users had no need for searching through their documents to find information. User's have had this need before vector search was used for ranking, so starting the story of how to search user's documents with vector databases is ignoring history.
That sentence is written as an introduction to an article about embeddings compression. So yeah, it does take a bit of a shortcut from "search" to "vector storage", but that's irrelevant to the rest of the article.
Re: Honey, I shrunk the embeddings: Matryoshka vs. PCA
#20The original MRL paper ( https://arxiv.org/pdf/2205.13147 ) reported an SVD baseline, which showed comparable performance (Table 1, top-1 accuracy) at d>=256, but much degraded performance at lower dims (d \in {8,32,64}). (Though Table 2, nearest-neighbor accuracy, doesn't show degradation until d In your conclusion, you report that PCA won on most dimensions. Did you investigate why you found that PCA outperforms MR…
I also found that interesting but our evaluation methodology is quite different, so I didn’t go too deep into it
Yeah...so I'm having difficulty conceptualizing your result. If you were publishing this I feel like it would be an important distinction to make.
Are the goals different, or should the original paper have done something more similar to your benchmark? Or something else?