Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A polynomial autoencoder beats PCA on transformer embeddings

ivanpleshkov.dev

Post reply on HN
Print view View original

21–30 of 39 posts

Re: A polynomial autoencoder beats PCA on transformer embeddings

#21

Post by pleshkov »

folderquestion wrote: post #12

This sound like projecting data into the linear space spanned by {x_i, x_i*x_j} where x_i are the features variables, and then applying standard regularization methods to remove noise and low value coefficients. Anisotropy and the cone ideas may explain why PCA underperforms, but it does not uniquely justify this particular quadratic decoder. The geometric story is not doing explanatory work beyond “data is nonlinear…

Author here. Fair characterization, and a fair critique on the geometric story. A few clarifications. I don't claim {x_i, x_i·x_j} is the right lift specifically — the post itself shows datasets where the quadratic decoder gives essentially no improvement over PCA. The contribution is empirical: "second-order is the simplest nonlinear decoder you can fit in closed form, and on anisotropic embeddings it picks up real signal that linear decoders miss." Whether degree 3 would help further is open. Degree 3 blows up fast: at d=100 that's 175K features, and the Ridge solve at that scale starts memorizing the corpus rather than generalizing (§7 in the post discusses this trap at d=256 already). So degree 2 is partly a choice, partly a practical ceiling for the closed-form route.

Re: A polynomial autoencoder beats PCA on transformer embeddings

#22

Post by wenc »

It sounds like this replaces the PCA reconstruction function with a quadratic.

The normal PCA encoding:

1) Given a mean-center-scaled X matrix, get the latent variable matrix T with X = T * P’ + e, where P = loadings and e = residuals. The P is your model, so for a new vector xnew, you can calculate tnew = xnew * P (because P’ * P = I).

This is the encoder —- nothing changes here. The original matrix is dimensionally reduced with residuals e discarded. This is why PCA is lossy.

The decoder is where things diverge

The usual PCA decoder reconstructs a given latent variable t_any by using the trained P loadings, like thus x_reconstructed = t_any * P’. This reconstructed data lies on a linear hyperplane, so if the original data did not lie on the hyperplane, reconstruction errors are potetially high.

In your proposal, instead of a linear decoder, you train a quadratic decoder (essentially a classic ridge regression using a quadratic) on the original X. So for your reconstruction, you have x_reconstructed = poly(t_new).

This achieves lower reconstruction error in-sample (naturally, because quadratic is higher order than linear), but your poly function is trained on a particular corpus. Which means that when you’re in-distribution within that corpus, you’re good but when you’re not, you can be very wrong in biased ways that PCA’s linear reconstruction is not.

SO this is not a better technique than PCA in a general sense. It’s a better reconstruction machine when your data is mostly in-sample. It’s a kind of computationally cheap “specialization” on a particular distribution of data, which can be useful if you’re mostly in-distribution but introduces new risks when out-of-distribution.

Whereas PCA just drops the residual and makes modest claims, a quadratic decoder is trying to predict the residual and on out-of-sample data, it can be wrong in biased ways that PCA is not. In other words, it can hallucinate.

But if on a large enough training corpus, chances are we’re going to be in-distribution most of the time, so maybe this could generalize well.

Re: A polynomial autoencoder beats PCA on transformer embeddings

#23

Post by pleshkov »

dbfclark wrote: post #18
pleshkov wrote: post #2

Author here — questions and pushback both welcome.

I think your per-axis std normalization is likely doing a big pile of the work —- it’s fairly well-known that “wrong” PCA, setting sigma=Id or just taking a square root, gives better embeddings than the un-normalized version. It would be worth showing a comparison to similarly-normalized PCA I think, if it’s not too hard?

Good catch, this is the obvious ablation I should have included. I'll re-run with per-axis normalized PCA as a separate baseline and post numbers in this thread tomorrow. Prior: I expect some of the gap to come from normalization, but not all — the no-improvement results on isotropic datasets (§4) suggest there's structural signal the polynomial cross-terms catch that linear projection structurally can't. But that's a prediction; let me actually run it.

Re: A polynomial autoencoder beats PCA on transformer embeddings

#24

Post by pleshkov »

Devilstro wrote: post #7
pleshkov wrote: post #2

Author here — questions and pushback both welcome.

In the article, you mention this approach requires no search over hyper-parameter, because the method comprises a closed-form solution with "simple" linear algebra. I agree with this, but do you not in think need to tune the L2-regularization strength? That would for me be a hyper-parameter you would need to do a CV over (or similarly).

Fair point — lam is technically a hyperparameter. In practice I used lam=1e-3 (the default in the code) across all four models without tuning, and the gap to PCA is robust enough that small variations don't change the conclusion. So more accurately: "one hyperparameter with a benign default" rather than "no hyperparameters" — you're right I overstated.

Re: A polynomial autoencoder beats PCA on transformer embeddings

#25

Post by aesthesia »

pleshkov wrote: post #2

Author here — questions and pushback both welcome.

Looking at your experiment code, it seems like the retrieval experiments are done with the reconstructed vectors of dimension D rather than the compressed vectors of dimension d, which doesn't have any direct performance improvements. Later on in the post you indicate that the real advantage is that the residuals are more isotropic and therefore you can quantize the pair (p, V_resid) with less quality degradation, but I don't see any experiments actually verifying that retrieval quality holds up in this setting. Also, it's not quite clear to me how you efficiently compute cosine similarity for vectors encoded in this form. Doesn't the V_resid part of the computation require something significantly more complex than a dot product?

Re: A polynomial autoencoder beats PCA on transformer embeddings

#26

Post by nickpsecurity »

There's also work in polynomial regression as an alternative to regular, neural networks.

Polynomial Regression As an Alternative to Neural Nets (2018) https://arxiv.org/abs/1806.06850

Π-nets: Deep Polynomial Neural Networks (2020) https://arxiv.org/abs/2003.03828

Re: A polynomial autoencoder beats PCA on transformer embeddings

#27

Post by [deleted] »

wenc wrote: post #22

It sounds like this replaces the PCA reconstruction function with a quadratic. The normal PCA encoding: 1) Given a mean-center-scaled X matrix, get the latent variable matrix T with X = T * P’ + e, where P = loadings and e = residuals. The P is your model, so for a new vector xnew, you can calculate tnew = xnew * P (because P’ * P = I). This is the encoder —- nothing changes here. The original matrix is dimensionally…

[deleted]
Post reply on HN