Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A polynomial autoencoder beats PCA on transformer embeddings

ivanpleshkov.dev

Post reply on HN
Print view View original

31–39 of 39 posts

Re: A polynomial autoencoder beats PCA on transformer embeddings

#31

Post by BobbyTables2 »

pleshkov wrote: post #21
folderquestion wrote: post #12

This sound like projecting data into the linear space spanned by {x_i, x_i*x_j} where x_i are the features variables, and then applying standard regularization methods to remove noise and low value coefficients. Anisotropy and the cone ideas may explain why PCA underperforms, but it does not uniquely justify this particular quadratic decoder. The geometric story is not doing explanatory work beyond “data is nonlinear…

Author here. Fair characterization, and a fair critique on the geometric story. A few clarifications. I don't claim {x_i, x_i·x_j} is the right lift specifically — the post itself shows datasets where the quadratic decoder gives essentially no improvement over PCA. The contribution is empirical: "second-order is the simplest nonlinear decoder you can fit in closed form, and on anisotropic embeddings it picks up real…

Is this similar Voltera series in signal processing?

Re: A polynomial autoencoder beats PCA on transformer embeddings

#33

Post by pleshkov »

mentalgear wrote: post #15

Geometric Algebra (GA) (Clifford Algebra) also has high potential to transform neural architectures. Models like the Geometric Algebra Transformer (GATr) and Versor (2026) demonstrate it can enhance or even make the Attention Mechanism obsolete. By representing data as multivectors, translational and rotational symmetries are encoded natively which allows them to handle geometric hierarchies with massive efficiency g…

The polynomial lift in this post originally came out of an unsuccessful experiment with hyperbolic embeddings. The idea was to embed corpora into a hyperbolic ball (anisotropic embeddings have a tree-like structure that hyperbolic space could exploit). The lift was a tool to go from hyperbolic latent back to Euclidean for retrieval. Hyperbolic part didn't work; the lift evaluated standalone kept showing real signal, and that became this post.

Re: A polynomial autoencoder beats PCA on transformer embeddings

#34

Post by gh0stmach1ne »

This was a good write-up but the author's claim that this is relatively unknown in machine learning is not quite accurate.

Back in grad school when we covered basic OLS methods expanding the feature space using a quadratic manifold was a common technique for teaching that regression can support nonlinear features and still remain "linear in the coefficients".

It is also implemented in widely used ML libraries. Scikit-learn, for example, has a KernelPCA object which supports using a polynomial kernel (in this instance a degree two) that computes the inner product on a explicit feature map which contains all second-order monomials.

Re: A polynomial autoencoder beats PCA on transformer embeddings

#36

Post by pleshkov »

dbfclark wrote: post #18
pleshkov wrote: post #2

Author here — questions and pushback both welcome.

I think your per-axis std normalization is likely doing a big pile of the work —- it’s fairly well-known that “wrong” PCA, setting sigma=Id or just taking a square root, gives better embeddings than the un-normalized version. It would be worth showing a comparison to similarly-normalized PCA I think, if it’s not too hard?

Just checked the normalization point. You were partially right, sqrt-normalization makes the difference x2 less. I'm updating the numbers in the post. Interesting moment. I did a smoke test of poly-AE without whitening, and the result didn't change. I won't mention it in the post cause right now I'm not sure if it's a random effect or really a polynomial lift compensates normalization

Re: A polynomial autoencoder beats PCA on transformer embeddings

#37

Post by pleshkov »

aesthesia wrote: post #25
pleshkov wrote: post #2

Author here — questions and pushback both welcome.

Looking at your experiment code, it seems like the retrieval experiments are done with the reconstructed vectors of dimension D rather than the compressed vectors of dimension d, which doesn't have any direct performance improvements. Later on in the post you indicate that the real advantage is that the residuals are more isotropic and therefore you can quantize the pair (p, V_resid) with less quality degradation, bu…

I agree that we don't want to reconstruct the whole vector while retrieval and it makes poly-AE toy-like at the current state non production ready. My main interest here in the just taking more recall pp in closed form. And then think about how to make it fast. In all threads I got a good intermediate thoughts about the topic which may help me to bring to closer to production form

Re: A polynomial autoencoder beats PCA on transformer embeddings

#38

Post by pleshkov »

ComplexSystems wrote: post #32

What makes this different from just kernel PCA with the quadratic kernel?

It's close but not the same. Kernel PCA lifts all D coordinates which gives M around 525k at D = 1024. In the post I do PCA first to reduce D to d = 256, then lift only those d coordinates, M = 33k. Much smaller, much faster Ridge solve.

Re: A polynomial autoencoder beats PCA on transformer embeddings

#39

Post by ComplexSystems »

pleshkov wrote: post #38
ComplexSystems wrote: post #32

What makes this different from just kernel PCA with the quadratic kernel?

It's close but not the same. Kernel PCA lifts all D coordinates which gives M around 525k at D = 1024. In the post I do PCA first to reduce D to d = 256, then lift only those d coordinates, M = 33k. Much smaller, much faster Ridge solve.

That makes sense. If you could magically just get the top d PCs in quadratic kernel space without having to compute the whole kernel matrix, and then just do top-d quadratic PCs -> ridge, would that be better than doing the PCA -> top-d -> quadratic kernel ridge as you are now?
Post reply on HN