Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The universal weight subspace hypothesis

arxiv.org

Post reply on HN
Print view View original

71–80 of 146 posts

Re: The universal weight subspace hypothesis

#71

Post by seeknotfind »

alyxya wrote: post #65

I’ve had a hard time parsing what exactly the paper is trying to explain. So far I’ve understood that their comparison seems to be models within the same family and same weight tensor dimensions, so they aren’t showing a common subspace when there isn’t a 1:1 match between weight tensors in a ViT and GPT2. The plots showing the distribution of principal component values presumably does this on every weight tensor, bu…

Yeah, it sounds platonic the way it's written, but it seems more like a hyped model compression technique.

Re: The universal weight subspace hypothesis

#72

Post by RandyOrion »

> From their project page:

> We analyze over 1,100 deep neural networks—including 500 Mistral-7B LoRAs and 500 Vision Transformers. We provide the first large-scale empirical evidence that networks systematically converge to shared, low-dimensional spectral subspaces, regardless of initialization, task, or domain.

I instantly thought of muon optimizer which provides high-rank gradient updates and Kimi-k2 which is trained using muon, and see no related references.

The 'universal' in the title is not that universal.

Re: The universal weight subspace hypothesis

#73

Post by lucid-dev »

Pretty funny if you ask me. Maybe we can start to realize now: "The common universal subspace between human individuals makes it easier for all of them to do 'novel' tasks so long as their ego and personality doesn't inhibit that basic capacity."

And that: "Defining 'novel' as 'not something that you've said before even though your using all the same words, concepts, linguistic tools, etc., doesn't actually make it 'novel'"

Point being, yeah duh, what's the difference between what any of these models are doing anyway? It would be far more surprising if they discovered a *different* or highly-unique subspace for each one!

Someone gives you a magic lamp and the genie comes out and says "what do you wish for"?

That's still the question. The question was never "why do all the genies seem to be able to give you whatever you want?"

Re: The universal weight subspace hypothesis

#74

Post by tsurba »

Many discriminative models converge to same representation space up to a linear transformation. Makes sense that a linear transformation (like PCA) would be able to undo that transformation.

https://arxiv.org/abs/2007.00810

Without properly reading the linked article, if thats all this is, not a particularly new result. Nevertheless this direction of proofs is imo at the core of understanding neural nets.

Re: The universal weight subspace hypothesis

#75

Post by mlpro »

tsurba wrote: post #74

Many discriminative models converge to same representation space up to a linear transformation. Makes sense that a linear transformation (like PCA) would be able to undo that transformation. https://arxiv.org/abs/2007.00810 Without properly reading the linked article, if thats all this is, not a particularly new result. Nevertheless this direction of proofs is imo at the core of understanding neural nets.

It's about weights/parameters, not representations.

Re: The universal weight subspace hypothesis

#76

Post by woopsn »

scotty79 wrote: post #51
altairprime wrote: post #19

For those trying to understand the most important parts of the paper, here's what I think is the most significant two statements, subquoted out of two (consecutive) paragraphs midway through the paper: > we selected five additional, previously unseen pretrained ViT models for which we had access to evaluation data. These models, considered out-of-domain relative to the initial set, had all their weights reconstructed…

16 seems like a suspiciously round number ... why not 17 or 13? ... is this just result of some bug in the code they used to do their science? or is it just that 16 was arbitrarily chosen by them as close enough to the actual minimal number of dimensions necessary?

It's a little arbitrary. Look at the graph on page 6, there's no steep gap in the spectrum there. 16 just about the balance point

Re: The universal weight subspace hypothesis

#77

Post by mlpro »

modeless wrote: post #58

This seems confusingly phrased. When they say things like "500 Vision Transformers", what they mean is 500 finetunes of the same base model, downloaded from the huggingface accounts of anonymous randos. These spaces are only "universal" to a single pretrained base model AFAICT. Is it really that surprising that finetunes would be extremely similar to each other? Especially LoRAs? I visited one of the models they refe…

Why would they be similar if they are trained on very different data? Also, trained from scratch models are also analyzed, imo.

Re: The universal weight subspace hypothesis

#78

Post by tsurba »

mlpro wrote: post #75
tsurba wrote: post #74

Many discriminative models converge to same representation space up to a linear transformation. Makes sense that a linear transformation (like PCA) would be able to undo that transformation. https://arxiv.org/abs/2007.00810 Without properly reading the linked article, if thats all this is, not a particularly new result. Nevertheless this direction of proofs is imo at the core of understanding neural nets.

It's about weights/parameters, not representations.

True, good point, maybe not a straightforward consequence to extend to weights.

Re: The universal weight subspace hypothesis

#80

Post by [deleted] »

mlpro wrote: post #77
modeless wrote: post #58

This seems confusingly phrased. When they say things like "500 Vision Transformers", what they mean is 500 finetunes of the same base model, downloaded from the huggingface accounts of anonymous randos. These spaces are only "universal" to a single pretrained base model AFAICT. Is it really that surprising that finetunes would be extremely similar to each other? Especially LoRAs? I visited one of the models they refe…

Why would they be similar if they are trained on very different data? Also, trained from scratch models are also analyzed, imo.

[deleted]
Post reply on HN