Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The universal weight subspace hypothesis

arxiv.org

Post reply on HN
Print view View original

41–50 of 146 posts

Re: The universal weight subspace hypothesis

#41

Post by farhanhubble »

westoncb wrote: post #31
altairprime wrote: post #19

For those trying to understand the most important parts of the paper, here's what I think is the most significant two statements, subquoted out of two (consecutive) paragraphs midway through the paper: > we selected five additional, previously unseen pretrained ViT models for which we had access to evaluation data. These models, considered out-of-domain relative to the initial set, had all their weights reconstructed…

> So, they found an underlying commonality among the post-training structures in 50 LLaMA3-8B models, 177 GPT-2 models, and 8 Flan-T5 models; and, they demonstrated that the commonality could in every case be substituted for those in the original models with no loss of function; and noted that they seem to be the first to discover this. Could someone clarify what this means in practice? If there is a 'commonality' wh…

It might we worth it to use that subset to initialize the weights of future models but more importantly you could save a huge number of computational cycles by using the lower dimensional weights at the time of inference.

Re: The universal weight subspace hypothesis

#42

Post by embedding-shape »

EvanAnderson wrote: post #12
VikingCoder wrote: post #5

I find myself wanting genetic algorithms to be applied to try to develop and improve these structures... But I always want Genetic Algorithms to show up in any discussion about neural networks...

I have a real soft spot for the genetic algorithm as a result of reading Levy's "Artificial Life" when I was a kid. The analogy to biological life is more approachable to my poor math education than neural networks. I can grok crossover and mutation pretty easily. Backpropagation is too much for my little brain to handle.

> Backpropagation is too much for my little brain to handle.

I just stumbled upon a very nice description of the core of it, right here: https://www.youtube.com/watch?v=AyzOUbkUf3M&t=133s

Almost all talks by Geoffrey Hinton (left side on https://www.cs.toronto.edu/~hinton/) are in very approachable if you're passingly familiar with some ML.

Re: The universal weight subspace hypothesis

#44

Post by altairprime »

VikingCoder wrote: post #5

I find myself wanting genetic algorithms to be applied to try to develop and improve these structures... But I always want Genetic Algorithms to show up in any discussion about neural networks...

That would be an excellent use of GA and all the other 'not based on training a network' methods, now that we have a target and can evaluate against it!

Re: The universal weight subspace hypothesis

#45

Post by crooked-v »

mlpro wrote: post #36
nothrowaways wrote: post #30

> Principal component analysis of 200 GPT2, 500 Vision Transformers, 50 LLaMA- 8B, and 8 Flan-T5 models reveals consistent sharp spectral decay - strong evidence that a small number of weight directions capture dominant variance despite vast differences in training data, objectives, and initialization. Isn't it obvious?

Not really. If the models are trained on different dataset - like one ViT trained on satellite images and another on medical X-rays - one would expect their parameters, which were randomly initialized to be completely different or even orthogonal.

Now I wonder how much this "Universal Subspace" corresponds to the same set of scraped Reddit posts and pirated books that apparently all the bigcorps used for model training. Is it 'universal' because it's universal, or because the same book-pirating torrents got reused all over?

Re: The universal weight subspace hypothesis

#46

Post by horsepatties »

I hope that this leads to more efficient models. And it’s intuitive- it seems as though you could find the essence of a good model and a model reduced to that essence would be more efficient. But, this is theoretical. I can also theorize flying cars- many have, it seems doable and achievable, but yet I see no flying cars on my way to work.

Re: The universal weight subspace hypothesis

#47

Post by scotty79 »

bigbuppo wrote: post #29
kacesensitive wrote: post #4

interesting.. this could make training much faster if there’s a universal low dimensional space that models naturally converge into, since you could initialize or constrain training inside that space instead of spending massive compute rediscovering it from scratch every time

Wouldn't this also mean that there's an inherent limit to that sort of model?

> Wouldn't this also mean that there's an inherent limit to that sort of model?

If all need just 16 dimensions if we ever make one that needs 17 we know we are making progress instead of running in circles.

Re: The universal weight subspace hypothesis

#49

Post by 100721 »

pagekicker wrote: post #43
100721 wrote: post #21

Earlier quoted context omitted.

Acting as a pass-through for LLMs is logically equivalent to wiring up a bot account.

No, it's not, unless you can argue that the bot would have thought of asking the same question I did, which is unlikely.

Let’s define the bot as one that asks LLMs to visualize concepts, then.

Now I’ve argued that the bot would very likely have thought of the same question you did, and my original assertion stands.

Re: The universal weight subspace hypothesis

#50

Post by scotty79 »

altairprime wrote: post #19

For those trying to understand the most important parts of the paper, here's what I think is the most significant two statements, subquoted out of two (consecutive) paragraphs midway through the paper: > we selected five additional, previously unseen pretrained ViT models for which we had access to evaluation data. These models, considered out-of-domain relative to the initial set, had all their weights reconstructed…

"16 dimensions is all you need" ... to do human achievable stuff at least
Post reply on HN