[flagged]
Acting as a pass-through for LLMs is logically equivalent to wiring up a bot account.
The universal weight subspace hypothesis
21–30 of 146 posts
Re: The universal weight subspace hypothesis
#22interesting.. this could make training much faster if there’s a universal low dimensional space that models naturally converge into, since you could initialize or constrain training inside that space instead of spending massive compute rediscovering it from scratch every time
Or an architecture chosen for that subspace or some of its properties as inductive biases.
Re: The universal weight subspace hypothesis
#23It's basically way better than LoRA under all respects and could even be used to speed up inference. I wonder whether the big models are not using it already... If not we'll see a blow up in capabilities very, very soon.
What they've shown is that you can find the subset of parameters responsible for transfer of capability to new tasks.
Does it apply to completely novel tasks? No, that would be magic. Tasks that need new features or representations break the method, but if it fits in the same domain then the answer is "YES".
Here's a very cool analogy from GPT 5.1 which hits the nail in the head in explaining the role of subspace in learning new tasks by analogy with 3d graphics.
Think of 3D character animation rigs:
• The mesh has millions of vertices (11M weights).
• Expressions are controlled via:
• “smile”
• “frown”
• “blink”
Each expression is just:
mesh += α_i \* basis_expression_i
Hundreds of coefficients modify millions of coordinates.Re: The universal weight subspace hypothesis
#24I find myself wanting genetic algorithms to be applied to try to develop and improve these structures... But I always want Genetic Algorithms to show up in any discussion about neural networks...
I have a real soft spot for the genetic algorithm as a result of reading Levy's "Artificial Life" when I was a kid. The analogy to biological life is more approachable to my poor math education than neural networks. I can grok crossover and mutation pretty easily. Backpropagation is too much for my little brain to handle.
I do too, and for the same reasons. Levy's book had a huge impact on me in general.
Re: The universal weight subspace hypothesis
#25Now that we know about this, that the calculations in the trained models follow some particular forms, is there an approximation algorithm to run the models without GPUs?
Re: The universal weight subspace hypothesis
#26What if all models are secretly just fine tunes of llama?
Re: The universal weight subspace hypothesis
#27The central claim, or "Universal Weight Subspace Hypothesis," is that deep neural networks, even when trained on completely different tasks (like image recognition vs. text generation) and starting from different random conditions, tend to converge to a remarkably similar, low-dimensional "subspace" in their massive set of weights.
Re: The universal weight subspace hypothesis
#28[deleted]
Re: The universal weight subspace hypothesis
#29interesting.. this could make training much faster if there’s a universal low dimensional space that models naturally converge into, since you could initialize or constrain training inside that space instead of spending massive compute rediscovering it from scratch every time
Wouldn't this also mean that there's an inherent limit to that sort of model?
Re: The universal weight subspace hypothesis
#30> Principal component analysis of 200 GPT2, 500 Vision Transformers, 50 LLaMA-
8B, and 8 Flan-T5 models reveals consistent sharp spectral decay - strong evidence that a small number of weight
directions capture dominant variance despite vast differences in training data, objectives, and initialization.
Isn't it obvious?