The universal weight subspace hypothesis
1–10 of 146 posts
Re: The universal weight subspace hypothesis
#2They compressed the compression? Or identified an embedding that can "bootstrap" training with a headstart ?
Not a technical person just trying to put it in other words.
Re: The universal weight subspace hypothesis
#3What's the relationship with the Platonic Representation Hypothesis?
Re: The universal weight subspace hypothesis
#4interesting.. this could make training much faster if there’s a universal low dimensional space that models naturally converge into, since you could initialize or constrain training inside that space instead of spending massive compute rediscovering it from scratch every time
Re: The universal weight subspace hypothesis
#5I find myself wanting genetic algorithms to be applied to try to develop and improve these structures...
But I always want Genetic Algorithms to show up in any discussion about neural networks...
Re: The universal weight subspace hypothesis
#6They compressed the compression? Or identified an embedding that can "bootstrap" training with a headstart ? Not a technical person just trying to put it in other words.
They identified that the compressed representation has structure to it that could potentially be discovered more quickly. It’s unclear if it would also make it easier to compress further but that’s possible.
Re: The universal weight subspace hypothesis
#7I immediately started thinking that if there are such patterns maybe they capture something about the deeper structure of the universe.
Re: The universal weight subspace hypothesis
#8What's the relationship with the Platonic Representation Hypothesis?
From what I can tell, they are very closely related (i.e. the shared representational structures would likely make good candidates for Platonic representations, or rather, representations of Platonic categories). In any case, it seems like there should be some sort of interesting mapping between the two.
Re: The universal weight subspace hypothesis
#9What's the relationship with the Platonic Representation Hypothesis?
I hope someone much smarter than I answers this. I’ve been noticing an uptick platonic and neo-platonic discourse in the zeitgeist and am wondering if we’re converging on something profound.
Re: The universal weight subspace hypothesis
#10(Finds a compression artifact) "Is this the meaning of consciousness???"