Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The universal weight subspace hypothesis

arxiv.org

Post reply on HN
Print view View original

101–110 of 146 posts

Re: The universal weight subspace hypothesis

#101

Post by swivelmaster »

augment_me wrote: post #93
daemonologist wrote: post #86

Earlier quoted context omitted.

I agree - the results on the finetunes are not very surprising. The trained-from-scratch ResNets (Figure 2 and Section 3.2.1) are definitely more interesting, though somewhat limited in scope. In any case, my impression is that this is not immediately more useful than a LoRA (and is probably not intended to be), but is maybe an avenue for further research.

I don't think its that surprising actually. And I think the paper in general completely oversells the idea. The ResNet results hold from scratch because strict local constraints (e.g., 3x3 convolutions) force the emergence of fundamental signal-processing features (Gabor/Laplacian filters) regardless of the dataset. The architecture itself enforces the subspace. The Transformer/ViT results rely on fine-tunes because…

You’ve explained this in plain and simple language far more directly than the linked study. Score yet another point for the theory that academic papers are deliberately written to be obtuse to laypeople rather than striving for accessibility.

Re: The universal weight subspace hypothesis

#102

Post by Havoc »

modeless wrote: post #58

This seems confusingly phrased. When they say things like "500 Vision Transformers", what they mean is 500 finetunes of the same base model, downloaded from the huggingface accounts of anonymous randos. These spaces are only "universal" to a single pretrained base model AFAICT. Is it really that surprising that finetunes would be extremely similar to each other? Especially LoRAs? I visited one of the models they refe…

Looks like both mistral and llamas per text but yeah incredibly underwhelming for „universal“

Re: The universal weight subspace hypothesis

#103

Post by bmacho »

swivelmaster wrote: post #101
augment_me wrote: post #93

Earlier quoted context omitted.

I don't think its that surprising actually. And I think the paper in general completely oversells the idea. The ResNet results hold from scratch because strict local constraints (e.g., 3x3 convolutions) force the emergence of fundamental signal-processing features (Gabor/Laplacian filters) regardless of the dataset. The architecture itself enforces the subspace. The Transformer/ViT results rely on fine-tunes because…

You’ve explained this in plain and simple language far more directly than the linked study. Score yet another point for the theory that academic papers are deliberately written to be obtuse to laypeople rather than striving for accessibility.

Vote for the Party that promises academic grants for people that write 1k character long forum posts for the laypeople instead of other experts of the field.

Re: The universal weight subspace hypothesis

#105

Post by swivelmaster »

bmacho wrote: post #103
swivelmaster wrote: post #101

Earlier quoted context omitted.

You’ve explained this in plain and simple language far more directly than the linked study. Score yet another point for the theory that academic papers are deliberately written to be obtuse to laypeople rather than striving for accessibility.

Vote for the Party that promises academic grants for people that write 1k character long forum posts for the laypeople instead of other experts of the field.

I’m not sure that’s something we get to vote on.

Re: The universal weight subspace hypothesis

#107

Post by rocqua »

bmacho wrote: post #103
swivelmaster wrote: post #101

Earlier quoted context omitted.

You’ve explained this in plain and simple language far more directly than the linked study. Score yet another point for the theory that academic papers are deliberately written to be obtuse to laypeople rather than striving for accessibility.

Vote for the Party that promises academic grants for people that write 1k character long forum posts for the laypeople instead of other experts of the field.

I don't think the parent post is complaining that academics are writing proposals (e.g as opposed to people with common sense). Instead, it seems to me that he is complaining that academics are writing proposals and papers to impress funding committees and journal editors, and to some extend to increase their own clout among their peers. Instead of writing to communicate clearly and honestly to their peers, or occasionally to laymen.

And this critique is likely not aimed at academics so much as the systems and incentives of academia. This is partially on the parties managing grants (caring much more about impact and visibility than actually moving science forwards, which means everyone is scrounging for or lying about low hanging fruit). It is partially on those who set (or rather maintain) the culture at academic institutions of gathering clout by getting 'impactful' publications. And those who manage journals also share blame, by trying to defend their moat, very much hamming up "high impact", and aggressively rent-seeking.

Re: The universal weight subspace hypothesis

#108

Post by moi2388 »

woopsn wrote: post #76
scotty79 wrote: post #51

Earlier quoted context omitted.

16 seems like a suspiciously round number ... why not 17 or 13? ... is this just result of some bug in the code they used to do their science? or is it just that 16 was arbitrarily chosen by them as close enough to the actual minimal number of dimensions necessary?

It's a little arbitrary. Look at the graph on page 6, there's no steep gap in the spectrum there. 16 just about the balance point

But there is a steep gap in the spectrum at 16 on page 7

Re: The universal weight subspace hypothesis

#109

Post by eru »

swivelmaster wrote: post #105
bmacho wrote: post #103

Earlier quoted context omitted.

Vote for the Party that promises academic grants for people that write 1k character long forum posts for the laypeople instead of other experts of the field.

I’m not sure that’s something we get to vote on.

On the margin, you can let anything influence your voting decision.

Re: The universal weight subspace hypothesis

#110

Post by statusfailed »

I saw a similar (I think!) paper "Grassmannian Optimization Drives Generalization in Overparameterized DNN" at OPT-ML at neurips last week[0]

This is a little outside my area, but I think the relevant part of that abstract is "Gradient-based optimization follows horizontal lifts across low-dimensional subspaces in the Grassmannian Gr(r, p), where r  p is the rank of the Hessian at the optimum"

I think this question is super interesting though: why can massively overparametrised models can still generalise?

[0]: https://opt-ml.org/papers/2025/paper90.pdf

Post reply on HN