Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Neural Network Diffusion

arxiv.org

Post reply on HN
Print view View original

11–20 of 92 posts

Re: Neural Network Diffusion

#11

Post by fnordpiglet »

mattnewton wrote: post #7
falcor84 wrote: post #4

Seems like we're getting very close to recursive self-improvement [0]. [0] https://www.lesswrong.com/tag/recursive-self-improvement

I upvoted because this was my first thought too, but reading the abstract and skimming the paper makes me think it’s not really an advance for general recursive improvement. I think the title makes people think this is a text -> model model, when it is really a bunch of model weights -> new model weights optimizer for a specific architecture and problem. Still a potentially very useful idea for learning from a bunch…

I suspect this is useful for porting one vector space to another which is an open problem when you’ve trained one model with one architecture and need to port it to another architecture without paying the full retraining cost.

Re: Neural Network Diffusion

#15

Post by goggy_googy »

"We synthesize 100 novel parameters by feeding random noise into the latent diffusion model and the trained decoder." Cool that patterns exist at this level, but also, 100 params means we have a long way to go before this process is efficient enough to synthesize more modern-sized models.

Re: Neural Network Diffusion

#16

Post by astrange »

falcor84 wrote: post #4

Seems like we're getting very close to recursive self-improvement [0]. [0] https://www.lesswrong.com/tag/recursive-self-improvement

No, this is an example of an existing technique called hypernetworks.

It's not "recursive self improvement", which is just a belief that magic is real and you can wish an AI into existence. In particular, this one needs too much training data, and you can't define "improvement" without knowing what to improve to.

Re: Neural Network Diffusion

#18

Post by vessenes »

I wasn't sure if this paper was parody on reading the abstract. It's not parody. Two things stand out to me: first is the idea of distilling these networks down into a smaller latent space, and then mucking around with that. That's interesting, and cross-sections a bunch of interesting topics like interpretability, compression, training, over- and under-.. The second is that they show the diffusion models don't just converge on similar parameters as the ones they train against/diffuse into, and that's also interesting.

I confess I'm not sure what I'd do with this in the random grab bag of Deep Learning knowledge I have, but I think it's pretty fascinating. I might like to see a trained latent encoder that works well on a bunch of different neural networks; maybe that thing would be a good tool for interpreting / inspecting.

Re: Neural Network Diffusion

#20

Post by hackerlight »

jackblemming wrote: post #17

The state of art neural net architecture, whether that be transformers or the like, trained on self play to optimize non-differentiable but highly efficient architectures is the way.

According to Hinton, before transformers were shown to work well, learning model architectures was Google's main focus
Post reply on HN