Neural Network Diffusion

Post by daxfohl » Sun, Feb 25, 2024, 1:00 AM UTC

vessenes wrote: post #18

I wasn't sure if this paper was parody on reading the abstract. It's not parody. Two things stand out to me: first is the idea of distilling these networks down into a smaller latent space, and then mucking around with that. That's interesting, and cross-sections a bunch of interesting topics like interpretability, compression, training, over- and under-.. The second is that they show the diffusion models don't just…

You could use this and try to tease out something similar to https://news.ycombinator.com/item?id=39487124, but for NNs instead of images. Maybe it's possible to have this NN diffusion model explain the pieces of the NN they generate and why parameters have those values.

If we can get that, then maybe we don't even need to train anymore; it'd be possible to start to generate NNs algorithmically.

Post by daxfohl » Sun, Feb 25, 2024, 8:15 AM UTC

SubiculumCode wrote: post #56

vessenes wrote: post #18

I wasn't sure if this paper was parody on reading the abstract. It's not parody. Two things stand out to me: first is the idea of distilling these networks down into a smaller latent space, and then mucking around with that. That's interesting, and cross-sections a bunch of interesting topics like interpretability, compression, training, over- and under-.. The second is that they show the diffusion models don't just…

Perhaps doing this to generate 10 similar but different versions of a model can then be fed into mixture of experts?

Or a good way to teleport out of local minima while training. Create a few clones and take the one with the steepest gradients.

Re: Neural Network Diffusion

Re: Neural Network Diffusion