Faster convergence for diffusion models
1–10 of 23 posts
Re: Faster convergence for diffusion models
#2Still waiting for a competitive diffusion llm
Re: Faster convergence for diffusion models
#3The title is not wrong, but it also doesn't feel correct either. What they do here is they use a pre-trained model to guide the training of a 2nd model. Of course, that massively speeds up training of the 2nd model. But it's not like you can now train a diffusion model from scratch 20x faster. Instead, this is a technique for transplanting an existing model onto a different architecture so that you don't have to start training from 0.
Re: Faster convergence for diffusion models
#4The title is not wrong, but it also doesn't feel correct either. What they do here is they use a pre-trained model to guide the training of a 2nd model. Of course, that massively speeds up training of the 2nd model. But it's not like you can now train a diffusion model from scratch 20x faster. Instead, this is a technique for transplanting an existing model onto a different architecture so that you don't have to star…
Yeah, I wonder whether this still saves compute if you include the compute used to train DINOV2/whatever representation model you'd like to use?
Re: Faster convergence for diffusion models
#5The title is not wrong, but it also doesn't feel correct either. What they do here is they use a pre-trained model to guide the training of a 2nd model. Of course, that massively speeds up training of the 2nd model. But it's not like you can now train a diffusion model from scratch 20x faster. Instead, this is a technique for transplanting an existing model onto a different architecture so that you don't have to star…
Yes, now it seems obvious, but before this it wasn't clear that that would be something that could speed things up, due to the fact that the pretrained model was trained on a separate objective. It's a brilliant idea that works amazingly.
Re: Faster convergence for diffusion models
#6Still waiting for a competitive diffusion llm
Why?
Re: Faster convergence for diffusion models
#7Still waiting for a competitive diffusion llm
Why?
Diffusion works significantly better for images than sequential pixel generation, there is a good chance it would work better for language as well.
Sequential generation used to be state of the art in 2016 and it's basically how current LLMs work:
Re: Faster convergence for diffusion models
#8Earlier quoted context omitted.
Why?
Diffusion works significantly better for images than sequential pixel generation, there is a good chance it would work better for language as well. Sequential generation used to be state of the art in 2016 and it's basically how current LLMs work: https://arxiv.org/abs/1601.06759
Neural LMs used to be based on recurrent architectures until the Transformer came along. That architecture is not recursive.
I am not sure that a diffusion approach is all that suitable for generating language. Word are much more discrete than pixels.
Re: Faster convergence for diffusion models
#9Earlier quoted context omitted.
Diffusion works significantly better for images than sequential pixel generation, there is a good chance it would work better for language as well. Sequential generation used to be state of the art in 2016 and it's basically how current LLMs work: https://arxiv.org/abs/1601.06759
Neural LMs used to be based on recurrent architectures until the Transformer came along. That architecture is not recursive. I am not sure that a diffusion approach is all that suitable for generating language. Word are much more discrete than pixels.
I meant sequential generation, I didn't mean using an RNN.
Diffusion doesn't work on pixels directly either, it works on a latent representation.
Re: Faster convergence for diffusion models
#10Earlier quoted context omitted.
Neural LMs used to be based on recurrent architectures until the Transformer came along. That architecture is not recursive. I am not sure that a diffusion approach is all that suitable for generating language. Word are much more discrete than pixels.
I meant sequential generation, I didn't mean using an RNN. Diffusion doesn't work on pixels directly either, it works on a latent representation.
All NNs work on latent representations.