Faster convergence for diffusion models • Hacker News Board

Re: Faster convergence for diffusion models

#21

Post by famouswaffles » Mon, Oct 14, 2024, 8:23 PM UTC

WithinReason wrote: post #7

kleiba wrote: post #6

Earlier quoted context omitted.

Why?

Diffusion works significantly better for images than sequential pixel generation, there is a good chance it would work better for language as well. Sequential generation used to be state of the art in 2016 and it's basically how current LLMs work: https://arxiv.org/abs/1601.06759

The most popular method using autoregression in image generation space is to predict image patches/tokens and not pixels, though that still scales worse than diffusion.

A fairly new but promising approach for autoregression that seems to scale as well as diffusion is predicting the next image scale/resolution rather than the next image patch.

https://arxiv.org/abs/2404.02905

Re: Faster convergence for diffusion models

#22

Post by orbital-decay » Wed, Oct 16, 2024, 4:57 PM UTC

nextaccountic wrote: post #20

gdiamos wrote: post #2

Still waiting for a competitive diffusion llm

So I can't find that paper that was posted on HN that said that, when viewed under the right theoretical framework, asserts that diffusion and transformers are doing the same thing under a different basis.. am I misrembering something?

https://sander.ai/2024/09/02/spectral-autoregression.html

Re: Faster convergence for diffusion models

#23

Post by byyoung3 » Thu, Oct 17, 2024, 11:00 AM UTC

psb217 wrote: post #18

byyoung3 wrote: post #5

Earlier quoted context omitted.

Yes, now it seems obvious, but before this it wasn't clear that that would be something that could speed things up, due to the fact that the pretrained model was trained on a separate objective. It's a brilliant idea that works amazingly.

It's a classic "Will it work? IDK, maybe. Let's try it and find out..." paper.

haha yeah I mean I think they are all like that to a certain extent