Beyond Diffusion: Inductive Moment Matching • Hacker News Board

Re: Beyond Diffusion: Inductive Moment Matching

#31

Post by littlestymaar » Wed, Mar 12, 2025, 5:42 PM UTC

bearobear wrote: post #9

programjames wrote: post #3

Anyone willing to give an intuitive summary of what they did mathwise? The math in the paper is super ugly to churn through.

Last author here (I also did the DDIM paper, https://arxiv.org/abs/2010.02502 ). I know this is going to be very tricky math-wise (and in the paper we just wrote the most general thing to make reviewers happy), so I tried to explain the idea more easily under the blog post ( https://lumalabs.ai/news/inductive-moment-matching ). If you look at how a single step of the DDIM sampler interacts with the target timestep, i…

Stupid question, what's a “timestep” in that context?

Re: Beyond Diffusion: Inductive Moment Matching

#32

Post by kadushka » Wed, Mar 12, 2025, 10:56 PM UTC

janalsncm wrote: post #30

kadushka wrote: post #26

Earlier quoted context omitted.

I don’t know which text diffusion models you’re talking about, the latest and greatest is this one: https://arxiv.org/abs/2502.09992 and it’s extremely slow – couple of orders of magnitude slower than a regular LLM, mainly because it does not support KV caching, and requires many full sequence processing steps per token.

I’m not familiar with that paper but it would probably be best to compare speeds with an unoptimized transformer decoder. The Vaswani paper came out 8 years ago so implementations will be pretty highly optimized at this point. On the other hand if there was a theoretical reason why text diffusion models could never be faster than autoregressive transformers it would be notable.

There’s not enough improvement over regular LLMs to motivate optimization effort. Recall that the original transformer was well received because it was fast and scalable compared to RNNs.

Re: Beyond Diffusion: Inductive Moment Matching

#33

Post by b2w » Thu, Mar 13, 2025, 5:23 AM UTC

Climatebamb wrote: post #23

b2w wrote: post #22

Earlier quoted context omitted.

So like intuitive photographic memory?

More like "Oh i remember what you roughly want, i rememeber basic steps of reaching it just not details, lets generate the details" vs. "learning x steps from noise to image". You make the way of reaching your target faster.

This helped me, thanks!