Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Beyond Diffusion: Inductive Moment Matching

lumalabs.ai

Post reply on HN
Print view View original

21–30 of 33 posts

Re: Beyond Diffusion: Inductive Moment Matching

#21

Post by bbminner »

bearobear wrote: post #11
bbminner wrote: post #4

Can anyone share insight into how this is different from consistency models? The insight seems quite similar?

Consistency models is a special case of IMM where you do moment matching with 1 sample from each distribution (i.e., you cannot match distributions properly). See Fig 5 for an ablation study, of course, adding more samples when you are doing moment matching makes it more stable during training :)

Makes sense. How can you even approximately estimate higher order differences in conditional moments in such a high dim space? Seems statistically impossible to get a reasonable estimate for a gradient. Moment matching in sample space has always been very hard.

Re: Beyond Diffusion: Inductive Moment Matching

#22

Post by b2w »

xela79 wrote: post #19

this went over my head quickly; read through it a few times, than asked GPT for a summary on my level of understanding, which does clear it up for me ,personally , to grasp the overall idea: Alright, imagine you have a big box of LEGO bricks, and you're trying to build a really cool spaceship. There are two main ways people usually build things like this: Step-by-step (Autoregressive Models) – Imagine you put one LEG…

So like intuitive photographic memory?

Re: Beyond Diffusion: Inductive Moment Matching

#23

Post by Climatebamb »

b2w wrote: post #22
xela79 wrote: post #19

this went over my head quickly; read through it a few times, than asked GPT for a summary on my level of understanding, which does clear it up for me ,personally , to grasp the overall idea: Alright, imagine you have a big box of LEGO bricks, and you're trying to build a really cool spaceship. There are two main ways people usually build things like this: Step-by-step (Autoregressive Models) – Imagine you put one LEG…

So like intuitive photographic memory?

More like "Oh i remember what you roughly want, i rememeber basic steps of reaching it just not details, lets generate the details" vs. "learning x steps from noise to image".

You make the way of reaching your target faster.

Re: Beyond Diffusion: Inductive Moment Matching

#24

Post by qumpis »

bearobear wrote: post #9
programjames wrote: post #3

Anyone willing to give an intuitive summary of what they did mathwise? The math in the paper is super ugly to churn through.

Last author here (I also did the DDIM paper, https://arxiv.org/abs/2010.02502 ). I know this is going to be very tricky math-wise (and in the paper we just wrote the most general thing to make reviewers happy), so I tried to explain the idea more easily under the blog post ( https://lumalabs.ai/news/inductive-moment-matching ). If you look at how a single step of the DDIM sampler interacts with the target timestep, i…

What happens if we don't add any moments matching objective? e.g. at train time just fit a diffusion model that predicts the target given any pair of timesteps (t, t')? Why is moment matching critical here?

Also regarding linearity, why is it inflexible? It seems quite convenient that a simple linear interpolation is used for reconstruction, besides, even in DDIM, the directions towards the final target changes at each step as the images become less noisy. In standard diffusion models or even flow matching, denoising is always equal to the prediction of the original data + direction from current timestep to the timestep t'. Just to be clear, it is intuitive that such models are inferior in few-step generations since they don't optimise for test time efficiency (in terms of the tradeoff of quality vs compute), but it's unclear what inflexibility exists there beyond this limitation.

Clearly there's no expected benefit in quality if all timesteps are used in denoising?

Re: Beyond Diffusion: Inductive Moment Matching

#25

Post by kadushka »

oofbey wrote: post #8
programjames wrote: post #3

Anyone willing to give an intuitive summary of what they did mathwise? The math in the paper is super ugly to churn through.

In normal diffusion you train a model to take lots of tiny steps, all the same small size. e.g. "You're gonna take 20 steps, at times [1.0, 0.95, 0.90, 0.85...]" and each time the model takes that small fixed-size step to make the image look better. Here they train a model to say "I'm gonna ask you to take a step from time B to A - might be a small step, might be a big step - but whatever size it is, make the image t…

No, we typically train any diffusion model on a single step (randomly chosen).

Re: Beyond Diffusion: Inductive Moment Matching

#26

Post by kadushka »

svachalek wrote: post #6
lukasb wrote: post #5

"Inference can generally be scaled along two dimensions: extending sequence length (in autoregressive models), and augmenting the number of refinement steps (in diffusion models)." Does this mean that diffusion models for text could scale inference compute to improve quality for a fixed-length output?

Yes, although so far it seems the main advantage of text diffusion models is that they're really, really fast. Iterations reach an asymptote very quickly.

I don’t know which text diffusion models you’re talking about, the latest and greatest is this one: https://arxiv.org/abs/2502.09992 and it’s extremely slow – couple of orders of magnitude slower than a regular LLM, mainly because it does not support KV caching, and requires many full sequence processing steps per token.

Re: Beyond Diffusion: Inductive Moment Matching

#28

Post by hyperbovine »

programjames wrote: post #3

Anyone willing to give an intuitive summary of what they did mathwise? The math in the paper is super ugly to churn through.

The math is totally standard if you've read recent important papers on score matching and flow matching. If you haven't, well, I can't see how you could possibly hope to understand this work at a technical level anyways.

Re: Beyond Diffusion: Inductive Moment Matching

#29

Post by azinman2 »

xela79 wrote: post #19

this went over my head quickly; read through it a few times, than asked GPT for a summary on my level of understanding, which does clear it up for me ,personally , to grasp the overall idea: Alright, imagine you have a big box of LEGO bricks, and you're trying to build a really cool spaceship. There are two main ways people usually build things like this: Step-by-step (Autoregressive Models) – Imagine you put one LEG…

Thank you, this is helpful framing. Obviously all the details are missing, but the blog post was impenetrable for me, and I’m quite technical.

Re: Beyond Diffusion: Inductive Moment Matching

#30

Post by janalsncm »

kadushka wrote: post #26
svachalek wrote: post #6

Earlier quoted context omitted.

Yes, although so far it seems the main advantage of text diffusion models is that they're really, really fast. Iterations reach an asymptote very quickly.

I don’t know which text diffusion models you’re talking about, the latest and greatest is this one: https://arxiv.org/abs/2502.09992 and it’s extremely slow – couple of orders of magnitude slower than a regular LLM, mainly because it does not support KV caching, and requires many full sequence processing steps per token.

I’m not familiar with that paper but it would probably be best to compare speeds with an unoptimized transformer decoder. The Vaswani paper came out 8 years ago so implementations will be pretty highly optimized at this point.

On the other hand if there was a theoretical reason why text diffusion models could never be faster than autoregressive transformers it would be notable.

Post reply on HN