Beyond Diffusion: Inductive Moment Matching
lumalabs.ai
Beyond Diffusion: Inductive Moment Matching
1–10 of 33 posts
Re: Beyond Diffusion: Inductive Moment Matching
#2I've been lightly following this type of research for a few years. I immediately recognized the broad idea as stemming from the lab of the ridiculously prolific Stefano Ermon. He's always taken a unique angle for generative models since the before times of GenAI. I was fortunate to get lunch with him in grad school after a talk he gave. Seeing the work from his lab in these modern days is compelling, I always figured his style of research would break out into the mainstream eventually. I'm hopeful the the future of ML improvements come from clever test-time algorithms like this article shows. I'm looking forward to when you can train a high quality generative model without needing a super cluster or webscale data.
Re: Beyond Diffusion: Inductive Moment Matching
#3Anyone willing to give an intuitive summary of what they did mathwise? The math in the paper is super ugly to churn through.
Re: Beyond Diffusion: Inductive Moment Matching
#4Can anyone share insight into how this is different from consistency models? The insight seems quite similar?
Re: Beyond Diffusion: Inductive Moment Matching
#5"Inference can generally be scaled along two dimensions: extending sequence length (in autoregressive models), and augmenting the number of refinement steps (in diffusion models)."
Does this mean that diffusion models for text could scale inference compute to improve quality for a fixed-length output?
Re: Beyond Diffusion: Inductive Moment Matching
#6"Inference can generally be scaled along two dimensions: extending sequence length (in autoregressive models), and augmenting the number of refinement steps (in diffusion models)." Does this mean that diffusion models for text could scale inference compute to improve quality for a fixed-length output?
Yes, although so far it seems the main advantage of text diffusion models is that they're really, really fast. Iterations reach an asymptote very quickly.
Re: Beyond Diffusion: Inductive Moment Matching
#7This is a gamechanger
Re: Beyond Diffusion: Inductive Moment Matching
#8Anyone willing to give an intuitive summary of what they did mathwise? The math in the paper is super ugly to churn through.
In normal diffusion you train a model to take lots of tiny steps, all the same small size. e.g. "You're gonna take 20 steps, at times [1.0, 0.95, 0.90, 0.85...]" and each time the model takes that small fixed-size step to make the image look better.
Here they train a model to say "I'm gonna ask you to take a step from time B to A - might be a small step, might be a big step - but whatever size it is, make the image that much better." You you might ask the model to improve the image from t=1.0 to t=0.25 and be almost done. It gets a side variable telling it how much improvement to make in each step.
I'm not sure this right, but that's what I got out of it by skimming the blog & paper.
Re: Beyond Diffusion: Inductive Moment Matching
#9Anyone willing to give an intuitive summary of what they did mathwise? The math in the paper is super ugly to churn through.
Last author here (I also did the DDIM paper, https://arxiv.org/abs/2010.02502). I know this is going to be very tricky math-wise (and in the paper we just wrote the most general thing to make reviewers happy), so I tried to explain the idea more easily under the blog post (https://lumalabs.ai/news/inductive-moment-matching).
If you look at how a single step of the DDIM sampler interacts with the target timestep, it is actually just a linear function. This is obviously quite inflexible if we want to use it to represent a flexible function where we can choose any target timestep. So just add this as an argument to the neural network and then train it with a moment matching objective.
In general, I feel that analyzing a method's inference-time properties before training it can be helpful to not only diffusion models, but also LLMs including various recent diffusion LLMs, which prompted me to write a position paper in the hopes that others develop cool new ideas (https://arxiv.org/abs/2503.07154).
Re: Beyond Diffusion: Inductive Moment Matching
#10Anyone willing to give an intuitive summary of what they did mathwise? The math in the paper is super ugly to churn through.
The authors own summary from the position paper is:
In particular, we examine the one-step iterative process of DDIM [39, 19, 21] and show that it has limited capacity with respect to the target timestep under the current denoising network design. This can be addressed by adding the target timestep to the inputs of the denoising network [15].
Interestingly, this one fix, plus a proper moment matching objective [5] leads to a stable, single-stage algorithm that surpasses diffusion models in sample quality while being over an order of magnitude more efficient at inference [50]. Notably, these ideas do not rely on denoising score matching [46] or the score-based stochastic differential equations [41] on which the foundations of diffusion models are built.