It is a good post, but is missing the connection to continuous normalizing flows. Diffusion models, flow matching, consistency models are biased approximations of continuous normalizing flows (which themselves have some slight biases, but less). Adversarial losses can somewhat help with bias (e.g. RL, GANs), but training those has issues.
Learning the Integral of a Diffusion Model
11–20 of 26 posts
Re: Learning the Integral of a Diffusion Model
#12This is way outside of my expertise, can anyone given a TL;DR or ai;dr?
Why not put it into an AI yourself? :) I'd rather we avoided a precedent of asking for it and N people replying with their own favorite AI version. The comments section would end up a ghost town. Extreme TL;DR: Diffusion models are like getting f(x) by calculating and summing f'(0), f'(1)...f'(x). Flow models are like just calculating f(x).
We've all seen that AI can give you plausible but incorrect answers. Having an expert read it or use AI on it and interpret and validate it before posting would be most welcome IMO.
Re: Learning the Integral of a Diffusion Model
#13It is a good post, but is missing the connection to continuous normalizing flows. Diffusion models, flow matching, consistency models are biased approximations of continuous normalizing flows (which themselves have some slight biases, but less). Adversarial losses can somewhat help with bias (e.g. RL, GANs), but training those has issues .
As explanation, something I wrote previously:
The most common approach to modeling continuous distributions is to train a reversible model f that maps it to another continuous distribution P that is already known. The original image can be recovered by tracking the bits needed to encode its latent, as well as the reverse path:
−log P(f(x)) − log|det ∂f/∂x (x)|
dx/dt = g(x)
log|det ∂f/∂x (x)| = ∫ Tr(∂g(x)/∂x) dt = ∫ E_{ε∼N(0,I)} [εᵀ ∂g(x)/∂x ε] dt
dx′ = g(x′)dt + ε(t)dW
d(δxᵀδx)/dt = 2δxᵀ ∂g(x)/∂x δx,
Re: Learning the Integral of a Diffusion Model
#14It is a good post, but is missing the connection to continuous normalizing flows. Diffusion models, flow matching, consistency models are biased approximations of continuous normalizing flows (which themselves have some slight biases, but less). Adversarial losses can somewhat help with bias (e.g. RL, GANs), but training those has issues .
I briefly covered that connection in an earlier blog post: https://sander.ai/2023/07/20/perspectives.html#flow
... but it's definitely something that might deserve a longer-form treatment at some point :)
Re: Learning the Integral of a Diffusion Model
#15Earlier quoted context omitted.
Diffusion and flow matching models generate samples by iterative denoising. Iterative denoising means passing input to the neural network, running a forward pass, and taking the output back as input and rerunning the neural network. Often you do this 100 times, which is slow and expensive. Flow maps / consistency models / shortcut models instead try to learn to compress this iterative work into 1 forward pass. This m…
Very helpful! Naïve question (I haven’t had a chance to read TFA at all and diffusion/flow models are not my area of expertise). Doesn’t learning the integral/solution of the diffusion process in a single pass just take us back to like OG generative CNN that we had before diffusion models took over? Surely the answer is “no” but would love to hear your framing as to why.
It kind of does! In the modern era of generative modelling, it seems like we rely on pre-training to capture the data distribution, and then on post-training (and various other tricks) to carve out a sliver of that distribution that we actually care about (i.e. what we want our model to generate).
To be able to specify that subset with relatively few examples, a good high-level understanding of the data distribution is necessary. The way I see this, is that training a diffusion model gets you to that point, and then once you've selected the part of the distribution you actually care about, you can distill it down quite aggressively, because you no longer need all of that computation to model a much simpler distribution (sometimes all the way to one step, but usually it's a few steps in practice).
Re: Learning the Integral of a Diffusion Model
#16Earlier quoted context omitted.
Why not put it into an AI yourself? :) I'd rather we avoided a precedent of asking for it and N people replying with their own favorite AI version. The comments section would end up a ghost town. Extreme TL;DR: Diffusion models are like getting f(x) by calculating and summing f'(0), f'(1)...f'(x). Flow models are like just calculating f(x).
HN is a place where it's legitimate to ask those kinds of questions. The site has a high concentration of advanced practitioners -- in my experience it is not uncommon for the creator of a technology or deep expert to reply. John Carmack has an account on the site for instance. :)
[dead]
Re: Learning the Integral of a Diffusion Model
#17Does anyone have good resources into a more practical approach toward building diffusion models? I found the book by Rashka for Building an LLM from Scratch really helpful in understanding a lot of concepts behind LLMs, and I am looking for a similar resource for diffusion models
Re: Learning the Integral of a Diffusion Model
#18Does anyone have good resources into a more practical approach toward building diffusion models? I found the book by Rashka for Building an LLM from Scratch really helpful in understanding a lot of concepts behind LLMs, and I am looking for a similar resource for diffusion models
I really liked Calvin Lou’s tutorial. It’s quite old at this point and probably not up to date, but I felt it was awesome for understanding the concepts
Re: Learning the Integral of a Diffusion Model
#19Does anyone have good resources into a more practical approach toward building diffusion models? I found the book by Rashka for Building an LLM from Scratch really helpful in understanding a lot of concepts behind LLMs, and I am looking for a similar resource for diffusion models
MIT's OCW is usually pretty reliable:
https://www.practical-diffusion.org/lectures/
There is more math-heavy https://diffusion.csail.mit.edu/2026/index.html
Re: Learning the Integral of a Diffusion Model
#20Earlier quoted context omitted.
Why not put it into an AI yourself? :) I'd rather we avoided a precedent of asking for it and N people replying with their own favorite AI version. The comments section would end up a ghost town. Extreme TL;DR: Diffusion models are like getting f(x) by calculating and summing f'(0), f'(1)...f'(x). Flow models are like just calculating f(x).
We've all seen that AI can give you plausible but incorrect answers. Having an expert read it or use AI on it and interpret and validate it before posting would be most welcome IMO.
[deleted]