Diffusion models from scratch, from a new theoretical perspective • Hacker News Board

Re: Diffusion models from scratch, from a new theoretical perspective

#41

Post by eutropia » Tue, Mar 12, 2024, 4:41 PM UTC

ycy wrote: post #4

Author here, when I tried to understand diffusion models I realized that the code and math can be greatly simplified, which led to me writing this blog post and diffusion library. Happy to answer any questions.

In your example images at the very end, the momentum term seems to have a deleterious effect on the digital painting of the house (the door is missing in the gamma = 2.0 image). I would like to know more details of that example to build an intuition for the effect of your gradient-informed ddim sampler.

As someone who's spent a little time experimenting with sampling procedures on stable diffusion, I was also hoping for a comparison to DDIM in terms of convergence time/steps. Is there a relationship between momentum, convergence and error? (i.e, your momentum sampler at 16 steps is ~equivalent to ddim at 20 steps ± %err_term)

thanks for the excellent post

Re: Diffusion models from scratch, from a new theoretical perspective

#42

Post by ycy » Tue, Mar 12, 2024, 5:04 PM UTC

eutropia wrote: post #41

ycy wrote: post #4

Author here, when I tried to understand diffusion models I realized that the code and math can be greatly simplified, which led to me writing this blog post and diffusion library. Happy to answer any questions.

In your example images at the very end, the momentum term seems to have a deleterious effect on the digital painting of the house (the door is missing in the gamma = 2.0 image). I would like to know more details of that example to build an intuition for the effect of your gradient-informed ddim sampler. As someone who's spent a little time experimenting with sampling procedures on stable diffusion, I was also hoping…

It is hard to quantify the performance of samplers on individual images, but we have quantitative experiments on pretrained models in our paper (https://arxiv.org/pdf/2306.04848.pdf). Table 2 has a comparison to other samplers and Figure 9 plots the effects of the momentum term on convergence.

Re: Diffusion models from scratch, from a new theoretical perspective

#43

Post by godelski » Tue, Mar 12, 2024, 5:47 PM UTC

zrm wrote: post #38

godelski wrote: post #35

Earlier quoted context omitted.

> How long before more legible code or code-like constructs (or markup?) start to displace arcane mathematical notation? My best guess is never. The problem is that math is a language itself. In fact, I'd more accurately call it a family of languages. People often confuse it for an inherent thing because most of the time we use it in applied ways for subjects like physics, and the sciences. But clearly those people h…

> Remember that a lot of mathematics is converting your system into another system Claude Shannon had something to say about this as I recall. > Do you have another suggestion? Because I don't see how code, English, or literally any other thing that can be written down is anything more meaningful than "a bunch of squiggles." The difference is that the squiggles appear on your keyboard. If you use modern language to c…

Shannon said a lot of things, you'll have to be a bit more specific to jog my memory.

Shannon would tell you that there can only be so many simple __distinguishable__ letters. It's not just Greek. Sounds like you're mentioning the current Chinese phenomena. Where people have an easier time reading and it helps that it's text, including traditional characters. But typing can still be hard. Because that's the problem, that there's only so many easy to distinguish squiggles you can put on a keyboard. Because I'm sure while every key in distinguishable, you wouldn't want to operate a keyboard that looked like this [0] (it's actually a car decorated in keys, in case you want to see [1]).

There's a balance that that's hard to strike. The reason many languages use alphabets is not because the complexity of writing hieroglyphics, but because hieroglyphics even when written well can be difficult to distinguish and be cumbersome. There's probably no universal language (sorry Chomsky) so there's no symbols you can put on those keys that are universally recognizable. If you're actually interested in this kind of topic there's two things I suggest you look into. The first is Korean (specifically Hangul) [2] which has an interesting history relevant here and the other is long term nuclear storage [3,4]. The former was specifically designed to increase literacy and was extremely effective. The later is the difficulties of universal innate communications where we think certain things are absolutely obvious but aren't.

The other problem with languages is just usage. There's many over loaded words and it's not like we're at a shortage of combinations in the English language. Even combination of short words! The word cb doesn't exist, nor cbra! This alone should make someone suspicious that the depth of the problem is so much more than symbol complexity and combinatorics.

I'm sure Shannon would have something to say about the tyranny about encoding and complexity... Your suggesting is just the other end of the problem.

[0] https://www.ocregister.com/wp-content/uploads/migration/m1m/...

[1] https://www.ocregister.com/2012/03/30/keyboard-decorated-car...

[2] https://en.m.wikipedia.org/wiki/Hangul

[3] https://en.m.wikipedia.org/wiki/Long-term_nuclear_waste_warn...

[4] https://en.m.wikipedia.org/wiki/Long-term_nuclear_waste_warn...

Re: Diffusion models from scratch, from a new theoretical perspective

#44

Post by godelski » Tue, Mar 12, 2024, 5:57 PM UTC

ycy wrote: post #40

godelski wrote: post #17

Earlier quoted context omitted.

As a researcher, there's a lot of diffusion blogs that I do not like. But I actually really do like this one! It does a great job at getting to the meat of things, showing some of the complexities (often missing) but without getting lost or distracted. I especially like the discussion of trajectories as this is motivating to many things I think a lot of people struggle with (e.g. with schedulers). That can be hard to…

Glad you liked it! Just curious, what else would you like to see to make the post more complete? I had focused on getting through the basics of diffusion training and sampling as concisely as possible, but perhaps in future posts I can delve deeper into other aspects of diffusion.

That's actually hard for me to answer. I'm afraid to taint it. Honestly, I'd be wary to touch it especially being advice coming from someone already well versed in the domain. It's too easy to accidentally complexify and then you end up losing your target audience. But maybe the connection to VAEs could be made clearer? Maybe a bit clearer or better introducing the topic of optimization? But again, it's a hard line to walk because you've successfully written something that is approachable to a novice. This is the same problem I suffer when trying to teach haha. But I think the fact that you're asking and looking to improve it's the biggest sign you'll continue to do well.

For later topics I think there's a lack on discussions about things like score modeling, the difficulties of understanding latent representations (so many misunderstand the curse of dimensionality), and Schrodinger Bridges. There's so much lol

The better advice I can give is to listen to anyone complaining. It's hard because you have to try to see beyond their words and find their intent (I think this also makes them more digestible and less emotionally draining/hurtful). Especially consider a novice isn't going to have to right words to correctly express their difficulties parsing and it may just come out looking like anger when it's frustration.

Just keep doing your best :)