Diffusion models from scratch, from a new theoretical perspective
31–40 of 44 posts
Re: Diffusion models from scratch, from a new theoretical perspective
#32That goes into a lot more mathematical detail but is also accompanied by a minimal, very easy to understand <500 line implementation.
Re: Diffusion models from scratch, from a new theoretical perspective
#33Author here, when I tried to understand diffusion models I realized that the code and math can be greatly simplified, which led to me writing this blog post and diffusion library. Happy to answer any questions.
This looks great! How long does it take (on what hardware) to train the toy models? Such as the `fashion_mnist.py` example? Thanks.
Re: Diffusion models from scratch, from a new theoretical perspective
#34Earlier quoted context omitted.
They're famous for lots of equations, but truth be told, most diffusion researchers I know have the exact same response. A lot of people repeat the same exact equations and they only are there for review purposes. On the other hand, if you want to really dig in, I'd suggest reading into works by Kingma, Gao, Ricky Tian Qi Chen, and honestly, any of Max Welling's students (Tomczak (was post doc), Hoogeboom, etc), and…
I’d probably say Karras has written the best papers on how to think about diffusion models.
Re: Diffusion models from scratch, from a new theoretical perspective
#35Earlier quoted context omitted.
They're famous for lots of equations, but truth be told, most diffusion researchers I know have the exact same response. A lot of people repeat the same exact equations and they only are there for review purposes. On the other hand, if you want to really dig in, I'd suggest reading into works by Kingma, Gao, Ricky Tian Qi Chen, and honestly, any of Max Welling's students (Tomczak (was post doc), Hoogeboom, etc), and…
How long before more legible code or code-like constructs (or markup?) start to displace arcane mathematical notation? It would take quite a long time but I could see this slowly happening for analogous reasons to how the printing press favored alphabetic languages over pictorial ones. Computers teach us to read, write, and express linguistically and are now thanks to LLMs capable of working much more fluently with l…
My best guess is never.
The problem is that math is a language itself. In fact, I'd more accurately call it a family of languages. People often confuse it for an inherent thing because most of the time we use it in applied ways for subjects like physics, and the sciences. But clearly those people have never taken abstract algebra or developed their own toy algebra (or group, ring (or rung), or field). You might say they have an _ideal_istic notion of mathematics.
That said, I am absolutely sympathetic to the complexity of mathematics and there is so much that is difficult to grasp. What's worse, is that when this abstract gobbledy goop finally clicks it's so abundantly obvious that it deceives you into thinking you've known it all along. I think this contributes to the difficulty of teaching the subject because those qualified have forgotten the struggles they themselves have had (to be fair, human psychology plays a big role here too because if we only remember the struggles we'd have difficulty finding the passion and beauty).
But I think our best hope is to find a unified mathematics. The obsession people have with things like set theory and category theory is that you can see this interconnectedness. You get this deep feeling that you're looking at the same things manifesting in different ways. Remember that a lot of mathematics is converting your system into another system (we might call this an isomorphic mapping) such that your problem is easier to solve in the other system (think about how we might want to go from Cartesian coordinates to polar, but now abstract that concept out more than your friend that took a few tabs of acid). You zoom out move around, and zoom back in, and maybe a few more times (but you have to keep track of your path).
> Squiggles and lots of Greek letters seem like an anachronism.
Do you have another suggestion? Because I don't see how code, English, or literally any other thing that can be written down is anything more meaningful than "a bunch of squiggles." There's only so many simple, and importantly, distinguishable squiggles that we can write down. Luckily we have a tool to exactly determine this ;) The shitty part of mathematics is also the best part, in that you have to embrace the abstractness of it all. But of course human brains weren't designed for this. (Yes, there are some of us trying to figure out how to get machines to think with that extreme level of abstractedness at its core. No, this is not how computers already "think.")
So yeah, I empathize with you. Quite a lot. There's so much more that I want to know about this thing too. But it is often seemingly impenetrable. All I can tell you is that it just looks that way and isn't. Everyone has the capacity but finding the right way up this mountain (that I'm nowhere near Terry, and he's nowhere near the summit) can be just as hard as the climb itself. The best way is persistence though. Just remember that you don't play video games because they are easy. Sometimes you gotta just trick yourself into pushing through. And some of the best advice I can give is revisiting old topics. So much of what you think you've known actually has a tremendous amount of depth to it. Coming back with fresh eyes and a better understanding of the larger picture can help you see all the beauty in it, even if this is sometimes fleeting.
I'm sorry to have vomited so much, but you know, Stockholm Syndrome is a bitch.
Re: Diffusion models from scratch, from a new theoretical perspective
#36Earlier quoted context omitted.
They're famous for lots of equations, but truth be told, most diffusion researchers I know have the exact same response. A lot of people repeat the same exact equations and they only are there for review purposes. On the other hand, if you want to really dig in, I'd suggest reading into works by Kingma, Gao, Ricky Tian Qi Chen, and honestly, any of Max Welling's students (Tomczak (was post doc), Hoogeboom, etc), and…
yeah fair enough :) i am unfortunately not driven enough on diffusion models yet to need to dive into those but i hope a future seeker finds your references here.
Re: Diffusion models from scratch, from a new theoretical perspective
#37Earlier quoted context omitted.
> FWIW, a friend of mine (and not "a friend of mine") What? Do you mean a coworker or something as compared to a friend?
They are clarifying that "a friend of mine" is not a euphemism for themselves, because it's more common to use it as a euphemism for yourself than it is to actually talk to strangers online about your friend's life, problems, opinions etc.
Re: Diffusion models from scratch, from a new theoretical perspective
#38Earlier quoted context omitted.
How long before more legible code or code-like constructs (or markup?) start to displace arcane mathematical notation? It would take quite a long time but I could see this slowly happening for analogous reasons to how the printing press favored alphabetic languages over pictorial ones. Computers teach us to read, write, and express linguistically and are now thanks to LLMs capable of working much more fluently with l…
> How long before more legible code or code-like constructs (or markup?) start to displace arcane mathematical notation? My best guess is never. The problem is that math is a language itself. In fact, I'd more accurately call it a family of languages. People often confuse it for an inherent thing because most of the time we use it in applied ways for subjects like physics, and the sciences. But clearly those people h…
Claude Shannon had something to say about this as I recall.
> Do you have another suggestion? Because I don't see how code, English, or literally any other thing that can be written down is anything more meaningful than "a bunch of squiggles."
The difference is that the squiggles appear on your keyboard. If you use modern language to construct a variable name, the name can be suggestive of what it is to someone already fluent in natural language. Multi-character names also reduce collisions -- there are only so many Greek letters. And then the research paper doesn't show an inscrutable glyph as a JPEG that you can't even copy and paste into a search engine if you don't already know what it represents.
Re: Diffusion models from scratch, from a new theoretical perspective
#39Author here, when I tried to understand diffusion models I realized that the code and math can be greatly simplified, which led to me writing this blog post and diffusion library. Happy to answer any questions.
I am curious if any of these concepts derive somehow from some physics principles? Like the same way neural networks are modeled after biological neural networks? Maybe you have some insights on that conception?
Re: Diffusion models from scratch, from a new theoretical perspective
#40Author here, when I tried to understand diffusion models I realized that the code and math can be greatly simplified, which led to me writing this blog post and diffusion library. Happy to answer any questions.
As a researcher, there's a lot of diffusion blogs that I do not like. But I actually really do like this one! It does a great job at getting to the meat of things, showing some of the complexities (often missing) but without getting lost or distracted. I especially like the discussion of trajectories as this is motivating to many things I think a lot of people struggle with (e.g. with schedulers). That can be hard to…