Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Illustrated Stable Diffusion

jalammar.github.io

Post reply on HN
Print view View original

21–30 of 128 posts

Re: The Illustrated Stable Diffusion

#21

Post by CrazyStat »

jw1224 wrote: post #19
coldcode wrote: post #13

I find SD to be amazing technology, but it still (mostly) sucks at producing "intelligent" images. It basically fancy math that turns noise into images (from the opposite it trained on) but still has no idea what it is producing. If you run it long enough you eventually get lucky and find a gem. I like to try "George Washington riding a Unicorn in Times Square"; I've so far never gotten anything a first year art stud…

> "George Washington riding a Unicorn in Times Square" The “secret” to Stable Diffusion (and other CLIP-based models) is being as descriptive as possible. This prompt, whilst easy for humans to imagine, actually has a whole lot of ambiguity baked in. How high is the unicorn flying? Is the unicorn even flying, or on the ground? How old is George Washington? What visual style is the image in? Is the image from the pers…

I love how the unicorn horn got stuck on a Washington's head in the bottom right instead of on the unicorn.

Re: The Illustrated Stable Diffusion

#22

Post by l33tman »

ilaksh wrote: post #20

It says the final output before pixel space is 64x64x4? How can that be enough information?

The autoencoder that maps between that and the 512x512x3 RGB space was trained together with the model, so it is specialized in upscaling the 64x64x4 info to pixel space for this particular purpose. It's "just" a factor of 48 (de)compression..

Re: The Illustrated Stable Diffusion

#23

Post by l33tman »

coldcode wrote: post #13

I find SD to be amazing technology, but it still (mostly) sucks at producing "intelligent" images. It basically fancy math that turns noise into images (from the opposite it trained on) but still has no idea what it is producing. If you run it long enough you eventually get lucky and find a gem. I like to try "George Washington riding a Unicorn in Times Square"; I've so far never gotten anything a first year art stud…

The reason you can't get the images you want from it is not because of the noise diffusion process (after all, this is probably the closest similarity to how a human gets a flash of creativity) but the lack of a large language model in SD - it was deliberately scaled down so the result could fit in consumer GPUs.

DALLE-2 uses a much larger language model and you can explain more complicated concepts to it. Googles Imagen likewise (not released though).

It's mostly a matter of scaling to get this better.

Re: The Illustrated Stable Diffusion

#24

Post by 6gvONxR4sf7o »

Waterluvian wrote: post #5

Closer. But I still get lost when words like “tensor” are used. “structured lists of numbers” really doesn’t seem to explain it usefully. This reminds me that explaining seemingly complex things in simple terms is one of the most valuable and rarest skills in engineering. Most people just can’t. And often because they no-longer remember what’s not general knowledge. You end up with a recursive Feynmannian “now explai…

You’re talking like using jargon makes something a bad explanation, but maybe you just aren’t the audience? Why not use words like that if it’s a super basic concept to your intended audience?

Re: The Illustrated Stable Diffusion

#25

Post by minism »

Great overview, I think the part for me which is still very unintuitive is the denoising process.

If the diffusion process is removing noise by predicting a final image and comparing it to the current one, why can't we just jump to the final predicted image? Or is the point that because its an iterative process, each noise step results in a different "final image" prediction?

Re: The Illustrated Stable Diffusion

#26

Post by dr_dshiv »

coldcode wrote: post #13

I find SD to be amazing technology, but it still (mostly) sucks at producing "intelligent" images. It basically fancy math that turns noise into images (from the opposite it trained on) but still has no idea what it is producing. If you run it long enough you eventually get lucky and find a gem. I like to try "George Washington riding a Unicorn in Times Square"; I've so far never gotten anything a first year art stud…

> I like to try "George Washington riding a Unicorn in Times Square"; I've so far never gotten anything a first year art student can draw.

Why the hell would a first year art student draw that? Flunk their ass. God damn dumb ass prompts I have to deal with.

—Stable Diffusion

Re: The Illustrated Stable Diffusion

#27

Post by [deleted] »

minism wrote: post #25

Great overview, I think the part for me which is still very unintuitive is the denoising process. If the diffusion process is removing noise by predicting a final image and comparing it to the current one, why can't we just jump to the final predicted image? Or is the point that because its an iterative process, each noise step results in a different "final image" prediction?

[deleted]

Re: The Illustrated Stable Diffusion

#28

Post by subdane »

yieldcrv wrote: post #15

What are you guys currently using for Stable Diffusion on OSX with M1? There are so many variants and forks that I don't know which one to install any more. Something that takes advantage of Metal and the CPU cores. Any that retains the "upload a sketch and then add a description" feature?

https://github.com/divamgupta/diffusionbee-stable-diffusion-...

Re: The Illustrated Stable Diffusion

#29

Post by Imnimo »

ilaksh wrote: post #20

It says the final output before pixel space is 64x64x4? How can that be enough information?

The way I think of it, we have a 512x512x3 target, so that's 48x the information. I don't think it's unreasonable to say that far less than 1/48th of the space of 512x512x3 outputs are natural images (meaning an image that might actually exist, rather than meaningless pixels). So if we think about that 64x64x4 tensor as telling us what in the smaller space of natural images we should draw, it seems like plenty of information. Especially since we have the information stored weights of the output network also.

Re: The Illustrated Stable Diffusion

#30

Post by [deleted] »

Waterluvian wrote: post #5

Closer. But I still get lost when words like “tensor” are used. “structured lists of numbers” really doesn’t seem to explain it usefully. This reminds me that explaining seemingly complex things in simple terms is one of the most valuable and rarest skills in engineering. Most people just can’t. And often because they no-longer remember what’s not general knowledge. You end up with a recursive Feynmannian “now explai…

[deleted]
Post reply on HN