Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Illustrated Stable Diffusion

jalammar.github.io

Post reply on HN
Print view View original

61–70 of 128 posts

Re: The Illustrated Stable Diffusion

#61

Post by mwambua »

uptown wrote: post #52

“We then compare the resulting embeddings using cosine similarity. When we begin the training process, the similarity will be low, even if the text describes the image correctly.” How is this training performed? How is accuracy rated?

From the paper on the CLIP embedder it appears that they use a form of contrastive loss that maximizes the cosine similarity between related images and prompts, but also minimizes the same between unrelated prompts & images.

See section 2.3 of the CLIP paper: https://arxiv.org/pdf/2103.00020.pdf

Also, the writeup on OpenAI's blog: https://openai.com/blog/clip/

Re: The Illustrated Stable Diffusion

#62

Post by dougabug »

minism wrote: post #25

Great overview, I think the part for me which is still very unintuitive is the denoising process. If the diffusion process is removing noise by predicting a final image and comparing it to the current one, why can't we just jump to the final predicted image? Or is the point that because its an iterative process, each noise step results in a different "final image" prediction?

If you removed all of the noise in a corrupted image in one step, you would have a denoising autoencoder, which has been around since the mid-aughts or perhaps earlier. Denoising diffusion models remove noise a little bit at a time. Think about an image which only has a slight amount of noise added to it. It’s generally easier to train a model to remove a tiny amount of noise than a large amount of noise. At the same time, we likely introduced a small amount of change to the actual contents of the image.

Typically, in generating the training data for diffusion models, we add noise incrementally to an image until it’s essentially all noise. Going backwards from almost all noise to the original images directly in one step is a pretty dubious proposition.

Re: The Illustrated Stable Diffusion

#63

Post by mkaic »

renewiltord wrote: post #12

Isn't it really cool? It's like the AI is asking itself what shapes the clouds are making and whether the moon has a face, over and over again.

This is one of the best analogies I've heard for denoising diffusion models. I'm totally stealing this the next time my parents ask me how it works :P

Re: The Illustrated Stable Diffusion

#64

Post by nicd »

amelius wrote: post #57
pvarangot wrote: post #43

Earlier quoted context omitted.

A Tensor is a mathematical object for symbolic manipulation of relationships between other objects that belong in conceptually similar universes or spaces. Literature on Deep Learning, like Goodfellow, call for the CS-minded reader to just assume it's a fancy word for a matrix of more than two dimensions. That makes matters confusing because mathematically you could have scalar or vectorial tensors. The classic mathe…

How are sum-types handled in deep learning? E.g., a type that holds "a 3x2 tensor OR a 4x6x2x5 tensor".

I'm not aware of tensor shape type safety (in general) being widely used in deep learning, let alone sum-types. I believe Pytorch and TensorFlow lack support for tensor shape type checking (looks like there is a Pytorch issue open: https://github.com/pytorch/pytorch/issues/26889).

Re: The Illustrated Stable Diffusion

#66

Post by pdntspa »

Waterluvian wrote: post #5

Closer. But I still get lost when words like “tensor” are used. “structured lists of numbers” really doesn’t seem to explain it usefully. This reminds me that explaining seemingly complex things in simple terms is one of the most valuable and rarest skills in engineering. Most people just can’t. And often because they no-longer remember what’s not general knowledge. You end up with a recursive Feynmannian “now explai…

I got lost at the word 'tensor' too, but then I just googled it and skilled up...

But simply put, a 'tensor' is a 3+-dimensional array of floats. Or a stack of matrices.

Re: The Illustrated Stable Diffusion

#68

Post by culi »

torbTurret wrote: post #7

Love the visual explainers for machine learning nowadays. The author has more here: https://jalammar.github.io/ Amazon has some highly interactive ones here: https://mlu-explain.github.io/ Google had: distill.pub Hope to see education in this space grow more.

> Google had: distill.pub

Wait I thought they were just taking a break. Don't tell me it's being killed

Re: The Illustrated Stable Diffusion

#69

Post by leereeves »

OkayPhysicist wrote: post #60
zestyping wrote: post #54

Earlier quoted context omitted.

> we use tensor because it allows for any number of dimensions "Vector" implies one dimension and "matrix" strongly implies two. But an array can have any number of dimensions, so "array" is the best word. We don't need the word "tensor"; when the context is programming, "tensor" is only confusing and doesn't really add any useful meaning.

Tensor does imply a set of operations that are expected. Multiplying two arrays together is an ambiguous operation; multiplying two tensors together is well-defined. And really, the context is math, not programming. The programming side of DL is approximately trivial, the interesting bits are all represented as mathematical operations on a set of tensors.

It's especially confusing as math, because the standard meaning of "tensor" in math includes a lot of baggage that's not used in deep learning (related to physics and differential geometry).

Re: The Illustrated Stable Diffusion

#70

Post by backpropaganda »

BrainVirus wrote: post #67

https://jalammar.github.io/images/stable-diffusion/article-F... Can you say with a straight face that this image (from the original paper) was intended to explain rather than obfuscate?

Yes, I can read it perfectly fine. It relies on a lot of notation that AI researchers are familiar with.
Post reply on HN