Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Illustrated Stable Diffusion

jalammar.github.io

Post reply on HN
Print view View original

81–90 of 128 posts

Re: The Illustrated Stable Diffusion

#81

Post by minism »

psb217 wrote: post #59
minism wrote: post #25

Great overview, I think the part for me which is still very unintuitive is the denoising process. If the diffusion process is removing noise by predicting a final image and comparing it to the current one, why can't we just jump to the final predicted image? Or is the point that because its an iterative process, each noise step results in a different "final image" prediction?

In the reverse diffusion process, the reason we can't directly jump from a noisy image at step t to a clean image at step 0 is that each possible noisy image at step t may be visited by potentially many real images during the forward diffusion process. Thus, our model which inverts the diffusion process by minimizing least-squares prediction error of a clean image given a noisy image at step t will learn to predict t…

Thank you! This is a great explanation.

Re: The Illustrated Stable Diffusion

#82

Post by kgwgk »

FeepingCreature wrote: post #76
zestyping wrote: post #54

Earlier quoted context omitted.

> we use tensor because it allows for any number of dimensions "Vector" implies one dimension and "matrix" strongly implies two. But an array can have any number of dimensions, so "array" is the best word. We don't need the word "tensor"; when the context is programming, "tensor" is only confusing and doesn't really add any useful meaning.

"Array" implies one dimension to me. Maybe n-array?

Array may imply 1d to you, or 2d to others, but it is more general that that: https://en.wikipedia.org/wiki/Array_(data_type)

Re: The Illustrated Stable Diffusion

#83

Post by sroussey »

mota7 wrote: post #58
minism wrote: post #25

Great overview, I think the part for me which is still very unintuitive is the denoising process. If the diffusion process is removing noise by predicting a final image and comparing it to the current one, why can't we just jump to the final predicted image? Or is the point that because its an iterative process, each noise step results in a different "final image" prediction?

The problem is that predicting a pixel requires knowing what the pixels around it looks like. But if we start with lots of noise, then the neighboring pixels are all just noise and have no signal. You could also think of this as: We start with a terrible signal to noise ratio. So we need to average over very large areas to get any reasonable signal. But as we increase the signal, we can average over a smaller area to…

I suppose that static plus a subliminal message would do the same thing to our own numeral networks. Or clouds. I can be convinced I’m seeing almost anything in clouds…

Re: The Illustrated Stable Diffusion

#84

Post by [deleted] »

Waterluvian wrote: post #5

Closer. But I still get lost when words like “tensor” are used. “structured lists of numbers” really doesn’t seem to explain it usefully. This reminds me that explaining seemingly complex things in simple terms is one of the most valuable and rarest skills in engineering. Most people just can’t. And often because they no-longer remember what’s not general knowledge. You end up with a recursive Feynmannian “now explai…

[deleted]

Re: The Illustrated Stable Diffusion

#85

Post by kgwgk »

Waterluvian wrote: post #18
netruk44 wrote: post #9

Earlier quoted context omitted.

If it helps you to understand at all, assuming you have a CS background, any time you see the word "tensor" you can replace it with "array" and you'll be 95% of the way to understanding it. Or "matrix" if you have a mathematical background. Whereas CS arrays tend to be 1 dimensional, and sometimes 2 dimensional, tensors can be as many dimensions as you need. A 256x256 photo with RGB channels would be stored as a [256…

This helps. Thank you. Any advice on where to look to understand why the word tensor was used?

It’s a cool name from the good ol’ “multi-dimensional arrays with a uniform type”

https://www.tensorflow.org/guide/tensor

Re: The Illustrated Stable Diffusion

#86

Post by mxkopy »

amelius wrote: post #57
pvarangot wrote: post #43

Earlier quoted context omitted.

A Tensor is a mathematical object for symbolic manipulation of relationships between other objects that belong in conceptually similar universes or spaces. Literature on Deep Learning, like Goodfellow, call for the CS-minded reader to just assume it's a fancy word for a matrix of more than two dimensions. That makes matters confusing because mathematically you could have scalar or vectorial tensors. The classic mathe…

How are sum-types handled in deep learning? E.g., a type that holds "a 3x2 tensor OR a 4x6x2x5 tensor".

In my experience (CV) you typically want 4 dims - (height, width, channels, batch). In Julia you can do element-wise operations on two arrays if their lengths (i.e. size after flattening) are the same. Obviously dimension-wise that sort of thing won't work (what's the dot product of a 3-vector by a 6-vector? Undefined!)

But really, the whitepaper should include this sort of detail, and you should be following the white paper.

Re: The Illustrated Stable Diffusion

#87

Post by kgwgk »

OkayPhysicist wrote: post #60
zestyping wrote: post #54

Earlier quoted context omitted.

> we use tensor because it allows for any number of dimensions "Vector" implies one dimension and "matrix" strongly implies two. But an array can have any number of dimensions, so "array" is the best word. We don't need the word "tensor"; when the context is programming, "tensor" is only confusing and doesn't really add any useful meaning.

Tensor does imply a set of operations that are expected. Multiplying two arrays together is an ambiguous operation; multiplying two tensors together is well-defined. And really, the context is math, not programming. The programming side of DL is approximately trivial, the interesting bits are all represented as mathematical operations on a set of tensors.

Interestingly the Wikipedia page

Tensor (disambiguation)

https://en.wikipedia.org/wiki/Tensor_(disambiguation)

has an entry

Tensor type (computing)

which is just a redirection to

Array (data type)

https://en.wikipedia.org/wiki/Array_(data_type)

Doesn’t seem to support the relevance of “tensor” as something distinct from “array”.

Re: The Illustrated Stable Diffusion

#88

Post by a1369209993 »

zestyping wrote: post #55
Imnimo wrote: post #29

Earlier quoted context omitted.

The way I think of it, we have a 512x512x3 target, so that's 48x the information. I don't think it's unreasonable to say that far less than 1/48th of the space of 512x512x3 outputs are natural images (meaning an image that might actually exist, rather than meaningless pixels). So if we think about that 64x64x4 tensor as telling us what in the smaller space of natural images we should draw, it seems like plenty of inf…

The amount of information in a 64x64x4 array would depend on the precision of the numbers in it, right? For example, a 512x512 image in 24-bit colour could be completely encoded in a 64x64x4 array if each of the 64 x 64 x 4 = 16,384 values had 384 bits of precision. So, I wonder — what's the minimum number of bits of precision in the 64x64x4 array that would be sufficient for this to work?

According to a anecdotal test on one of the images found elsewhere in this thread, JPEG comression at 80% quality can chop a factor of 16 off the size of a 24bpp .bmp file.

384 / 16 = 24bits per [64x64x4-] array value. Integer range of 32bit float is 2^24. So "literally just a jpeg packed into 16K floats" is a option.

Re: The Illustrated Stable Diffusion

#89

Post by wokwokwok »

psb217 wrote: post #59
minism wrote: post #25

Great overview, I think the part for me which is still very unintuitive is the denoising process. If the diffusion process is removing noise by predicting a final image and comparing it to the current one, why can't we just jump to the final predicted image? Or is the point that because its an iterative process, each noise step results in a different "final image" prediction?

In the reverse diffusion process, the reason we can't directly jump from a noisy image at step t to a clean image at step 0 is that each possible noisy image at step t may be visited by potentially many real images during the forward diffusion process. Thus, our model which inverts the diffusion process by minimizing least-squares prediction error of a clean image given a noisy image at step t will learn to predict t…

> But, as we take a bunch of small steps and gradually move back through the diffusion process...

...but, the question is, why can't we take a big step and be at the end in one step.

Obviously a series of small steps gets you there, but the question was why you need to take small steps.

I feel like this is just a 'intuitive explanation' that doesn't actually do anything other than rephrase the question; "You take a series of small steps to reduce the noise in each step and end up with a picture with no noise".

The real reason is that big steps result in worse results (1); the model was specifically designed to be a series of small steps because when you take big steps, you end up with over fitting, where the model just generates a few outputs from any input.

(1) - https://arxiv.org/pdf/1503.03585.pdf

Re: The Illustrated Stable Diffusion

#90

Post by auggierose »

jayalammar wrote: post #38
Waterluvian wrote: post #18

Earlier quoted context omitted.

This helps. Thank you. Any advice on where to look to understand why the word tensor was used?

I updated the post to say "multi-dimensional array". In a context like this, we use tensor because it allows for any number of dimensions (while vector/ array is only one, matrix is two). When you get into ML libraries, both popular packages PyTorch and TensorFlow use the "tensor" terminology. It's a good point. Hope it's clearer for devs with "array" terminology.

I think tensor is much shorter than "multi-dimensional array", and it is OK to use it. You could just mention that in the context of DL tensor is just a shorter way to say "multi-dimensional array", and everyone should be happy. On top of that, if they read another text about DL that uses the word tensor, there is no need to be scared anymore.
Post reply on HN