Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Illustrated Stable Diffusion

jalammar.github.io

Post reply on HN
Print view View original

91–100 of 128 posts

Re: The Illustrated Stable Diffusion

#91

Post by l33tman »

astrange wrote: post #39
l33tman wrote: post #23

Earlier quoted context omitted.

The reason you can't get the images you want from it is not because of the noise diffusion process (after all, this is probably the closest similarity to how a human gets a flash of creativity) but the lack of a large language model in SD - it was deliberately scaled down so the result could fit in consumer GPUs. DALLE-2 uses a much larger language model and you can explain more complicated concepts to it. Googles Im…

It's not just size but also model architecture. DALLE mini (craiyon.com) has the opposite priority because of its different architecture; you can enter a complex prompt and it will follow it, but it's much slower and the image quality is a lot worse. SD prefers to make aesthetic pictures over listening to everything you tell it. You can improve this in SD by raising cfg_scale at the cost of some weird "oversharpening…

Yeah true, I like dalle-mini :) It did seem to understand the prompts better.

The training set also affects it, as the guidance signal competes with the diffusion-model's priors it learned from the training set (the cfg_scale) and I've found situations where it seems the priors are just encoded too strong it seems - for example with very well-known celebs or objects it's difficult to make variations.

I guess it's interesting that these issues are kind of reflected in humans as well.

Re: The Illustrated Stable Diffusion

#92

Post by gpderetta »

ntr-- wrote: post #74
nicd wrote: post #64

Earlier quoted context omitted.

I'm not aware of tensor shape type safety (in general) being widely used in deep learning, let alone sum-types. I believe Pytorch and TensorFlow lack support for tensor shape type checking (looks like there is a Pytorch issue open: https://github.com/pytorch/pytorch/issues/26889 ).

https://docs.rs/ndarray/0.15.6/ndarray/ Rust to the rescue, as it always is for safety. (I don't know if this directly translates for tensors, I used it for manipulating point clouds and iirc it allows for arbitrary dimension containers)

How does this help with typechecking tensor types in python?

Re: The Illustrated Stable Diffusion

#93

Post by beernet »

Waterluvian wrote: post #5

Closer. But I still get lost when words like “tensor” are used. “structured lists of numbers” really doesn’t seem to explain it usefully. This reminds me that explaining seemingly complex things in simple terms is one of the most valuable and rarest skills in engineering. Most people just can’t. And often because they no-longer remember what’s not general knowledge. You end up with a recursive Feynmannian “now explai…

I think this post explains tensors fairly well: https://www.kdnuggets.com/2018/05/wtf-tensor.html

Quote: "A tensor is a container which can house data in N dimensions. Often and erroneously used interchangeably with the matrix (which is specifically a 2-dimensional tensor), tensors are generalizations of matrices to N-dimensional space."

Re: The Illustrated Stable Diffusion

#94

Post by TuringTest »

TigeriusKirk wrote: post #47
Waterluvian wrote: post #5

Closer. But I still get lost when words like “tensor” are used. “structured lists of numbers” really doesn’t seem to explain it usefully. This reminds me that explaining seemingly complex things in simple terms is one of the most valuable and rarest skills in engineering. Most people just can’t. And often because they no-longer remember what’s not general knowledge. You end up with a recursive Feynmannian “now explai…

One of my favorite moments in Geoffrey Hinton's otherwise pretty info-dense Coursera neural network class was when he said- "To deal with a 14-dimensional space, visualize a 3-D space and say 'fourteen' to yourself very loudly. Everyone does it."

Kudos to this. It also helps to think of a three-dimensional space bounded by the sides of a box, and to think of another 11 boxes stacked on top of each other.

Then, you can visualize orthogonal on dimensions higher than 3 by throwing wires between equivalent points in two boxes.

Re: The Illustrated Stable Diffusion

#95

Post by squeaky-clean »

yreg wrote: post #51
avereveard wrote: post #32

Earlier quoted context omitted.

To reduce it a little, matrix holds numbers, tensor holds whatever. Numbers. Vectors. Operations.

So it is math lingo for array.

In math tensor has a more specific meaning than just an array, but in machine learning lingo, tensor just means an array or multidimensional array. That base is true in math as well, but it can also be a single value (rank zero tensor, aka a scalar. Sometimes ML will call this a tensor as well) and has to obey certain rules about linearity for some operations applied to it.

Re: The Illustrated Stable Diffusion

#96

Post by kuu »

uptown wrote: post #52

“We then compare the resulting embeddings using cosine similarity. When we begin the training process, the similarity will be low, even if the text describes the image correctly.” How is this training performed? How is accuracy rated?

Cosine similarity is a fixed way of comparing two vectors, so we can think of it as making a difference: A-B = d

If d is close to 0, we say that both embeddings are similar.

If d is close to 1, we say that both embeddings are different.

Imagine we have the following data:

- Image A and its description A

- Image B and its description B

We would generate the following dataset:

- Image A & Description A. Expected label: 0

- Image B & Description B. Expected label: 0

- Image A & Description B. Expected label: 1

- Image B & Description A. Expected label: 1

The mixture of Image Y with Description Z with Y!=Z is what we call "negative sampling"

If the model predicts 1 but the expected value was 0 (or the other way around), it's a miss, and therefore the model is "penalized" and has to adjust the weights; if the prediction matches the expectation, it's a success, the model is not modified.

I hope this clears it

Re: The Illustrated Stable Diffusion

#97

Post by ilaksh »

astrange wrote: post #41
l33tman wrote: post #22

Earlier quoted context omitted.

The autoencoder that maps between that and the 512x512x3 RGB space was trained together with the model, so it is specialized in upscaling the 64x64x4 info to pixel space for this particular purpose. It's "just" a factor of 48 (de)compression..

And remember we only care about 8 bits of the output (actually less due to JPEG compression) but each latent value is a 32-bit float.

Oh so each dimension is measured in 64 bit numbers? So the total bytes is (64*8) * (64*8) * (4 *8 )?*

Re: The Illustrated Stable Diffusion

#98

Post by cl3misch »

wwarek wrote: post #79

Another very good (although less deep) explanation was published yesterday by Computerphile: https://www.youtube.com/watch?v=1CIpzeNxIhU

I also thought about this video while reading the HN post. The Computerphile video completely omits the latent space, right? But instead spends a lot of time on the iterative denoising. Even though I like Computerphile a lot, I don't think this was the best tradeoff.

Re: The Illustrated Stable Diffusion

#99

Post by dr_dshiv »

TuringTest wrote: post #94
TigeriusKirk wrote: post #47

Earlier quoted context omitted.

One of my favorite moments in Geoffrey Hinton's otherwise pretty info-dense Coursera neural network class was when he said- "To deal with a 14-dimensional space, visualize a 3-D space and say 'fourteen' to yourself very loudly. Everyone does it."

Kudos to this. It also helps to think of a three-dimensional space bounded by the sides of a box, and to think of another 11 boxes stacked on top of each other. Then, you can visualize orthogonal on dimensions higher than 3 by throwing wires between equivalent points in two boxes.

Or imagine a spreadsheet with 14 columns, amiright?

Re: The Illustrated Stable Diffusion

#100

Post by spywaregorilla »

kuu wrote: post #96
uptown wrote: post #52

“We then compare the resulting embeddings using cosine similarity. When we begin the training process, the similarity will be low, even if the text describes the image correctly.” How is this training performed? How is accuracy rated?

Cosine similarity is a fixed way of comparing two vectors, so we can think of it as making a difference: A-B = d If d is close to 0, we say that both embeddings are similar. If d is close to 1, we say that both embeddings are different. Imagine we have the following data: - Image A and its description A - Image B and its description B We would generate the following dataset: - Image A & Description A. Expected label:…

I'd be curious to see an example gallery of image generation of the same vector scaled to different magnitudes. That is, 100% cosine similarity, but still hitting different points of the embedding space.

The outcome vectors aren't normalized right? So there could be a hefty amount of difference in this space? Maybe not on concept, but perhaps on image quality?

Post reply on HN