Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Explaining the SDXL Latent Space

huggingface.co

Post reply on HN
Print view View original

21–30 of 34 posts

Re: Explaining the SDXL Latent Space

#21

Post by viraptor »

incrudible wrote: post #20
viraptor wrote: post #16

Earlier quoted context omitted.

> But the data trained on is usually in RGB format, so probably the reasoning. It's trivial to convert the values for training - basically 0% of the cost of the process. But there's likely more "meaning" in HSV than in RGB. So I don't think that would account for the difference.

ML systems generally do not care about human semantics, and they will not produce them naturally. The VAE works at 16 bits float per channel, so compression is not an issue either, but if it was, HSV would be a poor choice too.

ML systems don't care, but humans do and better semantically-meaningful representations in training data usually lead to better results for us. In images you often care about "different colours of similar brightness" rather than "matching levels of 3 colour components", so there's a non-zero chance HSV/HLS would do better than RGB. It's nothing to do with compression.

Re: Explaining the SDXL Latent Space

#22

Post by countWSS »

fpgaminer wrote: post #4
ttul wrote: post #3

I’ve been playing with diffusion a ton for the past few months, writing a new sampler that implements an iterative blending technique described in a recent paper. The latent space is rich in semantic information, so it can be a great place to apply various transformations rather than operating on the image directly. Yet it still has a significant spatial component, so things you do in one spatial area will affect tha…

> It can be less malleable than the older model and harder to work with to achieve a given desired result. That has been my experience as well. It's frustrating because SDXL can be exquisite, but SD 1.5 is more "fun" to work with and more creative. I can throw random ideas into a mish-mash of a prompt and SD 1.5 will output an array of interesting things while SDXL will just seem to fall back to something "reasonable…

SDXL(and possibly 2.1) switched to different CLIP implementation that is geared for sentence-level understanding, SD1.5 uses old CLIP that works with tag-cloud type prompts.

Re: Explaining the SDXL Latent Space

#23

Post by wruza »

ttul wrote: post #3

I’ve been playing with diffusion a ton for the past few months, writing a new sampler that implements an iterative blending technique described in a recent paper. The latent space is rich in semantic information, so it can be a great place to apply various transformations rather than operating on the image directly. Yet it still has a significant spatial component, so things you do in one spatial area will affect tha…

Entered this thread to write your comment. I find SDXL inferior to 1.5 and yes, much harder to work with.

My another issue is that sdxl images that you can see on the web always have that “from a movie/ads”-?ish? coating. Can’t explain it, but it feels even more uncanny than 1.5.

SDXL is too resource-hungry for what it produces. 3x+ model sizes, 12GB vram is barely enough for it, 40 steps is the minimum, and I don’t think training loras will turn out feasible at all. I can’t lower the resolution without distortions, and even proportions are hard to deal with. It feels much less flexible than 1.5 in this regard.

I’m sticking with 1.5, no sdxl plans.

Re: Explaining the SDXL Latent Space

#24

Post by l33tman »

01HNNWZ0MV43FF wrote: post #8

All the patterns and textures are expressed by only one dimension? Bizarre.

It's only patterns and textures for 8x8 images, so I guess it could make sense, you're not going to need every conceivable pattern of 8x8 pixels in normal images..

If you quantize those 4 floats per 8x8 block, is that encoding better than say the old venerable JPG 8x8 DCT + quant?

Re: Explaining the SDXL Latent Space

#25

Post by l33tman »

Just a terminology comment here. "Latent space" means a lot of different things in different models. For a GAN for example it actually means the "top concept" space where you can change the entire concept of the image by moving around in the latent space, which is notoriously difficult. For SD/SDXL it refers to the bottommost layer just above pixelspace, which expands the generated image from 64x64 to 512x512 pixels in the case of SD1.5.

This allows the rest of the network to be smaller while still generating a usable output resolution, so it's a performance "hack".

It's a really good idea to explore it and hack into it like in the article, to "remaster" the image so to speak!

Re: Explaining the SDXL Latent Space

#26

Post by shenberg »

Lerc wrote: post #7
nomel wrote: post #5

What's the reason for using RGB rather than, say, HSV? RGB seems like it would be fairly discontinuous. Or, do I have that backwards?

I think there might be an opinion that since most colour space conversions can be expressed with relatively small neural nets (since they are mostly accumulations of variously scaled values), the autoencoder can dedicate a negligible proportion of its parameters towards that job and that gives it the potential to choose whatever color space training dictates. I'm not entirely convinced by this idea myself. I have see…

I suspect that weight initializations are geared towards inputs being normal random variables with mean 0 and variance 1. Deviating from that makes the learning process unhappy.

Re: Explaining the SDXL Latent Space

#27

Post by incrudible »

viraptor wrote: post #21
incrudible wrote: post #20

Earlier quoted context omitted.

ML systems generally do not care about human semantics, and they will not produce them naturally. The VAE works at 16 bits float per channel, so compression is not an issue either, but if it was, HSV would be a poor choice too.

ML systems don't care, but humans do and better semantically-meaningful representations in training data usually lead to better results for us. In images you often care about "different colours of similar brightness" rather than "matching levels of 3 colour components", so there's a non-zero chance HSV/HLS would do better than RGB. It's nothing to do with compression.

Does it lead to better results though? For the system, the best representation would be one that it learned - which is the latent representation, 4 channels in this case. Would it learn a "better" representation when fed with HSL instead of RGB? If so, what's the intuition? RGB somewhat resembles human vision, whereas HSL exists for interactive editing, and YCbCr exists for compression. If anything, I would expect YCbCr to outperform.

Re: Explaining the SDXL Latent Space

#28

Post by nomel »

incrudible wrote: post #27
viraptor wrote: post #21

Earlier quoted context omitted.

ML systems don't care, but humans do and better semantically-meaningful representations in training data usually lead to better results for us. In images you often care about "different colours of similar brightness" rather than "matching levels of 3 colour components", so there's a non-zero chance HSV/HLS would do better than RGB. It's nothing to do with compression.

Does it lead to better results though? For the system, the best representation would be one that it learned - which is the latent representation, 4 channels in this case. Would it learn a "better" representation when fed with HSL instead of RGB? If so, what's the intuition? RGB somewhat resembles human vision, whereas HSL exists for interactive editing, and YCbCr exists for compression. If anything, I would expect YC…

> If so, what's the intuition?

HSV closer resembles physical properties, for most natural things. Hue and saturation variations are usually meaningful variations in the actual material. Brightness variations often end up being mostly about lighting, rather than the material. It can be surprisingly effective for simple segmentation [1], which is why it's usually the first one implemented in computer vision classes.

Our eyes have RGB sensors, but I would claim I perceive the colors in my surroundings in something like HSV (although, that could very well be from the way I learned colors). And, I think this makes sense: if you're looking for something, you want a color perception that's not overly sensitive to lighting conditions. RGB is directly related.

[1] https://medium.com/neurosapiens/segmentation-and-classificat...

Re: Explaining the SDXL Latent Space

#29

Post by ttul »

countWSS wrote: post #22
fpgaminer wrote: post #4

Earlier quoted context omitted.

> It can be less malleable than the older model and harder to work with to achieve a given desired result. That has been my experience as well. It's frustrating because SDXL can be exquisite, but SD 1.5 is more "fun" to work with and more creative. I can throw random ideas into a mish-mash of a prompt and SD 1.5 will output an array of interesting things while SDXL will just seem to fall back to something "reasonable…

SDXL(and possibly 2.1) switched to different CLIP implementation that is geared for sentence-level understanding, SD1.5 uses old CLIP that works with tag-cloud type prompts.

SDXL actually takes conditioning from either the old or the new CLIP, or both. The malleability of SDXL is not just down to the choice of the new CLIP; the UNet itself is more opinionated.

Re: Explaining the SDXL Latent Space

#30

Post by ttul »

SV_BubbleTime wrote: post #13
designium wrote: post #10

Earlier quoted context omitted.

With comfyui, you can do SDXL > SD1.5 or SD1.5 > SDXL, it makes more sense to generate basic image in SDXL Turbo and apply the effects of a checkpoint later.

Kind of blowing my mind here. Coming from Auto1111 for a year, I thought comfy was most like always using img2img, then I figured out it wasn’t that but laten2latent… which is cool, but using XL to get the better prompting and 1.5 to get checkpoints and Loras I want is making it all click now.

ComfyUI is insanely amazing. The learning curve is well worth the effort.
Post reply on HN