That was an excellently written article.
I for sure thought a discussion about latent spaces would instantly be over my head. It was, but took a few paragraphs.
Explaining the SDXL Latent Space
11–20 of 34 posts
Re: Explaining the SDXL Latent Space
#12Anyone know if the work shown here has been implemented in Automatic1111 or ComfyUI as an extension? If not, than that might be my first project to add since these are quite simple (relatively speaking) in the code to implement.
Re: Explaining the SDXL Latent Space
#13Earlier quoted context omitted.
> It can be less malleable than the older model and harder to work with to achieve a given desired result. That has been my experience as well. It's frustrating because SDXL can be exquisite, but SD 1.5 is more "fun" to work with and more creative. I can throw random ideas into a mish-mash of a prompt and SD 1.5 will output an array of interesting things while SDXL will just seem to fall back to something "reasonable…
With comfyui, you can do SDXL > SD1.5 or SD1.5 > SDXL, it makes more sense to generate basic image in SDXL Turbo and apply the effects of a checkpoint later.
Kind of blowing my mind here.
Coming from Auto1111 for a year, I thought comfy was most like always using img2img, then I figured out it wasn’t that but laten2latent… which is cool, but using XL to get the better prompting and 1.5 to get checkpoints and Loras I want is making it all click now.
Re: Explaining the SDXL Latent Space
#14What's the reason for using RGB rather than, say, HSV? RGB seems like it would be fairly discontinuous. Or, do I have that backwards?
The format isn't explicit to the network. But the data trained on is usually in RGB format, so probably the reasoning. I found a repo where someone tried different formats but it's wroth noting that this was for discrimination so just because it can discriminate doesn't mean it does the same thing. Maybe I'll run some experiments. You could use a UNet for classification and then look at the bottom layer and do the same thing. Be hard to do with SD (or SDXL) because you'd need to retrain with the format. Tuning could possibly work but the network would likely be biased to understand the RGB encoding.
Edit: ops, forgot the link
https://github.com/ducha-aiki/caffenet-benchmark/blob/master...
Re: Explaining the SDXL Latent Space
#15Thank you for the excellent article! Top notch work!
Re: Explaining the SDXL Latent Space
#16What's the reason for using RGB rather than, say, HSV? RGB seems like it would be fairly discontinuous. Or, do I have that backwards?
The format isn't explicit to the network. But the data trained on is usually in RGB format, so probably the reasoning. I found a repo where someone tried different formats but it's wroth noting that this was for discrimination so just because it can discriminate doesn't mean it does the same thing. Maybe I'll run some experiments. You could use a UNet for classification and then look at the bottom layer and do the sa…
> But the data trained on is usually in RGB format, so probably the reasoning.
It's trivial to convert the values for training - basically 0% of the cost of the process. But there's likely more "meaning" in HSV than in RGB. So I don't think that would account for the difference.
Re: Explaining the SDXL Latent Space
#17I’ve been playing with diffusion a ton for the past few months, writing a new sampler that implements an iterative blending technique described in a recent paper. The latent space is rich in semantic information, so it can be a great place to apply various transformations rather than operating on the image directly. Yet it still has a significant spatial component, so things you do in one spatial area will affect tha…
> It can be less malleable than the older model and harder to work with to achieve a given desired result. That has been my experience as well. It's frustrating because SDXL can be exquisite, but SD 1.5 is more "fun" to work with and more creative. I can throw random ideas into a mish-mash of a prompt and SD 1.5 will output an array of interesting things while SDXL will just seem to fall back to something "reasonable…
> But SDXL can draw hands consistently, so ... it's a tough choice.
Looking at the article photos it still has some way to go. I counted 3 cases of missing fingers, two cases of extra fingers (on the cartoon girl), and a few arm poses that in real life would need medical attention.
Re: Explaining the SDXL Latent Space
#18Enhance.
Re: Explaining the SDXL Latent Space
#19I’ve been playing with diffusion a ton for the past few months, writing a new sampler that implements an iterative blending technique described in a recent paper. The latent space is rich in semantic information, so it can be a great place to apply various transformations rather than operating on the image directly. Yet it still has a significant spatial component, so things you do in one spatial area will affect tha…
I am curious what do you mean by high tech?
Re: Explaining the SDXL Latent Space
#20Earlier quoted context omitted.
The format isn't explicit to the network. But the data trained on is usually in RGB format, so probably the reasoning. I found a repo where someone tried different formats but it's wroth noting that this was for discrimination so just because it can discriminate doesn't mean it does the same thing. Maybe I'll run some experiments. You could use a UNet for classification and then look at the bottom layer and do the sa…
> But the data trained on is usually in RGB format, so probably the reasoning. It's trivial to convert the values for training - basically 0% of the cost of the process. But there's likely more "meaning" in HSV than in RGB. So I don't think that would account for the difference.
ML systems generally do not care about human semantics, and they will not produce them naturally. The VAE works at 16 bits float per channel, so compression is not an issue either, but if it was, HSV would be a poor choice too.