Explaining the SDXL Latent Space
huggingface.co
Explaining the SDXL Latent Space
1–10 of 34 posts
Re: Explaining the SDXL Latent Space
#2That's very cool. I had no idea the latent space was that accessible and obviously manipulatable.
Also interesting is how the way sdxl structures latents affects how it thinks about images.
Re: Explaining the SDXL Latent Space
#3I’ve been playing with diffusion a ton for the past few months, writing a new sampler that implements an iterative blending technique described in a recent paper. The latent space is rich in semantic information, so it can be a great place to apply various transformations rather than operating on the image directly. Yet it still has a significant spatial component, so things you do in one spatial area will affect that same area of the image.
Stable Diffusion 1.5 may be quite old now, but it is an incredibly rich model that still yields shockingly good results. SDXL is newer and more high tech, but it’s not a revolutionary improvement. It can be less malleable than the older model and harder to work with to achieve a given desired result.
Re: Explaining the SDXL Latent Space
#4I’ve been playing with diffusion a ton for the past few months, writing a new sampler that implements an iterative blending technique described in a recent paper. The latent space is rich in semantic information, so it can be a great place to apply various transformations rather than operating on the image directly. Yet it still has a significant spatial component, so things you do in one spatial area will affect tha…
> It can be less malleable than the older model and harder to work with to achieve a given desired result.
That has been my experience as well. It's frustrating because SDXL can be exquisite, but SD 1.5 is more "fun" to work with and more creative. I can throw random ideas into a mish-mash of a prompt and SD 1.5 will output an array of interesting things while SDXL will just seem to fall back to something "reasonable", ignoring anything "weird" in the prompt. SDXL also seems to have a lot more position bias in the prompt. SD 1.5 had a bit of that, paying more attention to words earlier in the prompt, but SDXL takes that to a new level.
But SDXL can draw hands consistently, so ... it's a tough choice.
Re: Explaining the SDXL Latent Space
#5What's the reason for using RGB rather than, say, HSV? RGB seems like it would be fairly discontinuous. Or, do I have that backwards?
Re: Explaining the SDXL Latent Space
#6I don't think it's as simple as this naive approach suggests, but it's a good preliminary analysis. It's a good lesson that while being absolutely correct might be quite difficult, diving in and having a go might get you further than you think.
Re: Explaining the SDXL Latent Space
#7What's the reason for using RGB rather than, say, HSV? RGB seems like it would be fairly discontinuous. Or, do I have that backwards?
I think there might be an opinion that since most colour space conversions can be expressed with relatively small neural nets (since they are mostly accumulations of variously scaled values), the autoencoder can dedicate a negligible proportion of its parameters towards that job and that gives it the potential to choose whatever color space training dictates.
I'm not entirely convinced by this idea myself. I have seen a few networks where a range of -1..1 inputs do a lot better than inputs in the range of 0..2 even though translation should be an easyish step for the network to figure out. The benefit from preprocessing the inputs, to me seem to be more advantageous than my common sense tells me it should be.
Re: Explaining the SDXL Latent Space
#8All the patterns and textures are expressed by only one dimension? Bizarre.
Re: Explaining the SDXL Latent Space
#9What's the reason for using RGB rather than, say, HSV? RGB seems like it would be fairly discontinuous. Or, do I have that backwards?
It's more like YCbCr or YCgCo color space than RGB.
Re: Explaining the SDXL Latent Space
#10I’ve been playing with diffusion a ton for the past few months, writing a new sampler that implements an iterative blending technique described in a recent paper. The latent space is rich in semantic information, so it can be a great place to apply various transformations rather than operating on the image directly. Yet it still has a significant spatial component, so things you do in one spatial area will affect tha…
> It can be less malleable than the older model and harder to work with to achieve a given desired result. That has been my experience as well. It's frustrating because SDXL can be exquisite, but SD 1.5 is more "fun" to work with and more creative. I can throw random ideas into a mish-mash of a prompt and SD 1.5 will output an array of interesting things while SDXL will just seem to fall back to something "reasonable…
With comfyui, you can do SDXL > SD1.5 or SD1.5 > SDXL, it makes more sense to generate basic image in SDXL Turbo and apply the effects of a checkpoint later.