Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Direct pixel-space megapixel image generation with diffusion models

crowsonkb.github.io

Post reply on HN
Print view View original

31–40 of 50 posts

Re: Direct pixel-space megapixel image generation with diffusion models

#31

Post by sorenjan »

Birch-san wrote: post #30
sorenjan wrote: post #11

This is probably a stupid question, but what kind of image generation does this do? The architecture overview shows "input image", and I don't see anything about text to image. Is it super resolution? Does class-conditional mean that it takes a class like "car" or "face" and generate a new random image of that class?

> Is it super resolution? nope, we don't do Imagen-style super-resolution. we go direct to high resolution with a single-stage model.

I was referring to the input image in the diagram, what is that and how is the output image generated from it? Is it 256x256 noise that gets denoised into an image? I guess what I'm really asking is what guides the process into the final image if it's not text to image?

Re: Direct pixel-space megapixel image generation with diffusion models

#32

Post by bravura »

ttul wrote: post #18
Birch-san wrote: post #15

I'm one of the authors; happy to answer questions. this arch is of course nice for high-resolution synthesis, but there's some other cool stuff worth mentioning.. activations are small! so you can enjoy bigger batch sizes. this is due to the 4x patching we do on the ingress to the model, and the effectiveness of neighbourhood attention in joining patches at the seams. the model's inductive biases are pretty different…

Hi Alex Amazing work. I scanned the paper and dusted off my aging memories of Jeremy Howard’s course. Will your model live happily alongside the existing SD infrastructure such as ControlNet, IPAdapter, and the like? Obviously we will have to retrain these to fit onto your model, but conceptually, does your model have natural places where adapters of various kinds can be attached?

IPAdapter, I am curious if there are useful GUIs for this? Creating image masks through uploading to colab is not so cute.

Re: Direct pixel-space megapixel image generation with diffusion models

#33

Post by stefanbaumann »

sorenjan wrote: post #31
Birch-san wrote: post #30

Earlier quoted context omitted.

> Is it super resolution? nope, we don't do Imagen-style super-resolution. we go direct to high resolution with a single-stage model.

I was referring to the input image in the diagram, what is that and how is the output image generated from it? Is it 256x256 noise that gets denoised into an image? I guess what I'm really asking is what guides the process into the final image if it's not text to image?

The "input image" is just the noisy sample from the previous timestep, yes.

The overall architecture diagram does not explicitly show the conditioning mechanism, which is a small separate network. For this paper, we only trained on class-conditional ImageNet and completely unconditional megapixel-scale FFHQ.

Training large-scale text-to-image models with this architecture is something we have not yet attempted, although there's no indication that this shouldn't work with a few tweaks.

Re: Direct pixel-space megapixel image generation with diffusion models

#34

Post by orbital-decay »

bravura wrote: post #32
ttul wrote: post #18

Earlier quoted context omitted.

Hi Alex Amazing work. I scanned the paper and dusted off my aging memories of Jeremy Howard’s course. Will your model live happily alongside the existing SD infrastructure such as ControlNet, IPAdapter, and the like? Obviously we will have to retrain these to fit onto your model, but conceptually, does your model have natural places where adapters of various kinds can be attached?

IPAdapter, I am curious if there are useful GUIs for this? Creating image masks through uploading to colab is not so cute.

Here's one example: https://github.com/Acly/krita-ai-diffusion/

But generally, most other UIs support it. It has serious limitations though, for example it center-crops the input to 224x224px. (which is enough for a surprisingly large amount of uses, but not enough for many others)

Re: Direct pixel-space megapixel image generation with diffusion models

#35

Post by sorenjan »

stefanbaumann wrote: post #33
sorenjan wrote: post #31

Earlier quoted context omitted.

I was referring to the input image in the diagram, what is that and how is the output image generated from it? Is it 256x256 noise that gets denoised into an image? I guess what I'm really asking is what guides the process into the final image if it's not text to image?

The "input image" is just the noisy sample from the previous timestep, yes. The overall architecture diagram does not explicitly show the conditioning mechanism, which is a small separate network. For this paper, we only trained on class-conditional ImageNet and completely unconditional megapixel-scale FFHQ. Training large-scale text-to-image models with this architecture is something we have not yet attempted, altho…

Thank you, I'm not used to reading this kind of research papers but I think I got the gist of it now.

Can this architecture be used to distill models that need fewer timesteps like LCMs or SDXL turbo?

Re: Direct pixel-space megapixel image generation with diffusion models

#36

Post by l33tman »

fpgaminer wrote: post #26
nwoli wrote: post #7

Earlier quoted context omitted.

Which discord if its open to the public? I was on one woth kath in 2021 and loved her insights, would love to again

Same; a good ML focused discord would be great. Training ViTs all day is lonely work. I'm mostly locked into skimming the "Research" channels of image generation discords. LAION used to be decent with a good amount of interesting discussion, but it seems to have devolved into toxicity in the last year.

LAION is good

Re: Direct pixel-space megapixel image generation with diffusion models

#37

Post by stefanbaumann »

sorenjan wrote: post #35
stefanbaumann wrote: post #33

Earlier quoted context omitted.

The "input image" is just the noisy sample from the previous timestep, yes. The overall architecture diagram does not explicitly show the conditioning mechanism, which is a small separate network. For this paper, we only trained on class-conditional ImageNet and completely unconditional megapixel-scale FFHQ. Training large-scale text-to-image models with this architecture is something we have not yet attempted, altho…

Thank you, I'm not used to reading this kind of research papers but I think I got the gist of it now. Can this architecture be used to distill models that need fewer timesteps like LCMs or SDXL turbo?

Both Latent Consistency Models and Adversarial Diffusion Distillation (the method behind SDXL Turbo) are methods that do not depend on any specific properties of the backbone. So, as Hourglass Diffusion Transformers are just a new kind of backbone that can be used just like the Diffusion U-Nets in Stable Diffusion (XL), these methods should also be applicable to it.

Re: Direct pixel-space megapixel image generation with diffusion models

#38

Post by ttul »

orbital-decay wrote: post #34
bravura wrote: post #32

Earlier quoted context omitted.

IPAdapter, I am curious if there are useful GUIs for this? Creating image masks through uploading to colab is not so cute.

Here's one example: https://github.com/Acly/krita-ai-diffusion/ But generally, most other UIs support it. It has serious limitations though, for example it center-crops the input to 224x224px. (which is enough for a surprisingly large amount of uses, but not enough for many others)

Yes. I discussed this issue with the author of the ComfyUI IP-Adapter nodes. It would doubtless be handy if someone could end-to-end train a higher resolution IP-Adapter model that integrated its own variant of CLIPVision that is not subject to the 224px constraint. I have no idea what kind of horsepower would be required for that.

A latent space CLIPVision model would be cool too. Presumably you could leverage the semantic richness of the latent space to efficiently train a more powerful CLIPVision. I don’t know whether anyone has tried this. Maybe there is a good reason for that.

Re: Direct pixel-space megapixel image generation with diffusion models

#39

Post by sophrocyne »

Birch-san wrote: post #23
sophrocyne wrote: post #17

Earlier quoted context omitted.

Alex - I run Invoke (one of the popular OSS SD UIs for pros) Thanks for your work - it’s been impactful since the early days of the project. Excited to see where we get to this year.

ah, originally lstein/stable-diffusion? yeah that was an important fork for us Mac users in the early days. I have to confess I've still never used a UI. :) this year I'm hoping for efficiency and small models! even if it's proprietary. if our work can reduce some energy usage behind closed doors that'd still be a good outcome.

Yes, indeed. Lincoln's still an active maintainer.

Energy efficiency is key - Especially with some of these extremely inefficient (wasteful, even) features like real-time canvas.

Good luck - Let us know if/how we can help.

Re: Direct pixel-space megapixel image generation with diffusion models

#40

Post by SEGyges »

nwoli wrote: post #7
tbalsam wrote: post #2

I enjoyed this paper (I share a discord with the author so I read it a bit earlier). It's not entirely clear from the comparison numbers at the end, but I think the big argument here is efficiency for the amount of performance achieved. One can get lower FID numbers, but also with a ton of compute. I can't really speak technically to it as I've not given it a super in depth look, but this seems like a nice set of mot…

Which discord if its open to the public? I was on one woth kath in 2021 and loved her insights, would love to again

You and the guy below you in this thread should probably tag me on twitter, same tag as here, I can point you. I do not especially want to leave the discord link in a frontpage hn thread.
Post reply on HN