This is probably a stupid question, but what kind of image generation does this do? The architecture overview shows "input image", and I don't see anything about text to image. Is it super resolution? Does class-conditional mean that it takes a class like "car" or "face" and generate a new random image of that class?
Direct pixel-space megapixel image generation with diffusion models
11–20 of 50 posts
Re: Direct pixel-space megapixel image generation with diffusion models
#12This is probably a stupid question, but what kind of image generation does this do? The architecture overview shows "input image", and I don't see anything about text to image. Is it super resolution? Does class-conditional mean that it takes a class like "car" or "face" and generate a new random image of that class?
If it's Imagenet class-conditioned, FFHQ unconditioned.
>Does class-conditional mean that it takes a class like "car" or "face" and generate a new random image of that class?
Yup
Re: Direct pixel-space megapixel image generation with diffusion models
#13Earlier quoted context omitted.
You're actually talking to a bot, in this particular case. 12 minutes old with -2 karma. :berk:
Did you seriously just berk me on HN. What next, is a Walmart cashier going to do that RobloxNite "dab dance" or whatever on me? Discusting. Dusgraseful.
You can take the Katt Williams route in response, or the way of Zen. The choice is yours.
Re: Direct pixel-space megapixel image generation with diffusion models
#14I hope that all these insights about diffusion model training that have been explored in last few years will be used by Stability AI to train their large text-to-image models, because when it comes to that they just use to most basic pipeline you can imagine with plenty of problems that get "solved" by some workarounds, for example to train SDXL they used the scheduler used by the DDPM paper(2020), epsilon-objective and noise-offset, an ugly workaround that was created when people realized that SD v1.5 wasn't able to generate images that were too dark or bright, a problem related to the epsilon-objective that cause the model to always generate images with a mean close to 0 (the same as the gaussian noise).
A few people have finetuned Stable Diffusion models on v-objective and solved the problem from the root.
Re: Direct pixel-space megapixel image generation with diffusion models
#15I'm one of the authors; happy to answer questions.
this arch is of course nice for high-resolution synthesis, but there's some other cool stuff worth mentioning..
activations are small! so you can enjoy bigger batch sizes. this is due to the 4x patching we do on the ingress to the model, and the effectiveness of neighbourhood attention in joining patches at the seams.
the model's inductive biases are pretty different than (for example) a convolutional UNet's. the innermost levels seem to train easily, so images can have good global coherence early in training.
there's no convolutions! so you don't need to worry about artifacts stemming from convolution padding, or having canvas edge padding artifacts leak an implicit position bias.
we can finally see what high-resolution diffusion outputs look like _without_ latents! personally I think current latent VAEs don't _really_ achieve the high resolutions they claim (otherwise fine details like text would survive a VAE roundtrip faithfully); it's common to see latent diffusion outputs with smudgy skin or blurry fur. what I'd like to see in the future of latent diffusion is to listen to the Emu paper and use more channels, or a less ambitious upsample.
it's a transformer! so we can try applying to it everything we know about transformers, like sigma reparameterisation or multimodality. some tricks like masked training will require extra support in [NATTEN](https://github.com/SHI-Labs/NATTEN), but we're very happy with its featureset and performance so far.
but honestly I'm most excited about the efficiency. there's too little work on making pretraining possible at GPU-poor scale. so I was very happy to see HDiT could succeed at small-scale tasks within the resources I had at home (you can get nice oxford flowers samples at 256x256px with half an hour on a 4090). I think with models that are better fits for the problem, perhaps we can get good results with smaller models. and I'd like to see big tech go that direction too!
-Alex Birch
Re: Direct pixel-space megapixel image generation with diffusion models
#16Looking at the output image examples, very nice, although they seem a little blurry. But I guess that's a dataset issue?
Have you tried training anything above 1024x1024? Hope someone releases a model based on this since open source pixel space models are a rarity afaik
Re: Direct pixel-space megapixel image generation with diffusion models
#17I'm one of the authors; happy to answer questions. this arch is of course nice for high-resolution synthesis, but there's some other cool stuff worth mentioning.. activations are small! so you can enjoy bigger batch sizes. this is due to the 4x patching we do on the ingress to the model, and the effectiveness of neighbourhood attention in joining patches at the seams. the model's inductive biases are pretty different…
Alex - I run Invoke (one of the popular OSS SD UIs for pros)
Thanks for your work - it’s been impactful since the early days of the project.
Excited to see where we get to this year.
Re: Direct pixel-space megapixel image generation with diffusion models
#18I'm one of the authors; happy to answer questions. this arch is of course nice for high-resolution synthesis, but there's some other cool stuff worth mentioning.. activations are small! so you can enjoy bigger batch sizes. this is due to the 4x patching we do on the ingress to the model, and the effectiveness of neighbourhood attention in joining patches at the seams. the model's inductive biases are pretty different…
Hi Alex
Amazing work. I scanned the paper and dusted off my aging memories of Jeremy Howard’s course. Will your model live happily alongside the existing SD infrastructure such as ControlNet, IPAdapter, and the like? Obviously we will have to retrain these to fit onto your model, but conceptually, does your model have natural places where adapters of various kinds can be attached?
Re: Direct pixel-space megapixel image generation with diffusion models
#19Earlier quoted context omitted.
HN to SEO spam pipeline
Ah yes, the checks notes ever popular bank account hacking SEO spam line. (Legitimately I am confused but maybe it is a one-two scam or the like, lolz. <3 :')))) ;'PPPP ;'PPPP)
Sure. "pay me xx amount of money to hack your enemy's bank account", or possibly "pay me xx amount of money to get your money back after you've been hacked"
Re: Direct pixel-space megapixel image generation with diffusion models
#20I think NATTEN does not support cross attention, wonder if the authors have tried any text-conditioned cases? Does the cross-attention can only add to regular attention? Or added through adanorm?