Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Direct pixel-space megapixel image generation with diffusion models

crowsonkb.github.io

Post reply on HN
Print view View original

21–30 of 50 posts

Re: Direct pixel-space megapixel image generation with diffusion models

#21

Post by Birch-san »

ttul wrote: post #18
Birch-san wrote: post #15

I'm one of the authors; happy to answer questions. this arch is of course nice for high-resolution synthesis, but there's some other cool stuff worth mentioning.. activations are small! so you can enjoy bigger batch sizes. this is due to the 4x patching we do on the ingress to the model, and the effectiveness of neighbourhood attention in joining patches at the seams. the model's inductive biases are pretty different…

Hi Alex Amazing work. I scanned the paper and dusted off my aging memories of Jeremy Howard’s course. Will your model live happily alongside the existing SD infrastructure such as ControlNet, IPAdapter, and the like? Obviously we will have to retrain these to fit onto your model, but conceptually, does your model have natural places where adapters of various kinds can be attached?

regarding ControlNet: we have a UNet backbone, so the idea of "make trainable copies of the encoder blocks" sounds possible. the other part, "use a zero-inited dense layer to project the peer-encoder output and add it to the frozen-decoder output" also sounds fine. not quite sure what they do with the mid-block but I doubt there'd be any problem there.

regarding IPAdapter: I'm not familiar with it, but from the code it looks like they just run cross-attention again and sum the two attention outputs. feels a bit weird to me, because the attention probabilities add up to 2 instead of 1. and they scale the bonus attention output only instead of lerping. it'd make more sense to me to formulate it as a cross-cross attention (Q against cat([key0, key1]) and cat([val0, val1])), but maybe they wanted it to begin as a no-op at the start of training or something. anyway.. yes, all of that should work fine with HDiT. the paper doesn't implement cross-attention, but it can be added in the standard way (e.g. like stable-diffusion) or as self-cross attention (e.g. DeepFloyd IF or Imagen).

I'd recommend though to make use of HDiT's mapping network. in our attention blocks, the input gets AdaNormed against the condition from the mapping network. this is currently used to convey stuff like class conditions, Karras augmentation conditions and timestep embeddings. but it supports conditioning on custom (single-token) conditions of your choosing. so you could use this to condition on an image embed (this would give you the same image-conditioning control as IPAdapter but via a simpler mechanism).

Re: Direct pixel-space megapixel image generation with diffusion models

#22

Post by SEGyges »

GaggiX wrote: post #14

I hope that all these insights about diffusion model training that have been explored in last few years will be used by Stability AI to train their large text-to-image models, because when it comes to that they just use to most basic pipeline you can imagine with plenty of problems that get "solved" by some workarounds, for example to train SDXL they used the scheduler used by the DDPM paper(2020), epsilon-objective…

I have good news about who wrote this paper

Re: Direct pixel-space megapixel image generation with diffusion models

#23

Post by Birch-san »

sophrocyne wrote: post #17
Birch-san wrote: post #15

I'm one of the authors; happy to answer questions. this arch is of course nice for high-resolution synthesis, but there's some other cool stuff worth mentioning.. activations are small! so you can enjoy bigger batch sizes. this is due to the 4x patching we do on the ingress to the model, and the effectiveness of neighbourhood attention in joining patches at the seams. the model's inductive biases are pretty different…

Alex - I run Invoke (one of the popular OSS SD UIs for pros) Thanks for your work - it’s been impactful since the early days of the project. Excited to see where we get to this year.

ah, originally lstein/stable-diffusion? yeah that was an important fork for us Mac users in the early days. I have to confess I've still never used a UI. :)

this year I'm hoping for efficiency and small models! even if it's proprietary. if our work can reduce some energy usage behind closed doors that'd still be a good outcome.

Re: Direct pixel-space megapixel image generation with diffusion models

#24

Post by fpgaminer »

Seems like a solid paper from a skim through it. My rough summary:

The popular large scale diffusion models like StableDiffusion are CNN based at their heart, with attention layers sprinkled throughout. This paper builds on recent research exploring whether competitive image diffusion models can be built out of purely transformers, no CNN layers.

In this paper they build a similar U-Net like structure, but out of transformer layers, to improve efficiency compared to a straight Transformer. They also use local attention when the resolution is high to save on computational cost, but regular global attention in the middle to maintain global coherence.

Based on ablation studies this allows them to maintain or slightly improve FID score compared to Transformer-only diffusion models that don't do U-net like structures, but at 1/10th the computation cost. An incredible feat for sure.

There is a variety of details: RoPE positional encoding, GEGELU activations, RMSNorm, learnable skip connections, learnable cosine-sim attention, neighborhood attention for the local attention, etc.

The biggest gains in FID occur when the authors use "soft-min-snr" as the loss function; FID drops from 41 to 28!

Lots of ablation study was done across all their changes (see Table 1).

Training is otherwise completely standard AdamW, 5e-4, 0.01, 256 batch, constant LR, 400k steps for most experiments at 128x128 resolution.

So yeah, overall seems like solid work that combines a great mixture of techniques and pushes Transformer based diffusion forward.

If scaled up I'm not sure it would be "revolutionary" in terms of FID compared to SDXL or DALLE3, mostly because SD and DALLE already use attention obviating the scaling issue, and lots of tricks like diffusion based VAEs. But it's likely to provide a nice incremental improvement in FID, since in general Transformers perform better than CNNs unless the CNNs are _heavily_ tuned.

And being pixel based rather than latent based has many advantages.

Re: Direct pixel-space megapixel image generation with diffusion models

#25

Post by Birch-san »

gutianpei wrote: post #20

I think NATTEN does not support cross attention, wonder if the authors have tried any text-conditioned cases? Does the cross-attention can only add to regular attention? Or added through adanorm?

cross-attention doesn't need to involve NATTEN. there's no neighbourhood involved because it's not self-attention. so you can do it the stable-diffusion way: after self-attention, run torch sdp with Q=image and K=V=text.

I tried adding "stable-diffusion-style" cross-attn to HDiT, text-conditioning on small class-conditional datasets (oxford flowers), embedding the class labels as text prompts with Phi-1.5. trained it for a few minutes, and the images were relevant to the prompts, so it seemed to be working fine.

but if instead of a text condition you have a single-token condition (class label) then yeah the adanorm would be a simpler way.

Re: Direct pixel-space megapixel image generation with diffusion models

#26

Post by fpgaminer »

nwoli wrote: post #7
tbalsam wrote: post #2

I enjoyed this paper (I share a discord with the author so I read it a bit earlier). It's not entirely clear from the comparison numbers at the end, but I think the big argument here is efficiency for the amount of performance achieved. One can get lower FID numbers, but also with a ton of compute. I can't really speak technically to it as I've not given it a super in depth look, but this seems like a nice set of mot…

Which discord if its open to the public? I was on one woth kath in 2021 and loved her insights, would love to again

Same; a good ML focused discord would be great. Training ViTs all day is lonely work. I'm mostly locked into skimming the "Research" channels of image generation discords. LAION used to be decent with a good amount of interesting discussion, but it seems to have devolved into toxicity in the last year.

Re: Direct pixel-space megapixel image generation with diffusion models

#27

Post by Birch-san »

artninja1988 wrote: post #16

Looking at the output image examples, very nice, although they seem a little blurry. But I guess that's a dataset issue? Have you tried training anything above 1024x1024? Hope someone releases a model based on this since open source pixel space models are a rarity afaik

the FFHQ-1024 examples shouldn't be blurry. you can download the originals from the project page[0] — click any image in the teaser, or download our 50k samples.

the ImageNet-256 examples also aren't typically blurry (but they are 256x256 so your viewer may be bicubic scaling them or something). the ImageNet dataset _can_ have blurry, compressed or low resolution training samples, which can afflict some classes more than others, and we learn to produce samples like the training set.

[0] https://crowsonkb.github.io/hourglass-diffusion-transformers...

Re: Direct pixel-space megapixel image generation with diffusion models

#28

Post by Birch-san »

fpgaminer wrote: post #24

Seems like a solid paper from a skim through it. My rough summary: The popular large scale diffusion models like StableDiffusion are CNN based at their heart, with attention layers sprinkled throughout. This paper builds on recent research exploring whether competitive image diffusion models can be built out of purely transformers, no CNN layers. In this paper they build a similar U-Net like structure, but out of tra…

FID doesn't reward high-resolution detail. the inception feature size is 299x299! so we are forced to downsample our FFHQ-1024 samples to compute FID.

it also doesn't punish poor detail either! this advantages latent diffusion, which can claim to achieve a high resolution but without actually needing to have correct textures to get good metrics.

Re: Direct pixel-space megapixel image generation with diffusion models

#29

Post by GaggiX »

SEGyges wrote: post #22
GaggiX wrote: post #14

I hope that all these insights about diffusion model training that have been explored in last few years will be used by Stability AI to train their large text-to-image models, because when it comes to that they just use to most basic pipeline you can imagine with plenty of problems that get "solved" by some workarounds, for example to train SDXL they used the scheduler used by the DDPM paper(2020), epsilon-objective…

I have good news about who wrote this paper

Two authors are from Stability AI, that's the reason why I wrote the comment.

Re: Direct pixel-space megapixel image generation with diffusion models

#30

Post by Birch-san »

sorenjan wrote: post #11

This is probably a stupid question, but what kind of image generation does this do? The architecture overview shows "input image", and I don't see anything about text to image. Is it super resolution? Does class-conditional mean that it takes a class like "car" or "face" and generate a new random image of that class?

> Is it super resolution?

nope, we don't do Imagen-style super-resolution. we go direct to high resolution with a single-stage model.

Post reply on HN