Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DeepFloyd IF: open-source text-to-image model

github.com

Post reply on HN
Print view View original

161–170 of 237 posts

Re: DeepFloyd IF: open-source text-to-image model

#161

Post by lucidrains »

mkaic wrote: post #145
causality0 wrote: post #139

Is this intended to replace Stable Diffusion? Somebody want to give the eli5?

This does outperform Stable Diffusion 2.1, but uses a different architecture and requires more memory and compute. Stable Diffusion runs its denoising process in a compressed "latent space" which is how it was able to be so compute-efficient compared to other diffusion models. It also uses the (relatively) small text encoder from OpenAI's CLIP model to encode user prompts. Both of these optimizations meant that it co…

tldr: bigger text encoder is better. SD will catch up quickly, as conditioning on a new set of precomputed text embeddings is a trivial change

Re: DeepFloyd IF: open-source text-to-image model

#163

Post by mkaic »

lucidrains wrote: post #161
mkaic wrote: post #145

Earlier quoted context omitted.

This does outperform Stable Diffusion 2.1, but uses a different architecture and requires more memory and compute. Stable Diffusion runs its denoising process in a compressed "latent space" which is how it was able to be so compute-efficient compared to other diffusion models. It also uses the (relatively) small text encoder from OpenAI's CLIP model to encode user prompts. Both of these optimizations meant that it co…

tldr: bigger text encoder is better. SD will catch up quickly, as conditioning on a new set of precomputed text embeddings is a trivial change

I didn't think about that but you're totally right, assuming they have those embeddings cached it would be super easy to retrain SD using them. 11B parameter count is rather unfortunate though tbh, I've never been the biggest fan of "scale is all you need" even though it seems to ring irritatingly true most of the time.

Re: DeepFloyd IF: open-source text-to-image model

#164

Post by FL33TW00D »

mkaic wrote: post #149
srajabi wrote: post #140

Wow this does so well on text! The original model struggled a lot, it's impressive to see how far they've come.

I'm quite curious how much of the improvement on text rendering is from the switch to pixel-space diffusion vs. the switch to a much larger pretrained text encoder. I'm leaning towards the latter, which then raises the question of what happens when you try training Stable Diffusion with T5-XXL-1.1 as the text encoder instead of CLIP — does it gain the ability to do text well?

Check out figure 4A from the ImageGen paper: https://arxiv.org/pdf/2205.11487.pdf

From that - I would strongly suspect the answer to your question to be yes.

Re: DeepFloyd IF: open-source text-to-image model

#165

Post by tasty_freeze »

manv1 wrote: post #157
gjsman-1000 wrote: post #144

Earlier quoted context omitted.

This is the dumb part about open-source models. Criminals, governments, and propaganda spreaders need not worry about the license; but legitimate users do.

This is the same problem with laws. The only people that follow them are legitimate users.

I hear this complaint often, especially in regards to gun control. Yes, there are a subset of people who do what they are going to do irrespective of laws. There is also a middle ground of people where the laws might curtail unwanted behavior. But the main purpose is it provides a basis for punishing unwanted behavior.

To make it concrete, one could argue that bank robbers rob banks even though it is illegal, so why have a law against it since law abiding people aren't going to rob banks. Does anyone really think we should remove such laws?

Re: DeepFloyd IF: open-source text-to-image model

#166

Post by minimaxir »

mkaic wrote: post #149
srajabi wrote: post #140

Wow this does so well on text! The original model struggled a lot, it's impressive to see how far they've come.

I'm quite curious how much of the improvement on text rendering is from the switch to pixel-space diffusion vs. the switch to a much larger pretrained text encoder. I'm leaning towards the latter, which then raises the question of what happens when you try training Stable Diffusion with T5-XXL-1.1 as the text encoder instead of CLIP — does it gain the ability to do text well?

DeepFloyd IF is effectively the same architecture/text encoder as Imagen (https://imagen.research.google/), although that paper doesn't hypothesize why text works out a lot better.

Re: DeepFloyd IF: open-source text-to-image model

#167

Post by simonw »

srajabi wrote: post #140

Wow this does so well on text! The original model struggled a lot, it's impressive to see how far they've come.

It's much better, but it's not perfect. Here's what I got for:

> a photograph of raccoon in the woods holding a sign that says "I will eat your trash"

https://twitter.com/simonw/status/1651994059781832704

Re: DeepFloyd IF: open-source text-to-image model

#168

Post by [deleted] »

minimaxir wrote: post #152

GitHub: https://github.com/deep-floyd/IF Colab Notebook for running the model based on the diffusers library: https://colab.research.google.com/github/huggingface/noteboo... Hugging Face Space for testing the model: https://huggingface.co/spaces/DeepFloyd/IF Note that the model is substantially more compute-intensive than Stable Diffusion, so it may be slower even though that space is running on an A100.

[deleted]

Re: DeepFloyd IF: open-source text-to-image model

#169

Post by mkaic »

minimaxir wrote: post #166
mkaic wrote: post #149

Earlier quoted context omitted.

I'm quite curious how much of the improvement on text rendering is from the switch to pixel-space diffusion vs. the switch to a much larger pretrained text encoder. I'm leaning towards the latter, which then raises the question of what happens when you try training Stable Diffusion with T5-XXL-1.1 as the text encoder instead of CLIP — does it gain the ability to do text well?

DeepFloyd IF is effectively the same architecture/text encoder as Imagen ( https://imagen.research.google/ ), although that paper doesn't hypothesize why text works out a lot better.

Right, I'm aware of the Imagen architecture, just curious to see further research determining which aspect of it is responsible for the improved text rendering.

EDIT: According to the figure in the Imagen paper FL33TW00D's response referred me to, it looks like the text encoder size is the biggest factor in the improved model performance all-around.

Re: DeepFloyd IF: open-source text-to-image model

#170

Post by dragonwriter »

mkaic wrote: post #145
causality0 wrote: post #139

Is this intended to replace Stable Diffusion? Somebody want to give the eli5?

This does outperform Stable Diffusion 2.1, but uses a different architecture and requires more memory and compute. Stable Diffusion runs its denoising process in a compressed "latent space" which is how it was able to be so compute-efficient compared to other diffusion models. It also uses the (relatively) small text encoder from OpenAI's CLIP model to encode user prompts. Both of these optimizations meant that it co…

> For example, Stability is gearing up to release Stable Diffusion XL right now, a larger version of the original SD that does higher fidelity and higher resolution generations. I wouldn't be surprised if it takes the crown back from DeepFloyd when it releases, but I guess we'll have to see.

SDXL is available on StabilityAI’s hosted services already, so they can be compared head to head.

Post reply on HN