Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DeepFloyd IF: open-source text-to-image model

github.com

Post reply on HN
Print view View original

211–220 of 237 posts

Re: DeepFloyd IF: open-source text-to-image model

#211

Post by GaggiX »

bwesz wrote: post #172
minimaxir wrote: post #152

GitHub: https://github.com/deep-floyd/IF Colab Notebook for running the model based on the diffusers library: https://colab.research.google.com/github/huggingface/noteboo... Hugging Face Space for testing the model: https://huggingface.co/spaces/DeepFloyd/IF Note that the model is substantially more compute-intensive than Stable Diffusion, so it may be slower even though that space is running on an A100.

I tried the HF Space and it generates images of 64x64 resolution, which are basically useless.

There is a big "Upscale" button.

Re: DeepFloyd IF: open-source text-to-image model

#212

Post by marvinkennis »

jah242 wrote: post #193
marvinkennis wrote: post #156

Seeing a lot of text-to-image out there recently. Does anyone know what the current state of the art is on image-to-text? Thinking something similar to Midjourney's /describe command that they added in v5

This is text + image -> text but pretty cool and still might be of interest to you: https://llava-vl.github.io

Just entering "Describe this image" in the chat prompt got me exactly what I was looking for. Thanks!

Re: DeepFloyd IF: open-source text-to-image model

#213

Post by grungegun »

vitorgrs wrote: post #208

Tried using right now, and it's way better than Stable Diffusion (be it 1.5, 2.1 or SDXL). But is harder to get a good picture. This fine tuned with a good RLHF will be amazing.

What does this mean? Isn't the quality of a model determined by how easy it is to get a good picture?

Re: DeepFloyd IF: open-source text-to-image model

#214

Post by vitorgrs »

grungegun wrote: post #213
vitorgrs wrote: post #208

Tried using right now, and it's way better than Stable Diffusion (be it 1.5, 2.1 or SDXL). But is harder to get a good picture. This fine tuned with a good RLHF will be amazing.

What does this mean? Isn't the quality of a model determined by how easy it is to get a good picture?

Not necessarily. IMO a good model needs to follow your prompt well, and that was my problem with Stable Diffusion.

I've been trying to get a good portrait picture with "neon lights" on Stable Diffusion and it is almost impossible. Meanwhile with the new Dall-e, that was possible. The picture specially with SDXL is good, but it doesn't really have neon lights...

I tried now similar prompt on deepfloyd and managed to get there!

Re: DeepFloyd IF: open-source text-to-image model

#215

Post by causality0 »

mkaic wrote: post #145
causality0 wrote: post #139

Is this intended to replace Stable Diffusion? Somebody want to give the eli5?

This does outperform Stable Diffusion 2.1, but uses a different architecture and requires more memory and compute. Stable Diffusion runs its denoising process in a compressed "latent space" which is how it was able to be so compute-efficient compared to other diffusion models. It also uses the (relatively) small text encoder from OpenAI's CLIP model to encode user prompts. Both of these optimizations meant that it co…

Does the increased memory footprint mean it can't be run on a normal desktop like SD?

Re: DeepFloyd IF: open-source text-to-image model

#216

Post by dragonwriter »

causality0 wrote: post #215
mkaic wrote: post #145

Earlier quoted context omitted.

This does outperform Stable Diffusion 2.1, but uses a different architecture and requires more memory and compute. Stable Diffusion runs its denoising process in a compressed "latent space" which is how it was able to be so compute-efficient compared to other diffusion models. It also uses the (relatively) small text encoder from OpenAI's CLIP model to encode user prompts. Both of these optimizations meant that it co…

Does the increased memory footprint mean it can't be run on a normal desktop like SD?

The VRAM requirements are higher (14GB) so lots of things that can do SD won’t do this with thr existing toolchain. But some of that is “aftermarket” SD optimization, and this maybe could see some of that, too.

But there are consumer cards with 14GB+ VRAM, so its not, even before optimization, out of reach of consumer hardware.

Re: DeepFloyd IF: open-source text-to-image model

#219

Post by [deleted] »

mkaic wrote: post #154

I think this model will result in a massive new wave of meme culture. AI's already seen success in memes up to this point, but the ability for readable text to be incorporated into images totally changes the game. Going to be an interesting next few months on the interwebz, that's for sure. Exciting times!

[deleted]

Re: DeepFloyd IF: open-source text-to-image model

#220

Post by dragonwriter »

2bitencryption wrote: post #196
mkaic wrote: post #145

Earlier quoted context omitted.

This does outperform Stable Diffusion 2.1, but uses a different architecture and requires more memory and compute. Stable Diffusion runs its denoising process in a compressed "latent space" which is how it was able to be so compute-efficient compared to other diffusion models. It also uses the (relatively) small text encoder from OpenAI's CLIP model to encode user prompts. Both of these optimizations meant that it co…

thanks for the explanation! denoising in latent space certainly seems like the "correct" path. My (amateur) thinking is, the more you can do in latent space, the better.

I’m not sure why “denoise in latent space at 64x64 and decode to pixel space at target resolution” is fundamentally better than “denoise in pixel space at 64x64, then upscale to pixel space at target resolution and denoise some more”.

The former seems likely to be lower compute-for-resolution, but that’s not the only consideration for “better”...

Post reply on HN