I would be more interested in image-to-text models. Does someone know of any decent model? I saw the GPT4 demo, and they showed that they do image-to-text... but then that was actually a fake (i.e., the model was interpreting the image filename).
DeepFloyd IF: open-source text-to-image model
171–180 of 237 posts
Re: DeepFloyd IF: open-source text-to-image model
#172GitHub: https://github.com/deep-floyd/IF Colab Notebook for running the model based on the diffusers library: https://colab.research.google.com/github/huggingface/noteboo... Hugging Face Space for testing the model: https://huggingface.co/spaces/DeepFloyd/IF Note that the model is substantially more compute-intensive than Stable Diffusion, so it may be slower even though that space is running on an A100.
I tried the HF Space and it generates images of 64x64 resolution, which are basically useless.
Re: DeepFloyd IF: open-source text-to-image model
#173Earlier quoted context omitted.
I'm quite curious how much of the improvement on text rendering is from the switch to pixel-space diffusion vs. the switch to a much larger pretrained text encoder. I'm leaning towards the latter, which then raises the question of what happens when you try training Stable Diffusion with T5-XXL-1.1 as the text encoder instead of CLIP — does it gain the ability to do text well?
Check out figure 4A from the ImageGen paper: https://arxiv.org/pdf/2205.11487.pdf From that - I would strongly suspect the answer to your question to be yes.
Ah, yes, this seems to be pretty strong evidence. Thanks for pointing that figure out to me!
Re: DeepFloyd IF: open-source text-to-image model
#174GitHub: https://github.com/deep-floyd/IF Colab Notebook for running the model based on the diffusers library: https://colab.research.google.com/github/huggingface/noteboo... Hugging Face Space for testing the model: https://huggingface.co/spaces/DeepFloyd/IF Note that the model is substantially more compute-intensive than Stable Diffusion, so it may be slower even though that space is running on an A100.
> Note that the model is substantially more compute-intensive than Stable Diffusion, so it may be slower even though that space is running on an A100.
Any more specifics on this? Sampling has gotten better too. My primary concern is the amount of memory necessary for a generation (batch size = 1, or I guess "2" using classifier free guidance).
Similar to OpenAI's cascaded diffusion models and GLIDE, you can presumably run the models in sequence, unloading earlier models from memory to make room for the 2nd and 3rd stage models.
Right now, I really only need 256px resolution. So, will I be able to fit the first stage (64px) model in memory on its own with my 12 GB RTX 3060? What about the 2nd stage (64px -> 256px)? 3rd stage?
Re: DeepFloyd IF: open-source text-to-image model
#175Earlier quoted context omitted.
> Stability AI releases DeepFloyd IF, a powerful text-to-image model Hope not. This is a worse license.
As far as I can tell from Emad's discord and twitter discussion, the idea appears to be to make this a "research" release, and therefore the worse license. At a later point the model will be renamed "StableIf", and released with a similar license to StableDiffusion.
Well, its a better license than SDXL is available under right now (which is “you can’t have it, but you can use it on StabilityAI’s hosted services”.)
Re: DeepFloyd IF: open-source text-to-image model
#176The current license makes this largely unusable for nearly any purpose. Really disappointing release from SAI.
[deleted]
Re: DeepFloyd IF: open-source text-to-image model
#177Wow this does so well on text! The original model struggled a lot, it's impressive to see how far they've come.
I'm quite curious how much of the improvement on text rendering is from the switch to pixel-space diffusion vs. the switch to a much larger pretrained text encoder. I'm leaning towards the latter, which then raises the question of what happens when you try training Stable Diffusion with T5-XXL-1.1 as the text encoder instead of CLIP — does it gain the ability to do text well?
It is most likely due to the text encoder - see "Character-Aware Models Improve Visual Text Rendering". https://arxiv.org/abs/2212.10562
Re: DeepFloyd IF: open-source text-to-image model
#178Earlier quoted context omitted.
This does outperform Stable Diffusion 2.1, but uses a different architecture and requires more memory and compute. Stable Diffusion runs its denoising process in a compressed "latent space" which is how it was able to be so compute-efficient compared to other diffusion models. It also uses the (relatively) small text encoder from OpenAI's CLIP model to encode user prompts. Both of these optimizations meant that it co…
> For example, Stability is gearing up to release Stable Diffusion XL right now, a larger version of the original SD that does higher fidelity and higher resolution generations. I wouldn't be surprised if it takes the crown back from DeepFloyd when it releases, but I guess we'll have to see. SDXL is available on StabilityAI’s hosted services already, so they can be compared head to head.
I believe the version available on DreamStudio is heavily RLHF-tuned, no? I'm mostly interested to see how the raw weights perform out of the box compared to IF, which we have to wait for the release for.
Re: DeepFloyd IF: open-source text-to-image model
#179GitHub: https://github.com/deep-floyd/IF Colab Notebook for running the model based on the diffusers library: https://colab.research.google.com/github/huggingface/noteboo... Hugging Face Space for testing the model: https://huggingface.co/spaces/DeepFloyd/IF Note that the model is substantially more compute-intensive than Stable Diffusion, so it may be slower even though that space is running on an A100.
> Note that the model is substantially more compute-intensive than Stable Diffusion, so it may be slower even though that space is running on an A100. Any more specifics on this? Sampling has gotten better too. My primary concern is the amount of memory necessary for a generation (batch size = 1, or I guess "2" using classifier free guidance). Similar to OpenAI's cascaded diffusion models and GLIDE, you can presumabl…
They’ve said the whole pipeline (using the large models in each step with a choice, IIRC) can be run sequenced with 14GB VRAM.
Re: DeepFloyd IF: open-source text-to-image model
#180Wow this does so well on text! The original model struggled a lot, it's impressive to see how far they've come.
It's much better, but it's not perfect. Here's what I got for: > a photograph of raccoon in the woods holding a sign that says "I will eat your trash" https://twitter.com/simonw/status/1651994059781832704
It actually adds quite a nice charm by wording it not correctly