Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DeepFloyd IF: open-source text-to-image model

github.com

Post reply on HN
Print view View original

191–200 of 237 posts

Re: DeepFloyd IF: open-source text-to-image model

#191

Post by grumbel »

danwee wrote: post #171

I would be more interested in image-to-text models. Does someone know of any decent model? I saw the GPT4 demo, and they showed that they do image-to-text... but then that was actually a fake (i.e., the model was interpreting the image filename).

CLIP Interrogator[1], which is also build into AUTOMATIC1111, gives quite reasonable results, at least if all you need is a prompt, it can't handle complex interactions:

Image: https://i.imgur.com/husplYZ.png

Output: "a white horse with a sign that says rexel's in space, pixelperfect, inspired by Paul Kelpe, official simpsons movie artwork, alternate album cover, in style of nanospace, by Apelles, pickles, pespective, pop surrealism, ingame, in a space cadet outfit, sifi"

[1] https://huggingface.co/spaces/pharma/CLIP-Interrogator

Re: DeepFloyd IF: open-source text-to-image model

#192

Post by krisoft »

alex_sf wrote: post #181
itslennysfault wrote: post #151

This could be super cool for logos. I've tried using Stable Diffusion to generate logos and it does pretty good at helping brainstorm, but the text is always gibberish so you can use its idea, but you have to add your own text which basically means creating a logo from scratch using its designs as inspiration.

You can't use this to make logos for any commercial product, and it's not safe to use it for hobby projects either, based on the current model license.

> You can't use this to make logos for any commercial product

Yeah good luck figuring out that a particular logo was generated with this particular model. And if someone does good luck doing anything about it.

With this amount of fear one wouldn't dare to cross a road without three layers of bubble wrap, plus written authorisation from a lawyer plus a feasibility study from a traffic engineer.

Re: DeepFloyd IF: open-source text-to-image model

#193

Post by jah242 »

marvinkennis wrote: post #156

Seeing a lot of text-to-image out there recently. Does anyone know what the current state of the art is on image-to-text? Thinking something similar to Midjourney's /describe command that they added in v5

This is text + image -> text but pretty cool and still might be of interest to you:

https://llava-vl.github.io

Re: DeepFloyd IF: open-source text-to-image model

#194

Post by jah242 »

danwee wrote: post #171

I would be more interested in image-to-text models. Does someone know of any decent model? I saw the GPT4 demo, and they showed that they do image-to-text... but then that was actually a fake (i.e., the model was interpreting the image filename).

This is text + image -> text but pretty cool and still might be of interest to you:

https://llava-vl.github.io

Re: DeepFloyd IF: open-source text-to-image model

#195

Post by indymike »

simonw wrote: post #167
srajabi wrote: post #140

Wow this does so well on text! The original model struggled a lot, it's impressive to see how far they've come.

It's much better, but it's not perfect. Here's what I got for: > a photograph of raccoon in the woods holding a sign that says "I will eat your trash" https://twitter.com/simonw/status/1651994059781832704

This is not a problem. The sign was clearly made by the raccoon.

Re: DeepFloyd IF: open-source text-to-image model

#196

Post by 2bitencryption »

mkaic wrote: post #145
causality0 wrote: post #139

Is this intended to replace Stable Diffusion? Somebody want to give the eli5?

This does outperform Stable Diffusion 2.1, but uses a different architecture and requires more memory and compute. Stable Diffusion runs its denoising process in a compressed "latent space" which is how it was able to be so compute-efficient compared to other diffusion models. It also uses the (relatively) small text encoder from OpenAI's CLIP model to encode user prompts. Both of these optimizations meant that it co…

thanks for the explanation!

denoising in latent space certainly seems like the "correct" path. My (amateur) thinking is, the more you can do in latent space, the better.

Re: DeepFloyd IF: open-source text-to-image model

#197

Post by GaggiX »

mkaic wrote: post #149
srajabi wrote: post #140

Wow this does so well on text! The original model struggled a lot, it's impressive to see how far they've come.

I'm quite curious how much of the improvement on text rendering is from the switch to pixel-space diffusion vs. the switch to a much larger pretrained text encoder. I'm leaning towards the latter, which then raises the question of what happens when you try training Stable Diffusion with T5-XXL-1.1 as the text encoder instead of CLIP — does it gain the ability to do text well?

The CLIP text encoder is trained to align with the pooled image embedding (a single vector), which is why most text embeddings are not very meaningful on their own (but still convey the overall semantics of the text). With T5 every text embedding is important.

Re: DeepFloyd IF: open-source text-to-image model

#200

Post by dormento »

alex_sf wrote: post #181
itslennysfault wrote: post #151

This could be super cool for logos. I've tried using Stable Diffusion to generate logos and it does pretty good at helping brainstorm, but the text is always gibberish so you can use its idea, but you have to add your own text which basically means creating a logo from scratch using its designs as inspiration.

You can't use this to make logos for any commercial product, and it's not safe to use it for hobby projects either, based on the current model license.

You also cannot just ingest people's media to train a for-profit AI image generation service, and well, here we are.

PS: not an AI apologist, just pointing the irony. Feels like those fan sonic characters "original content do not steal".

Post reply on HN