Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DeepFloyd IF: open-source text-to-image model

github.com

Post reply on HN
Print view View original

21–30 of 237 posts

Re: DeepFloyd IF: open-source text-to-image model

#21

Post by thewataccount »

hunkins wrote: post #19

New restriction in their License suggests the software can't be modified. "2. All persons obtaining a copy or substantial portion of the Software, a modified version of the Software (or substantial portion thereof), or a derivative work based upon this Software (or substantial portion thereof) must not delete, remove, disable, diminish, or circumvent any inference filters or inference filter mechanisms in the Softwar…

> New restriction in their License suggests the software can't be modified.

It can be modified. That just says it can't be modified to bypass their filters.

Re: DeepFloyd IF: open-source text-to-image model

#22

Post by GaggiX »

hunkins wrote: post #19

New restriction in their License suggests the software can't be modified. "2. All persons obtaining a copy or substantial portion of the Software, a modified version of the Software (or substantial portion thereof), or a derivative work based upon this Software (or substantial portion thereof) must not delete, remove, disable, diminish, or circumvent any inference filters or inference filter mechanisms in the Softwar…

>New restriction in their License suggests the software can't be modified.

To remove filters.

"Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:"

Re: DeepFloyd IF: open-source text-to-image model

#23

Post by SekstiNi »

thewataccount wrote: post #13
zimpenfish wrote: post #4

16GB VRAM minimum is a bit steep. Sadly excludes my 3080 which is annoying because I'd like something better than Stable Diffusion locally.

Once these are quantized (I assume they can be), they should be ~1/4th the size. Can anyone explain why it needs so much ram in the first place though? 4.3B is only ~9GB at 16bit (I'm not as familiar with image models). I'm really happy to see that fits under 24GB - that's what I consider the limit for being able to run on "consumer hardware".

They took down the blogpost, but from what I remember the model is composite and consists of a text encoder as well as 3 "stages":

1. (11B) T5-XXL text encoder [1]

2. (4.3B) Stage 1 UNet

3. (1.3B) Stage 2 upscaler (64x64 -> 256x256)

4. (?B) Stage 3 upscaler (256x256 -> 1024x1024)

Resolution numbers could be off though. Also the third stage can apparently use the existing stable diffusion x4, or a new upscaler that they aren't releasing yet (ever?).

> Once these are quantized (I assume they can be)

Based on the success of LLaMA 4bit quantization, I believe the text encoder could be. As for the other modules, I'm not sure.

edit: the text encoder is 11B, not 4.5B as I initially wrote.

[1]: https://huggingface.co/google/t5-v1_1-xxl

Re: DeepFloyd IF: open-source text-to-image model

#24

Post by flangola7 »

GaggiX wrote: post #10
55555 wrote: post #7

So this one can create perfect text in images? If true, that’s insane

LDM-400M was already able to generate text (predecessor of Stable Diffusion), thanks to the fact that every token in the text encoder (trained from scratch) was available in the attention layer.

>thanks to the fact that every token in the text encoder (trained from scratch) was available in the attention layer.

>ChatGPT explain this like I'm 5

Re: DeepFloyd IF: open-source text-to-image model

#25

Post by [deleted] »

thewataccount wrote: post #21
hunkins wrote: post #19

New restriction in their License suggests the software can't be modified. "2. All persons obtaining a copy or substantial portion of the Software, a modified version of the Software (or substantial portion thereof), or a derivative work based upon this Software (or substantial portion thereof) must not delete, remove, disable, diminish, or circumvent any inference filters or inference filter mechanisms in the Softwar…

> New restriction in their License suggests the software can't be modified. It can be modified. That just says it can't be modified to bypass their filters.

[deleted]

Re: DeepFloyd IF: open-source text-to-image model

#26

Post by oh_sigh »

hunkins wrote: post #19

New restriction in their License suggests the software can't be modified. "2. All persons obtaining a copy or substantial portion of the Software, a modified version of the Software (or substantial portion thereof), or a derivative work based upon this Software (or substantial portion thereof) must not delete, remove, disable, diminish, or circumvent any inference filters or inference filter mechanisms in the Softwar…

You can't remove the filters per the license, but the weights will be available soon and so anyone can just reimplement this code using the weights

Re: DeepFloyd IF: open-source text-to-image model

#27

Post by ilaksh »

orra wrote: post #11

Neither the source code nor the weights are open source... This is actually worse than Stability AI's previous offering, in that regard.

They are technically open source. It's just that the model license prohibits commercial use and the code license prohibits bypassing the filters. So it's kind of worse than closed source in a way because it's like a tease. With no API apparently.

Theoretically large companies or rich people might be able to make a licensing agreement.

Re: DeepFloyd IF: open-source text-to-image model

#28

Post by RobotToaster »

hunkins wrote: post #19

New restriction in their License suggests the software can't be modified. "2. All persons obtaining a copy or substantial portion of the Software, a modified version of the Software (or substantial portion thereof), or a derivative work based upon this Software (or substantial portion thereof) must not delete, remove, disable, diminish, or circumvent any inference filters or inference filter mechanisms in the Softwar…

Then by definition it isn't open source, violating points 3, 4, and 6 of the open source definition. https://opensource.org/osd/

Re: DeepFloyd IF: open-source text-to-image model

#29

Post by Taek »

For anyone who doesn't know, DeepFloyd is a StableDiffusion style image model that more or less replaced CLIP with a full LLM (11b params). The result is that it is much better at responding to more complex prompts.

In theory, it is also smarter at learning from its training data.

Re: DeepFloyd IF: open-source text-to-image model

#30

Post by rgbrgb »

ilaksh wrote: post #27
orra wrote: post #11

Neither the source code nor the weights are open source... This is actually worse than Stability AI's previous offering, in that regard.

They are technically open source. It's just that the model license prohibits commercial use and the code license prohibits bypassing the filters. So it's kind of worse than closed source in a way because it's like a tease. With no API apparently. Theoretically large companies or rich people might be able to make a licensing agreement.

> model license prohibits commercial use

I thought that at first, but I think it only prohibits commercial use that breaks regional copyright or privacy laws.

Post reply on HN