DeepFloyd IF: open-source text-to-image model • Hacker News Board

Re: DeepFloyd IF: open-source text-to-image model

#51

Post by ronsor » Wed, Apr 26, 2023, 9:55 PM UTC

jacob019 wrote: post #42

Any web based front ends yet? I put together a system that runs a variety of web based open source AI image generation and editing tools on Vultr GPU instances. It spins up instances on demand, mounts an NFS filesystem with local caching and a COW layer, spawns the services, proxies the requests, and then spins down idle instances when I'm done. Would love to add this, suppose I could whip something up if none exists…

It'll probably be in the Auto1111 WebUI within a week.

Re: DeepFloyd IF: open-source text-to-image model

#52

Post by Taek » Wed, Apr 26, 2023, 9:59 PM UTC

atleastoptimal wrote: post #49

> Text > Hands good god it solves the two biggest meme issues with image models in one go. Will this be the new state of the art every other model is compared to?

There are good reasons to believe that this will be the new state of the art by a comfortable margin. Hard to know until we can actually play with it.

Re: DeepFloyd IF: open-source text-to-image model

#53

Post by gwern » Wed, Apr 26, 2023, 9:59 PM UTC

atleastoptimal wrote: post #49

> Text > Hands good god it solves the two biggest meme issues with image models in one go. Will this be the new state of the art every other model is compared to?

We already knew those were going to be solved by scale like using T5 instead of the really small bad text encoder SD used, because they were solved by Imagen etc.

Re: DeepFloyd IF: open-source text-to-image model

#54

Post by gwern » Wed, Apr 26, 2023, 10:03 PM UTC

SekstiNi wrote: post #23

thewataccount wrote: post #13

Earlier quoted context omitted.

Once these are quantized (I assume they can be), they should be ~1/4th the size. Can anyone explain why it needs so much ram in the first place though? 4.3B is only ~9GB at 16bit (I'm not as familiar with image models). I'm really happy to see that fits under 24GB - that's what I consider the limit for being able to run on "consumer hardware".

They took down the blogpost, but from what I remember the model is composite and consists of a text encoder as well as 3 "stages": 1. (11B) T5-XXL text encoder [1] 2. (4.3B) Stage 1 UNet 3. (1.3B) Stage 2 upscaler (64x64 -> 256x256) 4. (?B) Stage 3 upscaler (256x256 -> 1024x1024) Resolution numbers could be off though. Also the third stage can apparently use the existing stable diffusion x4, or a new upscaler that th…

You'll be able to optimize it a lot to make it fit on small systems if you are willing to modify your workflow a bit: instead of 1 prompt -> 1 image _n_ times, do 1 prompt -> _n_ images 1 time -> _m_ times... For a given prompt, run it through the T5 model and store; you can do that in CPU RAM if you have to because you only need the embedding once so you don't need a GPU which can run T5-XXL naively. Then you can get a large batch of samples from #2; 64px is enough to preview; only once you pick some do you run through #3, and then from those through #4. Your peak VRAM should be 1 image in #2 or #4 and that can be quantized or pruned down to something that will fit on many GPUs.

Re: DeepFloyd IF: open-source text-to-image model

#55

Post by jrm4 » Wed, Apr 26, 2023, 10:05 PM UTC

ilaksh wrote: post #27

orra wrote: post #11

Neither the source code nor the weights are open source... This is actually worse than Stability AI's previous offering, in that regard.

They are technically open source. It's just that the model license prohibits commercial use and the code license prohibits bypassing the filters. So it's kind of worse than closed source in a way because it's like a tease. With no API apparently. Theoretically large companies or rich people might be able to make a licensing agreement.

I am a lawyer, and as flimsy and wishy-washy as the term "open-source" already is, I can't even fathom what is meant by "open source" here?

Are people suggesting that "look at the code but don't touch" actually fits what some people think of as open source?

Re: DeepFloyd IF: open-source text-to-image model

#56

Post by dragonwriter » Wed, Apr 26, 2023, 10:09 PM UTC

Jackson__ wrote: post #45

oh_sigh wrote: post #26

Earlier quoted context omitted.

You can't remove the filters per the license, but the weights will be available soon and so anyone can just reimplement this code using the weights

There is a similar license clause for the weights[0] as well, so I'm not sure this would apply unless you write the code and train your model from scratch. [0] https://github.com/deep-floyd/IF/blob/main/LICENSE-MODEL#L54

Or unless, as seems to be fairly widely expected but untested, model weights are not actually copyrightable, so model licenses are superfluous.

Re: DeepFloyd IF: open-source text-to-image model

#57

Post by dragonwriter » Wed, Apr 26, 2023, 10:14 PM UTC

yellowapple wrote: post #34

rgbrgb wrote: post #30

Earlier quoted context omitted.

> model license prohibits commercial use I thought that at first, but I think it only prohibits commercial use that breaks regional copyright or privacy laws.

That's already prohibited by, you know, those very same copyright and privacy laws. Adding those same prohibitions to the license not only makes the software nonfree, but pointlessly does so.

Its not pointless, it means the model licensor has a claim against you, as well as whoever would for violating the referenced laws; it also means, and this is probably more important, that in some juridictions, the model licensor has a better defense against liability for contributory infringement if the licensee infringes.

EDIT: That said, it’s unambiguously not open source.

Re: DeepFloyd IF: open-source text-to-image model

#58

Post by NBJack » Wed, Apr 26, 2023, 10:21 PM UTC

TaylorAlexander wrote: post #6

zimpenfish wrote: post #4

16GB VRAM minimum is a bit steep. Sadly excludes my 3080 which is annoying because I'd like something better than Stable Diffusion locally.

If you don't mind the power consumption I noticed that older nvidia P6000's (24GB) are pretty cheap on ebay! My 16GB P5000 is pretty handy for this stuff.

An M40 24GB is less than $200, if you don't mind the trouble to get it's drivers installed, cooled, etc. It's also important to note your motherboard must support larger VRAM addressing; many older chipsets won't be able to boot with it (i.e. some, perhaps almost all, Zen 1 supporters).

Re: DeepFloyd IF: open-source text-to-image model

#59

Post by swyx » Wed, Apr 26, 2023, 10:46 PM UTC

lalaithion wrote: post #5

Has anyone tried the Scott Alexander AI bet prompts? 1. A stained glass picture of a woman in a library with a raven on her shoulder with a key in its mouth 2. An oil painting of a man in a factory looking at a cat wearing a top hat 3. A digital art picture of a child riding a llama with a bell on its tail through a desert 4. A 3D render of an astronaut in space holding a fox wearing lipstick 5. Pixel art of a farmer…

where are these prompts from?

Re: DeepFloyd IF: open-source text-to-image model

#60

Post by Mizza » Wed, Apr 26, 2023, 10:49 PM UTC

yellowapple wrote: post #36

RobotToaster wrote: post #28

Earlier quoted context omitted.

Then by definition it isn't open source, violating points 3, 4, and 6 of the open source definition. https://opensource.org/osd/

Yep. It's getting really exhausting seeing projects falsely advertising themselves as "open source". Either be FOSS or don't be; don't pretend to be while using some nonsense like the BSL or whatever adhocery is in play here.

In the README they even call it "Modified MIT", the modification being where they turned it from a very permissive license into a fully proprietary one. Very cool model though.