Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

FLUX is fast and it's open source

replicate.com

Post reply on HN
Print view View original

81–90 of 127 posts

Re: FLUX is fast and it's open source

#81

Post by sorenjan »

miki123211 wrote: post #73
sorenjan wrote: post #23

Text to image models feels inefficient to me. I wonder if it would be possible and better to do it in separate steps, like text to scene graph, scene graph to semantically segmented image, segmented image to final image. That way each step could be trained separately and be modular, and the image would be easier to edit instead of completely replace it with the output of a new prompt. That way it should be much easie…

In general, it has been shown time and time again that this approach fails for neural network based models. If you can train a neural network that goes from a to b and a network that goes from b to c, you can usually replace that combination with a simpler network that goes from a to c directly. This makes sense, as there might be information in a that we lose by a conversion to b. A single neural network will ensure…

Yes this is true, you do lose some information between the layers, and this increased expressibility is the big benefit of using ML instead of classic feature engineering. However, I think the gain would be worth it for some use cases. You could for instance take an existing image, run that through a semantic segmentation model, and then edit the underlying image description. You could add a yellow hat to a person without regenerating any other part of the image, you could edit existing text, change a person's pose, you could probably more easily convert images to 3D, etc.

It's probably not a viable idea, I just wish for more composable modules that lets us understand the models' representation better and change certain aspects of them, instead of these massive black boxes that mix all these tasks into one.

I would also like to add that the text2image models already have multiple interfaces between different parts. There's the text encoder, the latent to pixel space VAE decoder, controlnets, and sometimes there a separate img2imgstyle transfer at the end. Transformers already process images patchwise, but why does those patches have to be even square patches instead of semantically coherent areas?

Re: FLUX is fast and it's open source

#82

Post by sorenjan »

teh_infallible wrote: post #43
sorenjan wrote: post #23

Text to image models feels inefficient to me. I wonder if it would be possible and better to do it in separate steps, like text to scene graph, scene graph to semantically segmented image, segmented image to final image. That way each step could be trained separately and be modular, and the image would be easier to edit instead of completely replace it with the output of a new prompt. That way it should be much easie…

I am hoping that AI art tends towards a modular approach, where generating a character, setting, style, and camera movement each happens in its own step. It doesn’t make sense to describe everything at once and hope you like what you get.

Definitely, that would make much more sense seeing how content is produced by people. Adjust the technology to how people want to use it instead of forcing artists becoming prompt engineers and settling for something close enough what they want.

At the very least image generators should output layers, I think the style component is already possible with the img2img models.

Re: FLUX is fast and it's open source

#83

Post by sorenjan »

llm_trw wrote: post #32
ZoomZoomZoom wrote: post #28

Earlier quoted context omitted.

The issue with this is there's a false assumption that an image is a collection of objects. It's not (necessarily). I want a picture of frozen cyan peach fuzz.

https://imgur.com/ayAWSKr Prompt: frozen cyan peach fuzz, with default settings on a first generation SD model. People _seriously_ do not understand how good these tools have been for nearly two years already.

Running that image through Segment Anything you get this: https://imgur.com/a/XzCanxx

Imagine if instead of generating the RGB image directly the model would generate something like that, but with richer descriptive embeddings on each segment, and then having a separate model generating the final RGB image. Then it would be easy to change the background, rotate the peach, change color, add other fruits, etc, by editing this semantic representation of the image instead of wrestling with the prompt to try to do small changes without regenerating the entire image from scratch.

Re: FLUX is fast and it's open source

#84

Post by slig »

jsemrau wrote: post #30

My favorite thing to do with Flux is create images with a white background for my substack[1] because the text following is amazing and I can communicate something visually through the artwork as well. [1] https://substackcdn.com/image/fetch/w_1456,c_limit,f_webp,q_...

Could you share the prompt? Thanks.

Re: FLUX is fast and it's open source

#85

Post by uxhacker »

trickstra wrote: post #44

Non-commercial is not open-source, because if the original copyright holder stops maintaining it, nobody else can continue (or has to work like a slave for free). Open-source is about what happens if the original author stops working on it. Open-source gives everyone the license to continue developing it, which obviously means also the ability to get paid. Don't call it open-source if this aspect is missing. Only the…

There is OpenFLUX.1 which is a fine tune of the FLUX.1-schnell model that has had the distillation trained out of it. OpenFLUX.1 is licensed Apache 2.0. https://huggingface.co/ostris/OpenFLUX.1/

Re: FLUX is fast and it's open source

#86

Post by weird-eye-issue »

swyx wrote: post #35
weird-eye-issue wrote: post #20

Earlier quoted context omitted.

I mean it literally explains why in the second paragraph. It returns the actual file data in the response rather than a URL where you have to make a second request to get the file data

thats not "making the image models much faster", thats just making getting the image back slightly faster

In all practical senses it is the same thing

Re: FLUX is fast and it's open source

#87

Post by Kiboneu »

peterpans01 wrote: post #62
thomashop wrote: post #47

If you want to play with FLUX.schnell easily, type the prompt into a Pollinations URL: https://pollinations.ai/p/a_donkey_holding_a_sign_with_flux_... https://pollinations.ai/p/a_donkey_holding_a_sign_with_flux_... https://pollinations.ai/p/Minimalist%20and%20conceptual%20ar... It's incredible how fast it is. We generate 8000 images every 30 minutes for our users using only three L40S GPUs. Disclaimer: I'm behind Pol…

The "only" word sounds quite expensive for most of us.

He started a whole business to help pay the installments.

Re: FLUX is fast and it's open source

#88

Post by Scrapemist »

pajeets wrote: post #51
ruthmarx wrote: post #36

Earlier quoted context omitted.

That example you gave is a good reason why artists get pissed off IMO. The LLM is clearly aping some artists specific style, and now missing out on paid work as a result. Not sure I have an opinion on that, technology marches on etc, but it is interesting.

Dont care about artists opinion on rest of using AI tools instead of not paying them because I couldnt and wouldnt so theres no demand in the first place. All I wanna know is the prompt that was used to generate the art speaking of which i wanna know how to create cartoony images like that OP

[flagged]

Re: FLUX is fast and it's open source

#89

Post by nerdponx »

kqr wrote: post #40
sorenjan wrote: post #23

Text to image models feels inefficient to me. I wonder if it would be possible and better to do it in separate steps, like text to scene graph, scene graph to semantically segmented image, segmented image to final image. That way each step could be trained separately and be modular, and the image would be easier to edit instead of completely replace it with the output of a new prompt. That way it should be much easie…

Isn't this essemtially the approach to image recognition etc. that failed for ages until we brute forced it with bigger and deeper matrices? It seems sensible to extract features and reason about things the way a human would, but it turns out its easier to scale pattern matching purely done by computer.

You could imagine doing it with 2 specialized NNs, but then you have to figure out a huge labeled dataset of scene graphs. The problem fundamentally is that any "manual" feature engineering is not going to be supervised and fitted on a huge corpus, the way the self-learned features are.

Re: FLUX is fast and it's open source

#90

Post by Zambyte »

sorenjan wrote: post #23

Text to image models feels inefficient to me. I wonder if it would be possible and better to do it in separate steps, like text to scene graph, scene graph to semantically segmented image, segmented image to final image. That way each step could be trained separately and be modular, and the image would be easier to edit instead of completely replace it with the output of a new prompt. That way it should be much easie…

You seem to be describing ComfyUI to me. You can definitely do this kind of workflow with ComfyUI.
Post reply on HN