Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SuperPrompt: Better Text to Image Prompts in 77M Parameters

brianfitzgerald.xyz

Post reply on HN
Print view View original

11–20 of 33 posts

Re: SuperPrompt: Better Text to Image Prompts in 77M Parameters

#11

Post by roborovskis »

pstorm wrote: post #3

I'm surprised this isn't getting more love. I love the concept of finetuned, hyper-specific, tiny LLMs. Of course, the data is the most important part.

Thanks for the kind words! I started with the 780M param flan-t5-large model, and kept trying smaller and smaller base models - I was shocked at how good the output was at 77M. As you go smaller, though, it's much easier to accidentally overfit or collapse the model and produce gibberish. Had to be very careful with hyperparams and sanitizing / filtering the dataset.

Re: SuperPrompt: Better Text to Image Prompts in 77M Parameters

#13

Post by roborovskis »

lionkor wrote: post #12

> Left: Drawbench prompt "A rainbow penguin in a tuxedo". Right: SDXL output with SuperPrompt applied to the same input prompt. Neither is wearing a tuxedo.

Yup, the model will still forget details sometimes. This is a common issue with prompt upsampling methods, but I'm hoping to improve this with the next version.

Re: SuperPrompt: Better Text to Image Prompts in 77M Parameters

#14

Post by Lerc »

Is the lack of training data the only thing preventing this approach from being applied to both positive and negative prompts together?

What size data set is actually needed? Does it need to be machine generated or can you get away with something smaller, perhaps crowdsourced?

Re: SuperPrompt: Better Text to Image Prompts in 77M Parameters

#18

Post by roborovskis »

Lerc wrote: post #14

Is the lack of training data the only thing preventing this approach from being applied to both positive and negative prompts together? What size data set is actually needed? Does it need to be machine generated or can you get away with something smaller, perhaps crowdsourced?

You could definitely use this for upsampling negative prompts, though I haven't tested that much. In theory, future T2I models shouldn't need to be negatively prompted as much; I find it's better to focus on really high quality positive prompts, as that is closer to the captions the model was trained on.

You can take a look at the dataset here: https://huggingface.co/datasets/roborovski/upsampled-prompts... Roughly 5k samples were needed for the smaller ones at a minimum, filtered from the 95k total generated.

Re: SuperPrompt: Better Text to Image Prompts in 77M Parameters

#19

Post by ultrasaurus »

I was reading a blog today[1] that was pretty confident that "continual orders-of-magnitude increases in compute usage [by AI] will utterly drown any changes in efficiency" but this is just one of a million ways we can make AI more efficient. It doesn't seem like a foregone conclusion that the costs will get order-of-magnitudeS more expensive on every axis.

1: Paywalled: https://www.noahpinion.blog/p/three-threats-to-the-age-of-en...

Post reply on HN