Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Img2Prompt – Get prompts from stable diffusion generated images

img2prompt.io

Post reply on HN
Print view View original

21–30 of 78 posts

Re: Img2Prompt – Get prompts from stable diffusion generated images

#21

Post by jabiko »

OscarCunningham wrote: post #3

So I tried it with an image of a monkey that I often use for profile pictures ( https://mathstodon.xyz/@OscarCunningham ). This image wasn't made by Stable Diffusion. It gave me this prompt: > a monkey plushie on a white background, photograph taken by steve buscemi from a zoom lens, studio lighting, ultrarealistic Can someone tell me what Steve Buscemi is doing here?

Maybe it was trained on a image set of politicians. I put in a image Dr. Evil doing air quotes and it came up with "FirstName LastName from FirstName LastName in star trek the next generation ( 2005 ) ( 2 0 1 9 )".

FirstName LastName being the name of a politician.

Re: Img2Prompt – Get prompts from stable diffusion generated images

#22

Post by FeepingCreature »

batterseapower wrote: post #12

Related - here's a fun short sci-fi story about a savant ("the prompt whisperer") who is able to intuit the prompt that was used to generate things: https://interconnected.org/home/2022/08/03/whisperer

Hee, cute, but predictable once the ... showed up. Honestly I think the ending was unnecessary; it destroys all subtlety.

Of course, this is not actually how latent space works. It's the AI's understanding of concepts, not the inherent nature of concepts; that's why every model has its own version of "latent space". Though the understanding of latent space in the story is internally consistent; given a superintelligent image generator, you could do prompt engineering like this.

Re: Img2Prompt – Get prompts from stable diffusion generated images

#23

Post by sahil_chaudhary »

Der_Einzige wrote: post #10

How is this different from image captioning when the model used is a booru model? That's already a thing people do with making their training data for fine tuning these models.

It actually works on top of an image captioning model, SD takes in keywords as well like "artstation" and "octane render" which are not covered in standard captioning so that is why the difference between using an off-the-shelf captioning model vs this

Re: Img2Prompt – Get prompts from stable diffusion generated images

#24

Post by sahil_chaudhary »

totetsu wrote: post #8

interesting.. but my guess is it's using a big library of generated image and prompt pairs? So all its suggested prompts are right out of someones 'stable diffusion prompt cheatsheet.pdf' . That is to say overly outputting commonly known artists, and things like 'trending on deviant art'

It works by using an image-captioning model finetuned on SD prompts, so it may be outputting common known artists based on their occurrence in the training data

Re: Img2Prompt – Get prompts from stable diffusion generated images

#25

Post by sahil_chaudhary »

speedgoose wrote: post #13

Isn’t it just CLIP ? The model that made these image generation models possible. It’s good to describe a picture but it’s not reverse engineering. The predicted prompt usually has very little in common with the actual prompt. And it’s worse when you use embeddings or fine tuned models.

It is based on an image-captioning model, so a different approach then CLIP interrogator, though you are correct that aim is not to get the exact prompt back but actually get a prompt to generate similar styles of images

Re: Img2Prompt – Get prompts from stable diffusion generated images

#26

Post by sahil_chaudhary »

Sophira wrote: post #16
speedgoose wrote: post #13

Isn’t it just CLIP ? The model that made these image generation models possible. It’s good to describe a picture but it’s not reverse engineering. The predicted prompt usually has very little in common with the actual prompt. And it’s worse when you use embeddings or fine tuned models.

What's interesting to me is that it even tries to predict the prompt on images that came straight from Stable Diffusion with no editing - which is weird because such images actually do have the prompt embedded inside of them already. (At least, that's the case for me - the prompt and parameters are stored in a tEXt chunk in the PNG file, which can be read with, for example, "pngcheck -t".)

True, images generated through some UIs have prompts in meta data, aim here is to work on images people find online with no metadata. So it doesn't try to read the metadata but actually predict a similar prompt

Re: Img2Prompt – Get prompts from stable diffusion generated images

#28

Post by sahil_chaudhary »

hnarayanan wrote: post #15

Ok, this is very cool! I took one image from a series I generated, had it guess a prompt (very different from mine, but it doesn't matter) and have it regenerate an another image that captures the same feeling: https://imgur.com/a/Jz0mBej

Nice, that's the aim, not to get the actual prompt back but get a prompt which can generate same style of images

Re: Img2Prompt – Get prompts from stable diffusion generated images

#29

Post by sahil_chaudhary »

bheadmaster wrote: post #19

I usually have problems coming up with prompts to generate the kind of images I want. This tool is useful for reverse-engineering prompts from the kind of images I want, then generating new ones in the same style. Very cool.

Glad you like it

Re: Img2Prompt – Get prompts from stable diffusion generated images

#30

Post by sahil_chaudhary »

rf15 wrote: post #6
OscarCunningham wrote: post #3

So I tried it with an image of a monkey that I often use for profile pictures ( https://mathstodon.xyz/@OscarCunningham ). This image wasn't made by Stable Diffusion. It gave me this prompt: > a monkey plushie on a white background, photograph taken by steve buscemi from a zoom lens, studio lighting, ultrarealistic Can someone tell me what Steve Buscemi is doing here?

This sounds a lot like the results you get from CLIP interrogation. Maybe they just use that and made another online service for it?

It is actually based on a different approach, it uses an image-captioning model finetuned on image-prompt pairs
Post reply on HN