The basic idea is to use diffusion classifiers to caption the image to generate descriptive text and append the prompt.
The work part is getting the ensemble right since you'll need to use a general classifier, like BLIP, to identify say a bunch of text from a plant and then, in this example, use structured OCR and pl@ntnet to get more specific.
But it's not that hard - maybe a dozen models. The prompt context can help as well. Then you combine the output with qualifiers in a hierarchy with respect to the model pipeline and swap the text into the prompt
Using examples from the article, here's a PoC framework to prove it works
"[I have] (photo description) (prompt)"
---
Working Examples
---
- Plant:
Here's the flower photo from TFA: https://9ol.es/tmp/lily.jpg
Go to https://identify.plantnet.org/ and upload it. It hits "Spathiphyllum wallisii Regel/Peace lily" with extremely high confidence.
We got a match cropping a screenshot of a thumbnail!
Let's say you didn't have the word "plant" in the prompt. You can fall back on a universal image classifier, such as the diffusor based BLIP here: https://huggingface.co/Salesforce/blip-image-captioning-base (uploader is on the right)
Upload the same image. You'll get "a plant in a white pot" which then, because we use feed-forward networks these days, will lead you to pl@ntnet and you'll get the peace lily again.
Using our framework, ask GPT 3.5 " I have a Spathiphyllum wallisii Regel/Peace lily. What is that plant and how should I care for it?"
And you get a nearly identical reply to the one in the article.
- Penny:
Upload the penny image (from https://en.wikipedia.org/wiki/Penny_(United_States_coin)) to the BLIP classifier and you get "a penny coin with the face of abraham"
Let's go back to GPT 3.5 and use our format from above,
"I have a penny coin with the face of abraham. What coin is that?"
And of course you get: "A penny coin with the face of Abraham Lincoln is most likely a United States one-cent coin, commonly known as a "Lincoln penny"..."
And there we go. For a full FLOSS stack, you can ask llama2 70b https://stablediffusion.fr/llama2 and get "The face of Abraham Lincoln is featured on the United States one-cent coin, commonly known as the penny."
more complex photos:
You can use Facebooks SAM (segment anything) https://segment-anything.com/ to break up the image, BLIP caption the segments, then forward off to the specialized classifiers.
It's a fairly intensive pipeline that requires lots of modern hardware and requires you to have familiarity with a wide variety of models, then tweak them, test it, have some GANs maybe set up for refinement ... but this is well within reach of non-geniuses. I'm merely average on a good day and even I can see how to set this up.
They might be using a different approach but using SAM, BLIP and a few specialized classifiers covers all the examples in the articles without using any human discretion. For instance, the city one is way more powerful if they're using something like this: https://static.googleusercontent.com/media/research.google.c...
I'm trying to justify why bother cloning it. Maybe to have a free alternative? It's a bit of work but it's not new magic.