Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Are LLMs able to notice the “gorilla in the data”?

chiraaggohel.com

Post reply on HN
Print view View original

21–30 of 207 posts

Re: Are LLMs able to notice the “gorilla in the data”?

#21

Post by johnfn »

dartos wrote: post #15
johnfn wrote: post #5

GPT can't "see" the results of the scatterplot (unless prompted with an image), it only sees the code it wrote. If a human had the same constraints I doubt they'd identify there was a gorilla there. Take a screenshot of the scatterplot and feed it into multimodal GPT and it does a fine job at identifying it. EDIT: Sorry, as a few people pointed out, I missed the part where the author did feed a PNG into GPT. I kind o…

Before seeing Claude’s response, did you see where the author said > I asked the model to closely look at the plot, and also uploaded a png of the plot it had generated.

Hm, interesting. The way I tried it was by pasting an image into Claude directly as the start of the conversation, plus a simple prompt ("What do you see here?"). It got the specific image wrong (it thought it was baby yoda, lol), but it did understand that it was an image.

I wonder if the author got different results because they had been talking a lot about a data set before showing the image, which possibly predisposed AI to think that it was a normal data set. In any case, I think that "Your Ai Can't See Gorillas" isn't really a valid conclusion.

Re: Are LLMs able to notice the “gorilla in the data”?

#22

Post by sw1sh »

I got "The scatter plot appears to be arranged to resemble the character "Pepe the Frog," a popular internet meme ... " lol

Not sure whether multimodal embeddings have such a good pattern recognition accuracy in this case, probably most of information goes into attending to plot related features, like its labels and ticks.

Re: Are LLMs able to notice the “gorilla in the data”?

#23

Post by 4ndrewl »

badgersnake wrote: post #14
4ndrewl wrote: post #4

Earlier quoted context omitted.

You mean that whole post could have been written: "AI can't see gorilla because wokification"? /s Edit: Adding /s Thought "wokification" already signalled that.

You didn’t read it either.

I was /s commenting on the comment :) I did read it, could relate to the constant reframing of the questioner to just "look" at the graph. Like talking to a child.

Re: Are LLMs able to notice the “gorilla in the data”?

#24

Post by mapt »

luluthefirst wrote: post #18
zozbot234 wrote: post #2

That's done on purpose. The AI can't easily tell whether the drawing might be intended to be one of a human or a gorilla, so when in doubt it doesn't want to commit either way and just ignores the topic altogether. It's just another example of AI ethics influencing its behavior for alignment purposes.

It's not that at all... Similar drawings of non-humanoid shapes like an ostrich or the map of Europe would have resulted in the exact same 'blindness'.

This is not obvious to me, and nor should it be to anyone who didn't program these AIs (and it probably shouldn't be obvious even to the people who did). I think you both should try testing the hypothesis and present your results.

Re: Are LLMs able to notice the “gorilla in the data”?

#25

Post by vunderba »

johnfn wrote: post #21
dartos wrote: post #15

Earlier quoted context omitted.

Before seeing Claude’s response, did you see where the author said > I asked the model to closely look at the plot, and also uploaded a png of the plot it had generated.

Hm, interesting. The way I tried it was by pasting an image into Claude directly as the start of the conversation, plus a simple prompt ("What do you see here?"). It got the specific image wrong (it thought it was baby yoda, lol), but it did understand that it was an image. I wonder if the author got different results because they had been talking a lot about a data set before showing the image, which possibly predis…

Please read TFA. The conclusion of the article isn't nearly so simplistic, they're just suggesting that you have to be aware of the natural strengths and weaknesses of LLMs, even multi modal ones particularly around visual pattern recognition vs quantitative pattern recognition.

And yes, the idea that the initial context can sometimes predispose the LLM to consider things in a more narrow manner than a user might otherwise want is definitely well known.

Re: Are LLMs able to notice the “gorilla in the data”?

#26

Post by GaggiX »

badgersnake wrote: post #20
GaggiX wrote: post #8

If you give the graph as image to the model they will easily see the monkey: "I see a drawing of a monkey outlined with red and blue dots.", if you give them coordinates than they will much more struggle with it like a human would do.

Nope, any human when asked to plot that data would pretty quickly give up and (correctly) assume it was a wind-up.

They still don't see the gorilla anyway ahah

Re: Are LLMs able to notice the “gorilla in the data”?

#28

Post by runjake »

Only tangentially related to this story, I've been trying for months to train the YOLO models to recognize my Prussian blue cat, with its assorted white spots, as a cat rather than a dog or a person.

However, it refuses to cooperate. It's maddening.

As a result, I receive "There is a person at your front door" notifications at all hours of the night.

Re: Are LLMs able to notice the “gorilla in the data”?

#30

Post by GaggiX »

runjake wrote: post #28

Only tangentially related to this story, I've been trying for months to train the YOLO models to recognize my Prussian blue cat, with its assorted white spots, as a cat rather than a dog or a person. However, it refuses to cooperate. It's maddening. As a result, I receive "There is a person at your front door" notifications at all hours of the night.

Something is very wrong if the model cannot tell the difference between a Prussian blue cat and a person. I imagine you have inserted in training data the images of the cat from the camera and in similar quantities of a person from the same camera.
Post reply on HN