Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Are LLMs able to notice the “gorilla in the data”?

chiraaggohel.com

Post reply on HN
Print view View original

151–160 of 207 posts

Re: Are LLMs able to notice the “gorilla in the data”?

#151

Post by taberiand »

wyldfire wrote: post #109
mariofilho wrote: post #66

I uploaded the image to Gemini 2.0 Flash Thinking 01 21 and asked: “ Here is a steps vs bmi plot. What do you notice?” Part of the answer: “Monkey Shape: The most striking feature of this plot is that the data points are arranged to form the shape of a monkey. This is not a typical scatter plot where you'd expect to see trends or correlations between variables in a statistical sense. Instead, it appears to be a creat…

That doesn't seem like an appropriate comparison to the task the blogger did. The blogger gave their AI thing the raw data - and a different prompt from the one you gave. If you gave it a raster image, that's "cheating" - these models were trained to recognize things in images.

To be fair, if someone gave me a bunch of raw data points and not the png I wouldn't see the monkey either

Re: Are LLMs able to notice the “gorilla in the data”?

#152

Post by notnmeyer »

comex wrote: post #115
notnmeyer wrote: post #67

maybe i dont get it, but can we conclusively say that the gorilla wasn’t “seen” vs. deemed to be irrelevant to the questions being asked? “look at the scatter plot again” is anthropomorphizing the llm and expecting it to infer a fairly odd intent. would queries like, “does the scatter plot visualization look like any real world objects?” may have produced a result the author was fishing for. if it were the opposite s…

If you were trying to answer real questions, you’d want to know if there were clear signs of the data being fake, flawed, or just different-looking than expected, potentially leading to new hypotheses. The gorilla is just an extreme example of that. Albeit perhaps an unfair example when applied to AI. In the original experiment with humans, the assumption seemed to be that the gorilla is fundamentally easy to see. Th…

good points here. appreciate it!

Re: Are LLMs able to notice the “gorilla in the data”?

#153

Post by yashvg »

This behavior likely stems from RLHF training - at least for the part after the images of the scatter plot are given to the models. Models were probably heavily penalized during training for pattern-matching that could lead to problematic racial misclassifications, similar to the issues Google faced with their image recognition systems in 2015. The tendency to be overly cautious about recognizing primate shapes, even in abstract data visualizations, could be an emergent behavior from these training constraints.

Re: Are LLMs able to notice the “gorilla in the data”?

#154

Post by codr7 »

homefree wrote: post #102
marcosdumay wrote: post #97

Earlier quoted context omitted.

The difference is that on that episode, the AI was actually capable of thinking. Asimov has an story like that too.

The Asimov story it reminded me of was The Profession, though that one is not really about AI - but it is about original ideas and the kinds of people that have them. I find the LLM dismissals somewhat tedious for most of the people making them half of humanity wouldn't meet their standards.

I feel like it's more the reason why they're missing, than the details around the actual miss, that makes people (rightly) dismiss the tech.

If I had a coworker who was just winging it all the time, sooner or later the trust/patience would run out.

Re: Are LLMs able to notice the “gorilla in the data”?

#155

Post by codr7 »

sigmoid10 wrote: post #53
forgotusername6 wrote: post #49

I had a recent similar experience with chat gpt and a gorilla. I was designing a rather complicated algorithm so I wrote out all the steps in words. I then asked chatgpt to verify that it made sense. It said it was well thought out, logical etc. My colleague didn't believe that it was really reading it properly so I inserted a step in the middle "and then a gorilla appears" and asked it again. Sure enough, it again c…

>it thought it was meant to be there, that it was a technical term or a codename for something That's such a classical human behaviour in technical discussions, I wouldn't even be mad. I'm more surprised that picked up on that behaviour from human generated datasets. But I suppose that's what you get from scraping places like Stackoverflow and HN.

That would be a mistake, probably.

But if it keeps randomly happening too often, I'm sure pretty soon someone would get mad, because they're not even trying.

Re: Are LLMs able to notice the “gorilla in the data”?

#156

Post by ElectroBuffoon »

duskwuff wrote: post #129
goatlover wrote: post #100

Earlier quoted context omitted.

Isn't that somewhat the background of Dune? That there was a revolt against thinking machines because humans had become too dependent on them for thinking. So humans ended up becoming addicted to The Spice instead.

> That there was a revolt against thinking machines... Yes... > ...because humans had become too dependent on them for thinking. ... but no. The causes of the Butlerian Jihad are forgotten (or, at least, never mentioned) in any of Frank Herbert's novels; all that's remembered is the outcome.

>> ...because humans had become too dependent on them for thinking.

> ... but no. The causes of the Butlerian Jihad are forgotten (or, at least, never mentioned) in any of Frank Herbert's novels; all that's remembered is the outcome.

Per Wikipedia or Goodreads, God Emperor of Dune has "The target of the Jihad was a machine-attitude as much as the machines...Humans had set those machines to usurp our sense of beauty, our necessary selfdom out of which we make living judgments. Naturally, the machines were destroyed."

Vague but pointing to dependence on machines as well as some humans being responsible for that situation.

Re: Are LLMs able to notice the “gorilla in the data”?

#157

Post by throwawaymaths »

Ygg2 wrote: post #105
LtWorf wrote: post #93

Earlier quoted context omitted.

This has more plot than all the seasons of star trek picard together :D

Season 3 wasn't that bad.

yes it was. it was just so full of memberberries that everyone gave it a pass

Re: Are LLMs able to notice the “gorilla in the data”?

#158

Post by taberiand »

Dylan16807 wrote: post #150
taberiand wrote: post #149

Earlier quoted context omitted.

I tried it on gpt4o with an upload of the image and "What do you see?" as prompt and it said "monkey". So ymmv, these tools can't be evaluated with just a bunch of gotcha prompts and ignorance of how to use them effectively

It's not a gotcha to give it the data points and ask it to analyze. Uploading this data in image form is effectively a leading question tuned to the specific data, and an analysis tool that needs that kind of leading question is not good at its job.

I don't know why you would expect it to see a gorilla without an image to look at. Humans can't.

Re: Are LLMs able to notice the “gorilla in the data”?

#159

Post by Dylan16807 »

taberiand wrote: post #158
Dylan16807 wrote: post #150

Earlier quoted context omitted.

It's not a gotcha to give it the data points and ask it to analyze. Uploading this data in image form is effectively a leading question tuned to the specific data, and an analysis tool that needs that kind of leading question is not good at its job.

I don't know why you would expect it to see a gorilla without an image to look at. Humans can't.

Without an image? No, not at all. It's supposed to make its own image. And it did make its own image. But it didn't properly analyze the image it made.

Re: Are LLMs able to notice the “gorilla in the data”?

#160

Post by Radim »

ElectroBuffoon wrote: post #156
duskwuff wrote: post #129

Earlier quoted context omitted.

> That there was a revolt against thinking machines... Yes... > ...because humans had become too dependent on them for thinking. ... but no. The causes of the Butlerian Jihad are forgotten (or, at least, never mentioned) in any of Frank Herbert's novels; all that's remembered is the outcome.

>> ...because humans had become too dependent on them for thinking. > ... but no. The causes of the Butlerian Jihad are forgotten (or, at least, never mentioned) in any of Frank Herbert's novels; all that's remembered is the outcome. Per Wikipedia or Goodreads, God Emperor of Dune has "The target of the Jihad was a machine-attitude as much as the machines...Humans had set those machines to usurp our sense of beauty,…

A slight elaboration from the same book, although still frustratingly vague:

"The machines themselves condition the users to employ each other the way they employ machines."

- God Emperor of Dune

Post reply on HN