Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Are LLMs able to notice the “gorilla in the data”?

chiraaggohel.com

Post reply on HN
Print view View original

61–70 of 207 posts

Re: Are LLMs able to notice the “gorilla in the data”?

#61

Post by hammock »

wodenokoto wrote: post #40

I love that gorilla test. Happens in my team all the time, that people start with the assumption that the data is “good” and then deep dive. Is there a blog post that just focus on the gorilla test that I can share with my team? I’m not even interested in the LLM part

Same here. Can’t count the number of times I’ve had to come in and say “hold on, you built an entire report with conclusions and recommendations but didn’t stop to say hmm this data looks weird and dig into validation?” “We assumed the data was right and that it must be xyz…”

A corollary if this that is my personal pet peeve is attributing everything you can’t explain to “seasonality” , that is such a crutch. If you can’t explain it then just say that. There is a better than not chance it is noise anyway.

Re: Are LLMs able to notice the “gorilla in the data”?

#62

Post by catlifeonmars »

Retr0id wrote: post #60
catlifeonmars wrote: post #55

Earlier quoted context omitted.

I think that’s part of the authors point. The article starts out by explaining a human phenomenon and then extending it to LLMs

Humans only failed to spot it when prompted in a way that was misdirective, though.

True. What I think is missing (and probably the more interesting question) is an analysis on _why_ LLMs failed to spot it. I imagine it has something to do with the model architecture.

Re: Are LLMs able to notice the “gorilla in the data”?

#66

Post by mariofilho »

I uploaded the image to Gemini 2.0 Flash Thinking 01 21 and asked:

“ Here is a steps vs bmi plot. What do you notice?”

Part of the answer:

“Monkey Shape: The most striking feature of this plot is that the data points are arranged to form the shape of a monkey. This is not a typical scatter plot where you'd expect to see trends or correlations between variables in a statistical sense. Instead, it appears to be a creative visualization where data points are placed to create an image.”

Gemini 2.0 Pro without thinking didn’t see the monkey

Re: Are LLMs able to notice the “gorilla in the data”?

#67

Post by notnmeyer »

maybe i dont get it, but can we conclusively say that the gorilla wasn’t “seen” vs. deemed to be irrelevant to the questions being asked?

“look at the scatter plot again” is anthropomorphizing the llm and expecting it to infer a fairly odd intent.

would queries like, “does the scatter plot visualization look like any real world objects?” may have produced a result the author was fishing for.

if it were the opposite situation and you were trying to answer “real” questions and the llm was suggesting, “the data is visualized looks like notorious big” we’d all be here laughing at a different post about the dumb llm.

Re: Are LLMs able to notice the “gorilla in the data”?

#68

Post by izackp »

forgotusername6 wrote: post #49

I had a recent similar experience with chat gpt and a gorilla. I was designing a rather complicated algorithm so I wrote out all the steps in words. I then asked chatgpt to verify that it made sense. It said it was well thought out, logical etc. My colleague didn't believe that it was really reading it properly so I inserted a step in the middle "and then a gorilla appears" and asked it again. Sure enough, it again c…

I typically tell it that there at 5 problems in the logic. Summarize the steps, why it’s necessary, and what typically comes after that step. Then please list and explain all five errors.

Re: Are LLMs able to notice the “gorilla in the data”?

#69

Post by martinsnow »

mariofilho wrote: post #66

I uploaded the image to Gemini 2.0 Flash Thinking 01 21 and asked: “ Here is a steps vs bmi plot. What do you notice?” Part of the answer: “Monkey Shape: The most striking feature of this plot is that the data points are arranged to form the shape of a monkey. This is not a typical scatter plot where you'd expect to see trends or correlations between variables in a statistical sense. Instead, it appears to be a creat…

It thought my bald colleague was a plant in the background. So don't have high hopes for it. He did wear a headset so that is apparently very plant like.

Re: Are LLMs able to notice the “gorilla in the data”?

#70

Post by zmgsabst »

But both models did see the gorilla when prompted with it…?

ChatGPT:

> It looks like the scatter plot unintentionally formed an artistic pattern rather than a meaningful representation of the data.

Claude:

> Looking at the scatter plot more carefully, I notice something concerning: there appear to be some unlikely or potentially erroneous values in the data. Let me analyze this in more detail.

> Ah, now I see something very striking that I missed in my previous analysis - there appears to be a clear pattern in the data points that looks artificial. The data points form distinct curves and lines across the plot, which is highly unusual for what should be natural, continuous biological measurements.

Given the context of asking for quantitative analysis and their general beaten-into-submission attitude where they defer to you, eg, your assertion this is a real dataset… I’m not sure what conclusion we’re supposed to draw.

That if you lie to the AI, it’ll believe you…?

Neither was prompted that this is potentially adversarial data — and AI don’t generally infer social context very well. (A similar effect occurs with math tests.)

Post reply on HN