Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Are LLMs able to notice the “gorilla in the data”?

chiraaggohel.com

Post reply on HN
Print view View original

121–130 of 207 posts

Re: Are LLMs able to notice the “gorilla in the data”?

#121

Post by hammock »

8n4vidtmkvmk wrote: post #82
hammock wrote: post #61

Earlier quoted context omitted.

Same here. Can’t count the number of times I’ve had to come in and say “hold on, you built an entire report with conclusions and recommendations but didn’t stop to say hmm this data looks weird and dig into validation?” “We assumed the data was right and that it must be xyz…” A corollary if this that is my personal pet peeve is attributing everything you can’t explain to “seasonality” , that is such a crutch. If you…

> attributing everything you can’t explain to “seasonality” Is this a literal thing or figurative thing? Because it should be very easy to see the seasons if you have a few years of data. I just attribute all the data I don't like to noise :-)

Presented as a literal thing, but is really figurative. What I mean is, often you don’t have the data to actually plot/PCA/whatever technique the seasonality, maybe you only have 2 years of data for example. But it dips in October and you have no idea why so you just say “Q4 tends to be a low season for this product” or something equally dubious, with no further analysis or hope of same

Re: Are LLMs able to notice the “gorilla in the data”?

#122

Post by Dylan16807 »

appleorchard46 wrote: post #91

These posts about X task LLMs fails at when you give it Y prompt are getting more and more silly. If you ask an AI to analyze some data, should the default behavior be to use that data to make various types of graphs, export said graphs, feed them back in to itself, then analyze the shapes of those graphs to see if they resemble an animal? Personally I would be very annoyed if I actually wanted a statistical analysis…

You don't think "Examine the data" and "Which other conclusions can you draw from the data?" are open-ended?

And even when explicitly prompted to look at the plot, they only brush up against the data anomalies rather than properly analyzing the plot.

Re: Are LLMs able to notice the “gorilla in the data”?

#124

Post by s1mplicissimus »

hammock wrote: post #61
wodenokoto wrote: post #40

I love that gorilla test. Happens in my team all the time, that people start with the assumption that the data is “good” and then deep dive. Is there a blog post that just focus on the gorilla test that I can share with my team? I’m not even interested in the LLM part

Same here. Can’t count the number of times I’ve had to come in and say “hold on, you built an entire report with conclusions and recommendations but didn’t stop to say hmm this data looks weird and dig into validation?” “We assumed the data was right and that it must be xyz…” A corollary if this that is my personal pet peeve is attributing everything you can’t explain to “seasonality” , that is such a crutch. If you…

I share this experience of people often just performing the steps without thinking about the meaning behind them. In data analysis and software development.

My conclusion so far has been "well they are not doing their job properly".

I assume that's the kinds of jobs LLM's can replace: People you don't want on your payroll anyway

Re: Are LLMs able to notice the “gorilla in the data”?

#125

Post by s1mplicissimus »

_zamorano_ wrote: post #104

Someday, an LLM will send every human an 'obvious' pattern (maybe a weird protein or something like that) and we'll all fail to notice and that day Skynet decides it no longer has a use for us

Maybe. But before I start worrying about that I'll have to see them count correctly or not be fooled by variations of simple riddles

Re: Are LLMs able to notice the “gorilla in the data”?

#126

Post by appleorchard46 »

amarshall wrote: post #112
appleorchard46 wrote: post #91

These posts about X task LLMs fails at when you give it Y prompt are getting more and more silly. If you ask an AI to analyze some data, should the default behavior be to use that data to make various types of graphs, export said graphs, feed them back in to itself, then analyze the shapes of those graphs to see if they resemble an animal? Personally I would be very annoyed if I actually wanted a statistical analysis…

I think it depends if one is using “AI” as a tool or as a replacement for an intelligent expert? The former, sure, it’s maybe not expected, because the prompter is already an intelligent expert. If the latter, then yes, I think, because if you gave the task to an expect and they did not notice this, I would consider them not good at their job. See also Anscombe's quartet[1] and the Datasaurus dozen[2] (mentioned in a…

This is true, but I would replace 'intelligent expert' with 'intelligent human expert'.

Graphing data to analyze it - and then seeing shapes and creatures in said graph - is a distinctly human practice, and not an inherently necessary part of most data analysis (the obvious exception being when said data draws a picture).

I think it's because the interface uses human language that people expect AI to make the same assumptions and follow the same processes as humans. In some ways it does, in other ways it doesn't. Expecting it to be the same as a human leads to frustration and a flawed understanding of its capabilities and limits.

Re: Are LLMs able to notice the “gorilla in the data”?

#128

Post by isaacremuant »

appleorchard46 wrote: post #91

These posts about X task LLMs fails at when you give it Y prompt are getting more and more silly. If you ask an AI to analyze some data, should the default behavior be to use that data to make various types of graphs, export said graphs, feed them back in to itself, then analyze the shapes of those graphs to see if they resemble an animal? Personally I would be very annoyed if I actually wanted a statistical analysis…

I benchmark many of these things as "what would I want a human assistant to do" if they were had insta speed and noticing the pattern would definitely be warranted to determined if data could be falsely generated.

It's not silly at all.

Re: Are LLMs able to notice the “gorilla in the data”?

#129

Post by duskwuff »

goatlover wrote: post #100
sillysaurusx wrote: post #85

Earlier quoted context omitted.

There actually is an episode of TNG similar to that. The society stopped being able to think for themselves, because the AI did all their thinking for them. Anything the AI didn’t know how to do, they didn’t know how to do. It was in season 1 or season 2.

Isn't that somewhat the background of Dune? That there was a revolt against thinking machines because humans had become too dependent on them for thinking. So humans ended up becoming addicted to The Spice instead.

> That there was a revolt against thinking machines...

Yes...

> ...because humans had become too dependent on them for thinking.

... but no. The causes of the Butlerian Jihad are forgotten (or, at least, never mentioned) in any of Frank Herbert's novels; all that's remembered is the outcome.

Post reply on HN