I asked chatgpt (pro) why it thought that it missed it sometimes and not others, and it said when it's presented with a user input it takes time to decide it's approach, sometimes more "teacherly” sometimes more “investigative", if it took the investigative approach, it would read the code line by line, if it took a teacherly approach, it would treat it as a statistical interpretation exercise.
Are LLMs able to notice the “gorilla in the data”?
101–110 of 207 posts
Re: Are LLMs able to notice the “gorilla in the data”?
#102Earlier quoted context omitted.
There actually is an episode of TNG similar to that. The society stopped being able to think for themselves, because the AI did all their thinking for them. Anything the AI didn’t know how to do, they didn’t know how to do. It was in season 1 or season 2.
The difference is that on that episode, the AI was actually capable of thinking. Asimov has an story like that too.
The Asimov story it reminded me of was The Profession, though that one is not really about AI - but it is about original ideas and the kinds of people that have them.
I find the LLM dismissals somewhat tedious for most of the people making them half of humanity wouldn't meet their standards.
Re: Are LLMs able to notice the “gorilla in the data”?
#103Re: Are LLMs able to notice the “gorilla in the data”?
#104Someday, an LLM will send every human an 'obvious' pattern (maybe a weird protein or something like that) and we'll all fail to notice and that day Skynet decides it no longer has a use for us
Re: Are LLMs able to notice the “gorilla in the data”?
#105Earlier quoted context omitted.
Just imagining an episode of Star Trek where the inhabitants of a planet have been failing to progress in warp drive tech for several generations. The team beams down to discover that society's tech stopped progressing when they became addicted to pentesting their LLM for intelligence, only to then immediately patch the LLM in order to pass each particular pentest that it failed. Now the society's time and energy has…
This has more plot than all the seasons of star trek picard together :D
Season 3 wasn't that bad.
Re: Are LLMs able to notice the “gorilla in the data”?
#106I had a recent similar experience with chat gpt and a gorilla. I was designing a rather complicated algorithm so I wrote out all the steps in words. I then asked chatgpt to verify that it made sense. It said it was well thought out, logical etc. My colleague didn't believe that it was really reading it properly so I inserted a step in the middle "and then a gorilla appears" and asked it again. Sure enough, it again c…
Not to troubleshoot but unless you visually inspected the context that was provided to the model it is quite possible it never even had your change pulled in.
Lots of front ends will do tricks like partially loading the file or using a cached version or some other behavior. Plus if you presented the file to the same “thread” it is possible it got confused about which to look at.
These front ends do a pretty lousy job of communicating to you, the end user, precisely what they are pulling into the models context window at any given time. And what the model sees as its full context window might change during the conversation as the “front end” makes edits to part portions of the same session (like dropping large files it pulled in earlier that it determines aren’t relevant somehow).
In short what you see might not be what the model is seeing at all, thus it not returning the results you expect. Every front end plays games with the context it provides to the model in order to reduce token counts and improve model performance (however “performance gets defined and measured by the designers)
That all being said it’s also completely possible it missed the gorilla in the middle… so who really knows eh?
Re: Are LLMs able to notice the “gorilla in the data”?
#107How do LLMs calculate statistic metrics like average or standard deviation accurately in such experiments?
Re: Are LLMs able to notice the “gorilla in the data”?
#108I had a recent similar experience with chat gpt and a gorilla. I was designing a rather complicated algorithm so I wrote out all the steps in words. I then asked chatgpt to verify that it made sense. It said it was well thought out, logical etc. My colleague didn't believe that it was really reading it properly so I inserted a step in the middle "and then a gorilla appears" and asked it again. Sure enough, it again c…
« saying that it thought »
This is, quite literally, not something that ChatGPT has the capability to do -- reporting on its thought process, that is. This is a hallucination.
Re: Are LLMs able to notice the “gorilla in the data”?
#109I uploaded the image to Gemini 2.0 Flash Thinking 01 21 and asked: “ Here is a steps vs bmi plot. What do you notice?” Part of the answer: “Monkey Shape: The most striking feature of this plot is that the data points are arranged to form the shape of a monkey. This is not a typical scatter plot where you'd expect to see trends or correlations between variables in a statistical sense. Instead, it appears to be a creat…
That doesn't seem like an appropriate comparison to the task the blogger did. The blogger gave their AI thing the raw data - and a different prompt from the one you gave. If you gave it a raster image, that's "cheating" - these models were trained to recognize things in images.
Re: Are LLMs able to notice the “gorilla in the data”?
#110Earlier quoted context omitted.
Just imagining an episode of Star Trek where the inhabitants of a planet have been failing to progress in warp drive tech for several generations. The team beams down to discover that society's tech stopped progressing when they became addicted to pentesting their LLM for intelligence, only to then immediately patch the LLM in order to pass each particular pentest that it failed. Now the society's time and energy has…
That's actually a moderately decent pitch for an episode.
Strange New Worlds might still accept fan scripts. Star Trek used to be famous for their willingness to produce episodes based on mailed in scripts.