Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cargo Cult AI

queue.acm.org

Post reply on HN
Print view View original

141–150 of 191 posts

Re: Cargo Cult AI

#141

Post by TeMPOraL »

dirkt wrote: post #90
paulddraper wrote: post #85

Earlier quoted context omitted.

So....like a person?

The difference to a person is that most (though not all) people actually have an understanding if they know something, if they guess something, if they are making something up, or if they are outright lying. Which is about the first thing you train during a scientific education. And in an honest interaction, they will tell you. ChatGPT etc. does not. So basically it acts like a pathological liar (who happen to be rig…

> The difference to a person is that most (though not all) people actually have an understanding if they know something, if they guess something, if they are making something up, or if they are outright lying. Which is about the first thing you train during a scientific education.

Scientist? STEM workers? Maybe. People who value truth / consistent world model for its own sake? Sure.

Normies? Not so much. It's not that they can't - I think they never learned to pay enough attention. What I mean is, most people tend to say things in confidence, and maybe even believe them, regardless of how they acquired the information. They don't seem to process the distinction between "I read it in a book", "A colleague told me that their colleague heard on the radio that...", etc. They don't even track provenance of the information, which is a critical skill you need to not end up confidently making things up.

It's a learnable skill, and I think it's even quite easy to pick it up from osmosis - but someone has to make the person feel that it's important.

> So basically it acts like a pathological liar

I think it's more of a bullshitter than a liar, in the sense that it doesn't care about truth value of what it says.

Lying involve knowing the truth, or at least knowing that the thing you're saying ain't it. Making a mistake involves you thinking you're saying the truth, but actually being wrong about it. Bullshitting is just saying whatever helps you achieve your goal; the truth value of what you say doesn't even enter the picture.

Re: Cargo Cult AI

#142

Post by albertzeyer »

kerkeslager wrote: post #93
albertzeyer wrote: post #82

Earlier quoted context omitted.

> the idea that larger models can do more than smaller models, which should be a surprise to no one. Actually this was quite a surprise to a lot of people, since the whole race to scaling up began, with GPT2 or so. It was totally not obvious that you can scale up the model (and also training) and it would improve the performance. Many (most?) people thought there would be some limit, and we were close to that limit w…

You didn't read the post you're responding to, and shouldn't be responding. In fact, you didn't read the part of the post you quoted, where I said it should be a surprise to no one. But, unsurprisingly, the sort of people who stop reading at the first chance they see to correct something, are easily surprised, since actually understanding LLMs would require actually doing some nuanced reading.

I think you are misunderstanding sth. I did read your post. I'm also publishing peer-reviewed research articles related to this. I think I have some good understanding on this.

I was simply saying that I partly disagree with you. And I still do. It's wrong that this should be a surprise to no-one. In fact, I think it is reasonable that it is surprising. It was indeed really unexpected that scaling up such models leads to such behavior.

Now that we have such models, and see this behavior, sure you can say in hindsight, of course it's obvious, nothing unexpected. But this is wrong. It was unexpected to many people.

And saying "it should not have been unexpected", I'm not really sure what you want to say with that. Yes, it would be nice if everyone's prediction are always correct. Obviously that's not the case. Or you are saying you think this is a particular trivial case. I would disagree.

English is not my native language. Maybe I just understood sth wrong.

Re: Cargo Cult AI

#143

Post by TeMPOraL »

Jack000 wrote: post #74

There’s too much focus on AGI. Language models do not emulate human minds - they are models of language. The emergent behavior from these models are only a side effect of their main training task, which is to build a model of all meaningful sequences of words. We then use RFHL to bias the model toward a small area of the language latent space which conforms to our idea of intelligent behavior. Humans (a GI) have zero…

> Humans (a GI) have zero ability to do language modeling.

But perhaps they have a component that has this ability.

I maintain that LLMs are best compared not to the entire human mind/intelligence, but rather to the "inner voice" - that bit that sits between conscious and unconscious, having a part on each site, and uses natural language as an interface to the conscious side.

I.e. imagine someone hooked up electrodes to your brain and was able to eavesdrop on the thoughts that you consciously notice, and which are expressed in natural language. If they had the device print those thoughts out as it "hears" them, I think the output - and changes to it in response to what's going on in and around you - would quite resemble the way LLMs respond to prompts.

Re: Cargo Cult AI

#144

Post by mustacheemperor »

sebzim4500 wrote: post #111
tjr wrote: post #92

Earlier quoted context omitted.

I wonder if it's "fixed" or if it's just less obvious. It seems that an LLM would "hallucinate" a bogus answer if it didn't actually have a good answer somewhere in its training. Is GPT4 so much more trained that it rarely encounters something it doesn't have a reasonable answer for? In which case, it would still "hallucinate" if cornered on some more obscure matter? I mean, like, what would it mean to actually solve…

In my experience, GPT-4 is equally willing to make things up if it doesn't know something but it has so much more knowledge than GPT-3.5 that this happens less often in practice.

On that note, I’ve found that just including in the prompt a request for GPT4 to consider its confidence level in an answer and inform me of that confidence level, to reconsider its answer if its confidence is low, and that accuracy is critically important for the topic of the conversation, also can result in better steering it.

I mean, kind of works with humans too. In a high pressure work or school environment, people can confabulate the answers someone wants to hear to avoid discomfort “oh yes, we did the training exercise, the trucks and tanks are in great shape.” Sometimes people need to know they can admit they are wrong. I wonder if some aspect of current LLM tuning/training could be modified so LLMs are more “comfortable”, for lack of a non-anthropomorphized term coming to mind, with admitting they are unsure.

Re: Cargo Cult AI

#145

Post by TeMPOraL »

sandworm101 wrote: post #3

Every time I read about AI I am reminded of the mouse running a maze. Any AI algorithm can learn to complete a maze in record time. It can memorize every corner. It can run a search pattern perfectly and improve that pattern iteratively, to the point that it may create new search patterns, applying what appear to be novel ideas. But the mouse actually understands the concept of a maze. It knows that the cheese exists…

Here is me putting GPT-4 in a vaguely described maze, giving it an underspecified goal, making it a player in a game, myself acting as DM:

https://cloud.typingmind.com/share/c0a68cb2-5f59-4e83-b383-b...

I don't think GPT-4 is memorizing solutions here. I can see extrapolation and some degree of imagination in there, but of course you could say it's memorizing higher-level patterns. At some point though, you have to consider the mouse is also running hard-wired high-level patterns, and ask yourself if the difference here is really a matter of kind, or just degree.

Re: Cargo Cult AI

#146

Post by mustacheemperor »

kerkeslager wrote: post #134
pmoriarty wrote: post #117

Earlier quoted context omitted.

Please consider HN's Guidelines[1] when replying. In particular: - Please don't comment on whether someone read an article. "Did you even read the article? It mentions that" can be shortened to "The article mentions that". - Be kind. Don't be snarky. Converse curiously; don't cross-examine. Edit out swipes. - Please don't fulminate. Please don't sneer, including at the rest of the community. [1] - https://news.ycombi…

> Please don't comment on whether someone read an article. "Did you even read the article? It mentions that" can be shortened to "The article mentions that". This guideline is likely one of the main reasons Hacker News comments are so simultaneously overconfident and undereducated. If you want HN to be a safe place for people interrupting informed conversation with whatever nonsense pops into their head, fine, but I…

It seems that commenter did in fact read the article, so I for one as someone far less informed than both of you am interested to see the discussion carry on.

On that note,

>Please respond to the strongest plausible interpretation of what someone says, not a weaker one that's easier to criticize. Assume good faith.

“You didn’t even read it” is not the strongest plausible interpretation of that reply nor assumes good faith.

Re: Cargo Cult AI

#147

Post by sebzim4500 »

bmc7505 wrote: post #140
sebzim4500 wrote: post #95

Earlier quoted context omitted.

> The problems that AI will manifest are the result of human ambition and failings, no different as any other technology that empowers individuals. Yes, individuals and organizations will misuse AI for immoral purposes, but the popular belief that AI is inherently antihumanist launders accountability by pretending to remove human agency from the equation. How we use or misuse AI technology is entirely on us. So what?…

Although I think their hearts are in the right place, AI safety researchers are primarily driven by irrational instincts and misjudge the promise and perils of artificial intelligence. If humanity chooses to turn away from God and sacrifice each other worshipping false idols, that will be our fault alone, whether or not the technology exists to hasten our demise. We do have thousands of Einsteins today wielding untol…

For the record, if you had told me earlier in the conversation that your faith in your convictions arise from your religious faith you would have saved us both a fair amount of time.

Obviously, your arguments are never going to be convincing to someone who does not share your beliefs.

Re: Cargo Cult AI

#148

Post by TeMPOraL »

sandworm101 wrote: post #61
RC_ITR wrote: post #45

Earlier quoted context omitted.

An interesting fallacy I see emerging is 'this system doesn't have animal-style intelligence, therefore it is lesser' The entire point of neural nets is removing the biases of animal intelligence and letting the computer brute force solutions during training. We are now learning the early stages of the amazing things that can lead to. From an outside perspective, it's not crazy to argue that symbolic reasoning is a c…

Why do you assume that the animal is also not brute forcing the problem? Brute forcing is the basis of evolution. The algorithm running inside the head of the mouse is the survivor of a million iterative generations as the species brute-forced the entire "get to the food" survival problem. I encourage those touting computers as something new to comprehend the concept of deep time, that no matter how many times you ru…

> Brute forcing is the basis of evolution. The algorithm running inside the head of the mouse is the survivor of a million iterative generations as the species brute-forced the entire "get to the food" survival problem.

Here is an interesting corollary: the same is the case with humans. Evolution being what it is, whatever makes the mouse brain tick, and whatever makes our brains tick, must have been relatively easy for evolution to stumble upon and scale up, and it must have been delivering continuous improvements to survival. Otherwise, natural selection wouldn't be able to home in on the solution.

In short: the fundamentals of intelligence have to be something simple enough to be discoverable by accident.

This is one of the main reasons I believe that LLMs aren't unrelated to how our own brains work. We tried to make a simple yet generic system, somewhat emulating what we see in nature, and then we kept scaling it up, until it surprised us with suddenly gaining capabilities we consider to be advanced cognitive functions. Just how many such structures could be there in nature, so that evolution and AI researchers stumbled on two entirely unrelated ones? It's too much of a coincidence.

Re: Cargo Cult AI

#149

Post by throwaway675309 »

gyrovagueGeist wrote: post #116
Jtsummers wrote: post #66

Earlier quoted context omitted.

Probably, I mean I first studied ANNs over two decades ago and had conversations with people about them prior to starting college in the 90s who had developed solutions with them in the 80s (obviously severely computationally constrained in those days compared to today). So still not magic. To be very blunt: If you believe that GPT and the like are magic, then you're not thinking clearly. You're blinded by the result…

"It doesn't stop being magic just because you know how it works" - Terry Prachett

Just because we have a pithy quote doesn't detract from the fact that we shouldn't go all starry eyed and bandy about terms which obfuscate rather than explore the mechanics of how things work. Calling something "black magic" is too close to all of the anthropomorphism bullshit that seems to travel like a miasma around LLMs.

Re: Cargo Cult AI

#150

Post by sandworm101 »

TeMPOraL wrote: post #145
sandworm101 wrote: post #3

Every time I read about AI I am reminded of the mouse running a maze. Any AI algorithm can learn to complete a maze in record time. It can memorize every corner. It can run a search pattern perfectly and improve that pattern iteratively, to the point that it may create new search patterns, applying what appear to be novel ideas. But the mouse actually understands the concept of a maze. It knows that the cheese exists…

Here is me putting GPT-4 in a vaguely described maze, giving it an underspecified goal, making it a player in a game, myself acting as DM: https://cloud.typingmind.com/share/c0a68cb2-5f59-4e83-b383-b... I don't think GPT-4 is memorizing solutions here. I can see extrapolation and some degree of imagination in there, but of course you could say it's memorizing higher-level patterns. At some point though, you have to c…

A text maze. It is effectively brute-forcing a choose-your-own-adventure book. An aware mind would escape the maze by just skipping ahead to read the good parts of the book. That's what I did.
Post reply on HN