Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Are LLMs So Gullible?

amistrongeryet.substack.com

Post reply on HN
Print view View original

51–60 of 107 posts

Re: Why Are LLMs So Gullible?

#51

Post by orbital-decay »

Alchemista wrote: post #29
orbital-decay wrote: post #21

Earlier quoted context omitted.

What exactly makes you think those two are different in nature, not just in scale and training data? It seems like a lot of these discussions are walking in circles trying to compare ill-defined things (human cognition) with well-defined ones (prediction).

Don't you think the onus should be on the people making the fantastical claims to prove it? If human cognition is ill-defined, then define it before making grand claims like ML models being on some path of childhood development and is a few steps from being an adult.

I think that neither side has a convincing argument in these discussions, actually, I'm pointing out that both extremes - anthropomorphism and stochastic parrot - lack any foundation, so I wouldn't be so confident. The behavior of neither statistical models nor biological systems is well understood. It's entirely possible that every trait you consider human can naturally emerge from a dumb statistical model, and indeed certain processes are remarkably similar as the models get bigger, smarter, and have better training data. It's possible that it can't, however.

Re: Why Are LLMs So Gullible?

#52

Post by Jensson »

thfuran wrote: post #48
Jensson wrote: post #40

Earlier quoted context omitted.

To approximate a function with a loop you would need a close to infinitely large neural net. Humans do have loops in their thinking, we need a new architecture for LLMs to be able to think in loops.

I don't think anyone would say that being an RNN disqualifies an architecture from being considered an LLM.

But we don't know how to train those to be good yet. It would require some novel step there, and it is unclear if it would still be anything like current LLMs after that.

Re: Why Are LLMs So Gullible?

#53

Post by chx »

orbital-decay wrote: post #15
kingkongjaffa wrote: post #4

because the output isn't the result of cognitive reasoning, it's the result of a statistical optimization problem where the goal is maximum acceptance by the user. these tools and approaches are neither gullible nor not-gullble.

Can you quantify the difference between cognitive reasoning and statistical optimization?

Sure thing.

People really don't learn the history of AI any more apparently or this question wouldn't come up all the time.

There is basically any number of questions you can ask a two year old human who have never encountered that question nor anything even remotely similar to it and yet they can answer without fail. Meanwhile absolutely no AI can answer these unless the specific question / the rules underlying the questions were previously fed into it. The textbook example is "If Susan goes shopping will her head go with her?" Of course, since this specific question is literally a textbook one, you can't fool an LLM with it but it's easy to come up with brand new ones.

In the early 1980s this stopped Douglas Lenat who has worked very successfully on discovery systems and made him turn to assembling these facts and rules into CyC.

Re: Why Are LLMs So Gullible?

#54

Post by nomel »

Alchemista wrote: post #17
nomel wrote: post #14

Earlier quoted context omitted.

Comparing the intelligence of machine learning models that are designed to emulate human cognition and logical, to a well understood stage of human cognition and logic, is completely logical, and completely aligned with the purpose of the ML model's existence.

LLMs aren't designed to emulate human cognition, they are a statistical model designed to predict the next word in a sentence. It happens that they seem to exhibit some similarities to human cognition as a side effect, but that does not mean they are on some developmental path to a "full human" like a child. Again it is silly to try and compare the two.

> LLMs aren't designed to emulate human cognition

I'm not sure that's fair, or correct. The behavior of an LLM is set by the design of the combination of loss function and training data. Achieving it is the desired result of the selection of those.

> It happens that they seem to exhibit some similarities to human cognition as a side effect

Yes, the desired behavior of nearly all LLM projects is to emulate the capabilities of human cognition, and that stated goal is the justification that organizations are using for spending millions to train them.

> but that does not mean they are on some developmental path to a "full human" like a child.

You are the first to suggest any such thing, in this comment chain. But, the field of AI is objectively on that development path, since that is the stated goal of many of the orgs, with LLM existing on that path. If that path actually leads anywhere is anyone's guess.

Re: Why Are LLMs So Gullible?

#55

Post by iforgotpassword »

throwaway240219 wrote: post #30
kingkongjaffa wrote: post #4

because the output isn't the result of cognitive reasoning, it's the result of a statistical optimization problem where the goal is maximum acceptance by the user. these tools and approaches are neither gullible nor not-gullble.

Exactly correct. Article title would be better as, "Why are users of LLMs so gullible?" Because people implicitly treat AI as if it were conscious, and we keep forgetting that.

Because it still makes sense to reason about them that way, as weird as it may be. It's like the Chinese room problem, even though we know nobody is inside.

It's like when people say "our brain thinks that ..." when they talk about something we do subconsciously, or some illusion we fall for. What does that even mean? Is our brain suddenly a detached entity thinking on its own? Then what am I using to think? Yet, everybody understands what is meant by that.

So I don't think people keep forgetting that llms aren't conscious, at least as long as we talk about the target audience of articles like this, eg hn folks.

Re: Why Are LLMs So Gullible?

#56

Post by snewman »

kingkongjaffa wrote: post #4

because the output isn't the result of cognitive reasoning, it's the result of a statistical optimization problem where the goal is maximum acceptance by the user. these tools and approaches are neither gullible nor not-gullble.

(author here)

Of course LLMs are not people. But human metaphors can (sometimes!) be useful in understanding, explaining, and even enhancing their behavior. For instance, techniques such as Chain-of-Thought prompting explicitly apply techniques that work well for people to improve the reasoning ability of LLMs.

A point I attempt to make in this article is that one reason reason LLMs are so vulnerable to jailbreaks and prompt injection is that these types of attacks include non sequiturs that are not well represented in the training data. I would argue that "LLMs are gullible because they are naive [haven't had much past exposure to this form of trickery]" is a reasonable mental shorthand for explaining and internalizing this idea. It's especially helpful for readers who won't be familiar with terms like "out of distribution" or "adversarial examples", but who would benefit from being able to internalize the idea that LLMs are easily subverted.

In other words, I don't think it's helpful to reflexively dismiss any application of human metaphors to LLMs. It's easy to go wrong with metaphors, but they can also be valuable tools for conveying complex ideas. Did you read the article, and do you have any comments as to the substance of its content?

Re: Why Are LLMs So Gullible?

#57

Post by Alchemista »

nomel wrote: post #54
Alchemista wrote: post #17

Earlier quoted context omitted.

LLMs aren't designed to emulate human cognition, they are a statistical model designed to predict the next word in a sentence. It happens that they seem to exhibit some similarities to human cognition as a side effect, but that does not mean they are on some developmental path to a "full human" like a child. Again it is silly to try and compare the two.

> LLMs aren't designed to emulate human cognition I'm not sure that's fair, or correct. The behavior of an LLM is set by the design of the combination of loss function and training data. Achieving it is the desired result of the selection of those. > It happens that they seem to exhibit some similarities to human cognition as a side effect Yes, the desired behavior of nearly all LLM projects is to emulate the capabil…

> You are the first to suggest any such thing, in this comment chain.

I'm sorry, did you miss the original post: "Because they are at a child level of development. Give it a few years."

I extrapolated a little bit, but not much. They were clearly implying that it is on a similar developmental path as a child.

Re: Why Are LLMs So Gullible?

#58

Post by famouswaffles »

chx wrote: post #53
orbital-decay wrote: post #15

Earlier quoted context omitted.

Can you quantify the difference between cognitive reasoning and statistical optimization?

Sure thing. People really don't learn the history of AI any more apparently or this question wouldn't come up all the time. There is basically any number of questions you can ask a two year old human who have never encountered that question nor anything even remotely similar to it and yet they can answer without fail. Meanwhile absolutely no AI can answer these unless the specific question / the rules underlying the…

You've not quantified anything, nor have you even provided any valid example of these so abundant mystery questions.

If it's so easy, come up with one and show us.

Re: Why Are LLMs So Gullible?

#59

Post by nyrikki »

mrtksn wrote: post #34
kingkongjaffa wrote: post #4

because the output isn't the result of cognitive reasoning, it's the result of a statistical optimization problem where the goal is maximum acceptance by the user. these tools and approaches are neither gullible nor not-gullble.

I used to think like that but I'm not so sure anymore. The statistical optimisation thing is an analytical approach to Neural Networks but its similar to saying that love is just hormones.

Real question, do you really want to know where you may be mistaken, or will you hand wave away information in order to protect what you believe?

No judgement here but I am just tired of sharing information with people to explain why while interpretation of a complex model may be hard, we know the methods of how PAC learning works and some hard boundaries on what it can do.

Obviously we need people to push boundaries and assumptions.

But we have known about hard upper limits for a long time. Right now we are pushing up to those limits in what we can actually implement, but those hard limits haven't budged in decades.

Re: Why Are LLMs So Gullible?

#60

Post by [deleted] »

genman wrote: post #25
Affric wrote: post #18

Earlier quoted context omitted.

You mean these highly anthropomorphised programs? It’s important that technical people don’t anthropomorphise them? I agree but the creators of all the main LLMs have already crossed the line by a long way. E.g. It’s deeply troubling that it’s acceptable that LLMs deliver inline apologies.

These "apologies" are something that deeply irritates me.

[deleted]
Post reply on HN