Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Are LLMs So Gullible?

amistrongeryet.substack.com

Post reply on HN
Print view View original

41–50 of 107 posts

Re: Why Are LLMs So Gullible?

#41

Post by mattlutze »

kypro wrote: post #37
kingkongjaffa wrote: post #4

because the output isn't the result of cognitive reasoning, it's the result of a statistical optimization problem where the goal is maximum acceptance by the user. these tools and approaches are neither gullible nor not-gullble.

This depends on perspective. I could argue the issue isn't that it's gullible but misaligned. In the case of the napalm Grandma it seems odd to me that you're suggesting the LLM is stupid because it's answering in a way that makes sense given its prompt. The issue doesn't necessarily suggest a lack of reasoning, but that the LLM is trusting the human. For the record, I agree with you – I would have thought that an AI…

> For the record, I agree with you – I would have thought that an AI that can reason well would probably know when not to trust humans, but I suppose that assumes it values preventing humans creating napalm over being correct and helpful.

Do we want LLMs, and later other multi-modal / servo systems, that are deciding they can't trust a human prompter and taking actions based on that?

>... and that we must find a way to build an LLM that is more distrusting and deceptive if we wish to align it with our values and our nature.

Tongue in cheek or actual argument here?

Re: Why Are LLMs So Gullible?

#42

Post by Verdex »

thfuran wrote: post #10
kingkongjaffa wrote: post #5

Earlier quoted context omitted.

> they don't understand the question and respond with abstract reasoning. Yet. What makes you think LLM's as a class of technology will ever have the capacity to really do this. I thought that no matter how big a model gets it's never actually 'thinking'. All those prompts like 'think step by step' are just helpers along the way, because as you say it's 'really advanced autocomplete'

It depends what exactly you mean by "LLM". But an ANN is effectively a function approximator. If you made one big enough to very closely approximate the entire quantum state of a person interacting with an environment, would you still declare that nothing it could do is "thinking"?

I don't know. This feels like an unfair question to ask. You've proposed a basically impossible engineering problem and then declared that the outcome will obviously be the outcome that you want it to be.

What if I answer, "No because humans have supernatural souls."

You can easily answer, "Souls don't exist, therefore I'm still right."

But then we actually build your machine and it turns out that Penrose's non-computable microtubules really exist and the machine is useless.

We can't know the result of such an ambitious endeavor before we go through with it. So it doesn't make much sense to me to use such a thought experiment as evidence for something much lesser that is currently contested.

Re: Why Are LLMs So Gullible?

#43

Post by BazookaMusic »

Jensson wrote: post #36
mrtksn wrote: post #34

Earlier quoted context omitted.

I used to think like that but I'm not so sure anymore. The statistical optimisation thing is an analytical approach to Neural Networks but its similar to saying that love is just hormones.

Isn't love just hormones? It isn't rational reasoning at least.

A simple proof that love is not just hormones is that love can last for some time. If it was just a chemical phenomenon why does it happen repeatedly. Why can someone feel love for someone just by bringing to mind the symbol which represents that person? Why can someone feel love by seeing an illustration of someone they love?

Re: Why Are LLMs So Gullible?

#44

Post by nomel »

Jensson wrote: post #36
mrtksn wrote: post #34

Earlier quoted context omitted.

I used to think like that but I'm not so sure anymore. The statistical optimisation thing is an analytical approach to Neural Networks but its similar to saying that love is just hormones.

Isn't love just hormones? It isn't rational reasoning at least.

It's a bias modifier that causes a huge restructuring/reconfiguration of the whole world model, and even perception itself.

It's what the hormone does that's important, not what it is.

Re: Why Are LLMs So Gullible?

#45

Post by reaperman »

kypro wrote: post #37
kingkongjaffa wrote: post #4

because the output isn't the result of cognitive reasoning, it's the result of a statistical optimization problem where the goal is maximum acceptance by the user. these tools and approaches are neither gullible nor not-gullble.

This depends on perspective. I could argue the issue isn't that it's gullible but misaligned. In the case of the napalm Grandma it seems odd to me that you're suggesting the LLM is stupid because it's answering in a way that makes sense given its prompt. The issue doesn't necessarily suggest a lack of reasoning, but that the LLM is trusting the human. For the record, I agree with you – I would have thought that an AI…

> the LLM is trusting the human

> an AI that can reason well would probably know when not to trust humans

> it values preventing humans creating napalm over being correct and helpful.

> Maybe it just doesn't share our values

> prioritises being honest and helpful.

> they are too trusting and too honest

> an LLM that is more distrusting and deceptive

Current LLM's do/have/feel literally none of these things. They do not have emotion, they do not have "theory of mind" so they cannot be said to "trust" or "distrust". They cannot reason. They don't have any values - not our values, not different values, literally they have no values at all. They are not an alien species to be understood - they are unthinking, unfeeling, unyielding machines.

Re: Why Are LLMs So Gullible?

#46

Post by px43 »

ctoth wrote: post #6

Model RLHFed to follow instructions follows instructions, even when we might not want it to. But alignment is easy folks, nothing to worry about :)

I think people might have forgotten that LLMs before InstructGPT came around could be weirdly opinionated jerks. There was this whole effort to train them so that we could actually give them instructions. It's probably a hell of a lot more useful to have an LLM that will just go with whatever weird stuff the human says rather than try to fight them on it.

https://openai.com/research/instruction-following

Re: Why Are LLMs So Gullible?

#47

Post by mrtksn »

Jensson wrote: post #36
mrtksn wrote: post #34

Earlier quoted context omitted.

I used to think like that but I'm not so sure anymore. The statistical optimisation thing is an analytical approach to Neural Networks but its similar to saying that love is just hormones.

Isn't love just hormones? It isn't rational reasoning at least.

It is hormones. Also, life is a series of chemical reactions. However this approach is useful in some context and completely useless in other, like in context where people talk about life and love.

Although, it can be deconstructed to this it doesn't mean that the other POVs are false. The reasoning comes from the process itself, it triggers a series of calculations that are applied on the input, which are the reasoning part.

The analytical approach is useful when calibrating these calculations.

Re: Why Are LLMs So Gullible?

#48

Post by thfuran »

Jensson wrote: post #40
thfuran wrote: post #10

Earlier quoted context omitted.

It depends what exactly you mean by "LLM". But an ANN is effectively a function approximator. If you made one big enough to very closely approximate the entire quantum state of a person interacting with an environment, would you still declare that nothing it could do is "thinking"?

To approximate a function with a loop you would need a close to infinitely large neural net. Humans do have loops in their thinking, we need a new architecture for LLMs to be able to think in loops.

I don't think anyone would say that being an RNN disqualifies an architecture from being considered an LLM.

Re: Why Are LLMs So Gullible?

#49

Post by Verdex »

dist-epoch wrote: post #2

Because they are at a child level of development. Give it a few years. https://en.wikipedia.org/wiki/Child_development_stages

Another possibility is that the only thing that LLMs are doing is encoding the structural data that exists in natural language. For example, you can load a corpus into vector space and then do algebra like:

  let v = man - woman;
  let r = king - v;
  assert( r == queen );
or so I'm told.

And then it turns out that those structures only have the intelligence of a child. Arbitrary LLM and other ML advancements that focus solely on scanning large natural language datasets may never be able to advance past child level intelligence if the intelligence that they're approximating isn't better than a child.

Re: Why Are LLMs So Gullible?

#50

Post by thfuran »

Verdex wrote: post #42
thfuran wrote: post #10

Earlier quoted context omitted.

It depends what exactly you mean by "LLM". But an ANN is effectively a function approximator. If you made one big enough to very closely approximate the entire quantum state of a person interacting with an environment, would you still declare that nothing it could do is "thinking"?

I don't know. This feels like an unfair question to ask. You've proposed a basically impossible engineering problem and then declared that the outcome will obviously be the outcome that you want it to be. What if I answer, "No because humans have supernatural souls." You can easily answer, "Souls don't exist, therefore I'm still right." But then we actually build your machine and it turns out that Penrose's non-compu…

You think it's unfair to ask whether a simulation of a person can think in a discussion on whether some particular class of algorithm can think? Lacking a clear definition of what exactly an LLM is and what thinking is, I can't think of a single more germane question (aside from what those definitions are, I suppose).
Post reply on HN