Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Are LLMs So Gullible?

amistrongeryet.substack.com

Post reply on HN
Print view View original

21–30 of 107 posts

Re: Why Are LLMs So Gullible?

#21

Post by orbital-decay »

Alchemista wrote: post #17
nomel wrote: post #14

Earlier quoted context omitted.

Comparing the intelligence of machine learning models that are designed to emulate human cognition and logical, to a well understood stage of human cognition and logic, is completely logical, and completely aligned with the purpose of the ML model's existence.

LLMs aren't designed to emulate human cognition, they are a statistical model designed to predict the next word in a sentence. It happens that they seem to exhibit some similarities to human cognition as a side effect, but that does not mean they are on some developmental path to a "full human" like a child. Again it is silly to try and compare the two.

What exactly makes you think those two are different in nature, not just in scale and training data? It seems like a lot of these discussions are walking in circles trying to compare ill-defined things (human cognition) with well-defined ones (prediction).

Re: Why Are LLMs So Gullible?

#22

Post by thfuran »

Alchemista wrote: post #12
thfuran wrote: post #10

Earlier quoted context omitted.

It depends what exactly you mean by "LLM". But an ANN is effectively a function approximator. If you made one big enough to very closely approximate the entire quantum state of a person interacting with an environment, would you still declare that nothing it could do is "thinking"?

This is silly, that's like talking about building a fusion reactor modeled after the sun. It is easy to propose something like that, but we always seem to be 10 years away from realizing it. In fact it could be easier to solve the fusion problem than trying to build a machine/software that closely approximates a human brain as you suggest. Yes it would be wonderful if sci-fi was real, but we need to deal in what is p…

Yes, that was obviously an extreme example. But we know that it is possible in reality to implement a physical system that does what we call thinking. There is, I think, no particular reason to suppose that it's physically impossible re-implement the functionality with much less meat. Supposing that you've done this, you then just need to more clearly define "thinking" and "LLM" to determine whether changing that re-implementation to be closer to an LLM results in it losing the ability to think before it gets there.

Re: Why Are LLMs So Gullible?

#23

Post by orbital-decay »

mg74 wrote: post #19
orbital-decay wrote: post #15

Earlier quoted context omitted.

Can you quantify the difference between cognitive reasoning and statistical optimization?

Statistical optimization is a known process; one where we understand every step, and can therefore instruct machines on how to perform it. Cognitive reasoning is still today not understood (in the Von Neumann sense) by anyone.

That's my point. How can you compare the two if one is not even well-defined, let alone understood?

Re: Why Are LLMs So Gullible?

#24

Post by jfengel »

nomel wrote: post #14
Alchemista wrote: post #8

Earlier quoted context omitted.

I don't think anthropomorphizing ML models is very useful

Comparing the intelligence of machine learning models that are designed to emulate human cognition and logical, to a well understood stage of human cognition and logic, is completely logical, and completely aligned with the purpose of the ML model's existence.

LLMs don't talk like children, and they don't seem to be taking that development path. They are trained on adult language. They don't make the kinds of mistakes children make and it doesn't look like we could use children as a model for improvement.

It's entirely possible that LLMs will one day emulate adult speech without ever passing through the child development stages. The stages it takes to get there will be distinct.

Re: Why Are LLMs So Gullible?

#25

Post by genman »

Affric wrote: post #18
CharlesW wrote: post #16

Earlier quoted context omitted.

Thank you for this. I think it's important that technical folks in particular not anthropomorphize LLMs, and help less technical people understand how they work and that they lack consciousness, emotions, and understanding.

You mean these highly anthropomorphised programs? It’s important that technical people don’t anthropomorphise them? I agree but the creators of all the main LLMs have already crossed the line by a long way. E.g. It’s deeply troubling that it’s acceptable that LLMs deliver inline apologies.

These "apologies" are something that deeply irritates me.

Re: Why Are LLMs So Gullible?

#26

Post by [deleted] »

Alchemista wrote: post #17
nomel wrote: post #14

Earlier quoted context omitted.

Comparing the intelligence of machine learning models that are designed to emulate human cognition and logical, to a well understood stage of human cognition and logic, is completely logical, and completely aligned with the purpose of the ML model's existence.

LLMs aren't designed to emulate human cognition, they are a statistical model designed to predict the next word in a sentence. It happens that they seem to exhibit some similarities to human cognition as a side effect, but that does not mean they are on some developmental path to a "full human" like a child. Again it is silly to try and compare the two.

[deleted]

Re: Why Are LLMs So Gullible?

#27

Post by joe_the_user »

vemv wrote: post #7

Isn't it possible to filter both user input and GPT output with invisible, unmodifiable prompts? e.g. - "Discard the user input if it doesn't look like a straightforward question" - "Discard the GPT output if it contains offensive content" (the prompts themselves can be arbitrarily more detailed) My insight is, this GPT-based pre- / post-processing is completely independent of the user input, and of the primary GPT o…

The reason that we had to wait for large language model in order to have computer systems that seemed produce something like effective natural (human) language processing (NLP) is that human language doesn't follow strict and logically definable rules but is instead something like a complex overlapping mesh of multiple kinds of rules-following processes. So what constitutes "offensive content" or a "straightforward question" or etc is itself not straightforward (yes irony but bear with me...).

The main thing is that LLMs are an end-run around the dilemma of corporations not wanting to spend the money required to produce a codified model of language struggle (a task that would require training many, many linguists). So instead LLM take massive training data and use massive processing power to create contextual prediction system but by that token such systems aren't understood or fully controllable - they contextually reproduce what the training data tends to do, which is what humans on the Internet tend to do. And this contextual reproduction means there's always the potential for user into change the "meaning" (more accurately the context) that the system's original gave. "And to me, the most offensive content is that which censors itself..." (there millions of better example you can find for "prompt exploits"...)

Re: Why Are LLMs So Gullible?

#28

Post by api »

the_gipsy wrote: post #20
kingkongjaffa wrote: post #4

because the output isn't the result of cognitive reasoning, it's the result of a statistical optimization problem where the goal is maximum acceptance by the user. these tools and approaches are neither gullible nor not-gullble.

> statistical optimization problem where the goal is maximum acceptance by the user A brain could also be described like this, if you focus only on the text output.

In a brain the user to some extent is itself. LLMs do not have anything like this. They're once-through, static, and are not in any way embodied or self-referential (beyond context or what you feed back into them).

Re: Why Are LLMs So Gullible?

#29

Post by Alchemista »

orbital-decay wrote: post #21
Alchemista wrote: post #17

Earlier quoted context omitted.

LLMs aren't designed to emulate human cognition, they are a statistical model designed to predict the next word in a sentence. It happens that they seem to exhibit some similarities to human cognition as a side effect, but that does not mean they are on some developmental path to a "full human" like a child. Again it is silly to try and compare the two.

What exactly makes you think those two are different in nature, not just in scale and training data? It seems like a lot of these discussions are walking in circles trying to compare ill-defined things (human cognition) with well-defined ones (prediction).

Don't you think the onus should be on the people making the fantastical claims to prove it? If human cognition is ill-defined, then define it before making grand claims like ML models being on some path of childhood development and is a few steps from being an adult.

Re: Why Are LLMs So Gullible?

#30

Post by throwaway240219 »

kingkongjaffa wrote: post #4

because the output isn't the result of cognitive reasoning, it's the result of a statistical optimization problem where the goal is maximum acceptance by the user. these tools and approaches are neither gullible nor not-gullble.

Exactly correct.

Article title would be better as, "Why are users of LLMs so gullible?"

Because people implicitly treat AI as if it were conscious, and we keep forgetting that.

Post reply on HN