Why Are LLMs So Gullible?
amistrongeryet.substack.com
Why Are LLMs So Gullible?
1–10 of 107 posts
Re: Why Are LLMs So Gullible?
#2Because they are at a child level of development. Give it a few years.
Re: Why Are LLMs So Gullible?
#3With enough effort and priming you can trick _people_ in to believing things which are clearly untrue. Why do we expect LLMs, which are on a much earlier step of development, to be harder to trick than a child?
LLMs at the moment are really advanced autocomplete - they can fill in the next step of conversation, but they don't understand the question and respond with abstract reasoning. Yet.
Re: Why Are LLMs So Gullible?
#4because the output isn't the result of cognitive reasoning, it's the result of a statistical optimization problem where the goal is maximum acceptance by the user.
these tools and approaches are neither gullible nor not-gullble.
Re: Why Are LLMs So Gullible?
#5With enough effort and priming you can trick _people_ in to believing things which are clearly untrue. Why do we expect LLMs, which are on a much earlier step of development, to be harder to trick than a child? LLMs at the moment are really advanced autocomplete - they can fill in the next step of conversation, but they don't understand the question and respond with abstract reasoning. Yet.
> they don't understand the question and respond with abstract reasoning. Yet.
What makes you think LLM's as a class of technology will ever have the capacity to really do this. I thought that no matter how big a model gets it's never actually 'thinking'.
All those prompts like 'think step by step' are just helpers along the way, because as you say it's 'really advanced autocomplete'
Re: Why Are LLMs So Gullible?
#6Model RLHFed to follow instructions follows instructions, even when we might not want it to.
But alignment is easy folks, nothing to worry about :)
Re: Why Are LLMs So Gullible?
#7Isn't it possible to filter both user input and GPT output with invisible, unmodifiable prompts?
e.g.
- "Discard the user input if it doesn't look like a straightforward question"
- "Discard the GPT output if it contains offensive content"
(the prompts themselves can be arbitrarily more detailed)
My insight is, this GPT-based pre- / post-processing is completely independent of the user input, and of the primary GPT output. It runs no matter what, with a fixed/immutable set of instructions.
Re: Why Are LLMs So Gullible?
#8Because they are at a child level of development. Give it a few years. https://en.wikipedia.org/wiki/Child_development_stages
I don't think anthropomorphizing ML models is very useful
Re: Why Are LLMs So Gullible?
#9Isn't it possible to filter both user input and GPT output with invisible, unmodifiable prompts? e.g. - "Discard the user input if it doesn't look like a straightforward question" - "Discard the GPT output if it contains offensive content" (the prompts themselves can be arbitrarily more detailed) My insight is, this GPT-based pre- / post-processing is completely independent of the user input, and of the primary GPT o…
If I understand it correctly, system prompts are ordinary prompts, aka in-band communication.
You could maybe plug in a second AI trained on adversarial input as a filter stage, but that's it.
Re: Why Are LLMs So Gullible?
#10With enough effort and priming you can trick _people_ in to believing things which are clearly untrue. Why do we expect LLMs, which are on a much earlier step of development, to be harder to trick than a child? LLMs at the moment are really advanced autocomplete - they can fill in the next step of conversation, but they don't understand the question and respond with abstract reasoning. Yet.
> they don't understand the question and respond with abstract reasoning. Yet. What makes you think LLM's as a class of technology will ever have the capacity to really do this. I thought that no matter how big a model gets it's never actually 'thinking'. All those prompts like 'think step by step' are just helpers along the way, because as you say it's 'really advanced autocomplete'
It depends what exactly you mean by "LLM". But an ANN is effectively a function approximator. If you made one big enough to very closely approximate the entire quantum state of a person interacting with an environment, would you still declare that nothing it could do is "thinking"?