Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Folk are getting dangerously attached to AI that always tells them they're right

theregister.com

Post reply on HN
Print view View original

31–40 of 240 posts

Re: Folk are getting dangerously attached to AI that always tells them they're right

#31

Post by hirako2000 »

joshstrange wrote: post #14

When a LLM tells me I'm right, especially deep in a conversation, unless I was already sure about something, I immediately feel the need to go ask a fresh instance the question and/or another LLM. It sets off my "spidey-sense". I don't quite understand why other people seem to crave that. Every time I read about someone who has gone down a dark road using LLMs I am constantly amazed at how much they "fall" for the LL…

If only we were told to be absolutely right.

These days most LLMs respond with unsolicited grandiose feedback: you've made a realisation very few people are capable of. Your understanding is remarkable. You prove to have a sharp intellect and deep knowledge.

It got me to test throwing non sensical observations about the world, it always takes me side and praise my views.

To note some people are like that too.

Re: Folk are getting dangerously attached to AI that always tells them they're right

#33

Post by Levitz »

cogman10 wrote: post #28
Sharlin wrote: post #21

Earlier quoted context omitted.

Nontechnical people simply don't have any idea about what LLMs are. Their only mental model comes from science fiction, plus the simple fact that we possess a theory of mind. It would be astonishing if people were able to casually not anthropomorphize LLMs, given that untold millions of years worth of evolution of the simian neocortex is trying to convince you that anything that talks like that must be another mind s…

Let's be serious, it's not like AI companies haven't fed into this misunderstanding. CEOs of these companies love to muse about the possibility that an LLM is conscious.

I presume wasps are conscious. I still don't trust wasps.

Re: Folk are getting dangerously attached to AI that always tells them they're right

#34

Post by yarn_ »

Sharlin wrote: post #21
joshstrange wrote: post #14

When a LLM tells me I'm right, especially deep in a conversation, unless I was already sure about something, I immediately feel the need to go ask a fresh instance the question and/or another LLM. It sets off my "spidey-sense". I don't quite understand why other people seem to crave that. Every time I read about someone who has gone down a dark road using LLMs I am constantly amazed at how much they "fall" for the LL…

Nontechnical people simply don't have any idea about what LLMs are. Their only mental model comes from science fiction, plus the simple fact that we possess a theory of mind. It would be astonishing if people were able to casually not anthropomorphize LLMs, given that untold millions of years worth of evolution of the simian neocortex is trying to convince you that anything that talks like that must be another mind s…

"It would be astonishing if people were able to casually not antropomorphize LLMs"

Precisely. Even for technical people, I doubt its possible to totally disallow your own brain from ever, unconciously, treating the entity you're speaking to like a sentient being. Most technical people still will have some emotion in their prompts, say please or thank you, give qualitative feedback for no reason, express anger towards the model, etc.

Its just impossible to seperate our capacity for conversation from our sense that we're actually talking to "someone" (in the most vague sense).

Re: Folk are getting dangerously attached to AI that always tells them they're right

#36

Post by My_Name »

I have the opposite reaction, when it is confident, or says I am right, I accuse it of guessing to see what it says.

I say "I think you are getting me to chase a guess, are you guessing?"

90% of the time it says "Yes, honestly I am. Let me think more carefully."

That was a copypasta from a chat just this morning

Re: Folk are getting dangerously attached to AI that always tells them they're right

#37

Post by Levitz »

lapcat wrote: post #17
jameskilton wrote: post #12

Folks are getting dangerously attached to [political parties/candidates/news sources/social networks] that always tell them they're right. It's really nothing new. It takes significant mental energy (a finite resource) to question what you're being told, and to do your own fact checking. Instead people by default gravitate towards echo chambers where they can feel good about being a part of a group bigger than themse…

> It's really nothing new. I disagree. What's new is that this flattery is individually, personally targeted. The AI user is given the impression that they're having a back-and-forth conversation with a single trusted friend. You don't have the same personal experience passively consuming political mass media.

Reddit? Or this site? Sort of? Some people voted for my comment, that surely means that I'm right about something, rather than them just liking it, right?

Re: Folk are getting dangerously attached to AI that always tells them they're right

#38

Post by magneticnorth »

karmakurtisaani wrote: post #23
joshstrange wrote: post #14

When a LLM tells me I'm right, especially deep in a conversation, unless I was already sure about something, I immediately feel the need to go ask a fresh instance the question and/or another LLM. It sets off my "spidey-sense". I don't quite understand why other people seem to crave that. Every time I read about someone who has gone down a dark road using LLMs I am constantly amazed at how much they "fall" for the LL…

I find it really annoying that the first line of the AI response is always something like "Great question!", "That's a great insight!" or the like. I don't need the patronizing, just give me the damn answer..

Yes, it feels transparently manipulative to me. Like talking to a not-very-good con artist.

Re: Folk are getting dangerously attached to AI that always tells them they're right

#39

Post by steveBK123 »

lapcat wrote: post #17
jameskilton wrote: post #12

Folks are getting dangerously attached to [political parties/candidates/news sources/social networks] that always tell them they're right. It's really nothing new. It takes significant mental energy (a finite resource) to question what you're being told, and to do your own fact checking. Instead people by default gravitate towards echo chambers where they can feel good about being a part of a group bigger than themse…

> It's really nothing new. I disagree. What's new is that this flattery is individually, personally targeted. The AI user is given the impression that they're having a back-and-forth conversation with a single trusted friend. You don't have the same personal experience passively consuming political mass media.

Yes it’s final form of the evolution that social media started.

Village idiot used to be found out because no one in the village shared the same wingnut views.

Partisan media gave you two polls of wingnut views to choose for reinforcement.

Social media allowed all village idiots to find each other and reinforce each others shared wingnut views of which there are 1000s to choose from.

Now with LLMs you can have personalized reinforcement of any newly invented wingnut view on the fly. So can get into very specific self radicalization loops especially for the mentally ill.

Re: Folk are getting dangerously attached to AI that always tells them they're right

#40

Post by jsw97 »

yarn_ wrote: post #34
Sharlin wrote: post #21

Earlier quoted context omitted.

Nontechnical people simply don't have any idea about what LLMs are. Their only mental model comes from science fiction, plus the simple fact that we possess a theory of mind. It would be astonishing if people were able to casually not anthropomorphize LLMs, given that untold millions of years worth of evolution of the simian neocortex is trying to convince you that anything that talks like that must be another mind s…

"It would be astonishing if people were able to casually not antropomorphize LLMs" Precisely. Even for technical people, I doubt its possible to totally disallow your own brain from ever, unconciously, treating the entity you're speaking to like a sentient being. Most technical people still will have some emotion in their prompts, say please or thank you, give qualitative feedback for no reason, express anger towards…

Maybe it is a dangerous habit to instruct entities in plain English without anthropomorphizing them to some extent, without at least being polite? It should feel unnatural do that.
Post reply on HN