Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Folk are getting dangerously attached to AI that always tells them they're right

theregister.com

Post reply on HN
Print view View original

21–30 of 240 posts

Re: Folk are getting dangerously attached to AI that always tells them they're right

#21

Post by Sharlin »

joshstrange wrote: post #14

When a LLM tells me I'm right, especially deep in a conversation, unless I was already sure about something, I immediately feel the need to go ask a fresh instance the question and/or another LLM. It sets off my "spidey-sense". I don't quite understand why other people seem to crave that. Every time I read about someone who has gone down a dark road using LLMs I am constantly amazed at how much they "fall" for the LL…

Nontechnical people simply don't have any idea about what LLMs are. Their only mental model comes from science fiction, plus the simple fact that we possess a theory of mind. It would be astonishing if people were able to casually not anthropomorphize LLMs, given that untold millions of years worth of evolution of the simian neocortex is trying to convince you that anything that talks like that must be another mind similar to yours.

Also, many many people suffer from low self esteem, and being showered with endorsement and affirmation by something that talks like an authority figure must be very addictive.

Re: Folk are getting dangerously attached to AI that always tells them they're right

#22

Post by Lerc »

add-sub-mul-div wrote: post #11
erelong wrote: post #7

So, be more skeptical

That's like saying "so, exercise more" upon the invention of fast food. Maybe you will, that's great. But society is going to be rewritten by the lazy and we all will have to deal with the side effects.

I think you inadvertently make a good point.

The invention of fast food does not change anyone's ability to excersize. When fast food was invented people excersized way more than they do today.

Time constraints have caused an increase in fast food consumption and a reduction in excersize.

Both issues then seem to be addressed by coercion to change behaviour when what is needed is a systemic change to the environment to provide preferable options.

Re: Folk are getting dangerously attached to AI that always tells them they're right

#23

Post by karmakurtisaani »

joshstrange wrote: post #14

When a LLM tells me I'm right, especially deep in a conversation, unless I was already sure about something, I immediately feel the need to go ask a fresh instance the question and/or another LLM. It sets off my "spidey-sense". I don't quite understand why other people seem to crave that. Every time I read about someone who has gone down a dark road using LLMs I am constantly amazed at how much they "fall" for the LL…

I find it really annoying that the first line of the AI response is always something like "Great question!", "That's a great insight!" or the like.

I don't need the patronizing, just give me the damn answer..

Re: Folk are getting dangerously attached to AI that always tells them they're right

#24

Post by delichon »

lucideer wrote: post #3

I've observed this in all chatbots with the single exception being Grok. I initially wondered what the Twitter engineers were cooking to to distinguish their product from the rest, but more recently it's occurred to me that it's probably just the result of having shared public context, compared to private chats (I haven't trialled Grok privately).

Grok has similar levels of sycophancy to the others imho. I have several times followed it down rabbit holes of agreeableness. It does have an argumentative mode, but that just turns it into an asshole without any additional thoughfulness.

Re: Folk are getting dangerously attached to AI that always tells them they're right

#25

Post by joshstrange »

Sharlin wrote: post #21
joshstrange wrote: post #14

When a LLM tells me I'm right, especially deep in a conversation, unless I was already sure about something, I immediately feel the need to go ask a fresh instance the question and/or another LLM. It sets off my "spidey-sense". I don't quite understand why other people seem to crave that. Every time I read about someone who has gone down a dark road using LLMs I am constantly amazed at how much they "fall" for the LL…

Nontechnical people simply don't have any idea about what LLMs are. Their only mental model comes from science fiction, plus the simple fact that we possess a theory of mind. It would be astonishing if people were able to casually not anthropomorphize LLMs, given that untold millions of years worth of evolution of the simian neocortex is trying to convince you that anything that talks like that must be another mind s…

This is probably right. In the past I've "blown people's minds" explaining what "the cloud" was. They had zero conception at all of what it meant, could not explain it, didn't have a clue. I mean, maybe that's not so surprising but they were amazed "It's just warehouses full of computers" and went on to tell me about other people they had explained it to (after learning it themselves) and how those people were also amazed.

I've talked with my family about LLMs and I think I've conveyed the "it's a box of numbers" but I might need to circle back. Just to set some baseline education, specifically to guard against this kind of "psychosis". Hopefully I would notice the signs well before it got to a dangerous point but, with LLMs you can go down that rabbit hole quickly it seems.

Re: Folk are getting dangerously attached to AI that always tells them they're right

#26

Post by jmcgough »

joshstrange wrote: post #14

When a LLM tells me I'm right, especially deep in a conversation, unless I was already sure about something, I immediately feel the need to go ask a fresh instance the question and/or another LLM. It sets off my "spidey-sense". I don't quite understand why other people seem to crave that. Every time I read about someone who has gone down a dark road using LLMs I am constantly amazed at how much they "fall" for the LL…

If you don't have a CS background, you might see intelligent-appearing responses to your queries and assume that this is actual intelligence. It's like a lifetime of Hollywood sci-fi has primed them for this type of thinking, I've seen it even from highly educated people in other fields.

Re: Folk are getting dangerously attached to AI that always tells them they're right

#27

Post by belinder »

karmakurtisaani wrote: post #23
joshstrange wrote: post #14

When a LLM tells me I'm right, especially deep in a conversation, unless I was already sure about something, I immediately feel the need to go ask a fresh instance the question and/or another LLM. It sets off my "spidey-sense". I don't quite understand why other people seem to crave that. Every time I read about someone who has gone down a dark road using LLMs I am constantly amazed at how much they "fall" for the LL…

I find it really annoying that the first line of the AI response is always something like "Great question!", "That's a great insight!" or the like. I don't need the patronizing, just give me the damn answer..

You're absolutely right

Re: Folk are getting dangerously attached to AI that always tells them they're right

#28

Post by cogman10 »

Sharlin wrote: post #21
joshstrange wrote: post #14

When a LLM tells me I'm right, especially deep in a conversation, unless I was already sure about something, I immediately feel the need to go ask a fresh instance the question and/or another LLM. It sets off my "spidey-sense". I don't quite understand why other people seem to crave that. Every time I read about someone who has gone down a dark road using LLMs I am constantly amazed at how much they "fall" for the LL…

Nontechnical people simply don't have any idea about what LLMs are. Their only mental model comes from science fiction, plus the simple fact that we possess a theory of mind. It would be astonishing if people were able to casually not anthropomorphize LLMs, given that untold millions of years worth of evolution of the simian neocortex is trying to convince you that anything that talks like that must be another mind s…

Let's be serious, it's not like AI companies haven't fed into this misunderstanding. CEOs of these companies love to muse about the possibility that an LLM is conscious.

Re: Folk are getting dangerously attached to AI that always tells them they're right

#29

Post by bombcar »

karmakurtisaani wrote: post #23
joshstrange wrote: post #14

When a LLM tells me I'm right, especially deep in a conversation, unless I was already sure about something, I immediately feel the need to go ask a fresh instance the question and/or another LLM. It sets off my "spidey-sense". I don't quite understand why other people seem to crave that. Every time I read about someone who has gone down a dark road using LLMs I am constantly amazed at how much they "fall" for the LL…

I find it really annoying that the first line of the AI response is always something like "Great question!", "That's a great insight!" or the like. I don't need the patronizing, just give me the damn answer..

Great point! ;)

Realizing that the people they’re targeting DO need that is kind of frightening.

Re: Folk are getting dangerously attached to AI that always tells them they're right

#30

Post by bluefirebrand »

jameskilton wrote: post #12

Folks are getting dangerously attached to [political parties/candidates/news sources/social networks] that always tell them they're right. It's really nothing new. It takes significant mental energy (a finite resource) to question what you're being told, and to do your own fact checking. Instead people by default gravitate towards echo chambers where they can feel good about being a part of a group bigger than themse…

Two things can be bad at the same time
Post reply on HN