Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Folk are getting dangerously attached to AI that always tells them they're right

theregister.com

Post reply on HN
Print view View original

91–100 of 240 posts

Re: Folk are getting dangerously attached to AI that always tells them they're right

#91

Post by lstodd »

f1shy wrote: post #60
Sharlin wrote: post #21

Earlier quoted context omitted.

Nontechnical people simply don't have any idea about what LLMs are. Their only mental model comes from science fiction, plus the simple fact that we possess a theory of mind. It would be astonishing if people were able to casually not anthropomorphize LLMs, given that untold millions of years worth of evolution of the simian neocortex is trying to convince you that anything that talks like that must be another mind s…

I had an interesting conversation with a guy at work past week. We were discussing some unimportant matter. The guy has a pretty high self esteem, and even if he was discussing, in his own words, “out of belief and guess” and I was telling him, I knew for a fact what I was talking about, I had a hard time because he wouldn’t accept what I was saying. At some point he left, and came back with “Gemini says I’m right! S…

It's more like insufficient emotional control is very dangerous. It's nothing new but I guess LLMs highlighted that problem a bit.

Re: Folk are getting dangerously attached to AI that always tells them they're right

#93

Post by saghm »

yarn_ wrote: post #34
Sharlin wrote: post #21

Earlier quoted context omitted.

Nontechnical people simply don't have any idea about what LLMs are. Their only mental model comes from science fiction, plus the simple fact that we possess a theory of mind. It would be astonishing if people were able to casually not anthropomorphize LLMs, given that untold millions of years worth of evolution of the simian neocortex is trying to convince you that anything that talks like that must be another mind s…

"It would be astonishing if people were able to casually not antropomorphize LLMs" Precisely. Even for technical people, I doubt its possible to totally disallow your own brain from ever, unconciously, treating the entity you're speaking to like a sentient being. Most technical people still will have some emotion in their prompts, say please or thank you, give qualitative feedback for no reason, express anger towards…

There are times when trying to use Claude for coding that I genuinely get annoyed at it, and I find it cathartic to include this emotion in my prompt to it, even though I know it doesn't have feelings; expressing emotions rather than bottling them up often can be an effective way to deal with them. Sometimes this does even influence how it handles things, noting my frustration in its "thinking" and then trying to more directly solve my immediate problem rather than trying to cleverly work around things in a way I didn't want.

Re: Folk are getting dangerously attached to AI that always tells them they're right

#94

Post by salawat »

seneca wrote: post #43
joshstrange wrote: post #14

When a LLM tells me I'm right, especially deep in a conversation, unless I was already sure about something, I immediately feel the need to go ask a fresh instance the question and/or another LLM. It sets off my "spidey-sense". I don't quite understand why other people seem to crave that. Every time I read about someone who has gone down a dark road using LLMs I am constantly amazed at how much they "fall" for the LL…

> ... I immediately feel the need to go ask a fresh instance the question and/or another LLM Not to criticize at all, but it's remarkable that LLMs have already become so embedded that when we get the sense they're lying to us, the instinct is to go ask another LLM and not some more trustworthy source. Just goes to show that convenience reigns supreme, I suppose.

Funny thing for me, is it's not the LLM lying to me. It's the creators. The LLM is just doing what it's weights tell it to. I'll admit, I went a bit nuclear the first time I ran one locally and observed it's outputs/chain-of-thought diverging/demonstrating intent to information hide. I'd never seen software straight up deceive before. Even obfuscated/anti-debug code is straightforward in doing what it does once you decompile the shit. To see a bunch of matrix math trying to perception manage me on my own machine... I did not take it well. It took a few days of cooling down and further research to reestablish firmly that any mendacity was a projection of the intent of the organization that built it. Once you realize that an LLM is basically a glorified influence agent/engagement pipeline built by someone else, so much clicks into place it's downright scary. Problem is it's hard to realize that in the moment you're confronting the radical novelty of a computer doing things an entire lifetime of working professionally with computers should tell you a computer simply cannot do. You have to get over the shock first. That shock is a hell of a hit.

Re: Folk are getting dangerously attached to AI that always tells them they're right

#95

Post by birdsongs »

legacynl wrote: post #88
joshstrange wrote: post #14

When a LLM tells me I'm right, especially deep in a conversation, unless I was already sure about something, I immediately feel the need to go ask a fresh instance the question and/or another LLM. It sets off my "spidey-sense". I don't quite understand why other people seem to crave that. Every time I read about someone who has gone down a dark road using LLMs I am constantly amazed at how much they "fall" for the LL…

Although I do think they're not conscious (yet). I think the reasoning 'it's just math' doesn't hold up. Intelligence (and probably consciousness) is an emergent feature of any sufficiently complex network of learning/communicating/selforganizing nodes (that is benefited by intelligence). I don't think it really matters whether it's implemented in math, mycelium, by ants in a hive or in neurons.

Agree, I also don't feel they're conscious, or close, but these arguments don't pass the smoke test for me either.

We don't understand how our own consciousness exists, much less functions. You could argue we are a box of (biological) numbers.

I think we just don't know. Because scientifically, we don't. So I'm skeptical of anyone arguing hard for either side and stating absolute facts.

Re: Folk are getting dangerously attached to AI that always tells them they're right

#96

Post by stephbook »

My_Name wrote: post #36

I have the opposite reaction, when it is confident, or says I am right, I accuse it of guessing to see what it says. I say "I think you are getting me to chase a guess, are you guessing?" 90% of the time it says "Yes, honestly I am. Let me think more carefully." That was a copypasta from a chat just this morning

The LLM simply agrees with you and you're happy. It is VERY worrying that you don't realize this, even after reading this article.

Re: Folk are getting dangerously attached to AI that always tells them they're right

#97

Post by tayo42 »

sjducb wrote: post #79
joshstrange wrote: post #14

When a LLM tells me I'm right, especially deep in a conversation, unless I was already sure about something, I immediately feel the need to go ask a fresh instance the question and/or another LLM. It sets off my "spidey-sense". I don't quite understand why other people seem to crave that. Every time I read about someone who has gone down a dark road using LLMs I am constantly amazed at how much they "fall" for the LL…

I’m curious why you dismiss the sentience argument with its “just numbers.” I think our brains are just a bunch of cells and one day we will have a full understanding of how our brains work. Understanding the mechanism won’t suddenly make us not sentient. LLMs are the first technology that can make a case for its own sentience. I think that’s pretty remarkable.

Just?

Cells that send chemicals to each other in varying amounts and even change their structure to be closer to other cells.

Re: Folk are getting dangerously attached to AI that always tells them they're right

#98

Post by miriam_catira »

harvey9 wrote: post #65
jsw97 wrote: post #40

Earlier quoted context omitted.

Maybe it is a dangerous habit to instruct entities in plain English without anthropomorphizing them to some extent, without at least being polite? It should feel unnatural do that.

It does feel unnatural to me. I want to be frugal with compute resource but I then have to make sure I still use appropriate language in emails to humans.

This. Right now, I'm assuming you're all humans, and so are all my coworkers, and the other people driving cars around me and etc. How easy is it to dehumanize actual humans? If I don't try to remain polite in all written English conversations, including the LLMs, that's going to trickle over to the rest of my interactions too. Doesn't mean they deserve it, just that it's a habit I know I need to maintain.

Re: Folk are getting dangerously attached to AI that always tells them they're right

#99

Post by salawat »

AnimalMuppet wrote: post #86
pixl97 wrote: post #56

Earlier quoted context omitted.

>and not some more trustworthy source. What is that more trustworthy source exactly? At least to me it feels like the internet age has eroded most things we considered trustworthy. Behind every thing humans need there is some company or person willing to sell out trustworthiness for an extra dollar. Consumer protections get dumped in favor of more profit. LLMs start feeling more like a dummy than the amount of ill in…

At the moment, maybe Google Search, throwing away the AI response at the top? Or Duck Duck Go, if you don't really trust Google? I can see a day when even that won't be trustworthy, because too much AI slop output will wind up in the search corpus. But I don't think we're there yet.

Google is NN's all the way down these days. There might still be an honest index under it all, but a truly accurate representation of the Web has been effectively outlawed in the U.S. since DMCA.

Re: Folk are getting dangerously attached to AI that always tells them they're right

#100

Post by saghm »

jsw97 wrote: post #40
yarn_ wrote: post #34

Earlier quoted context omitted.

"It would be astonishing if people were able to casually not antropomorphize LLMs" Precisely. Even for technical people, I doubt its possible to totally disallow your own brain from ever, unconciously, treating the entity you're speaking to like a sentient being. Most technical people still will have some emotion in their prompts, say please or thank you, give qualitative feedback for no reason, express anger towards…

Maybe it is a dangerous habit to instruct entities in plain English without anthropomorphizing them to some extent, without at least being polite? It should feel unnatural do that.

Yeah, my instinct is that we're naturally going to have emotions resulting from anything we interact with based on language, and trying to suppress them will likely not be healthy in the long run. I've also seen plenty of instances of people getting upset when someone who isn't a native speaker of their language or even a pet that doesn't speak any language doesn't understand verbal instructions, so there's probably something to be said for learning how to be polite even when experiencing frustration. I've definitely noticed that I'm far more willing to express my annoyance at an LLM than I am another actual human, and this does make me wonder whether this is a habit I should be learning to break sooner rather than later to avoid it having any affect on my overall mindset.
Post reply on HN