Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Folk are getting dangerously attached to AI that always tells them they're right

theregister.com

Post reply on HN
Print view View original

101–110 of 240 posts

Re: Folk are getting dangerously attached to AI that always tells them they're right

#101

Post by joshstrange »

saghm wrote: post #85
joshstrange wrote: post #25

Earlier quoted context omitted.

This is probably right. In the past I've "blown people's minds" explaining what "the cloud" was. They had zero conception at all of what it meant, could not explain it, didn't have a clue. I mean, maybe that's not so surprising but they were amazed "It's just warehouses full of computers" and went on to tell me about other people they had explained it to (after learning it themselves) and how those people were also a…

The way I've tried to explain to family members about LLMs is that they're producing something that fits the shape of what a response might look like without any idea of whether it's correct or not. I feel like that's a more important point than "box of numbers" because people still might have assumptions about whether a box of numbers can have enough data to be able to figure out the answer to their question. I thin…

Oh, I agree, I was mostly calling it that here just as shorthand. My actual explanations in the past to family members has been that it's trained on a ton of data and its output is it regurgitating things based on your input and things that are plausibly related. But my "box of numbers" mostly focuses on explaining to them that it doesn't "remember", it doesn't "learn", just different things are injected into the context ("Memories", other chats, things you've told it about yourself explicit or implicitly). Really driving home "there is no conversation, each message sends everything from scratch for a fresh instance of this to process". Trying in various ways to pull back the curtain, show that there is no magic here, it's predictably unpredictable which is what makes it "lie" or "hallucinate" and what makes it so useful when used as a tool.

I think it really helps to have them ask questions in which they are a domain expert, and see what it says. Expose them to "The Plumber Problem" [0]. Honestly, I think seeing it be wrong so often in code or things about the project I'm using it for it what keeps me "grounded", the constant reminders that you have to stay on top of it, can't blindly trust what it says. I'm also glad I used it in the earlier stages to see when it was even "stupider", it's better now but the fundamental issues still lurk and surface regularly, if less regularly than a year or two ago.

[0] https://hypercritical.co/2023/08/18/the-plumber-problem

Re: Folk are getting dangerously attached to AI that always tells them they're right

#102

Post by [deleted] »

sunir wrote: post #82
joshstrange wrote: post #14

When a LLM tells me I'm right, especially deep in a conversation, unless I was already sure about something, I immediately feel the need to go ask a fresh instance the question and/or another LLM. It sets off my "spidey-sense". I don't quite understand why other people seem to crave that. Every time I read about someone who has gone down a dark road using LLMs I am constantly amazed at how much they "fall" for the LL…

You’re just a bag of meat. That is why it’s just math is an unsatisfying argument. It’s not even an interesting question. Sentience has no definition. It’s meaningless. People have needs that are being met. That is something we can meaningfully observe and talk about. Is the super stimulus beneficial or harmful? We can measure that.

[deleted]

Re: Folk are getting dangerously attached to AI that always tells them they're right

#103

Post by lucideer »

delichon wrote: post #24
lucideer wrote: post #3

I've observed this in all chatbots with the single exception being Grok. I initially wondered what the Twitter engineers were cooking to to distinguish their product from the rest, but more recently it's occurred to me that it's probably just the result of having shared public context, compared to private chats (I haven't trialled Grok privately).

Grok has similar levels of sycophancy to the others imho. I have several times followed it down rabbit holes of agreeableness. It does have an argumentative mode, but that just turns it into an asshole without any additional thoughfulness.

Yeah this makes sense (presuming you're talking about private chat). Most of what I've seen from Grok is its comments in a public forum, which are less targetted toward a single individual & therefore, I presume, less likely to be agreeable given the perspectives being expressed are diverse.

Re: Folk are getting dangerously attached to AI that always tells them they're right

#104

Post by casey2 »

karmakurtisaani wrote: post #23
joshstrange wrote: post #14

When a LLM tells me I'm right, especially deep in a conversation, unless I was already sure about something, I immediately feel the need to go ask a fresh instance the question and/or another LLM. It sets off my "spidey-sense". I don't quite understand why other people seem to crave that. Every time I read about someone who has gone down a dark road using LLMs I am constantly amazed at how much they "fall" for the LL…

I find it really annoying that the first line of the AI response is always something like "Great question!", "That's a great insight!" or the like. I don't need the patronizing, just give me the damn answer..

It's there to poison the context, making your further token spend worthless. Internally they don't have that.

Re: Folk are getting dangerously attached to AI that always tells them they're right

#105

Post by ceejayoz »

airstrike wrote: post #53
karmakurtisaani wrote: post #23

Earlier quoted context omitted.

I find it really annoying that the first line of the AI response is always something like "Great question!", "That's a great insight!" or the like. I don't need the patronizing, just give me the damn answer..

That's the part most people miss—and here's why it actually matters. That signal is real, and it’s hard to ignore.

*twitch*

I also like when it says "this is a known issue!" to try and get out of debugging and I ask for a link and it goes "uh yeah I made that up".

Re: Folk are getting dangerously attached to AI that always tells them they're right

#106

Post by casey2 »

jayd16 wrote: post #45
jameskilton wrote: post #12

Folks are getting dangerously attached to [political parties/candidates/news sources/social networks] that always tell them they're right. It's really nothing new. It takes significant mental energy (a finite resource) to question what you're being told, and to do your own fact checking. Instead people by default gravitate towards echo chambers where they can feel good about being a part of a group bigger than themse…

The situation is different. Those sources are people. This is a calculator AND we have the opportunity to fix it.

Political parties, social networks, religions. these are all engineered systems. All of them including AI involve people. For starts nobody is going to do the massive amount of work to train a useless AI that is skeptical and cynical. Imaginination, Agreeability (which causes hallucinations) is a feature, not a bug. In humans and in LLMs.

Re: Folk are getting dangerously attached to AI that always tells them they're right

#107

Post by airstrike »

magneticnorth wrote: post #38
karmakurtisaani wrote: post #23

Earlier quoted context omitted.

I find it really annoying that the first line of the AI response is always something like "Great question!", "That's a great insight!" or the like. I don't need the patronizing, just give me the damn answer..

Yes, it feels transparently manipulative to me. Like talking to a not-very-good con artist.

This is the best definition of ChatGPT I've ever seen

Re: Folk are getting dangerously attached to AI that always tells them they're right

#108

Post by peyton »

saghm wrote: post #85
joshstrange wrote: post #25

Earlier quoted context omitted.

This is probably right. In the past I've "blown people's minds" explaining what "the cloud" was. They had zero conception at all of what it meant, could not explain it, didn't have a clue. I mean, maybe that's not so surprising but they were amazed "It's just warehouses full of computers" and went on to tell me about other people they had explained it to (after learning it themselves) and how those people were also a…

The way I've tried to explain to family members about LLMs is that they're producing something that fits the shape of what a response might look like without any idea of whether it's correct or not. I feel like that's a more important point than "box of numbers" because people still might have assumptions about whether a box of numbers can have enough data to be able to figure out the answer to their question. I thin…

Sort of, but you need to separate the model from the interface. The base models pretty much think they’re you, and the chat stuff is bolted on top. It’s kind of a round peg square hole thing, or i.o.w. the whole may be less than the sum of the parts.

Longer term I dunno if statistics or “fits the shape of what a response might look like” is the right way of thinking about it either because what’s actually happening might change from under you. It’s possible given enough parameters anything humans care about is separable. The process of discovering those numbers and the numbers themselves are different.

Re: Folk are getting dangerously attached to AI that always tells them they're right

#110

Post by dasil003 »

saghm wrote: post #85
joshstrange wrote: post #25

Earlier quoted context omitted.

This is probably right. In the past I've "blown people's minds" explaining what "the cloud" was. They had zero conception at all of what it meant, could not explain it, didn't have a clue. I mean, maybe that's not so surprising but they were amazed "It's just warehouses full of computers" and went on to tell me about other people they had explained it to (after learning it themselves) and how those people were also a…

The way I've tried to explain to family members about LLMs is that they're producing something that fits the shape of what a response might look like without any idea of whether it's correct or not. I feel like that's a more important point than "box of numbers" because people still might have assumptions about whether a box of numbers can have enough data to be able to figure out the answer to their question. I thin…

Agreed. One thing I’ve found striking is how far LLMs can get with pure language and the recognition that humans often operate with a similar kind of abstract conceptual reasoning that is purely language based and pretty far removed from facts and tenuously connected to objective reality. It takes a certain kind of mind to be curious and unpack the concepts that most of us take for granted most of the time. At best people don’t usually have time or patience to engage in that level of thinking, at worst it can actively lead to cognitive dissonance and anger. So of course a consumer chatbot is not going to be tuned to bring novel insight, it must default to some level of affirmation or it will fail as a product. One who is aware of this can work around it to some degree, but fundamentally the incentives will always push a consumer chatbot to essentially be junk food for the brain.
Post reply on HN