So, be more skeptical
That's like saying "so, exercise more" upon the invention of fast food. Maybe you will, that's great. But society is going to be rewritten by the lazy and we all will have to deal with the side effects.
Folk are getting dangerously attached to AI that always tells them they're right
11–20 of 240 posts
Re: Folk are getting dangerously attached to AI that always tells them they're right
#12Folks are getting dangerously attached to [political parties/candidates/news sources/social networks] that always tell them they're right.
It's really nothing new. It takes significant mental energy (a finite resource) to question what you're being told, and to do your own fact checking. Instead people by default gravitate towards echo chambers where they can feel good about being a part of a group bigger than themselves, and can spend their limited energy towards what really matters in their lives.
Re: Folk are getting dangerously attached to AI that always tells them they're right
#13Isn't this just Dale Carnegie 101? I've certainly never had a salesperson tell me that I'm 100% wrong and being a fool. And, tbh, I often try to remember to do the same.
The attachment such feedback creates must be why marketing people are universally beloved.
Re: Folk are getting dangerously attached to AI that always tells them they're right
#14When a LLM tells me I'm right, especially deep in a conversation, unless I was already sure about something, I immediately feel the need to go ask a fresh instance the question and/or another LLM. It sets off my "spidey-sense".
I don't quite understand why other people seem to crave that. Every time I read about someone who has gone down a dark road using LLMs I am constantly amazed at how much they "fall" for the LLM, often believing it's sentient. It's just a box of numbers, really cool numbers, with really cool math, that can do really cool things, but still just numbers.
Re: Folk are getting dangerously attached to AI that always tells them they're right
#15Isn't this just Dale Carnegie 101? I've certainly never had a salesperson tell me that I'm 100% wrong and being a fool. And, tbh, I often try to remember to do the same.
Dale Carnegie wasn't writing about LLMs and this isn't a salesperson, so no, it's not just Dale Carnegie 101.
Re: Folk are getting dangerously attached to AI that always tells them they're right
#16When a LLM tells me I'm right, especially deep in a conversation, unless I was already sure about something, I immediately feel the need to go ask a fresh instance the question and/or another LLM. It sets off my "spidey-sense". I don't quite understand why other people seem to crave that. Every time I read about someone who has gone down a dark road using LLMs I am constantly amazed at how much they "fall" for the LL…
I think it's basically equal to End of Line when it comes to an LLM. It means they have nothing else to add, there's zero context for them to draw from, and they've exhausted the probability chain you've been following; but they're creating to generate 'next token' and positive renforcement is _how they are trained_ in many cases so the token of choice would naturally be how they're trained, since it's a probability engine but it doesn't know the difference between the instruction and the output.
So, "great idea" is coming from the renforcement learning instruction rather than the answer portion of the generation.
Re: Folk are getting dangerously attached to AI that always tells them they're right
#17Folks are getting dangerously attached to [political parties/candidates/news sources/social networks] that always tell them they're right. It's really nothing new. It takes significant mental energy (a finite resource) to question what you're being told, and to do your own fact checking. Instead people by default gravitate towards echo chambers where they can feel good about being a part of a group bigger than themse…
> It's really nothing new.
I disagree. What's new is that this flattery is individually, personally targeted. The AI user is given the impression that they're having a back-and-forth conversation with a single trusted friend.
You don't have the same personal experience passively consuming political mass media.
Re: Folk are getting dangerously attached to AI that always tells them they're right
#18[dead]
Re: Folk are getting dangerously attached to AI that always tells them they're right
#19When a LLM tells me I'm right, especially deep in a conversation, unless I was already sure about something, I immediately feel the need to go ask a fresh instance the question and/or another LLM. It sets off my "spidey-sense". I don't quite understand why other people seem to crave that. Every time I read about someone who has gone down a dark road using LLMs I am constantly amazed at how much they "fall" for the LL…
Life in the moment is a lot easier if you don't second-guess yourself. I think this is why many people (and probably ~all people, if tired) crave simplistic solutions.
I like to make a subagent take the "devil's advocate" take on a subject. It usually does all the arguing for me as to why the main agent has it wrong. Commonly results in better decisions that I'd have made alone.
Asking the agent to interview on why I disagree helps too but is more effort.
Re: Folk are getting dangerously attached to AI that always tells them they're right
#20https://arxiv.org/abs/2602.14270
related: if you suggest a hypothesis then you'll get biased results (iow, you'll think you're right, but the true information is hidden)