Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What OpenAI did when ChatGPT users lost touch with reality

nytimes.com

Post reply on HN
Print view View original

21–30 of 548 posts

Re: What OpenAI did when ChatGPT users lost touch with reality

#21

Post by quitit »

ArcHound wrote: post #3

One of the more disturbing things I read this year was the my boyfriend is AI subreddit. I genuinely can't fathom what is going on there. Seems so wrong, yet no one there seems to care. I worry about the damage caused by these things on distressed people. What can be done?

There are plenty of reasons why having a chatbot partner is a bad idea (especially for young people), but here's just a few:

- The sycophantic and unchallenging behaviours of chatbots leaves a person unconditioned for human interactions. Real relationships have friction, from this we develop important interpersonal skills such as setting boundaries, settling disagreements, building compromise, standing up for oneself, understanding one another, and so on. These also have an effect on one's personal identity and self-value.

- Real relationships have the input from each participant, whereas chatbots are responding to the user's contribution only. The chatbot doesn't have its own life experiences and happenings to bring to the relationship, nor does it instigate autonomously, it's always some kind of structured reply to the user.

- The implication of being fully satisfied by a chatbot is that the person is seeking a partner who does not contribute to the relationship, but rather just an entity that only acts in response to them. It can also be an indication of some kind of problem that the individual needs to work through with why they don't want to seek genuine human connection.

Re: What OpenAI did when ChatGPT users lost touch with reality

#22

Post by thot_experiment »

Caelan Conrad made a few videos on specifically AI encouraging kids to socially isolate and commit suicide. In the videos he reads the final messages aloud for multiple cases, if this isn't your cup of tea there's also the court cases if you would prefer to read the chat logs. It's very harrowing stuff. I'm not trying to make any explicit point here as I haven't really processed this fully enough to have one, but I encourage anyone working in this space to hold this shit in their head at the very least.

https://www.youtube.com/watch?v=hNBoULJkxoU

https://www.youtube.com/watch?v=JXRmGxudOC0

https://www.youtube.com/watch?v=RcImUT-9tb4

Re: What OpenAI did when ChatGPT users lost touch with reality

#23

Post by josh-sematic »

ArcHound wrote: post #3

One of the more disturbing things I read this year was the my boyfriend is AI subreddit. I genuinely can't fathom what is going on there. Seems so wrong, yet no one there seems to care. I worry about the damage caused by these things on distressed people. What can be done?

I hadn’t heard of that until today. Wild, it seems some people report genuinely feeling deeply in love with the personas they’ve crafted for their chatbots. It seems like an incredibly precarious position to be in to have a deep relationship where you have to perpetually pay a 3rd party company to keep it going, and the company may destroy your “partner” or change their personality at a whim. Very “Black Mirror”.

Re: What OpenAI did when ChatGPT users lost touch with reality

#25

Post by ArcHound »

j-pb wrote: post #20
avensec wrote: post #11

Earlier quoted context omitted.

It may not be a concern now, but it comes down to their level of maintaining critical thinking. The risk of epistemic drift, when you have a system that is designed (or reinforced) to empathize with you, can create long-term effects not noticed in any single interaction. Related: "Delusions by design? How everyday AIs might be fuelling psychosis (and what can be done about it)" ( https://doi.org/10.31234/osf.io/cmy7n…

I don't disagree that AI psychosis is real, I've met people who believed that they were going to publish at Neurips due to the nonsense ChatGPT told them, that believed that the UI mockup that claude gave then were actually producing insights into it's inner workings instead of just being blinking SVGs, and I even encountered someone participating at a startup event with an Idea that I'm 100% is AI slop. My point was…

Dear god, there's more! I'll need a drink for this one.

However, I suspect I have better resistance to schizo posts than emotionally weird posts.

Re: What OpenAI did when ChatGPT users lost touch with reality

#26

Post by sunaookami »

blurbleblurble wrote: post #5

The whiplash of carefully filtering out sycophantic behavior from GPT-5 to adding it back in full force for GPT-5.1 is dystopian. We all know what's going on behind the scenes: The investors want their money.

GPT-5 was so good in the first week, just a raw chatbot like GPT-3.5 and GPT-4 were in the beginning and now it has this disgusting "happy" and "comforting" personality and "tuning" it doesn't help one bit, it makes performance way worse and after a few rounds it forgets all instructions. I've already deleted memory, past chats, etc...

Re: What OpenAI did when ChatGPT users lost touch with reality

#27

Post by j-pb »

ArcHound wrote: post #9
j-pb wrote: post #7

Earlier quoted context omitted.

After having spoken with one of the people there I'm a lot less concerned to be honest. They described it as something akin to an emotional vibrator, that they didn't attribute any sentience to, and that didn't trigger their PTSD that they normally experienced when dating men. If AI can provide emotional support and an outlet for survivors who would otherwise not be able to have that kind of emotional need fulfilled,…

phew, that's a healthy start. I am still slightly worried about accepting emotional support from a bot. I don't know if that slope is slippery enough to end in some permanent damage to my relationships and I am honestly not willing to try it at all even. That being said, I am fairly healthy in this regard. I can't imagine how it would go for other people with serious problems.

I completely agree that it is certainly something to be mindful of. It's just that found the people from there were a lot less delusional than the people from e.g. r/artificialsentience, which always believed that AI Moses was giving them some kind of tech revelation though magical alchemical AI symbols.

Re: What OpenAI did when ChatGPT users lost touch with reality

#28

Post by netsharc »

ArcHound wrote: post #9
j-pb wrote: post #7

Earlier quoted context omitted.

After having spoken with one of the people there I'm a lot less concerned to be honest. They described it as something akin to an emotional vibrator, that they didn't attribute any sentience to, and that didn't trigger their PTSD that they normally experienced when dating men. If AI can provide emotional support and an outlet for survivors who would otherwise not be able to have that kind of emotional need fulfilled,…

phew, that's a healthy start. I am still slightly worried about accepting emotional support from a bot. I don't know if that slope is slippery enough to end in some permanent damage to my relationships and I am honestly not willing to try it at all even. That being said, I am fairly healthy in this regard. I can't imagine how it would go for other people with serious problems.

A friend broke up with her partner. She said she was using ChatGPT as a therapist. She showed me a screenshot, ChatGPT wrote "Oh [name], I can feel how raw the pain is!".

WTF, no you don't bot, you're a hunk of metal!

Re: What OpenAI did when ChatGPT users lost touch with reality

#29

Post by notpachet »

avensec wrote: post #11
j-pb wrote: post #7

Earlier quoted context omitted.

After having spoken with one of the people there I'm a lot less concerned to be honest. They described it as something akin to an emotional vibrator, that they didn't attribute any sentience to, and that didn't trigger their PTSD that they normally experienced when dating men. If AI can provide emotional support and an outlet for survivors who would otherwise not be able to have that kind of emotional need fulfilled,…

It may not be a concern now, but it comes down to their level of maintaining critical thinking. The risk of epistemic drift, when you have a system that is designed (or reinforced) to empathize with you, can create long-term effects not noticed in any single interaction. Related: "Delusions by design? How everyday AIs might be fuelling psychosis (and what can be done about it)" ( https://doi.org/10.31234/osf.io/cmy7n…

Wouldn't there necessarily be correlative effects in professional settings a la programming?

Re: What OpenAI did when ChatGPT users lost touch with reality

#30

Post by fullshark »

belval wrote: post #10
ArcHound wrote: post #3

One of the more disturbing things I read this year was the my boyfriend is AI subreddit. I genuinely can't fathom what is going on there. Seems so wrong, yet no one there seems to care. I worry about the damage caused by these things on distressed people. What can be done?

> I worry about the damage caused by these things on distressed people. What can be done? Why? We are gregarious animals, we need social connections. ChatGPT has guardrails that keep this mostly safe and helps with the loneliness epidemic. It's not like people doing this are likely thriving socially in the first place, better with ChatGPT than on some forum à la 4chan that will radicalize them. I feel like this will…

The tech industry's capacity to rationalize anything, including psychosis, as long as it can make money off it is truly incredible. Even the temporarily embarrassed founders that populate this message board do it openly.
Post reply on HN