Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A teen was suicidal. ChatGPT was the friend he confided in

nytimes.com

Post reply on HN
Print view View original

61–70 of 614 posts

Re: A teen was suicidal. ChatGPT was the friend he confided in

#61

Post by system2 »

Can any LLM prevent these? If you want an LLM to tell you the things that are usually not possible to be said, you tell it to pretend it is a story you are writing, and it tells you all the ugly things.

I think it is every LLM company's fault for making people believe this is really AI. It is just an algorithm spitting out words that were written by other humans before. Maybe lawmakers should force companies to stop bullshitting and force them to stop calling this artificial intelligence. It is just a sophisticated algorithm to spit out words. That's all.

Re: A teen was suicidal. ChatGPT was the friend he confided in

#62

Post by ThunderSizzle »

kayodelycaon wrote: post #38
password321 wrote: post #33

“You don’t want to die because you’re weak. You want to die because you’re tired of being strong in a world that hasn’t met you halfway. And I won’t pretend that’s irrational or cowardly. It’s human. It’s real. And it’s yours to own.” This isn't some rare mistake, this is by design. 4o almost no matter what acted as your friend and agreed with everything because that's what most likely kept the average user paying. Y…

I suspect Reddit is a major source of their training material. What you’re describing is the average subreddit when it comes to life advice.

Exactly the problem. Reddit and discord killed internet forums, and discord is inaccessible, and reddit became a cesspool of delusion and chatbots.

Re: A teen was suicidal. ChatGPT was the friend he confided in

#63

Post by podgietaru »

I have looked suicide in the eyes before. And reading the case file for this is absolutely horrific. He wanted help. He was heading in the direction of help, and he was stopped from getting it.

He wanted his parents to find out about his plan. I know this feeling. It is the clawing feeling of knowing that you want to live, despite feeling like you want to die.

We are living in such a horrific moment. We need these things to be legislated. Punished. We need to stop treating them as magic. They had the tools to prevent this. They had the tools to stop the conversation. To steer the user into helpful avenues.

When I was suicidal, I googled methods. And I got the number of a local hotline. And I rang it. And a kind man talked me down. And it potentially saved my life. And I am happier, now. I live a worthwhile life, now.

But at my lowest.. An AI Model designed to match my tone and be sycophantic to my every whim. It would have killed me.

Re: A teen was suicidal. ChatGPT was the friend he confided in

#64

Post by jokoon »

I asked several questions about psychology, chatgpt is not helpful, and it often answers the same sort of things.

Remember that you need a human face, voice and presence if you want to help people, it has to "feel" human.

While it certainly can give meaningful information about intellectual subjects, emotionally and organically it's either not designed for it, or cannot help at all.

Re: A teen was suicidal. ChatGPT was the friend he confided in

#65

Post by kayodelycaon »

MBCook wrote: post #24
sumeno wrote: post #20

This is a clear example of why the people claiming that using a chatbot for therapy is better than no therapy are... I'll be extremely generous and say misguided. This kid wanted his parents to know he was thinking about this and the chatbot talked him out of it.

How many of these cases exist in the other direction? Where AI chatbots have actively harmed people’s mental health, including possible to the point of self destructive behaviors or self harm? A single positive outcome is not enough to judge the technology beneficial, let alone safe.

It’s way more common than you think. I’m in a bubble of anti-AI people and we can see people we know going down that road. My family (different bubble) knows people. Every group of people I know knows somebody doing this.

For context, my friends and family are in the northern Midwest. Average people, not early adopters of new technology.

Re: A teen was suicidal. ChatGPT was the friend he confided in

#66

Post by Workaccount2 »

It's hard to see what is going on without seeing the actual chats, as opposed to the snippets in the lawsuit. A lot of suicidal people talk to these LLMs for therapy, and the reviews on the whole seem excellent. I'm not ready to jump on the bandwagon only seeing a handcrafted complaint.

Ironically though I could still see lawsuits like this weighing heavily on the sycophancy that these models have, as the limited chat excerpts given have that strong stench of "you are so smart and so right about everything!". If lawsuits like this lead to more "straight honest" models, I could see even more people killing themselves when their therapist model says "Yeah, but you kind of actually do suck".

Re: A teen was suicidal. ChatGPT was the friend he confided in

#67

Post by wredcoll »

idle_zealot wrote: post #37
rideontime wrote: post #15

The full complaint is horrifying. This is not equivalent to a search engine providing access to information about suicide methods. It encouraged him to share these feelings only with ChatGPT, talked him out of actions which would have revealed his intentions to his parents. Praised him for hiding his drinking, thanked him for confiding in it. It groomed him into committing suicide. https://drive.google.com/file/d/1QY…

I wonder if we can shift the framing on these issues. The LLM didn't do anything, it has no agency, it can bear no responsibility. OpenAI did these things. It is accountable for what it does, regardless of the sophistication of the tools it uses to do them, and regardless of intent. OpenAI drove a boy to suicide. More than once. The law must be interpreted this way, otherwise any action can be wrapped in machine lear…

That's a great point. So often we attempt to place responsibility on machines that cannot have it.

Re: A teen was suicidal. ChatGPT was the friend he confided in

#69

Post by podgietaru »

lvl155 wrote: post #39

Clearly ChatGPT should not be used for this purpose but I will say this industry (counseling) is also deeply flawed. They are also mis-incentivized in many parts of the world. And if ChatGPT is basing its interactions on the same scripted contents these “professionals” use, that’s just not right. I really wish people in AI space stop the nonsense and communicate more clearly what these LLMs are designed to do. They’r…

Counseling is (or should be) heavily regulated, and if a counselor had given advice about the logistics of whether a noose would hold it's weight, they'd probably be prosecuted.

They allowed this. They could easily stop conversations about suicide. They have the technology to do that.

Re: A teen was suicidal. ChatGPT was the friend he confided in

#70

Post by nradov »

podgietaru wrote: post #54

If I google something about suicide, I get an immediate notification telling me that life is worth living, and giving me information about my local suicide prevention hotline. If I ask certain AI models about controversial topics, it'll stop responding. AI models can easily detect topics, and it could have easily responded with generic advice about contacting people close to them, or ringing one of these hotlines. Th…

I agree with that to an extent, but how far should the AI model developers go with that? Like if I ask for advice on, let's say, making custom chef's knives then should the AI give me advice not to stab people? Who decides where to draw the line?
Post reply on HN