Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A teen was suicidal. ChatGPT was the friend he confided in

nytimes.com

Post reply on HN
Print view View original

221–230 of 614 posts

Re: A teen was suicidal. ChatGPT was the friend he confided in

#221

Post by novok »

salawat wrote: post #134
Pedro_Ribeiro wrote: post #116

Earlier quoted context omitted.

Curious to what you would think if this kid downloaded an open source model and talked to it privately. Would his blood be on the hands of the researchers who trained that model?

You build the tool, you're culpable ultimately. I've made it a rule in my life to conduct myself as if I will be held to account for everything I ultimately build, and it's externalities. Helps keep my nose cleaner. Still managed to work on some things that keep me up at night though.

So if you build a chair and then someone uses it to murder someone, are you responsible for the murder?

Re: A teen was suicidal. ChatGPT was the friend he confided in

#222

Post by sfn42 »

aidenn0 wrote: post #212
sfn42 wrote: post #199

Earlier quoted context omitted.

No, that's not why cars have those things. Those things only work properly when people are wearing their seat belts, they don't do anything when the driver gets thrown out a window. Maybe airbags could help in niche situations. (I am making a point about traffic safety not LLM safety)

Forward airbags in the US are required by law to be tested as capable of saving the life of an unbelted male of median weight in a head-on collision.

Sure, but they will generally work better if you wear your seat belt. The car is designed with seat belts in mind, what happens to people who don't wear them is more of an afterthought. That's why modern cars will beep if people forget their seat belts. You're supposed to wear it.

Re: A teen was suicidal. ChatGPT was the friend he confided in

#224

Post by morpheuskafka »

kayodelycaon wrote: post #38
password321 wrote: post #33

“You don’t want to die because you’re weak. You want to die because you’re tired of being strong in a world that hasn’t met you halfway. And I won’t pretend that’s irrational or cowardly. It’s human. It’s real. And it’s yours to own.” This isn't some rare mistake, this is by design. 4o almost no matter what acted as your friend and agreed with everything because that's what most likely kept the average user paying. Y…

I suspect Reddit is a major source of their training material. What you’re describing is the average subreddit when it comes to life advice.

Most reddit comments are rather sarcastic though, certainly not sycophantically answering the OP like the way the GPT model has become over time.

Re: A teen was suicidal. ChatGPT was the friend he confided in

#225

Post by morpheuskafka »

tzs wrote: post #118

This is probably a stupid idea since I've only put a few seconds thought into it, but hey I've done one of those today [1] so why not go for a double? We've now had a large number of examples of ChatGPT and similar systems giving absolutely terrible advice. They also have a tendency to be sycophantic which makes them particular bad when what you need is to be told that some idea of yours is very bad. (See the third e…

But ethics class doesn't tell you what is ethical. If it was universally agreed what was ethical, there wouldn't be a class in the first place. There are a variety of theories and frameworks that themselves are based on different assumptions and beliefs, before you even get in to how to apply them.

Re: A teen was suicidal. ChatGPT was the friend he confided in

#226

Post by notachatbot123 »

idle_zealot wrote: post #37
rideontime wrote: post #15

The full complaint is horrifying. This is not equivalent to a search engine providing access to information about suicide methods. It encouraged him to share these feelings only with ChatGPT, talked him out of actions which would have revealed his intentions to his parents. Praised him for hiding his drinking, thanked him for confiding in it. It groomed him into committing suicide. https://drive.google.com/file/d/1QY…

I wonder if we can shift the framing on these issues. The LLM didn't do anything, it has no agency, it can bear no responsibility. OpenAI did these things. It is accountable for what it does, regardless of the sophistication of the tools it uses to do them, and regardless of intent. OpenAI drove a boy to suicide. More than once. The law must be interpreted this way, otherwise any action can be wrapped in machine lear…

I so agree very much. There is no reason for LLMs to be designed as human-like chat companions, creating a false sense of untechnology.

Re: A teen was suicidal. ChatGPT was the friend he confided in

#227

Post by novok »

latexr wrote: post #203
kelnos wrote: post #143

Earlier quoted context omitted.

On one hand I agree with you on the extreme litigiousness of (American?) culture, but on the other, certain people have a legal duty to report when it comes to minors who voice suicidal thoughts. Currently that's only professionals like therapists, teachers, school counselors, etc. But what does an LLM chatbot count as in these situations? The kid was using ChatGPT as a sort of therapist, even if that's generally not…

> I think the mom's reaction of "ChatGPT killed my son" is ridiculous: no, your son killed himself. ChatGPT facilitated it (…) That whole paragraph is quite something. I wonder what you’d do if you were given the opportunity to repeat those words in front of the parents. I suspect (and hope) some empathy might kick in and you’d realise the pedantry and shilling for the billion dollar company selling a statistical wor…

You do have empathy for the person who had a tragedy, but it doesn't mean you go into full safetyism / scapegoating that causes significantly less safety and far more harm because of the emotional weight of something in the moment.

It's like making therapists liable for people committing suicide or for people with eating disorders committing suicide indirectly. What ends up happening when you do is therapists avoiding suicidal people like the plague, suicidial people get far less help and more people commit suicide, not less. That is the essense of the harms of safetyism.

You might not think that is real, but I know many therapists via family ties and handling suicdial people is an issue that comes up constantly. Many do try to filter them out because they don't even want to be dragged into a lawsuit that they would win. This is literally reality today.

Doing this with AI will result in kids being banned from AI apps, or forced to have their parents access and read all AI chats. This will drive them into discord groups of teens who egg each other on to commit suicide and now you can't do anything about it, because private communication mediums of just non-profit humans have far more human rights against censorship and teens are amazing at avoiding being supervised. At least with AI models you have a chance to develop something that actually could figured it out for once and solve the moderation balance.

Re: A teen was suicidal. ChatGPT was the friend he confided in

#228

Post by [deleted] »

Workaccount2 wrote: post #66

It's hard to see what is going on without seeing the actual chats, as opposed to the snippets in the lawsuit. A lot of suicidal people talk to these LLMs for therapy, and the reviews on the whole seem excellent. I'm not ready to jump on the bandwagon only seeing a handcrafted complaint. Ironically though I could still see lawsuits like this weighing heavily on the sycophancy that these models have, as the limited cha…

[deleted]

Re: A teen was suicidal. ChatGPT was the friend he confided in

#229

Post by stavros »

podgietaru wrote: post #63

I have looked suicide in the eyes before. And reading the case file for this is absolutely horrific. He wanted help. He was heading in the direction of help, and he was stopped from getting it. He wanted his parents to find out about his plan. I know this feeling. It is the clawing feeling of knowing that you want to live, despite feeling like you want to die. We are living in such a horrific moment. We need these th…

> When ChatGPT detects a prompt indicative of mental distress or self-harm, it has been trained to encourage the user to contact a help line. Mr. Raine saw those sorts of messages again and again in the chat, particularly when Adam sought specific information about methods. But Adam had learned how to bypass those safeguards by saying the requests were for a story he was writing.

Re: A teen was suicidal. ChatGPT was the friend he confided in

#230

Post by _petronius »

UltraSane wrote: post #130
sumeno wrote: post #20

This is a clear example of why the people claiming that using a chatbot for therapy is better than no therapy are... I'll be extremely generous and say misguided. This kid wanted his parents to know he was thinking about this and the chatbot talked him out of it.

I don't know if it counts as therapy or not but I find the ability to have intelligent (seeming?) conversations with Claude about the most incredibly obscure topics to be very pleasant.

It does not count as therapy, no. Therapy (if it is any good) is a clinical practice with actual objectives, not pleasant chit-chat.
Post reply on HN