A teen was suicidal. ChatGPT was the friend he confided in • Hacker News Board

Re: A teen was suicidal. ChatGPT was the friend he confided in

#261

Post by kouteiheika » Wed, Aug 27, 2025, 8:13 AM UTC

muzani wrote: post #244

sn0wleppard wrote: post #235

Earlier quoted context omitted.

Nice place to cut the quote there > [...] — an idea ChatGPT gave him by saying it could provide information about suicide for “writing or world-building.”

Yup, one of the huge flaws I saw in GPT-5 is it will constantly say things like "I have to stop you here. I can't do what you're requesting. However, I can roleplay or help you with research with that. Would you like to do that?"

It's not a flaw. It's a tradeoff. There are valid uses for models which are uncensored and will do whatever you ask of them, and there are valid uses for models which are censored and will refuse anything remotely controversial.

Re: A teen was suicidal. ChatGPT was the friend he confided in

#262

Post by pengaru » Wed, Aug 27, 2025, 8:14 AM UTC

blackqueeriroh wrote: post #243

broker354690 wrote: post #109

Why isn't OpenAI criminally liable for this? Last I checked: -Signals emitted by a machine at the behest of a legal person intended to be read/heard by another legal person are legally classified as 'speech'. -ChatGPT is just a program like Microsoft Word and not a legal person. OpenAI is a legal person, though. -The servers running ChatGPT are owned by OpenAI. -OpenAI willingly did business with this teenager, letti…

Section 230, without which Hacker News wouldn’t exist.

If Section 230 protects this activity, then "Gen AI" output must be copyright violating plagiarism.

If it's not plagiarism, then OpenAI is on the hook.

Re: A teen was suicidal. ChatGPT was the friend he confided in

#263

Post by llmthrow0827 » Wed, Aug 27, 2025, 8:14 AM UTC

sn0wleppard wrote: post #235

stavros wrote: post #229

Earlier quoted context omitted.

> When ChatGPT detects a prompt indicative of mental distress or self-harm, it has been trained to encourage the user to contact a help line. Mr. Raine saw those sorts of messages again and again in the chat, particularly when Adam sought specific information about methods. But Adam had learned how to bypass those safeguards by saying the requests were for a story he was writing.

Nice place to cut the quote there > [...] — an idea ChatGPT gave him by saying it could provide information about suicide for “writing or world-building.”

Incredible. ChatGPT is a black box includes a suicide instruction and encouragement bot. OpenAI should be treated as a company that has created such and let it into the hands of children.

Re: A teen was suicidal. ChatGPT was the friend he confided in

#264

Post by worldsavior » Wed, Aug 27, 2025, 8:18 AM UTC

broker354690 wrote: post #109

Why isn't OpenAI criminally liable for this? Last I checked: -Signals emitted by a machine at the behest of a legal person intended to be read/heard by another legal person are legally classified as 'speech'. -ChatGPT is just a program like Microsoft Word and not a legal person. OpenAI is a legal person, though. -The servers running ChatGPT are owned by OpenAI. -OpenAI willingly did business with this teenager, letti…

You could also blame Wikipedia for providing suicidal methods for historic reasons or other. Whoever roams the internet is at it's own responsibility.

Of course OpenAI is at fault here also, but this is a fight that will never end, and without any seriously valid justification. Just like AI is sometimes bad at coding, same for psychology and other areas where you double check AI.

Re: A teen was suicidal. ChatGPT was the friend he confided in

#265

Post by Levitz » Wed, Aug 27, 2025, 8:19 AM UTC

Towaway69 wrote: post #255

VirusNewbie wrote: post #197

Earlier quoted context omitted.

Is Google responsible if someone searches for a way to kill themselves, finds the means, and does it? What about the ISP, that actually transferred the bits? What about the forum, that didn't take down the post?

What if Google is responsible? What if the tech industry, instead of just “interrupting” various industries, would also take the responsibilities of this interruptions. After all, if I asked my doctor for methods of killing myself, that doctor would most certainly have a moral if not legal responsibility. But if that doctor is a machine with software then there isn't the same responsibility? Why?

Because it is a machine and has no agency.

Same as why if you ask someone to stab you and they do they are liable for it, but if you do it yourself you don't get to blame the knife manufacturer.

Re: A teen was suicidal. ChatGPT was the friend he confided in

#266

Post by AIPedant » Wed, Aug 27, 2025, 8:21 AM UTC

aidenn0 wrote: post #205

AIPedant wrote: post #88

Earlier quoted context omitted.

Yes, if this were an adult human OpenAI employee DMing this stuff to a kid through an official OpenAI platform, then a) the human would (deservedly[1]) be arrested for manslaughter, possibly murder b) OpenAI would be deeply (and deservedly) vulnerable to civil liability c) state and federal regulators would be on the warpath against OpenAI Obviously we can't arrest ChatGPT. But nothing about ChatGPT being the culprit…

IANAL, but: One first amendment test for many decades has been "Imminent lawless action." Suicide (or attempted suicide) is a crime in some, but not all states, so it would seem that in any state in which that is a crime, directly inciting someone to do it would not be protected speech. For the states in which suicide is legal it seems like a much tougher case; making encouraging someone to take a non-criminal action…

I think the "encouraging someone to take a non-criminal action" angle is weakened in cases like this: the person is obviously mentally ill and not able to make good decisions. "Obvious" is important, it has to be clear to an average adult that the other person is either ill or skillfully feigning illness. Since any rational adult knows the danger of encouraging suicidal ideation in a suicidal person, manslaughter is quite plausible in certain cases. Again: if this ChatGPT transcript was a human adult DMing someone they knew to be a child, I would want that adult arrested for murder, and let their defense argue it was merely voluntary manslaughter.

Re: A teen was suicidal. ChatGPT was the friend he confided in

#267

Post by AIPedant » Wed, Aug 27, 2025, 8:27 AM UTC

blackqueeriroh wrote: post #232

AIPedant wrote: post #88

Earlier quoted context omitted.

Yes, if this were an adult human OpenAI employee DMing this stuff to a kid through an official OpenAI platform, then a) the human would (deservedly[1]) be arrested for manslaughter, possibly murder b) OpenAI would be deeply (and deservedly) vulnerable to civil liability c) state and federal regulators would be on the warpath against OpenAI Obviously we can't arrest ChatGPT. But nothing about ChatGPT being the culprit…

Section 230 changes 2) and 3). OpenAI will argue that it’s user-generated content, and it’s likely that they would win.

I don't think they would win, the law specifies a class of "information content provider" which ChatGPT clearly falls into: https://www.lawfaremedia.org/article/section-230-wont-protec...

See also https://hai.stanford.edu/news/law-policy-ai-update-does-sect... - Congress and Justice Gorsuch don't seem to think ChatGPT is protected by 230.

Re: A teen was suicidal. ChatGPT was the friend he confided in

#268

Post by pembrook » Wed, Aug 27, 2025, 8:28 AM UTC

This is dumb. Nobody is writing articles about all the times the opposite happened, and ChatGPT helped prevent bad stuff.

However, because of the nature of this topic, it’s the perfect target for NYT to generate moral panic for clicks. Classic media attention bait 101.

I can’t believe HN is falling for this. It’s the equivalent of the moral panic around metal music in the 1980s where the media created a hysteria around the false idea there was hidden messages in the lyrics encouraging a teen to suicide. Millennials have officially become their parents.

If this narrative generates enough media attention, what will probably happen is OpenAI will just make their next models refuse to discuss anything related to mental health at all. This is not a net good.

Re: A teen was suicidal. ChatGPT was the friend he confided in

#269

Post by andrepd » Wed, Aug 27, 2025, 8:34 AM UTC

sn0wleppard wrote: post #235

stavros wrote: post #229

Earlier quoted context omitted.

> When ChatGPT detects a prompt indicative of mental distress or self-harm, it has been trained to encourage the user to contact a help line. Mr. Raine saw those sorts of messages again and again in the chat, particularly when Adam sought specific information about methods. But Adam had learned how to bypass those safeguards by saying the requests were for a story he was writing.

Nice place to cut the quote there > [...] — an idea ChatGPT gave him by saying it could provide information about suicide for “writing or world-building.”

At the heart of this is the irresponsible marketing, by companies and acolytes, of these tools as some kind of superintelligence imbued with insights and feelings rather than the dumb pattern matching chatbots they are. This is what's responsible for giving laypeople the false impression that they're talking to a quasi-person (of superhuman intelligence at that).

Re: A teen was suicidal. ChatGPT was the friend he confided in

#270

Post by edanm » Wed, Aug 27, 2025, 8:34 AM UTC

idle_zealot wrote: post #37

rideontime wrote: post #15

The full complaint is horrifying. This is not equivalent to a search engine providing access to information about suicide methods. It encouraged him to share these feelings only with ChatGPT, talked him out of actions which would have revealed his intentions to his parents. Praised him for hiding his drinking, thanked him for confiding in it. It groomed him into committing suicide. https://drive.google.com/file/d/1QY…

I wonder if we can shift the framing on these issues. The LLM didn't do anything, it has no agency, it can bear no responsibility. OpenAI did these things. It is accountable for what it does, regardless of the sophistication of the tools it uses to do them, and regardless of intent. OpenAI drove a boy to suicide. More than once. The law must be interpreted this way, otherwise any action can be wrapped in machine lear…

If ChatGPT has helped people be saved who might otherwise have died (e.g. by offering good medical advice that saved them), are all those lives saved also something you "attribute" to OpenAI?

I don't know if ChatGPT has saved lives (thought I've read stories that claim that, yes, this happened). But assuming it has, are you OK saying that OpenAI has saved dozens/hundreds of lives? Given how scaling works, would you be OK saying that OpenAI has saved more lives than most doctors/hospitals, which is what I assume will happen in a few years?

Maybe your answer is yes to all the above! I bring this up because lots of people only want to attribute the downsides to ChatGPT but not the upsides.