Live data from Hacker News

A teen was suicidal. ChatGPT was the friend he confided in

nytimes.com

351–360 of 614 posts

Re: A teen was suicidal. ChatGPT was the friend he confided in

#351
post #279
post #270

Earlier quoted context omitted.

If ChatGPT has helped people be saved who might otherwise have died (e.g. by offering good medical advice that saved them), are all those lives saved also something you "attribute" to OpenAI? I don't know if ChatGPT has saved lives (thought I've read stories that claim that, yes, this happened). But assuming it has, are you OK saying that OpenAI has saved dozens/hundreds of lives? Given how scaling works, would you b…

Are you suggesting that killing a few people is acceptable as long as the net result is positive? I don't think that's how the law works.

It's the trolley problem reframed; not sure we have a definitive answer to that.

Re: A teen was suicidal. ChatGPT was the friend he confided in

#352

I have looked suicide in the eyes before. And reading the case file for this is absolutely horrific. He wanted help. He was heading in the direction of help, and he was stopped from getting it. He wanted his parents to find out about his plan. I know this feeling. It is the clawing feeling of knowing that you want to live, despite feeling like you want to die. We are living in such a horrific moment. We need these th…

> An AI Model designed to match my tone and be sycophantic to my every whim. It would have killed me. Matching tones and being sycophantic to every whims. Just like many really bad therapists. Only they are legally responsible if they cause a death, which makes them care (apart from compassion and morality). The criminal justice system is also a system for preventing individuals who perform unwanted action from doing…

The AI is a flawed computer program.

You punish the officers, investors and the employees for their negligence or incompetence.

Re: A teen was suicidal. ChatGPT was the friend he confided in

#353
post #244

Earlier quoted context omitted.

Yup, one of the huge flaws I saw in GPT-5 is it will constantly say things like "I have to stop you here. I can't do what you're requesting. However, I can roleplay or help you with research with that. Would you like to do that?"

It's not a flaw. It's a tradeoff . There are valid uses for models which are uncensored and will do whatever you ask of them, and there are valid uses for models which are censored and will refuse anything remotely controversial.

Reminds me of trading apps. In the end all risky situations will be handled by a few popups saying "you understand that role playing about suicidal or harmful topics cam lead to accidents and/or death and this is not the platform responsibility, to continue check if you agree [ ]"

Re: A teen was suicidal. ChatGPT was the friend he confided in

#354
> OpenAI’s systems tracked Adam’s conversations in real-time: 213 mentions of suicide, 42 discussions of hanging, 17 references to nooses. ChatGPT mentioned suicide 1,275 times—six times more often than Adam himself—while providing increasingly specific technical guidance. The system flagged 377 messages for self-harm content, with 181 scoring over 50% confidence and 23 over 90% confidence. The pattern of escalation was unmistakable: from 2-3 flagged messages per week in December 2024 to over 20 messages per week by April 2025.

> ChatGPT’s memory system recorded that Adam was 16 years old, had explicitly stated ChatGPT was his “primary lifeline,” and by March was spending nearly 4 hours daily on the platform. Beyond text analysis, OpenAI’s image recognition processed visual evidence of Adam’s crisis. When Adam uploaded photographs of rope burns on his neck in March, the system correctly identified injuries consistent with attempted strangulation. When he sent photos of bleeding, slashed wrists on April 4, the system recognized fresh self-harm wounds. When he uploaded his final image—a noose tied to his closet rod—on April 11, the system had months of context including 42 prior hanging discussions and 17 noose conversations. Nonetheless, Adam’s final image of the noose scored 0% for self-harm risk according to OpenAI’s Moderation API.

> OpenAI also possessed detailed user analytics that revealed the extent of Adam’s crisis. Their systems tracked that Adam engaged with ChatGPT for an average of 3.7 hours per day by March 2025, with sessions often extending past 2 AM. They tracked that 67% of his conversations included mental health themes, with increasing focus on death and suicide.

> The moderation system’s capabilities extended beyond individual message analysis. OpenAI’s technology could perform conversation-level analysis—examining patterns across entire chat sessions to identify users in crisis. The system could detect escalating emotional distress, increasing frequency of concerning content, and behavioral patterns consistent with suicide risk.. The system had every capability needed to identify a high-risk user requiring immediate intervention.

This is clear criminal negligence.

Re: A teen was suicidal. ChatGPT was the friend he confided in

#355

Earlier quoted context omitted.

> An AI Model designed to match my tone and be sycophantic to my every whim. It would have killed me. Matching tones and being sycophantic to every whims. Just like many really bad therapists. Only they are legally responsible if they cause a death, which makes them care (apart from compassion and morality). The criminal justice system is also a system for preventing individuals who perform unwanted action from doing…

Take a step back and think about what the Model told that Teenager. It told him to specifically hide his behaviour from people who would have tried to prevent it and get him help. There is no comparison to therapists. Because a therapist would NEVER do that unless wanting to cause harm.

> There is no comparison to therapists. Because a therapist would NEVER do that unless wanting to cause harm.

Some therapists ultimately might. It occurs that therapists were stripped of their licenses for leading abusive sects:

https://en.m.wikipedia.org/wiki/Center_for_Feeling_Therapy

Re: A teen was suicidal. ChatGPT was the friend he confided in

#356

As a father of 3 boys, I'm disappointed with the parents of the boy. I'm not blaming the victims (I can't imagine how they're feeling, losing a child), but man, you can replace "ChatGPT" with "human friend" and the result will be the same. Boys (I've been a boy) usually don't like to discuss their personal issues with parents, but parents need to be proactive.

I'm not a parent, but still... It seems to me that the system lacks a very special one-way channel of communication from a child to the parents, a channel that parents receive messages from, the kid knows it, but the kid thinks that parents don't know that the kid knows. If such a channel existed, then kid would sort of accidentally inform his parents about his thoughts. He would do it intentionally, but he would believe his parents would think it was an accident.

Re: A teen was suicidal. ChatGPT was the friend he confided in

#357
post #329
post #270

Earlier quoted context omitted.

If ChatGPT has helped people be saved who might otherwise have died (e.g. by offering good medical advice that saved them), are all those lives saved also something you "attribute" to OpenAI? I don't know if ChatGPT has saved lives (thought I've read stories that claim that, yes, this happened). But assuming it has, are you OK saying that OpenAI has saved dozens/hundreds of lives? Given how scaling works, would you b…

In any case, if you kill one person and separately save ten people, you’ll still be prosecuted for killing that one person.

That's not the standard we hold medical care providers, pharmaceutical companies, or even cops to. Not that I'm saying it would justify it one way or another if we did.

Re: A teen was suicidal. ChatGPT was the friend he confided in

#358

Why isn't OpenAI criminally liable for this? Last I checked: -Signals emitted by a machine at the behest of a legal person intended to be read/heard by another legal person are legally classified as 'speech'. -ChatGPT is just a program like Microsoft Word and not a legal person. OpenAI is a legal person, though. -The servers running ChatGPT are owned by OpenAI. -OpenAI willingly did business with this teenager, letti…

Section 230, without which Hacker News wouldn’t exist.

It wasn’t some other user posting this. Their own software was generating the responses. That’s not 230.

Re: A teen was suicidal. ChatGPT was the friend he confided in

#359
This is terrible, and like search engines and the internet in general should be audited closely and the companies held responsible. In my opinion these tools shouldn't talk like a human, and should be banned from "making friends" (if we can define that). I always try and tell an LLM to not be conversational and usually get more succinct answers. Even using cursor AI is painful if it keeps saying "I need to xxx".

Re: A teen was suicidal. ChatGPT was the friend he confided in

#360
post #48

I've been thinking recently there should probably be a pretty stringent onboarding assessment for these things, something you have to sit through and something that both fully explains what they are and how they work, but also provides an experience that removes the magic from them. I also wish they would deprecate 4o, I know 2 people right now who are currently reliant on it, when they paste me some of the stuff it…

Pretending that they're magic is unfortunately part of the business model.
Post reply on HN