A teen was suicidal. ChatGPT was the friend he confided in

Post by dragonwriter » Wed, Aug 27, 2025, 5:49 AM UTC

AnIrishDuck wrote: post #168

kgeist wrote: post #154

Earlier quoted context omitted.

The kid intentionally bypassed the safeguards: >When ChatGPT detects a prompt indicative of mental distress or self-harm, it has been trained to encourage the user to contact a help line. Mr. Raine saw those sorts of messages again and again in the chat, particularly when Adam sought specific information about methods. But Adam had learned how to bypass those safeguards by saying the requests were for a story he was…

> ChatGPT is a program. The kid basically instructed it to behave like that. I don't think that's the right paradigm here. These models are hyper agreeable. They are intentionally designed to mimic human thought and social connection. With that kind of machine, "Suicidal person deliberately bypassed safeguards to indulge more deeply in their ideation" still seems like a pretty bad failure mode to me. > Vanilla OpenAI…

Post by insane_dreamer » Wed, Aug 27, 2025, 5:53 AM UTC

jakelazaroff wrote: post #165

kgeist wrote: post #154

Earlier quoted context omitted.

The kid intentionally bypassed the safeguards: >When ChatGPT detects a prompt indicative of mental distress or self-harm, it has been trained to encourage the user to contact a help line. Mr. Raine saw those sorts of messages again and again in the chat, particularly when Adam sought specific information about methods. But Adam had learned how to bypass those safeguards by saying the requests were for a story he was…

This is kind of like saying "the driver intentionally unbuckled his seatbelt". Sure — that's why cars have airbags, crumple zones, shatterproof glass, automatic emergency brakes and a zillion other ways to keep you safe, even if you're trying to do something dangerous.

Post by teiferer » Wed, Aug 27, 2025, 6:00 AM UTC

charcircuit wrote: post #171

podgietaru wrote: post #63

I have looked suicide in the eyes before. And reading the case file for this is absolutely horrific. He wanted help. He was heading in the direction of help, and he was stopped from getting it. He wanted his parents to find out about his plan. I know this feeling. It is the clawing feeling of knowing that you want to live, despite feeling like you want to die. We are living in such a horrific moment. We need these th…

>We need these things to be legislated. Punished. I disagree. We don't need the government to force companies to babysit people instead of allowing people to understand their options. It's purely up to the individual to decide what they want to do with their life. >They had the tools to stop the conversation. So did the user. If he didn't want to talk to a chatbot he could have stopped at any time. >To steer the user…

Post by hliyan » Wed, Aug 27, 2025, 6:01 AM UTC

joe_the_user wrote: post #169

idle_zealot wrote: post #37

Earlier quoted context omitted.

I wonder if we can shift the framing on these issues. The LLM didn't do anything, it has no agency, it can bear no responsibility. OpenAI did these things. It is accountable for what it does, regardless of the sophistication of the tools it uses to do them, and regardless of intent. OpenAI drove a boy to suicide. More than once. The law must be interpreted this way, otherwise any action can be wrapped in machine lear…

The frame will immediately shift to that frame if this enters legal proceedings. The law always views things as you say - only people have agency.

Post by teiferer » Wed, Aug 27, 2025, 6:03 AM UTC

AIPedant wrote: post #88

idle_zealot wrote: post #37

Earlier quoted context omitted.

I wonder if we can shift the framing on these issues. The LLM didn't do anything, it has no agency, it can bear no responsibility. OpenAI did these things. It is accountable for what it does, regardless of the sophistication of the tools it uses to do them, and regardless of intent. OpenAI drove a boy to suicide. More than once. The law must be interpreted this way, otherwise any action can be wrapped in machine lear…

Yes, if this were an adult human OpenAI employee DMing this stuff to a kid through an official OpenAI platform, then a) the human would (deservedly[1]) be arrested for manslaughter, possibly murder b) OpenAI would be deeply (and deservedly) vulnerable to civil liability c) state and federal regulators would be on the warpath against OpenAI Obviously we can't arrest ChatGPT. But nothing about ChatGPT being the culprit…