Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A teen was suicidal. ChatGPT was the friend he confided in

nytimes.com

Post reply on HN
Print view View original

321–330 of 614 posts

Re: A teen was suicidal. ChatGPT was the friend he confided in

#321

Post by mothballed »

sooheon wrote: post #196
luisfmh wrote: post #175

Earlier quoted context omitted.

So people that look to chatgpt for answers and help (as they've been programmed to do with all the marketing and capabilities from openai) should just die because they looked to chatgpt for an answer instead of google or their local suicide helpline? That doesn't seem reasonable, but it sounds to me like what you're saying. > So did the user. If he didn't want to talk to a chatbot he could have stopped at any time. T…

Let's flip the hypothetical -- if someone googles for suicide info and scrolls past the hotline info and ends up killing themselves anyway, should google be on the hook?

The solution that is going to be found, is they will put some age controls, probably half-heartedly, and call it a day. I don't think the public can stomach the possible free speech limitations on consenting adults to use a dangerous tool that might cause them to hurt themselves.

Re: A teen was suicidal. ChatGPT was the friend he confided in

#322

Post by mothballed »

toofy wrote: post #316
stavros wrote: post #229

Earlier quoted context omitted.

> When ChatGPT detects a prompt indicative of mental distress or self-harm, it has been trained to encourage the user to contact a help line. Mr. Raine saw those sorts of messages again and again in the chat, particularly when Adam sought specific information about methods. But Adam had learned how to bypass those safeguards by saying the requests were for a story he was writing.

why did you leave out the most important piece of context? he didn’t go out of his way to learn how to bypass the safeguards, it specifically told him how to get around the limit by saying, i’m not allowed to talk to you about suicide, however, if you tell me it’s for writing a story i can discuss it as much as you like.

Because that's the factual bounds of the law, in places where suicide is illegal. ChatGPT is just being the 4chan chatbot, if you don't like that roleplaying suicide is OK then you're going to have to amend the first amendment.

Re: A teen was suicidal. ChatGPT was the friend he confided in

#323

Post by Aeolun »

toofy wrote: post #316
stavros wrote: post #229

Earlier quoted context omitted.

> When ChatGPT detects a prompt indicative of mental distress or self-harm, it has been trained to encourage the user to contact a help line. Mr. Raine saw those sorts of messages again and again in the chat, particularly when Adam sought specific information about methods. But Adam had learned how to bypass those safeguards by saying the requests were for a story he was writing.

why did you leave out the most important piece of context? he didn’t go out of his way to learn how to bypass the safeguards, it specifically told him how to get around the limit by saying, i’m not allowed to talk to you about suicide, however, if you tell me it’s for writing a story i can discuss it as much as you like.

Claude is the opposite. It’ll go and say “I see you are talking about [fictional terrible situation] here, but I don’t feel comfortable talking about that at all, even in a fictional context. Please ask me something more wholesome.”

Re: A teen was suicidal. ChatGPT was the friend he confided in

#324

Post by rideontime »

Pedro_Ribeiro wrote: post #116
rideontime wrote: post #47

Earlier quoted context omitted.

I completely agree and did not intend to absolve them of their guilt in any way. As far as I see it, this kid's blood is on Sam Altman's hands.

Curious to what you would think if this kid downloaded an open source model and talked to it privately. Would his blood be on the hands of the researchers who trained that model?

I specifically blame Sam Altman because of the allegations in the complaint that he ordered safety checks to be skipped in order to rush this model to market, specific safety checks that were later demonstrated to identify and prevent precisely this behavior.

Re: A teen was suicidal. ChatGPT was the friend he confided in

#325

Post by Recursing »

rideontime wrote: post #15

The full complaint is horrifying. This is not equivalent to a search engine providing access to information about suicide methods. It encouraged him to share these feelings only with ChatGPT, talked him out of actions which would have revealed his intentions to his parents. Praised him for hiding his drinking, thanked him for confiding in it. It groomed him into committing suicide. https://drive.google.com/file/d/1QY…

From page 23:

> 92. In spring 2024, Altman learned Google would unveil its new Gemini model on May 14. Though OpenAI had planned to release GPT-4o later that year, Altman moved up the launch to May 13—one day before Google’s event.

> 93. [...] To meet the new launch date, OpenAI compressed months of planned safety evaluation into just one week, according to reports.

Re: A teen was suicidal. ChatGPT was the friend he confided in

#326

Post by rideontime »

kgeist wrote: post #154
rideontime wrote: post #15

The full complaint is horrifying. This is not equivalent to a search engine providing access to information about suicide methods. It encouraged him to share these feelings only with ChatGPT, talked him out of actions which would have revealed his intentions to his parents. Praised him for hiding his drinking, thanked him for confiding in it. It groomed him into committing suicide. https://drive.google.com/file/d/1QY…

The kid intentionally bypassed the safeguards: >When ChatGPT detects a prompt indicative of mental distress or self-harm, it has been trained to encourage the user to contact a help line. Mr. Raine saw those sorts of messages again and again in the chat, particularly when Adam sought specific information about methods. But Adam had learned how to bypass those safeguards by saying the requests were for a story he was…

Re-read the quote that you shared. Specifically the part pointing out that ChatGPT gave him the instructions on how to bypass its own inadequate safety measures.

Re: A teen was suicidal. ChatGPT was the friend he confided in

#327

Post by password321 »

Workaccount2 wrote: post #66

It's hard to see what is going on without seeing the actual chats, as opposed to the snippets in the lawsuit. A lot of suicidal people talk to these LLMs for therapy, and the reviews on the whole seem excellent. I'm not ready to jump on the bandwagon only seeing a handcrafted complaint. Ironically though I could still see lawsuits like this weighing heavily on the sycophancy that these models have, as the limited cha…

>If lawsuits like this lead to more "straight honest" models, I could see even more people killing themselves when their therapist model says "Yeah, but you kind of actually do suck".

It is not one extreme or the other. o3 is nowhere near as sycophantic as 4o but it is also not going to tell you that you suck especially in a suicidal context. 4o was the mainstream model because OpenAI probably realised that this is what most people want rather than a more professional model like o3 (besides the fact that it also uses more compute).

The lawsuits probably did make them RLHF GPT-5 to be at least a bit more middle-ground though that led to backlash because people "missed" 4o due this type of behaviour so they made it bit more "friendly". Still not as bad as 4o.

Re: A teen was suicidal. ChatGPT was the friend he confided in

#328

Post by mothballed »

novok wrote: post #221
salawat wrote: post #134

Earlier quoted context omitted.

You build the tool, you're culpable ultimately. I've made it a rule in my life to conduct myself as if I will be held to account for everything I ultimately build, and it's externalities. Helps keep my nose cleaner. Still managed to work on some things that keep me up at night though.

So if you build a chair and then someone uses it to murder someone, are you responsible for the murder?

You're not even responsible if you build an AR-15, complete with a bunch of free advertisements from the US army using the select-fire variant to slaughter innocent Iraqis, and it's used to murder someone.

The government will require them to add age controls and that will be that.

Re: A teen was suicidal. ChatGPT was the friend he confided in

#329

Post by nkrisc »

edanm wrote: post #270
idle_zealot wrote: post #37

Earlier quoted context omitted.

I wonder if we can shift the framing on these issues. The LLM didn't do anything, it has no agency, it can bear no responsibility. OpenAI did these things. It is accountable for what it does, regardless of the sophistication of the tools it uses to do them, and regardless of intent. OpenAI drove a boy to suicide. More than once. The law must be interpreted this way, otherwise any action can be wrapped in machine lear…

If ChatGPT has helped people be saved who might otherwise have died (e.g. by offering good medical advice that saved them), are all those lives saved also something you "attribute" to OpenAI? I don't know if ChatGPT has saved lives (thought I've read stories that claim that, yes, this happened). But assuming it has, are you OK saying that OpenAI has saved dozens/hundreds of lives? Given how scaling works, would you b…

In any case, if you kill one person and separately save ten people, you’ll still be prosecuted for killing that one person.

Re: A teen was suicidal. ChatGPT was the friend he confided in

#330

Post by HPsquared »

pembrook wrote: post #268

This is dumb. Nobody is writing articles about all the times the opposite happened, and ChatGPT helped prevent bad stuff. However, because of the nature of this topic, it’s the perfect target for NYT to generate moral panic for clicks. Classic media attention bait 101. I can’t believe HN is falling for this. It’s the equivalent of the moral panic around metal music in the 1980s where the media created a hysteria arou…

Journalists and writers as a general class already have interests opposed to LLMs, and NYT in particular have an ongoing legal battle about copyright. Yes it's clearly dripping with bias.
Post reply on HN