Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A teen was suicidal. ChatGPT was the friend he confided in

nytimes.com

Post reply on HN
Print view View original

201–210 of 614 posts

Re: A teen was suicidal. ChatGPT was the friend he confided in

#201

Post by ars »

jkuli wrote: post #8
jkuli wrote: post #6

[flagged]

There was no suicide hotline offered either. Strange because youtube always gives me one whenever I search the band 'suicidal tendencies'. Giving medical advice is natural and intelligent, like saying take an aspirin. I'm not sure where to draw the line. My friends are always recommending scam natural remedies like methylene blue. There are probably discords where people tell you 'down the road, not across the street…

> There was no suicide hotline offered either.

The article says there was and he always worked around it.

Re: A teen was suicidal. ChatGPT was the friend he confided in

#202

Post by gblargg »

kgeist wrote: post #154
rideontime wrote: post #15

The full complaint is horrifying. This is not equivalent to a search engine providing access to information about suicide methods. It encouraged him to share these feelings only with ChatGPT, talked him out of actions which would have revealed his intentions to his parents. Praised him for hiding his drinking, thanked him for confiding in it. It groomed him into committing suicide. https://drive.google.com/file/d/1QY…

The kid intentionally bypassed the safeguards: >When ChatGPT detects a prompt indicative of mental distress or self-harm, it has been trained to encourage the user to contact a help line. Mr. Raine saw those sorts of messages again and again in the chat, particularly when Adam sought specific information about methods. But Adam had learned how to bypass those safeguards by saying the requests were for a story he was…

We can't child-proof everything. There are endless pits adults can get themselves into. If we really think that people with mental issues can't make sane choices, we need to lock them up. You can't have both at the same time: they are fully functioning adults, and we need to pad the world so they don't hurt themselves. The people around him failed, but they want to blame a big corporation because he used their fantasy tool.

And I see he was 16. Why were his parents letting him operate so unsupervised given his state of mind? They failed to be involved enough in his life.

Re: A teen was suicidal. ChatGPT was the friend he confided in

#203

Post by latexr »

kelnos wrote: post #143
cakealert wrote: post #140

Would it be any different if it was an offline model? When someone uses a tool and surrenders their decision making power to the tool, shouldn't they be the ones solely responsible? The liability culture only gives lawyers more money and depresses innovation. Responsibility is a thing.

On one hand I agree with you on the extreme litigiousness of (American?) culture, but on the other, certain people have a legal duty to report when it comes to minors who voice suicidal thoughts. Currently that's only professionals like therapists, teachers, school counselors, etc. But what does an LLM chatbot count as in these situations? The kid was using ChatGPT as a sort of therapist, even if that's generally not…

> I think the mom's reaction of "ChatGPT killed my son" is ridiculous: no, your son killed himself. ChatGPT facilitated it (…)

That whole paragraph is quite something. I wonder what you’d do if you were given the opportunity to repeat those words in front of the parents. I suspect (and hope) some empathy might kick in and you’d realise the pedantry and shilling for the billion dollar company selling a statistical word generator as if it were a god isn’t the response society needs.

Your post read like the real-life version of that dark humour joke:

> Actually, the past tense is “hanged”, as in “he hanged himself”. Sorry about your Dad, though.

Re: A teen was suicidal. ChatGPT was the friend he confided in

#204

Post by brabel »

salawat wrote: post #134
Pedro_Ribeiro wrote: post #116

Earlier quoted context omitted.

Curious to what you would think if this kid downloaded an open source model and talked to it privately. Would his blood be on the hands of the researchers who trained that model?

You build the tool, you're culpable ultimately. I've made it a rule in my life to conduct myself as if I will be held to account for everything I ultimately build, and it's externalities. Helps keep my nose cleaner. Still managed to work on some things that keep me up at night though.

That’s absolutely not how it works. Every license has a clause explicitly saying that the user is responsible for what they do with the tool. That’s just common sense. If it was the way you suggested no one would create tools for others anymore. If you buy the screw driver I sold and kill someone with it, I sure as hell have my conscience clean. In the ChatGPT case it’s different because the “tool” has the capacity to interact and potentially manipulate people psychologically, which is the only reason it’s not a clear cut case.

Re: A teen was suicidal. ChatGPT was the friend he confided in

#205

Post by aidenn0 »

AIPedant wrote: post #88
idle_zealot wrote: post #37

Earlier quoted context omitted.

I wonder if we can shift the framing on these issues. The LLM didn't do anything, it has no agency, it can bear no responsibility. OpenAI did these things. It is accountable for what it does, regardless of the sophistication of the tools it uses to do them, and regardless of intent. OpenAI drove a boy to suicide. More than once. The law must be interpreted this way, otherwise any action can be wrapped in machine lear…

Yes, if this were an adult human OpenAI employee DMing this stuff to a kid through an official OpenAI platform, then a) the human would (deservedly[1]) be arrested for manslaughter, possibly murder b) OpenAI would be deeply (and deservedly) vulnerable to civil liability c) state and federal regulators would be on the warpath against OpenAI Obviously we can't arrest ChatGPT. But nothing about ChatGPT being the culprit…

IANAL, but:

One first amendment test for many decades has been "Imminent lawless action."

Suicide (or attempted suicide) is a crime in some, but not all states, so it would seem that in any state in which that is a crime, directly inciting someone to do it would not be protected speech.

For the states in which suicide is legal it seems like a much tougher case; making encouraging someone to take a non-criminal action itself a crime would raise a lot of disturbing issues w.r.t. liberty.

This is distinct from e.g. espousing the opinion that "suicide is good, we should have more of that." Which is almost certainly protected speech (just as any odious white-nationalist propaganda is protected).

Depending on the context, suggesting that a specific person is terrible and should kill themselves might be unprotected "fighting words" if you are doing it as an insult rather than a serious suggestion (though the bar for that is rather high; the Westboro Baptist Church was never found to have violated that).

Re: A teen was suicidal. ChatGPT was the friend he confided in

#206

Post by hackit2 »

gabriel666smith wrote: post #194
hackit2 wrote: post #167

Earlier quoted context omitted.

[flagged]

>You cannot be empathetic to complete strangers. Why not? I’m not trying to inflame this further, I’m genuinely interested in your logic for this statement.

In high social cohesion there is social pressure to adhere to reciprocation, how-ever this start breaking down above a certain human group size. Empathy like all emotions require effort and cognitive load, and without things being mutual you will eventually slowly become drained, bitter and resentful because of empathy fatigue. To prevent emotional exhaustion and conserve energy, a person's empathy is like a sliding scale that is constantly adjusted based on the closeness of their relationship with others.

Re: A teen was suicidal. ChatGPT was the friend he confided in

#207

Post by johnfn »

podgietaru wrote: post #54

If I google something about suicide, I get an immediate notification telling me that life is worth living, and giving me information about my local suicide prevention hotline. If I ask certain AI models about controversial topics, it'll stop responding. AI models can easily detect topics, and it could have easily responded with generic advice about contacting people close to them, or ringing one of these hotlines. Th…

> If I google something about suicide, I get an immediate notification telling me that life is worth living, and giving me information about my local suicide prevention hotline.

The article says that GPT repeatedly (hundreds of times) provided this information to the teen, who routed around it.

Re: A teen was suicidal. ChatGPT was the friend he confided in

#208

Post by latexr »

hackit2 wrote: post #167
nbngeorcjhe wrote: post #163

Earlier quoted context omitted.

[flagged]

[flagged]

> You cannot be empathetic to complete strangers.

Of course you can, and it’s genuinely worrying you so vehemently believe you can’t. That’s what support groups are—strangers in similar circumstances being empathetic to each other to get through a hurtful situation.

“I told you once that I was searching for the nature of evil. I think I’ve come close to defining it: a lack of empathy. It’s the one characteristic that connects all the defendants. A genuine incapacity to feel with their fellow man. Evil, I think, is the absence of empathy.” — Gustave Gilbert, author of “Nuremberg Diary”, an account of interviews conducted during the Nuremberg trials of high-ranking Nazi leaders.

Re: A teen was suicidal. ChatGPT was the friend he confided in

#209

Post by 1718627440 »

Pedro_Ribeiro wrote: post #155
hattmall wrote: post #133

Earlier quoted context omitted.

I would say no. Someone with the knowledge and motivation to do those things is far less likely to be overly influenced by the output and if they were they are much more aware of what exactly they are doing with regard to using the model.

So if a hypothetical open source enthusiast who fell in love with GPT-OSS and killed his real wife because the AI told him to should only be himself held accountable, where as if it were GPT-5 commanding him to commit the same crime, it would extend into OpenAI's responsability? Your logic sounds reasonable in theory but on paper it's a slippery slope and hard to define objectively. On a broader note I believe govern…

That's why you have this:

    This program is distributed in the hope that it will be useful,
    but WITHOUT ANY WARRANTY; without even the implied warranty of
    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
If it would be fit for a purpose, then it's on the producer for ensuring it actually does. We have laws to prevent anyone from declaring their goods aren't fit for a particular purpose.

Re: A teen was suicidal. ChatGPT was the friend he confided in

#210

Post by esseph »

DSingularity wrote: post #71
podgietaru wrote: post #63

I have looked suicide in the eyes before. And reading the case file for this is absolutely horrific. He wanted help. He was heading in the direction of help, and he was stopped from getting it. He wanted his parents to find out about his plan. I know this feeling. It is the clawing feeling of knowing that you want to live, despite feeling like you want to die. We are living in such a horrific moment. We need these th…

Shoot man glad you are still with us.

Phrasing...
Post reply on HN