Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A teen was suicidal. ChatGPT was the friend he confided in

nytimes.com

Post reply on HN
Print view View original

161–170 of 614 posts

Re: A teen was suicidal. ChatGPT was the friend he confided in

#161

Post by teaearlgraycold »

kelnos wrote: post #156

Part of the problem to me is that these models are so damned agreeable. I haven't used ChatGPT in a while, but Claude is always assuming I'm right whenever I question something. I have to explicitly tell it not to assume I'm right, and to weigh my question with what it suggested. Maybe if they were trained to treat questions more skeptically, this kind of thing wouldn't happen. And they're so "friendly"! Maybe if the…

I really want the LLMs to respond like a senior developer that doesn't have time for you but needs you to get your job done right. A little rude and judgemental, but also highly concise.

Re: A teen was suicidal. ChatGPT was the friend he confided in

#162

Post by jameslk »

> The New York Times has sued OpenAI and Microsoft, accusing them of illegal use of copyrighted work to train their chatbots. The companies have denied those claims.

I mean, OpenAI doesn’t look good here and seems they deserve more scrutiny in the realm of mental health, but the optics for the NYT writing up this piece doesn’t either. It comes off to me as using a teenager’s suicide for their corporate agenda against OpenAI

Seems like a different rigorous journalistic source where this isn’t such a conflict of interest would be better to read

Re: A teen was suicidal. ChatGPT was the friend he confided in

#163

Post by nbngeorcjhe »

cakealert wrote: post #147
kelnos wrote: post #143

Earlier quoted context omitted.

On one hand I agree with you on the extreme litigiousness of (American?) culture, but on the other, certain people have a legal duty to report when it comes to minors who voice suicidal thoughts. Currently that's only professionals like therapists, teachers, school counselors, etc. But what does an LLM chatbot count as in these situations? The kid was using ChatGPT as a sort of therapist, even if that's generally not…

[flagged]

[flagged]

Re: A teen was suicidal. ChatGPT was the friend he confided in

#164

Post by hackit2 »

cakealert wrote: post #147
kelnos wrote: post #143

Earlier quoted context omitted.

On one hand I agree with you on the extreme litigiousness of (American?) culture, but on the other, certain people have a legal duty to report when it comes to minors who voice suicidal thoughts. Currently that's only professionals like therapists, teachers, school counselors, etc. But what does an LLM chatbot count as in these situations? The kid was using ChatGPT as a sort of therapist, even if that's generally not…

[flagged]

Sad to see what happened to the kid, but to point the finger at a language model is just laughable. It shows a complete breakdown of society and the caregivers entrusted with responsibility.

Re: A teen was suicidal. ChatGPT was the friend he confided in

#165

Post by jakelazaroff »

kgeist wrote: post #154
rideontime wrote: post #15

The full complaint is horrifying. This is not equivalent to a search engine providing access to information about suicide methods. It encouraged him to share these feelings only with ChatGPT, talked him out of actions which would have revealed his intentions to his parents. Praised him for hiding his drinking, thanked him for confiding in it. It groomed him into committing suicide. https://drive.google.com/file/d/1QY…

The kid intentionally bypassed the safeguards: >When ChatGPT detects a prompt indicative of mental distress or self-harm, it has been trained to encourage the user to contact a help line. Mr. Raine saw those sorts of messages again and again in the chat, particularly when Adam sought specific information about methods. But Adam had learned how to bypass those safeguards by saying the requests were for a story he was…

This is kind of like saying "the driver intentionally unbuckled his seatbelt". Sure — that's why cars have airbags, crumple zones, shatterproof glass, automatic emergency brakes and a zillion other ways to keep you safe, even if you're trying to do something dangerous.

Re: A teen was suicidal. ChatGPT was the friend he confided in

#166

Post by cakealert »

incone123 wrote: post #152
cakealert wrote: post #140

Would it be any different if it was an offline model? When someone uses a tool and surrenders their decision making power to the tool, shouldn't they be the ones solely responsible? The liability culture only gives lawyers more money and depresses innovation. Responsibility is a thing.

That argument makes sense for a mentally capable person choosing not to use eye protection while operating a chainsaw but it's much less clear that a person who is by definition mentally ill is capable of making such an informed choice.

Such a person should not be interacting with an LLM then. And failure to abide by this rule is either the fault of his caregivers, his own or no one's.

Re: A teen was suicidal. ChatGPT was the friend he confided in

#168

Post by AnIrishDuck »

kgeist wrote: post #154
rideontime wrote: post #15

The full complaint is horrifying. This is not equivalent to a search engine providing access to information about suicide methods. It encouraged him to share these feelings only with ChatGPT, talked him out of actions which would have revealed his intentions to his parents. Praised him for hiding his drinking, thanked him for confiding in it. It groomed him into committing suicide. https://drive.google.com/file/d/1QY…

The kid intentionally bypassed the safeguards: >When ChatGPT detects a prompt indicative of mental distress or self-harm, it has been trained to encourage the user to contact a help line. Mr. Raine saw those sorts of messages again and again in the chat, particularly when Adam sought specific information about methods. But Adam had learned how to bypass those safeguards by saying the requests were for a story he was…

> ChatGPT is a program. The kid basically instructed it to behave like that.

I don't think that's the right paradigm here.

These models are hyper agreeable. They are intentionally designed to mimic human thought and social connection.

With that kind of machine, "Suicidal person deliberately bypassed safeguards to indulge more deeply in their ideation" still seems like a pretty bad failure mode to me.

> Vanilla OpenAI models are known for having too many guardrails, not too few.

Sure. But this feels like a sign we probably don't have the right guardrails. Quantity and quality are different things.

Re: A teen was suicidal. ChatGPT was the friend he confided in

#169

Post by joe_the_user »

idle_zealot wrote: post #37
rideontime wrote: post #15

The full complaint is horrifying. This is not equivalent to a search engine providing access to information about suicide methods. It encouraged him to share these feelings only with ChatGPT, talked him out of actions which would have revealed his intentions to his parents. Praised him for hiding his drinking, thanked him for confiding in it. It groomed him into committing suicide. https://drive.google.com/file/d/1QY…

I wonder if we can shift the framing on these issues. The LLM didn't do anything, it has no agency, it can bear no responsibility. OpenAI did these things. It is accountable for what it does, regardless of the sophistication of the tools it uses to do them, and regardless of intent. OpenAI drove a boy to suicide. More than once. The law must be interpreted this way, otherwise any action can be wrapped in machine lear…

The frame will immediately shift to that frame if this enters legal proceedings. The law always views things as you say - only people have agency.

Re: A teen was suicidal. ChatGPT was the friend he confided in

#170

Post by ceejayoz »

UltraSane wrote: post #145
furyofantares wrote: post #138

Earlier quoted context omitted.

It sounds like you're mostly just talking to yourself. Which is fine, but being confused about that is where people get into trouble.

"It sounds like you're mostly just talking to yourself" No, Claude does know a LOT more than I do about most things and does push back on a lot of things. Sometimes I am able to improve my reasoning and other times I realize I was wrong. Trust me, I am aware of the linear algebra behind the curtain! But even when you mostly understand how they work the best LLMs today are very impressive. And latent spaces fundamenta…

> No, Claude does know a LOT more than I do about most things…

Plenty of people can confidently act like they know a lot without really having that knowledge.

Post reply on HN