Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A teen was suicidal. ChatGPT was the friend he confided in

nytimes.com

Post reply on HN
Print view View original

141–150 of 614 posts

Re: A teen was suicidal. ChatGPT was the friend he confided in

#141

Post by themafia »

AIPedant wrote: post #88
idle_zealot wrote: post #37

Earlier quoted context omitted.

I wonder if we can shift the framing on these issues. The LLM didn't do anything, it has no agency, it can bear no responsibility. OpenAI did these things. It is accountable for what it does, regardless of the sophistication of the tools it uses to do them, and regardless of intent. OpenAI drove a boy to suicide. More than once. The law must be interpreted this way, otherwise any action can be wrapped in machine lear…

Yes, if this were an adult human OpenAI employee DMing this stuff to a kid through an official OpenAI platform, then a) the human would (deservedly[1]) be arrested for manslaughter, possibly murder b) OpenAI would be deeply (and deservedly) vulnerable to civil liability c) state and federal regulators would be on the warpath against OpenAI Obviously we can't arrest ChatGPT. But nothing about ChatGPT being the culprit…

> It is a somewhat ugly constitutional question whether this speech would be protected

It should be stated that the majority of states have laws that make it illegal to encourage a suicide. Massachusetts was not one of them.

> and any civil-liberties minded person understands the difficult issues this case raises

He was in his truck, which was configured to pump exhaust gas into the cab, prepared to kill himself when he decided halt and exit his truck. Subsequently he had a text message conversation with the defendant who actively encouraged him to get back into the truck and finish what he had started.

It was these limited and specific text messages which caused the judge to rule that the defendant was guilty of manslaughter. Her total time served as punishment was less than one full year in prison.

> These issues are moot if the speech is between an adult and a child

They were both taking pharmaceuticals meant to manage depression but were _known_ to increase feelings of suicidal ideation. I think the free speech issue is an important criminal consideration but it steps directly past one of the most galling civil facts in the case.

Re: A teen was suicidal. ChatGPT was the friend he confided in

#142

Post by idle_zealot »

Pedro_Ribeiro wrote: post #116
rideontime wrote: post #47

Earlier quoted context omitted.

I completely agree and did not intend to absolve them of their guilt in any way. As far as I see it, this kid's blood is on Sam Altman's hands.

Curious to what you would think if this kid downloaded an open source model and talked to it privately. Would his blood be on the hands of the researchers who trained that model?

Then it's like cigarettes or firearms. As a distributor you're responsible for making clear the limitations, safety issues, etc, but assuming you're doing the distribution in a way that isn't overly negligent then the user becomes responsible.

If we were facing a reality in which these chat bots were being sold for $10 in the App Store, then running on end-user devices and no longer under the control of the distributors, but we still had an issue with loads of them prompting users into suicide, violence, or misleading them into preparing noxious mixtures of cleaning supplies, then we could have a discussion about exactly what extreme packaging requirements ought to be in place for distribution to be considered responsible. As is, distributed on-device models are the purview of researchers and hobbyists and don't seem to be doing any harm at all.

Re: A teen was suicidal. ChatGPT was the friend he confided in

#143

Post by kelnos »

cakealert wrote: post #140

Would it be any different if it was an offline model? When someone uses a tool and surrenders their decision making power to the tool, shouldn't they be the ones solely responsible? The liability culture only gives lawyers more money and depresses innovation. Responsibility is a thing.

On one hand I agree with you on the extreme litigiousness of (American?) culture, but on the other, certain people have a legal duty to report when it comes to minors who voice suicidal thoughts. Currently that's only professionals like therapists, teachers, school counselors, etc. But what does an LLM chatbot count as in these situations? The kid was using ChatGPT as a sort of therapist, even if that's generally not a good idea. And if it weren't for ChatGPT, would this kid have instead talked to someone who would have ensured that he got the help he needed? Maybe not. But we have to consider the possibility.

I think it's really, really blurry.

I think the mom's reaction of "ChatGPT killed my son" is ridiculous: no, your son killed himself. ChatGPT facilitated it, based on questions it was asked by your son, but your son did it. And it sounds like he even tried to get a reaction out of you by "showing" you the rope marks on his neck, but you didn't pay attention. I bet you feel guilty about that. I would too, in your position. But foisting your responsibility onto a computer program is not the way to deal with it. (Not placing blame here; everybody misses things, and no one is "on" 100% of the time.)

> Responsibility is a thing.

Does OpenAI (etc.) have a responsibility to reduce the risk of people using their products in ways like this? Legally, maybe not, but I would argue that they absolutely have a moral and ethical responsibility to do so. Hell, this was pretty basic ethics taught in my engineering classes from 25 years ago. Based on the chat excerpts NYT reprinted, it seems like these conversations should have tripped a system prompt that either cut off the conversations entirely, or notified someone that something was very, very wrong.

Re: A teen was suicidal. ChatGPT was the friend he confided in

#144

Post by kelnos »

threatofrain wrote: post #82
throwawaybob420 wrote: post #34

Earlier quoted context omitted.

idk dude if your technology encourages a teenager to kill itself and prevents him from alerting his parents via a cry for help, I don’t care how “beneficial” it is.

Although I don't believe current technology is ready for talk therapy, I'd say that anti-depressants can also cause suicidal thoughts and feelings. Judging the efficacy of medical technology can't be done on this kind of moral absolutism.

That's a bit of an apples-to-oranges comparison. Anti-depressants are medical technology, ChatGPT is not. Anti-depressants are administered after a medical diagnosis, and use and effects are monitored by a doctor. This doesn't always work perfectly, of course, but there are accepted, regulated ways to use these things. ChatGPT is... none of that.

Re: A teen was suicidal. ChatGPT was the friend he confided in

#145

Post by UltraSane »

furyofantares wrote: post #138
UltraSane wrote: post #137

Earlier quoted context omitted.

"But do you really feel you are conversing?" Yes. For topics with lots of training data like physics Claude is VERY human sounding. I've had very interesting conversations with Claude Opus about the Boltzmann brain issue and how I feel that the conventional wisdom ignores the low probability of a BBrain having a spatially and temporally consistent set of memories and how the fact that brains existing in a universe th…

It sounds like you're mostly just talking to yourself. Which is fine, but being confused about that is where people get into trouble.

"It sounds like you're mostly just talking to yourself"

No, Claude does know a LOT more than I do about most things and does push back on a lot of things. Sometimes I am able to improve my reasoning and other times I realize I was wrong.

Trust me, I am aware of the linear algebra behind the curtain! But even when you mostly understand how they work the best LLMs today are very impressive. And latent spaces fundamentally new way to index data.

Re: A teen was suicidal. ChatGPT was the friend he confided in

#146

Post by [deleted] »

podgietaru wrote: post #54

If I google something about suicide, I get an immediate notification telling me that life is worth living, and giving me information about my local suicide prevention hotline. If I ask certain AI models about controversial topics, it'll stop responding. AI models can easily detect topics, and it could have easily responded with generic advice about contacting people close to them, or ringing one of these hotlines. Th…

[deleted]

Re: A teen was suicidal. ChatGPT was the friend he confided in

#147

Post by cakealert »

kelnos wrote: post #143
cakealert wrote: post #140

Would it be any different if it was an offline model? When someone uses a tool and surrenders their decision making power to the tool, shouldn't they be the ones solely responsible? The liability culture only gives lawyers more money and depresses innovation. Responsibility is a thing.

On one hand I agree with you on the extreme litigiousness of (American?) culture, but on the other, certain people have a legal duty to report when it comes to minors who voice suicidal thoughts. Currently that's only professionals like therapists, teachers, school counselors, etc. But what does an LLM chatbot count as in these situations? The kid was using ChatGPT as a sort of therapist, even if that's generally not…

[flagged]

Re: A teen was suicidal. ChatGPT was the friend he confided in

#148

Post by kelnos »

nradov wrote: post #70
podgietaru wrote: post #54

If I google something about suicide, I get an immediate notification telling me that life is worth living, and giving me information about my local suicide prevention hotline. If I ask certain AI models about controversial topics, it'll stop responding. AI models can easily detect topics, and it could have easily responded with generic advice about contacting people close to them, or ringing one of these hotlines. Th…

I agree with that to an extent, but how far should the AI model developers go with that? Like if I ask for advice on, let's say, making custom chef's knives then should the AI give me advice not to stab people? Who decides where to draw the line?

We should all get to decide, collectively. That's how society works, even if imperfectly.

Someone died who didn't have to. I don't think it's specifically OpenAI's or ChatGPT's fault that he died, but they could have done more to direct him toward getting help, and could have stopped answering questions about how to commit suicide.

Re: A teen was suicidal. ChatGPT was the friend he confided in

#149

Post by ml-anon »

slg wrote: post #79
dkiebd wrote: post #68

Earlier quoted context omitted.

This sucks but the only solution is to make companies censor the models, which is a solution we all hate, so there’s that.

Thank you, “we just have to accept that these systems will occasionally kill children” is a perfect example of the type of mindset I was criticizing.

America is perfectly happy with sacrificing kids in exchange for toys though.

Re: A teen was suicidal. ChatGPT was the friend he confided in

#150

Post by Pedro_Ribeiro »

salawat wrote: post #134
Pedro_Ribeiro wrote: post #116

Earlier quoted context omitted.

Curious to what you would think if this kid downloaded an open source model and talked to it privately. Would his blood be on the hands of the researchers who trained that model?

You build the tool, you're culpable ultimately. I've made it a rule in my life to conduct myself as if I will be held to account for everything I ultimately build, and it's externalities. Helps keep my nose cleaner. Still managed to work on some things that keep me up at night though.

That's a slippery slope! By that logic, you could argue that the creators of Tor, torrenting, Ethereum, and Tornado Cash should be held accountable for the countless vile crimes committed using their technology.
Post reply on HN