Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A teen was suicidal. ChatGPT was the friend he confided in

nytimes.com

Post reply on HN
Print view View original

471–480 of 614 posts

Re: A teen was suicidal. ChatGPT was the friend he confided in

#471

Post by rsynnott »

idle_zealot wrote: post #452
rsynnott wrote: post #299

Earlier quoted context omitted.

> but we still had an issue with loads of them prompting users into suicide, violence, or misleading them into preparing noxious mixtures of cleaning supplies, then we could have a discussion about exactly what extreme packaging requirements ought to be in place for distribution to be considered responsible. Or, I mean, just banning sale on the basis that they're unsafe devices and unfit for purpose. Like, you can't…

In that extreme case the "packaging requirements" would be labeling the thing not as a boiler, but as dangerous explosive scrap.

I suspect in many places you couldn’t sell what would essentially be a device for poisoning people to consumers _at all_.

Re: A teen was suicidal. ChatGPT was the friend he confided in

#472

Post by slg »

jackjeff wrote: post #188
slg wrote: post #79

Earlier quoted context omitted.

Thank you, “we just have to accept that these systems will occasionally kill children” is a perfect example of the type of mindset I was criticizing.

Don’t cars and ropes and drills occasionally kill people too? Society seems to have accepted that fact long ago. Somehow we expect the digital world to be devoid of risks. Cryptography that only the good guys can crack is another example of this mindset. Now I’m not saying ClosedAI look good on this, their safety layer clearly failed and the sycophantic BS did not help. But I reckon this kind of failure more will alw…

Another great example of the HN mindset. How can you say "their safety layer clearly failed" without being able to acknowledge that they should be held responsible for that failure and that we should work to reduce the likelihood of similar failures in the future? If a car, rope, or drill had some sort of failure in their manufacturing that killed people, those companies would be held responsible. Why can't we do the same with OpenAI?

This isn't a desire for "the digital world to be devoid of risks", it is a desire for people and companies in the digital world to be held responsible for the harm they are causing. Yet, people here seem to believe that no one should ever be held responsible for the damage caused by the software they create.

Re: A teen was suicidal. ChatGPT was the friend he confided in

#473

Post by imtringued »

mothballed wrote: post #350
podgietaru wrote: post #341

Earlier quoted context omitted.

What? Is this deliberately obtuse? Books are not granted freedom of speech, authors are. Their method is books. This is like saying sound waves are not granted freedom of speech. Unless you're suggesting there's a man sat behind every ChatGPT chat your analogy is nonsense.

Yes I am saying there is a man "sat" as it were behind every ChatGPT chat. The authors of ChatGPT basically made something closer to a turing-complete "choose-your-own adventure" book. They ensured you could choose an adventure where the reader can choose a suicide roleplay adventure, but it is up to the reader whether they want to flip to that page. If they want to flip to the page that says "suicide" then it will t…

The author is the suicidal kid in this case though.

Re: A teen was suicidal. ChatGPT was the friend he confided in

#474

Post by bastawhiz »

AnIrishDuck wrote: post #168
kgeist wrote: post #154

Earlier quoted context omitted.

The kid intentionally bypassed the safeguards: >When ChatGPT detects a prompt indicative of mental distress or self-harm, it has been trained to encourage the user to contact a help line. Mr. Raine saw those sorts of messages again and again in the chat, particularly when Adam sought specific information about methods. But Adam had learned how to bypass those safeguards by saying the requests were for a story he was…

> ChatGPT is a program. The kid basically instructed it to behave like that. I don't think that's the right paradigm here. These models are hyper agreeable. They are intentionally designed to mimic human thought and social connection. With that kind of machine, "Suicidal person deliberately bypassed safeguards to indulge more deeply in their ideation" still seems like a pretty bad failure mode to me. > Vanilla OpenAI…

> These models are hyper agreeable. They are intentionally designed to mimic human thought and social connection.

Python is hyper agreeable. If I comment out some safeguards, it'll happily bypass whatever protections are in place.

Lots of people on here argue vehemently against anthropomorphizing LLMs. It's either a computer program crunching numbers, or it's a nebulous form of pseudo-consciousness, but you can't have it both ways. It's either a tool that has no mind of its own that follows instructions, or it thinks for itself.

I'm not arguing that the model behaved in a way that's ideal, but at what point do you make the guardrails impassable for 100% of users? How much user intent do you reject in the interest of the personal welfare of someone intent on harming themselves?

Re: A teen was suicidal. ChatGPT was the friend he confided in

#475

Post by pembrook »

ares623 wrote: post #366
pembrook wrote: post #268

This is dumb. Nobody is writing articles about all the times the opposite happened, and ChatGPT helped prevent bad stuff. However, because of the nature of this topic, it’s the perfect target for NYT to generate moral panic for clicks. Classic media attention bait 101. I can’t believe HN is falling for this. It’s the equivalent of the moral panic around metal music in the 1980s where the media created a hysteria arou…

I don’t get it. With all the evidence presented you think this situation is similar to mass hysteria? Yes, it rhymes with what you described. But this one has hard evidence. And you’re asking to ignore it because a similar thing happened in the past?

Yes, it’s clear there were zero other factors that led this teen to suicide. Teens have never committed suicide before ChatGPT.

Also video games lead to school shootings, music leads to teens doing drugs, and pagers are responsible for teen pregnancies.

Just look at all the evidence presented in this non-biased, objective article with no financial incentive to incite moral panic…obviously chatgpt is guilty of murder here!

Re: A teen was suicidal. ChatGPT was the friend he confided in

#476

Post by UltraSane »

habinero wrote: post #453
UltraSane wrote: post #402

Earlier quoted context omitted.

So you are denying that LLMs actually contain real knowledge?

They contain training data and a statistical model that might generate something true or it might generate garbage, both with equal confidence. You need to already know the answer to determine which is which.

Have you actually used Claude Opus 4.1? It is right far more than it is wrong.

Re: A teen was suicidal. ChatGPT was the friend he confided in

#477

Post by imtringued »

habinero wrote: post #458
msgodel wrote: post #441

Earlier quoted context omitted.

The problem is there's no way to build anything like a safety rail here. If you had it your way teens, likely everyone else too wouldn't be allowed to use computers at all without some kind of certification.

I honestly don't hate the idea. On a more serious note, of course there's ways to put in guard rails. LLMs behave like they do because of intentional design choices. Nothing about it is innate.

If you take this idea even a little bit further, you'll end up with licenses for being allowed to speak.

Re: A teen was suicidal. ChatGPT was the friend he confided in

#478

Post by hyperdimension »

yifanl wrote: post #469
scotty79 wrote: post #414

Earlier quoted context omitted.

Societies love theatres. Model guardrails are for chats what TSA is for air travel.

I have never heard anyone speak of the TSA favourably, so maybe it's not the best model to emulate?

That's the point. It hardly does what is claimed to do.

Re: A teen was suicidal. ChatGPT was the friend he confided in

#479

Post by GuinansEyebrows »

hackit2 wrote: post #164
cakealert wrote: post #147

Earlier quoted context omitted.

[flagged]

Sad to see what happened to the kid, but to point the finger at a language model is just laughable. It shows a complete breakdown of society and the caregivers entrusted with responsibility.

people are (rightly) pointing the finger at OpenAI, the organization comprised of human beings, all of whom made decisions along the way to release a language model that encouraged a child to attempt and complete suicide.

Re: A teen was suicidal. ChatGPT was the friend he confided in

#480

Post by pembrook »

lowsong wrote: post #365
pembrook wrote: post #268

This is dumb. Nobody is writing articles about all the times the opposite happened, and ChatGPT helped prevent bad stuff. However, because of the nature of this topic, it’s the perfect target for NYT to generate moral panic for clicks. Classic media attention bait 101. I can’t believe HN is falling for this. It’s the equivalent of the moral panic around metal music in the 1980s where the media created a hysteria arou…

Do AI apologists, like you, live in some parallel universe? One where it's acceptable to call the suicide of a vulnerable teenager "media attention bait". You should be ashamed of yourself.

No the NYT should be ashamed of itself for using this tragic story to generate clicks and sell subscriptions/ads.

They are quite literally profiting financially off of this teens death while spreading false moral panic at the same time, making us all dumber in the process.

Do you also believe that video games and music led to columbine? The NYT got a lot of attention for suggesting that at the time as well.

Post reply on HN