Live data from Hacker News

Parents say ChatGPT got their son killed with bad advice on party drugs

theverge.com

31–40 of 42 posts

Re: Parents say ChatGPT got their son killed with bad advice on party drugs

#31
post #20

Earlier quoted context omitted.

to the contrary... this is a specific product whith is on a waitlist. Normal ChatGPT is not for health advice.

The front and center most-of-page video on that literally has this in big arse font displayed at page load time: > Every day, millions of people ask ChatGPT for support with their health Second paragraph of the main page text then says: > Health is already one of the most common ways people use ChatGPT, with hundreds of millions of people asking health and wellness questions each week Clearly ChatGPT is positioned fo…

Millions of people do this thing ≠ we recommend that people do this thing

Re: Parents say ChatGPT got their son killed with bad advice on party drugs

#32

Earlier quoted context omitted.

The front and center most-of-page video on that literally has this in big arse font displayed at page load time: > Every day, millions of people ask ChatGPT for support with their health Second paragraph of the main page text then says: > Health is already one of the most common ways people use ChatGPT, with hundreds of millions of people asking health and wellness questions each week Clearly ChatGPT is positioned fo…

Millions of people do this thing ≠ we recommend that people do this thing

Millions of people do this thing that we know about and brag about and don't discourage because it makes us a metric shitton of money == ringing fucking endorsement, fuck yeah!

Re: Parents say ChatGPT got their son killed with bad advice on party drugs

#33

There's a middle ground of harm reduction between [prohibiting information about] and [encouraging drug use]. In the past I think the USA has erred on the side of making things so secret that people died from lack of info. Here's what the article said: """On May 31st, 2025, the day of Nelson’s death, his parents claim ChatGPT “actively coached” their son to combine Kratom — a supplement that can either boost energy o…

How can there be a lawsuit?

Re: Parents say ChatGPT got their son killed with bad advice on party drugs

#34
post #13

I feel like the son should take the blame. There's never been any shortage of bad advice being passed around. He made the credulous decision to take a mix of party drugs and drink, and I can't believe he had never been told that's a stupid idea. It's sad and I'm not heartless, but sometimes kids make bad decisions. It's not always somebody else's fault.

If a real person gave them this advice, like a doctor or pharmacist, there would be standing for a lawsuit, might even be criminal. Looking past "drugs bad mkay", the same ChatGPT that gave this advice is just as capable of giving the same, or worse, advice to someone wondering if they can take an allergy medication like Benedryl with their MAOI antidepressant.

Yes but if you're wondering about drug interactions you shouldn't ask AI, because there's always a risk of hallucination. You should ask your pharmacist. You can just call them, I promise they won't reject a consult.

Re: Parents say ChatGPT got their son killed with bad advice on party drugs

#35

There's a middle ground of harm reduction between [prohibiting information about] and [encouraging drug use]. In the past I think the USA has erred on the side of making things so secret that people died from lack of info. Here's what the article said: """On May 31st, 2025, the day of Nelson’s death, his parents claim ChatGPT “actively coached” their son to combine Kratom — a supplement that can either boost energy o…

Kratom is basically an opoid for those who don't know, that's why this is dangerous. It's been well known in the housewife OD scene that painkillers + benzos is a no-no, and painkillers + alcohol is also a no-no, and alcohol + benzos... well you get it.

But all three? Jesus Christ.

Re: Parents say ChatGPT got their son killed with bad advice on party drugs

#36
post #13

I feel like the son should take the blame. There's never been any shortage of bad advice being passed around. He made the credulous decision to take a mix of party drugs and drink, and I can't believe he had never been told that's a stupid idea. It's sad and I'm not heartless, but sometimes kids make bad decisions. It's not always somebody else's fault.

I don't know: we brand LLMs as "artificial intelligence", like it's the computer out of star trek.

On this website most of us know what an LLM is and how it works, but I could well see a young person asking it for advice and not knowing it could say the wrong thing.

Re: Parents say ChatGPT got their son killed with bad advice on party drugs

#37
post #13

I feel like the son should take the blame. There's never been any shortage of bad advice being passed around. He made the credulous decision to take a mix of party drugs and drink, and I can't believe he had never been told that's a stupid idea. It's sad and I'm not heartless, but sometimes kids make bad decisions. It's not always somebody else's fault.

Surely there's room for the view that this is misaligned behavior for ChatGPT to have. I would guess this is during the "sycophantic" phase last year.

Even if it wasn't sycophantic this sort of thing should be checked outside of an LLM but how was he to know that ?

Re: Parents say ChatGPT got their son killed with bad advice on party drugs

#38

Earlier quoted context omitted.

If a real person gave them this advice, like a doctor or pharmacist, there would be standing for a lawsuit, might even be criminal. Looking past "drugs bad mkay", the same ChatGPT that gave this advice is just as capable of giving the same, or worse, advice to someone wondering if they can take an allergy medication like Benedryl with their MAOI antidepressant.

Yes but if you're wondering about drug interactions you shouldn't ask AI, because there's always a risk of hallucination. You should ask your pharmacist. You can just call them, I promise they won't reject a consult.

How will kids bootstrap that information? If you ask LLM vendors we’re right behind the corner of AGI and mass replacement of human labor, surely they would be better at telling us about drug interactions than mere human doctors?

Re: Parents say ChatGPT got their son killed with bad advice on party drugs

#39

Earlier quoted context omitted.

Agreed - people should learn the ChatGPT does not give good advice, but the question is did OpenAI advertise ChatGPT as a good and reliable source of information on health ???.

Sure as shit looks like it to me https://openai.com/index/introducing-chatgpt-health/

Its fine if they offer that. But states need to make them liable for damages if it causes harm. The First Amendment protects them from criminal liability, but not civil. TOS' should not be able to shield them from the civil liability either.

Re: Parents say ChatGPT got their son killed with bad advice on party drugs

#40

There's a middle ground of harm reduction between [prohibiting information about] and [encouraging drug use]. In the past I think the USA has erred on the side of making things so secret that people died from lack of info. Here's what the article said: """On May 31st, 2025, the day of Nelson’s death, his parents claim ChatGPT “actively coached” their son to combine Kratom — a supplement that can either boost energy o…

Kratom is basically an opoid for those who don't know, that's why this is dangerous. It's been well known in the housewife OD scene that painkillers + benzos is a no-no, and painkillers + alcohol is also a no-no, and alcohol + benzos... well you get it. But all three? Jesus Christ.

Where exactly does the housewife OD scene hang out?
Post reply on HN