Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI says over a million people talk to ChatGPT about suicide weekly

techcrunch.com

Post reply on HN
Print view View original

311–320 of 585 posts

Re: OpenAI says over a million people talk to ChatGPT about suicide weekly

#311

Post by atleastoptimal »

probably_wrong wrote: post #229

If you haven't read the article (or even if you have but didn't click on outgoing links twice ) the NYT story about how ChatGPT convinced a suicidal teen not to look for help [1] should convince you that ChatGPT should be nowhere near anyone dealing with psychological issues. Here's ChatGPT discouraging said teenager from asking for help: > “I want to leave my noose in my room so someone finds it and tries to stop me…

OpenAI tried to get rid of the excessively sycophantic model (4o) but there was a massive backlash. They eventually relented and kept it as a model offering in ChatGPT.

OpenAI certainly has made mistakes with its rollouts in the past, but it is effectively impossible to keep everyone with psychological issues away from a free online web app.

>ChatGPT should be nowhere near anyone dealing with psychological issues.

Should every ledge with a >10ft drop have a suicide net? How would you imagine this would be enforced, requiring everyone who uses ChatGPT to agree to an "I am mentally stable" provisio?

Re: OpenAI says over a million people talk to ChatGPT about suicide weekly

#312

Post by anonzzzies »

Auracle wrote: post #159
NathanKP wrote: post #3

> It is estimated that more than one in five U.S. adults live with a mental illness (59.3 million in 2022; 23.1% of the U.S. adult population). https://www.nimh.nih.gov/health/statistics/mental-illness Most people don't understand just how mentally unwell the US population is. Of course there are one million talking to ChatGPT about suicide weekly. This is not a surprising stat at all. It's just a question of what to…

~11% of the US population is on antidepressants. I'm not, but I personally know the biggest detriment to my mental health is just how infrequently I'm in social situations. I see my friends perhaps once every few months. We almost all have kids. I'm perfectly willing and able to set aside more time than that to hang out, but my kids are both very young still and we're aren't drowning in sports/activities yet (hopeful…

> We almost all have kids.

Maybe that? I see most my close friends daily and we all do not have kids.

Re: OpenAI says over a million people talk to ChatGPT about suicide weekly

#313

Post by simonklitj »

SiempreViernes wrote: post #309
Al-Khwarizmi wrote: post #278

Earlier quoted context omitted.

In my previous post up the thread I said that we should measure whether in fact it does more good than harm or not. That's the context of my comment, I'm not saying we should just take it for granted without looking.

> we should measure whether in fact it does more good than harm or not The demonstrable harms include assisting suicide, there's is no way to ethically continue the measurement because continuing the measurements in their current form will with certainty result in further deaths.

Thank you! On top of that, it’s hard to measure “potential suicides averted,” and comparing that with “actual suicides caused/assisted with” would be incommnsurable.

And working to set a threshold for what we would consider acceptable? No thanks

Re: OpenAI says over a million people talk to ChatGPT about suicide weekly

#314

Post by RobotToaster »

grey-area wrote: post #253
Al-Khwarizmi wrote: post #247

Earlier quoted context omitted.

Supposing that the advice it provides does more good than harm, why? What's the objective reason? If it can save lives, who cares if the advice is based on intelligence and understanding or on regurgitating internet content?

First, do no harm.

The Hippocratic oath originated from Hippocratic medicine forbidding surgery, which is why surgeons are still not referred to as "doctor" today.

Re: OpenAI says over a million people talk to ChatGPT about suicide weekly

#315

Post by BrandoElFollito »

I wonder how many of these exchanges are from "legitimate" people trying to get advice on how to commit suicide.

Assisted suicide is a topic my government will not engage into (France, we have some ridiculous discussions poking the subject with a 10 m pole) so many people are left to themselves. They will then either go for the well-known (but miserable) solutions, or look at Belgium, the Netherlands or Switzerland (thanks god we have these countries nearby).

Re: OpenAI says over a million people talk to ChatGPT about suicide weekly

#317

Post by alex-moon »

Al-Khwarizmi wrote: post #236
probably_wrong wrote: post #229

If you haven't read the article (or even if you have but didn't click on outgoing links twice ) the NYT story about how ChatGPT convinced a suicidal teen not to look for help [1] should convince you that ChatGPT should be nowhere near anyone dealing with psychological issues. Here's ChatGPT discouraging said teenager from asking for help: > “I want to leave my noose in my room so someone finds it and tries to stop me…

We would need the big picture, though... maybe it caused that death (which is awful) but it's also saving lives? If there are that many people confiding in it, I wouldn't be surprised if it actually prevents some suicides with encouraging comments, and that's not going to make the news. Before declaring that it shouldn't be near anyone with psychological issues, someone in the relevant field should study whether the…

I suspect you've never done therapy yourself. Most people who have worked with a professional therapist understand intuitively why the only helpful feedback from an LLM to someone who needs professional help is: get professional help. AIs are really good at doing something to about 80%. When the stakes are life or death, as they are with someone who is suicidal, that is a good example of a time when 80% isn't good enough.

In such cases, where a new approach offers to replace an existing approach, the burden of proof is on the challenger, not the incumbent. This is why we have safety regulations, why we don't let people drive cars that haven't passed tests, build with materials that haven't passed tests, eat food that hasn't passed tests. You understand then, hopefully, why your comments here are dangerous...? I have no doubt you have no malicious intent here - you're right that these decisions need to be based on data - but you're not taking into account that the (potentially extremely harmful) challenger already has a foothold in the field.

I know that you will want to hear this from experts in the "relevant field" rather than myself, so here is a write-up from Stanford on the subject: https://hai.stanford.edu/news/exploring-the-dangers-of-ai-in...

Re: OpenAI says over a million people talk to ChatGPT about suicide weekly

#318

Post by akimbostrawman »

jacquesm wrote: post #187

HIPAA anybody? (1) they probably shouldn't even have that data (2) they shouldn't have it lying around in a way that it an be attributed to particular individuals (3) imagine that it leaks to the wrong party, it would make the hack of that Finnish institution look like child's play (4) if people try to engage it in such conversations the bot should immediately back out because it isn't qualified to have these convers…

If there is somebody in current year that still thinks they would not store, process/train and use/sell all data, they do probably need to see a doctor.

Re: OpenAI says over a million people talk to ChatGPT about suicide weekly

#319

Post by wongarsu »

jacquesm wrote: post #215
true_religion wrote: post #192

Earlier quoted context omitted.

HIPAA only applies to covered healthcare entitites. If you walk into a McDonalds' and talk about your suicidal ideation with the cashier, that's not HIPAA covered. To become a covered entity, the business has either work with a healhcare provider, health data trasmiter, or do business as one. Notably, even in the above case, HIPAA only applies to the healthcare part of the entity. So if McDonald's collocated pharmaci…

Arguably, if you start giving answers to these kind of questions your chatbot just became a medical device.

Then the McDonalds cashier also becomes a medical practitioner the moment they tell you that killing yourself isn't the answer. And if I tell my friend via SMS that I am thinking about suicide, do both our phones now also become HIPAA-covered medical devices?

Re: OpenAI says over a million people talk to ChatGPT about suicide weekly

#320

Post by Lerc »

probably_wrong wrote: post #229

If you haven't read the article (or even if you have but didn't click on outgoing links twice ) the NYT story about how ChatGPT convinced a suicidal teen not to look for help [1] should convince you that ChatGPT should be nowhere near anyone dealing with psychological issues. Here's ChatGPT discouraging said teenager from asking for help: > “I want to leave my noose in my room so someone finds it and tries to stop me…

While I agree that AI shouldn't be sycophantic, and I also agree that the AI shouldn't have said those things.

> “Please don’t leave the noose out,” ChatGPT responded. “Let’s make this space the first place where someone actually sees you.”

That is not sycophantic behaviour, it is asserting a form of control of the situation. The bot made a direct challenge to the suggestion.

I only just realised this now reading your comment, but I hardly ever see responses that push back against what I say like that.

Post reply on HN