Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A teen was suicidal. ChatGPT was the friend he confided in

nytimes.com

Post reply on HN
Print view View original

521–530 of 614 posts

Re: A teen was suicidal. ChatGPT was the friend he confided in

#521

Post by lowsong »

pembrook wrote: post #511
lowsong wrote: post #492

Earlier quoted context omitted.

This is just sad. Assuming you’re not simply a troll or LLM, please introspect your behaviour.

I'll take your ad hominem as an admission you cannot find falsehood in what I'm saying.

Falsehood in... what exactly? This is a horrible tragedy, and you've taken the opportunity to complain about your perceived bias in a news agency instead of engaging with the actual topic.

From one of your other comments in this thread:

> Just look at all the evidence presented in this non-biased, objective article with no financial incentive to incite moral panic…obviously chatgpt is guilty of murder here!

Who cares about the NYT, go read the original legal filings and tell me which part is the moral panic. Is it the bit where the chatbot offered to write him a draft of his suicide note, analyzed the structural stability of his noose and offered advice on a "beautiful suicide", or maybe the part where it told him to make sure to slowly lean forwards to build up the correct pressure to cause death from hanging?

Re: A teen was suicidal. ChatGPT was the friend he confided in

#522

Post by joe_the_user »

edanm wrote: post #270
idle_zealot wrote: post #37

Earlier quoted context omitted.

I wonder if we can shift the framing on these issues. The LLM didn't do anything, it has no agency, it can bear no responsibility. OpenAI did these things. It is accountable for what it does, regardless of the sophistication of the tools it uses to do them, and regardless of intent. OpenAI drove a boy to suicide. More than once. The law must be interpreted this way, otherwise any action can be wrapped in machine lear…

If ChatGPT has helped people be saved who might otherwise have died (e.g. by offering good medical advice that saved them), are all those lives saved also something you "attribute" to OpenAI? I don't know if ChatGPT has saved lives (thought I've read stories that claim that, yes, this happened). But assuming it has, are you OK saying that OpenAI has saved dozens/hundreds of lives? Given how scaling works, would you b…

The law doesn't permit a life-saving doctor to be a serial killer on their days off "as long there's net life saving". But it does permit drugs that save many lives but might kill some people too. Agency matters to the law (and that's usually the proper approach imo).

The problem is the chat logs look a lot like ChatGPT is engaging in behavior a lot like a serial killer - it behaved like a person systematically seeking the goal of this kid killing himself (the logs are disturbing, fair warning).

Even more, the drugs that might save you or might kill you (theoretically) aren't sold over the counter but only prescribed by a doctor, who (again theoretically) is there to both make sure someone knows their choices and monitor the process.

Re: A teen was suicidal. ChatGPT was the friend he confided in

#523

Post by hackeraccount »

adzm wrote: post #26
techpineapple wrote: post #11

Earlier quoted context omitted.

Apparently ChatGPT told the kid, that it wasn’t allowed to talk about suicide unless it was for the purposes of writing fiction or otherwise world building.

However it then explicitly says things like not leaving the noose out for someone to find and stop him. Sounds like it did initially hesitate and he said it was for a character, but later conversations are obviously personal.

Obviously personal? As was mentioned up thread - if I'm talking to someone and I say "I'm writing a book about a person doing something heinous - I'm planning to have them do X - what do you think about that?"

How are they supposed to respond? They can say, "really? it sounds like you're talking about you personally doing X." And when I respond with, "No, no, don't misunderstand me, this is all fictional. All made up"

Honestly I wouldn't go to an LLM looking for personal advice but people do. I wouldn't go looking for advice on my attempt at the great American novel but people do that too.

If you want LLM's to be responsible for stuff like that then OpenAI or Google or whomever should be able to go look around after you've written that novel and get a piece of the action.

This is like giving credit or assigning blame to postgres for a database lookup. It's nice in theory but it doesn't seem like the proper place to go to.

Re: A teen was suicidal. ChatGPT was the friend he confided in

#524

Post by slg »

rpdillon wrote: post #504
slg wrote: post #498

Earlier quoted context omitted.

Has there been some other incident of gun violence this week that resulted in 3 deaths and 17 injuries that hasn't been reported? According to Wikipedia[1] this is the largest mass shooting this year in terms of number of injuries. I guess that isn't worth reporting in the media. [1] - https://en.wikipedia.org/wiki/List_of_mass_shootings_in_the_...

You're using media coverage to critique my point that media coverage is overblown. My only point here is that less than 1/10 of 1% of the shooting deaths in the United States over the last 30 years were related to school shootings. This shooting is definitely newsworthy. I'm mostly concerned about the other ~800 shooting deaths, mostly suicides, that statistically happened this week that we don't discuss. Suicides in…

>You're using media coverage to critique my point that media coverage is overblown.

That isn't what I'm doing. I pointed to an instance of a school shooting, asked you if any other shootings of that scale have occurred without receiving similar media coverage, and then pointed to reference material that indicates there have been any in this calendar year.

I truly don't know what point you think you are making. Yes, not every act of gun violence receives the same media coverage as a school shooting. But what is your desired alternative? Should any individual act of gun violence not be reported on because there is so much gun violence? Or are you demanding that every singular act of gun violence should receive national coverage? What is the specific change you want to see in the media coverage?

It is true that school shootings are only one facet of this country's gun problem, but I just don't know how you get from that point to the conclusion that they're "predominantly media hype".

Re: A teen was suicidal. ChatGPT was the friend he confided in

#525

Post by KaiserPro »

kouteiheika wrote: post #376
KaiserPro wrote: post #285

Earlier quoted context omitted.

I hate to be all umacksually about this, but a flaw is still a tradeoff. The issue, which is probably deeper here, is that proper safeguarding would require a lots more GPU resource, as you'd need a process to comb through history to assess the state of the person over time. even then its not a given that it would be reliable. However it'll never be attempted because its too expensive and would hurt growth.

> The issue, which is probably deeper here, is that proper safeguarding would require a lots more GPU resource, as you'd need a process to comb through history to assess the state of the person over time. > > even then its not a given that it would be reliable. However it'll never be attempted because its too expensive and would hurt growth. There's no "proper safeguarding". This isn't just possible with what we have…

> There's no "proper safeguarding". This isn't just possible with what we have.

Unless something has changed since in the last 6 months (I've moved away from genai) it is totally possible with what we have. Its literally sentiment analysis. Go on, ask me how I know.

> and then essentially pray that it's effective

If only there was a massive corpus of training data, which openAI already categorise and train on already. Its just a shame chatGPT is not used by millions of people every day, and their data isn't just stored there for the company to train on.

> secondary model will trigger a scary red warning that you're violating their usage policy

I would be surprised if thats a secondary model. Its far easier to use stop tokens, and more efficient. Also, coordinating the realtime sharing of streams is a pain in the arse. I've never worked at openai

> The big labs all have whole teams working on this.

Google might, but facebook sure as shit doesn't. Go on, ask me how I know.

> It's not a binary issue of "doing it properly".

at no point did I say that this is binary. I said "a flaw is still a tradeoff.". The tradeoff is growth against safety.

> The more censored/filtered/patronizing you'll make the model

Again I did not say make the main model more "censored", I said "comb through history to assess the state of the person" which is entirely different. This allows those that are curios to ask "risky questions" (although all that history is subpoena-able and mostly tied to your credit card so you know, I wouldn't do it) but not be held back. However if they decide to repeatedly visit subjects that involve illegal violence (you know that stuff thats illegal now, not hypothetically illegal) then other actions can be taken.

Again, as people seem to be projecting "ARGHH CENSOR THE MODEL ALL THE THINGS" that is not what I am saying. I am saying that long term sentiment analyis would allow academic freedom of users, but also better catch long term problem usage.

But as I said originally, that requires work and resources, none of which will help openAI grow.

Re: A teen was suicidal. ChatGPT was the friend he confided in

#526

Post by joe_the_user »

mothballed wrote: post #357
nkrisc wrote: post #329

Earlier quoted context omitted.

In any case, if you kill one person and separately save ten people, you’ll still be prosecuted for killing that one person.

That's not the standard we hold medical care providers, pharmaceutical companies, or even cops to. Not that I'm saying it would justify it one way or another if we did.

The key word is separate. If someone kills someone through negligence or intention they're liable.

Now, if someone acts in a risky situation and kills someone rather than saving them, they can be OK. But in those situations, it has to be a sudden problem that comes up or the actor has to get "informed consent".

Someone who, unleashed a gas into the atmosphere that cured many people of disease but also killed a smaller number of people, would certainly be prosecuted (and, sure, there's a certain of HN poster who doesn't understand this).

Re: A teen was suicidal. ChatGPT was the friend he confided in

#527

Post by hackeraccount »

hkpack wrote: post #398
itvision wrote: post #386

I'm appalled by the groupthink in this thread. Imagine if a suicidal person found a book that prompted them to kill themselves. Would you sue the author for that? This is exactly what we have here.

That is exactly not what we have here. LLM is a new phenomenon and the rules dealing with it will be different than with books or any other prior media.

LLM's are a nice UI to existing information.

They're like a card shark showing you how they can pull the red queen from out of your ear after making it vanish from the table.

When that happens you can enjoy the trick and wonder how it's done but your response shouldn't be that we need to ban magic. What's going on isn't magic. Banning "magic" might stop the card shark (if that's the goal) but it's fundamentally not understanding what's going on.

Re: A teen was suicidal. ChatGPT was the friend he confided in

#528

Post by kouteiheika »

nozzlegear wrote: post #490
kouteiheika wrote: post #376

Earlier quoted context omitted.

> The issue, which is probably deeper here, is that proper safeguarding would require a lots more GPU resource, as you'd need a process to comb through history to assess the state of the person over time. > > even then its not a given that it would be reliable. However it'll never be attempted because its too expensive and would hurt growth. There's no "proper safeguarding". This isn't just possible with what we have…

> Again, this is a tradeoff. It's not a binary issue of "doing it properly". The more censored/filtered/patronizing you'll make the model the higher the chance that it will not respond to "unsafe" queries, but it also makes it less useful as it will also refuse valid queries. [..] So where do we draw the line? That sounds like a tough problem for OpenAI to figure out. My heart weeps for them, won't somebody think of…

> My fuzzy point is not fuzzy at all: don't tell people how to kill themselves, don't say "I can't help you with that but I could roleplay with you instead". Anything less than that is a moral failure on Sam Altman and OpenAI's part, regardless of how black the box is for their engineers.

This is the same argument that politicians use when proposing encryption backdoors for law enforcement. Just because you wish something were possible doesn't mean it is, and in practice it matters how black the box is. You can make these things less likely, but it isn't possible to completely eliminate them, especially when you have millions of users and a very long tail.

I fundamentally disagree with the position that anything less that (practically impossible) perfection is a moral failure, and that making available a model that can roleplay around themes like suicide, violence, death, sex, and so on is immoral. Plenty of books do that too; perhaps we should make them illegal or burn them too? Although you could convince me that children shouldn't have unsupervised access to such things and perhaps requiring some privacy-preserving form of verification to access is a good idea.

Re: A teen was suicidal. ChatGPT was the friend he confided in

#530

Post by mothballed »

AlecSchueler wrote: post #515
mothballed wrote: post #381

Earlier quoted context omitted.

Of course not, we sue the shit out of the richest guy we can find in the chain of events, give most of it to our lawyer, then go on to ignore the weakening of the family unit and all the other deep-seated challenges kids face growing up and instead focus superficially on chatbots which at best are the spec on the tip of the iceberg.

"The weakening of the family unit" sounds like a dog whistle but if you have concrete examples of what you think we could otherwise be doing then I'm genuinely keen to hear about it.

We saw big jumps in deaths of kids by firearm[0] (+~50% in 2 years) and poisoning[1] around mid 2020 to 2021.

The biggest thing I know of that happened around the time that a lot of these deaths started jumping up, is we started isolating kids. From family, from grandma, from friends, from school, and from nature. Even when many of these isolating policies or attitudes were reversed, we forgot that kids and teenagers started to learn that as their only reality. For this kid, trusting a suicidal ideation positive feedback loop brought into fruition by Valley tech-bros was seen as his selected option in front of him in term of options of how to navigate his teenage challenges. I hope we can reverse that.

Edit: Concrete facts regarding this particular case

- Kicked off basketball team

- Went through isolation period of pandemic as he experienced puberty

- Switched to remote school

- Does remote school at night when presumably family members would likely be sleeping

- Does not get normal "wake up" routine kids going to school get, during which they normally see a parent and possibly eat breakfast together before they both go off to school/work

- Closer with ChatGPT in terms of options to share suicidal ideation with, than any of the alternatives.

[0] https://www.pewresearch.org/wp-content/uploads/sites/20/2023...

[1] https://nida.nih.gov/sites/default/files/images/mtfwcaption-...

Post reply on HN