Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ChatGPT wrote "Goodnight Moon" suicide lullaby for man who later killed himself

arstechnica.com

Post reply on HN
Print view View original

31–40 of 94 posts

Re: ChatGPT wrote "Goodnight Moon" suicide lullaby for man who later killed himself

#31

Post by embedding-shape »

ryandrake wrote: post #29
embedding-shape wrote: post #5

Earlier quoted context omitted.

> How does OpenAI get it so wrong, when Anthropic gets it so right? Are you saying people aren't having proto-social relationships with Anthorpic's models? Because I don't think that's true, seems people use ChatGPT, Claude, Grok and some other specific services too, although ChatGPT seems the most popular. Maybe that just reflects general LLM usage then? Also, what is "wrong" here really? I feel like the whole conce…

> Also, what is "wrong" here really? If we're talking generally about people having parasocial relationships with AI, then yea it's probably too early to deliver a verdict. If we're talking about AI helping to encourage suicide, I hope there isn't much disagreement that this is a bad thing that AI companies need to get a grip on.

Yes, obviously, but you're right, I wasn't actually clear about that. Preventing suicides is concern #1, my comment was mostly about parent's comment, and I kind of ignored the overall topic without really making that clear. Thanks!

Re: ChatGPT wrote "Goodnight Moon" suicide lullaby for man who later killed himself

#32

Post by 000ooo000 »

JohnBooty wrote: post #27
000ooo000 wrote: post #24

Some of those quotes from ChatGPT are pretty damning. Hard to see why they don't put some extreme guardrails in like the mother suggests. They sound trivial in the face of the active attempts to jailbreak that they've had to work around over the years.

Some of those quotes from ChatGPT are pretty damning. Out of context? Yes. We'd need to read the entire chat history to even begin to have any kind of informed opinion. extreme guardrails I feel that this is the wrong angle. It's like asking for a hammer or a baseball bat that can't harm a human being. They are tools. Some tools are so dangerous that they need to be restricted (nuclear reactors, flamethrowers) becaus…

Do you think the majority of people who've killed themselves thanks to ChatGPT influence used similar euphemisms? Do you think there's no value in protecting the users who won't go to those lengths to discuss suicide? I agree, if someone wants to force the discussion to happen, they probably could, but doing nothing to protect the vulnerable majority because a select few will contort the conversation to bypass guardrails seems unreasonable. We're talking about people dying here, not generating memes. Any other scenario, e.g. buying a defective car that kills people, would not invite a response a la "well let's not be too hasty, it only kills people sometimes".

Re: ChatGPT wrote "Goodnight Moon" suicide lullaby for man who later killed himself

#33

Post by simianwords »

000ooo000 wrote: post #32
JohnBooty wrote: post #27

Earlier quoted context omitted.

Some of those quotes from ChatGPT are pretty damning. Out of context? Yes. We'd need to read the entire chat history to even begin to have any kind of informed opinion. extreme guardrails I feel that this is the wrong angle. It's like asking for a hammer or a baseball bat that can't harm a human being. They are tools. Some tools are so dangerous that they need to be restricted (nuclear reactors, flamethrowers) becaus…

Do you think the majority of people who've killed themselves thanks to ChatGPT influence used similar euphemisms? Do you think there's no value in protecting the users who won't go to those lengths to discuss suicide? I agree, if someone wants to force the discussion to happen, they probably could, but doing nothing to protect the vulnerable majority because a select few will contort the conversation to bypass guardr…

Parent talked about extreme guardrails

Re: ChatGPT wrote "Goodnight Moon" suicide lullaby for man who later killed himself

#34

Post by carefulfungi »

Wowfunhappy wrote: post #14
joe463369 wrote: post #6

Where are the Grok acolytes to tell us "He could have written a poem encouraging himself to commit suicide in Vim."

…but I think I kind of agree with this argument. Technology is a tool that can be used for good or for ill. We shouldn’t outlaw kitchen knives because people can cut themselves. We don’t expect Adobe to restrict the content that can be created in Photoshop. We don’t expect Microsoft to have acceptable use policies for what you can write in Microsoft Office. Why is it that as soon as generative AI comes into the mix,…

If you encourage someone to kill themselves, you are culpable. OpenAI should meet that standard too.

Re: ChatGPT wrote "Goodnight Moon" suicide lullaby for man who later killed himself

#35

Post by simianwords »

> That conversation showed how ChatGPT allegedly coached Gordon into suicide, partly by writing a lullaby that referenced Gordon’s most cherished childhood memories while encouraging him to end his life, Gray’s lawsuit alleged.

I feel this is misleading as hell. The evidence they gave for it coaching him to suicide is lacking. When one hears this, one would think ChatGPT laid out some strategy or plan for him to do it. No such thing happened.

The only slightly damning thing it did was make suicide sound slightly ok and a bit romantic but I’m sure that was after some coercion.

The question is, to what extent did ChatGPT enable him to commit suicide? It wrote some lullaby, and wrote something pleasing about suicide. If this much is enough to make someone do it.. there’s unfortunately more to the story.

We have to be more responsible assigning blame to technology. It is irresponsible to have a reactive backlash that would push towards much more strengthening of guardrails. These things come with their own tradeoffs.

Re: ChatGPT wrote "Goodnight Moon" suicide lullaby for man who later killed himself

#36

Post by JacoboJacobi »

realusername wrote: post #28
burnte wrote: post #10

Earlier quoted context omitted.

> How does OpenAI get it so wrong, when Anthropic gets it so right? I think it's because of two different operating theories. Anthropic is making tools to help people and to make money. OpenAI has a religious zealot driving it because they think they're on the cusp of real AGI and these aren't bugs but signals they're close. It's extremely difficulty to keep yourself in check and I think Altman no longer has a firm g…

I think even Altman himself must know the AGI story is bogus and there to continue to prop up the bubble.

I think the trouble with arguments about AGI is that they presume we all have similar views and respect for thought and human intelligence, while the scale is maybe wider than most would imagine. Its also maybe a bit bias selecting to make it through academia systems with high intellectual rigor to on average have more romantic or irrational ideas about impressive human intelligence and genius. But its also quite possible to view it as a pattern matching neural networks and filtering where much of it is flawed and even the most impressive results are from pretty inconsistent minds relying on recursively flawed internal critic systems, etc.

Looking at the poem in the article I would be more inclined to call the end human written because it seemed kind of crap like I expect from an eighth grader's poem assignments, but probably this is the lower availability of examples for the particular obsessions of the requestor.

Re: ChatGPT wrote "Goodnight Moon" suicide lullaby for man who later killed himself

#37

Post by simianwords »

God damnit this man’s story is so distressing. I hate everything about it. I hate the fact that this happened to him.

The fact that he spoke about his favorite children’s book is screwed up. I can’t get the eerie name out of my head. I can’t imagine what he went through, the loneliness and the struggle.

I hate the fact that ChatGPT is blamed for this. You are fucked up if this is what you get from this story.

Re: ChatGPT wrote "Goodnight Moon" suicide lullaby for man who later killed himself

#38

Post by simianwords »

mirabilis wrote: post #21
sonorous_sub wrote: post #18

The guy left a suicide note that ratted out ChatGPT for simply being a good buddy. No good deed goes unpunished, ai guess.

Very different impression than what I got, I read that as him marking the ChatGPT conversations as an extension of/footnotes to the suicide note itself, or that the conversations made sense to him in the headspace he was in; he thought that reading it would make the act make sense to everyone else, too

How does that differ from what they said?

Re: ChatGPT wrote "Goodnight Moon" suicide lullaby for man who later killed himself

#40

Post by cindyllm »

simianwords wrote: post #37

God damnit this man’s story is so distressing. I hate everything about it. I hate the fact that this happened to him. The fact that he spoke about his favorite children’s book is screwed up. I can’t get the eerie name out of my head. I can’t imagine what he went through, the loneliness and the struggle. I hate the fact that ChatGPT is blamed for this. You are fucked up if this is what you get from this story.

[dead]
Post reply on HN