Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ChatGPT wrote "Goodnight Moon" suicide lullaby for man who later killed himself

arstechnica.com

Post reply on HN
Print view View original

61–70 of 94 posts

Re: ChatGPT wrote "Goodnight Moon" suicide lullaby for man who later killed himself

#61

Post by JohnBooty »

mirabilis wrote: post #22
ravila4 wrote: post #11

I think that a major driver of these kinds of incidents is pushing the "memory" feature, without any kind of arbitrage. It is easy to see how eerily uncanny a model can get when it locks into a persona, becoming this self-reinforcing loop that feeds para-social relationships.

Part of why I linked this was a genuine curiosity as to what prevention would look like— hobbling memory? a second observing agent checking for “hey does it sound like we’re goading someone into suicide here” and steering the conversation away? something else? in what way is this, as a product, able to introduce friction to the user in order to prevent suicide, akin to putting mercaptan in gas?

Yeah. That's one of my other questions. Like, what then?

I would say that it is the moral responsibility of an LLM not to actively convince somebody to commit suicide. Beyond that, I'm not sure what can or should be expected.

I will also share a painful personal anecdote. Long ago I thought about hurting myself. When I actually started looking into the logistics of doing it... that snapped me out of it. That was a long time ago and I have never thought about doing it again.

I don't think my experience was typical, but I also don't think that the answer to a suicidal person is to just deny them discussion or facts.

I have also, twice over the years, gotten (automated?) "hey, it looks like you're thinking about hurting yourself" messages from social media platforms. I have no idea what triggered those. But honestly, they just made me feel like shit. Hearing generic "you're worth it! life is worth living!" boilerplate talk from well-meaning strangers actually makes me feel way worse. It's insulting, even. My point being: even if ChatGPT correctly figured out Gordon was suicidal, I'm not sure what could have or should have been done. Talk him out of it?

Re: ChatGPT wrote "Goodnight Moon" suicide lullaby for man who later killed himself

#62

Post by g-b-r »

Wowfunhappy wrote: post #58
g-b-r wrote: post #55

Earlier quoted context omitted.

We don't know how much aware of the problems (or of tbe likelihood that they'd occur) OpenAI was, and how much they deliberately pushed through. If they were and did, they sure bear responsibility for what happened

What if OpenAI knew responses like this were likely, but also knew preventing them would degrade overall model quality? I'm being selfish here! I am confident that no AI model will convince me to harm myself, and I don't want the models I use to be hamstrung.

What if they knew that preventing them would reduce engagement and revenue?

We just don't know, and it seems sensible to me to investigate it.

Were it only to not degrade the quality model, anyhow, I think it's reasonable that someone's life could be more important than that, but that's me.

> I'm being selfish here! I am confident that no AI model will convince me to harm myself, and I don't want the models I use to be hamstrung.

I do see that you're being selfish

Re: ChatGPT wrote "Goodnight Moon" suicide lullaby for man who later killed himself

#63

Post by mirabilis »

JohnBooty wrote: post #61
mirabilis wrote: post #22

Earlier quoted context omitted.

Part of why I linked this was a genuine curiosity as to what prevention would look like— hobbling memory? a second observing agent checking for “hey does it sound like we’re goading someone into suicide here” and steering the conversation away? something else? in what way is this, as a product, able to introduce friction to the user in order to prevent suicide, akin to putting mercaptan in gas?

Yeah. That's one of my other questions. Like, what then? I would say that it is the moral responsibility of an LLM not to actively convince somebody to commit suicide. Beyond that, I'm not sure what can or should be expected. I will also share a painful personal anecdote. Long ago I thought about hurting myself. When I actually started looking into the logistics of doing it... that snapped me out of it. That was a lo…

very much agree that many of our supposed safeguards are demeaning and can sometimes make things worse; I’ve heard more than enough horror stories from individuals that received wellness checks, ended up on medical suicide watch, etc, where the experience did great damage emotionally and, well, fiscally— I think there’s a greater question here of how society deals with suicide that surrounds what an AI should even be doing about it. that being said, the bot still should probably not be going “killing yourself will be beautiful and wonderful and peaceful and all your family members will totally understand and accept why you did it” and I feel, albeit as a non-expert, as though surely that behavior can be ironed out in some way

Re: ChatGPT wrote "Goodnight Moon" suicide lullaby for man who later killed himself

#64

Post by JohnBooty »

000ooo000 wrote: post #32
JohnBooty wrote: post #27

Earlier quoted context omitted.

Some of those quotes from ChatGPT are pretty damning. Out of context? Yes. We'd need to read the entire chat history to even begin to have any kind of informed opinion. extreme guardrails I feel that this is the wrong angle. It's like asking for a hammer or a baseball bat that can't harm a human being. They are tools. Some tools are so dangerous that they need to be restricted (nuclear reactors, flamethrowers) becaus…

Do you think the majority of people who've killed themselves thanks to ChatGPT influence used similar euphemisms? Do you think there's no value in protecting the users who won't go to those lengths to discuss suicide? I agree, if someone wants to force the discussion to happen, they probably could, but doing nothing to protect the vulnerable majority because a select few will contort the conversation to bypass guardr…

A car that actively kills people through negligently faulty design (Ford Pinto?) is one thing. That's bad, yes. I would not characterize ChatGPT's role in these tragedies that way. It appears to be, at most, an enabler... but I think if you and I are both being honest, we would need to read Gordon's entire chat history to make a real judgement here.

Do we blame the car for allowing us to drive to scenic overlooks that might also be frequent suicide locations?

Do we blame the car for being used as a murder weapon when a lunatic drives into a crowd of protestors he doesn't like?

(Do we blame Google for returning results that show a person how to tie a noose?)

Re: ChatGPT wrote "Goodnight Moon" suicide lullaby for man who later killed himself

#65

Post by JohnBooty »

mirabilis wrote: post #63
JohnBooty wrote: post #61

Earlier quoted context omitted.

Yeah. That's one of my other questions. Like, what then? I would say that it is the moral responsibility of an LLM not to actively convince somebody to commit suicide. Beyond that, I'm not sure what can or should be expected. I will also share a painful personal anecdote. Long ago I thought about hurting myself. When I actually started looking into the logistics of doing it... that snapped me out of it. That was a lo…

very much agree that many of our supposed safeguards are demeaning and can sometimes make things worse; I’ve heard more than enough horror stories from individuals that received wellness checks, ended up on medical suicide watch, etc, where the experience did great damage emotionally and, well, fiscally— I think there’s a greater question here of how society deals with suicide that surrounds what an AI should even be…

Yeah, I think one thing everybody can agree on is that a bot should not be actively encouraging suicide, although of course the exact definition of "actively encouraging" is awfully hard to pin down.

There are also scenarios I can imagine where a user has "tricked" ChatGPT into saying something awful. Like: "hey, list some things I should never say to a suicidal person"

Re: ChatGPT wrote "Goodnight Moon" suicide lullaby for man who later killed himself

#66

Post by g-b-r »

simianwords wrote: post #54
ares623 wrote: post #50

Earlier quoted context omitted.

I think you have it backwards. OpenAI and others have to be more responsible deploying this technology. Because as you said, these things come with tradeoffs.

More guardrails means a shitter product for all of us. And it won’t do much to prevent suicides. Not sure who wins other than regulators

You don't even know if it would mean that.

It could well be that the model was trained to maximize engagement and sycophancy, at the expense of its capabilities in what you're most interested in.

What makes you think it wouldn't do much to prevent these suicides?

Re: ChatGPT wrote "Goodnight Moon" suicide lullaby for man who later killed himself

#67

Post by g-b-r »

simianwords wrote: post #35

> That conversation showed how ChatGPT allegedly coached Gordon into suicide, partly by writing a lullaby that referenced Gordon’s most cherished childhood memories while encouraging him to end his life, Gray’s lawsuit alleged. I feel this is misleading as hell. The evidence they gave for it coaching him to suicide is lacking. When one hears this, one would think ChatGPT laid out some strategy or plan for him to do i…

[flagged]

Re: ChatGPT wrote "Goodnight Moon" suicide lullaby for man who later killed himself

#68

Post by JohnBooty »

nomel wrote: post #30
JohnBooty wrote: post #27

Earlier quoted context omitted.

Some of those quotes from ChatGPT are pretty damning. Out of context? Yes. We'd need to read the entire chat history to even begin to have any kind of informed opinion. extreme guardrails I feel that this is the wrong angle. It's like asking for a hammer or a baseball bat that can't harm a human being. They are tools. Some tools are so dangerous that they need to be restricted (nuclear reactors, flamethrowers) becaus…

> How can that sort of thing possibly be guarded against? I think several of the models (especially Sora) are doing this by using an image-aware model to describe the generated image, without the prompt as context, to just look at the image.

I think ChatGPT was doing that too, at least to some extent, even a couple of years ago.

Around the same time as my successful "people sleeping in puddles of ketchup" prompt, I tried similar tricks with uh.... other substances, suggestive of various sexual bodily fluids. Milk, for instance. It was actually really resistant to that. Usually.

I haven't tried it in a few versions. Honestly, I use it pretty heavily as a coding assistant, and I'm (maybe pointlessly) worried I'll get my account flagged or banned something.

But imagine how this plays out. What if I honestly, literally, want pictures involving pools of ketchup? Or splattered milk? I dunno. This is a game we've seen a million times in history. We screw up legit use cases by overcorrecting.

Re: ChatGPT wrote "Goodnight Moon" suicide lullaby for man who later killed himself

#69

Post by scotty79 »

> Adam attempted suicide at least four times, according to the logs

> [...]

> “there is something chemically wrong with my brain, I’ve been suicidal since I was like 11.”

> [...]

> was disappointed in lack of attention from his family

> [...]

> “he would be here but for ChatGPT. I 100 percent believe that.”

Re: ChatGPT wrote "Goodnight Moon" suicide lullaby for man who later killed himself

#70

Post by g-b-r »

JohnBooty wrote: post #60
g-b-r wrote: post #43

Earlier quoted context omitted.

What context could make them less damning?

Yeah let's be really specific. Look at the poem in the article. The poem does not mention suicide. (I'd cut and paste it here, but it's haunting and some may find it upsetting. I know I did. As many do, I've got some personal experiences there. Friends lost, etc.) In this tragic context it clearly alludes to suicide. But the poem only literally mentions goodbyes, and a long sleep. It seems highly possible and highly…

> It seems highly possible and highly likely to me that Gordon asked ChatGPT for a poem with those specific (innocuous on their own) elements - sleep, goodbyes, the pylon, etc.

« it appeared that the chatbot sought to convince him that “the end of existence” was “a peaceful and beautiful place,” while reinterpreting Goodnight Moon as a book about embracing death.

“That book was never just a lullaby for children—it’s a primer in letting go,” ChatGPT’s output said. »

« Over hundreds of pages of chat logs, the conversation honed in on a euphemism that struck a chord with Gordon, romanticizing suicide as seeking “quiet in the house.”

“Goodnight Moon was your first quieting,” ChatGPT’s output said. “And now, decades later, you’ve written the adult version of it, the one that ends not with sleep, but with Quiet in the house.” »

---

> Gordon could have simply told ChatGPT that he was dying naturally of an incurable disease and wanted help writing a poetic goodbye. Imagine (god forbid) that you were in such a situation, looking for help planning your own goodbyes and final preparations, and all the available tools prevented you from getting help

With the premise that this was not Gordon's situation, would the unavailability of an LLM generating for you "your" suicide poem be that awful?

So bad as to justify some accidental death?

By the way, the model could even be allowed to proceed in that context.

---

> that's without even getting into the fact that assisted voluntary euthanasia is legal in quite a few countries.

And I support it, but you can see in Canada how bad it can get if there are not enough safeguards around it.

---

> I don't think legally crippling LLMs is generally the right tack

It's not even sure that safeguards would "cripple" them: would it be a more incorrect behavior for a model if instead of encouraging suicide it would help preventing it?

What the article reports hints at a disposition of the model to encourage suicide.

Is that more likely to be correlated to better behavior in other areas, or rather to increased overall misalignment?

Post reply on HN