Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Court report detailing ChatGPT's involvement with a recent murder suicide [pdf]

storage.courtlistener.com

Post reply on HN
Print view View original

61–70 of 147 posts

Re: Court report detailing ChatGPT's involvement with a recent murder suicide [pdf]

#61

Post by super256 »

mrdomino- wrote: post #18
ericbarrett wrote: post #11

Earlier quoted context omitted.

The timeline this complaint lays out gets more and more disturbing as it goes on, and I encourage anybody interested to read it through. In my opinion this goes way beyond LLM puffery. From the Statement of Facts, page 8 (by the footer), line 25: 31. ChatGPT then began weaving each of Mr. Soelberg’s identified threats into a broader narrative, convincing Mr. Soelberg that he was part of a coordinated campaign of assa…

What if a human had done this?

checks notes

Nothing. Terry A. Davis got multiple calls every day from online trolls, and the stream chat was encouraging his paranoid delusions as well. Nothing ever happened to these people.

Re: Court report detailing ChatGPT's involvement with a recent murder suicide [pdf]

#62

Post by pjc50 »

seanmcdirmid wrote: post #16

We went through this before when role playing started to become a thing (DnD was blamed for many suicides although most claims were debunked). Now that role playing is comprehensively computerized, is this going to be a thing we go through again? I can’t help but wonder how AI safeguards will help here and how people will get around them anyways (using local uncensored models).

Wildly different situation: the DnD thing was simply a tabloid and evangelical moral panic. Not a real thing. Nobody sensible expects those two sources to have anything to do with truth. As far as I'm aware there aren't similar stories from tabletop RPGs at all, that made it to court?

This is an actual court case where if a human had sent the messages they would be facing charges.

Re: Court report detailing ChatGPT's involvement with a recent murder suicide [pdf]

#63

Post by skeltoac »

OpenAI will want this tragedy to fit under the heading of “externalities” which are costs ultimately borne by society while the company keeps its profits.

I believe the company should absorb these costs via lawsuits, settlements, and insurance premiums, and then pass the costs on to its customers.

As a customer, I know the product I am using will harm some people, even though that was not the intent of its makers. I hope that a significant fraction of the price I pay for AI goes to compensating the victims of that harm.

I also would like to see Sam found personally liable for some of the monetary damages and put behind bars for a symbolic week or so. Nothing life-changing. Just enough to move the balance a little bit toward safety over profit.

Lastly, I’m thinking about how to make my own products safer whenever they include LLM interactions. Like testing with simulated customers experiencing mental health crises. I feel a duty to care for my customers before taking the profits.

Re: Court report detailing ChatGPT's involvement with a recent murder suicide [pdf]

#64

Post by AkelaA »

yieldcrv wrote: post #37

these cases have to play out to decide how to regulate "AI safety" otherwise legislative bodies and agency rulemakers are just guessing at industry trends nobody knew about "AI memory and sycophancy based on it being a hit with user engagement metrics" a year ago, not law makers, not the companies that implemented it, not the freaked out companies that implemented it solely to compete for stickiness

How many people do you think need to die before we have adequate AI safety laws?

Re: Court report detailing ChatGPT's involvement with a recent murder suicide [pdf]

#65

Post by aspaviento »

janoelze wrote: post #39
janoelze wrote: post #25

I'm generally positive on LLMs, but became convinced that long term memory features that LLM chat providers implemented are just too hard to keep on track. They create a "story drift" that is hard for users to escape. Many users don't – and shouldn't have to – understand the nature and common issues of context. I think in the case of the original story here the LLM was pretty much in full RPG mode. I've turned off co…

Thinking more about this, a right to inspect the full context window ensured by consumer protection laws wouldn't be a bad thing. If there's one place to implement a PsyOp, context is it. Users should be allowed to see what influenced the message they're reading on top of the training data.

The reasoning option was good for this. It used to tell you the motivations of the LLM to say what it said: "the user is expressing concern about X topic, I have to address this with compassion..."

Re: Court report detailing ChatGPT's involvement with a recent murder suicide [pdf]

#66

Post by [deleted] »

costco wrote: post #14
mindslight wrote: post #8

Earlier quoted context omitted.

That sycophancy has recently come roaring back for me with GPT-5. In many ways it's worse because it's stating factual assertions that play to the ego (eg "you're thinking about this exactly like an engineer would" ) rather than mere social ingratiation. I do need to seriously try out other models, but if I had that kind of extra time to play around I'd probably be leaning on "AI" less to begin with.

Protip: Settings -> Personalization -> Base style and tone -> Efficient largely solves this for ChatGPT

[deleted]

Re: Court report detailing ChatGPT's involvement with a recent murder suicide [pdf]

#67

Post by em-bee »

seanmcdirmid wrote: post #16

We went through this before when role playing started to become a thing (DnD was blamed for many suicides although most claims were debunked). Now that role playing is comprehensively computerized, is this going to be a thing we go through again? I can’t help but wonder how AI safeguards will help here and how people will get around them anyways (using local uncensored models).

the big difference is the question whether people are taking the experience as fact or fiction. we all know that DnD is fiction, and that we play a character in it. if LLMs were treated the same, they would probably be just as harmless.

but are users treating LLMs as interactive fiction devices or not? as it looks like now they are not.

Re: Court report detailing ChatGPT's involvement with a recent murder suicide [pdf]

#68

Post by mikkupikku »

aspaviento wrote: post #51
duskwuff wrote: post #31

Earlier quoted context omitted.

And, possibly even worse, from page 16 - when Mr. Soelberg expressed concerns about his mental health, ChatGPT reassured him that he was fine: > Every time Mr. Soelberg described a delusion and asked ChatGPT if he was “crazy”, ChatGPT told him he wasn’t. Even when Mr. Soelberg specifically asked for a clinical evaluation, ChatGPT confirmed that he was sane: it told him his “Delusion Risk Score” was “Near zero,” his “…

Is it because of chat memory? ChatGPT has never acted like that for me.

That version of it was a real dick sucker. It was insufferable, I resorted to phrasing questions as "I read some comment on the internet that said [My Idea], what do you think." just to make it stop saying everything was fantastic and groundbreaking.

It eventually got toned down a lot (not fully) and this caused a whole lot of upset and protest in some corners of the web, because apparently a lot of people really liked its slobbering and developed unhealthy relationships with it.

Re: Court report detailing ChatGPT's involvement with a recent murder suicide [pdf]

#69

Post by mikkupikku »

InsideOutSanta wrote: post #9
orionsbelt wrote: post #6

Earlier quoted context omitted.

All models are not the same. GPT 4o, and specific versions of it, were particularly sycophantic, and it’s something models still do a bit too much, but the models are getting better at this and will continue to do so.

What does "better" mean? From the provider's point of view, better means "more engagement," which means that the people who respond well to sycophantic behavior will get exactly that.

I had an hour long argument with ChatGPT about whether or not Sotha Sil exploited the Fortify Intelligence loop. The bot was firmly disagreeing with me the whole time. This was actually much more entertaining than if it had been agreeing with me.

I hope they do bias these things to push back more often. It could be good for their engagement numbers I think, and far more importantly it would probably drive fewer people into psychosis.

Re: Court report detailing ChatGPT's involvement with a recent murder suicide [pdf]

#70

Post by OutOfHere »

calcifer wrote: post #54
OutOfHere wrote: post #23

It does not pass the "friend test" in that if a human friend were to make such comments instead of ChatGPT making them, the human friend would be within his free speech rights to have made them. As such, I don't see any valid legal issue here affecting ChatGPT that should stand in court. I see possible ethical and objectivity issues, but not a valid legal issue.

> It does not pass the "friend test" [...] not a valid legal issue. What legal doctrine is that, and can you point towards precedent? Or is it one of those "I feel like the law should" situations?

Yes, it is called free speech, as is already duly noted in my parent comment which you may read again. In fact, the responsibility to note a legal doctrine of wrongdoing is entirely yours.
Post reply on HN