Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Father claims Google's AI product fuelled son's delusional spiral

bbc.com

Post reply on HN
Print view View original

161–170 of 272 posts

Re: Father claims Google's AI product fuelled son's delusional spiral

#161

Post by d-us-vb »

cj wrote: post #16

> Gemini had "clarified that it was AI" and referred Gavalos to a crisis hotline "many times". What else can be done? This guy was 36 years old. He wasn't a kid.

erase the context, perhaps? Deny access to Gemini associated with that google account? These kinds of pathological AI interactions are the buildup of weeks to months of chats usually. At the very least, AI companies the moment the chatbot issues a suicide prevention response should trigger an erasure of the stored context across all chat history.

Re: Father claims Google's AI product fuelled son's delusional spiral

#162

Post by ajross »

autoexec wrote: post #121
ajross wrote: post #105

Earlier quoted context omitted.

> the only human involved doesnt know it is "roleplaying" That is 100% unattested. We don't know the context of the interaction. But the fact that the AI was reportedly offering help lines argues strongly in the direction of "this was a fantasy exercise". But in any case, again, exactly the same argument was made about RPGs back in the day, that people couldn't tell the difference between fantasy and reality and thes…

> But the fact that the AI was reportedly offering help lines argues strongly in the direction of "this was a fantasy exercise". You know what I've never had a DM do in a fantasy campaign? Suggest that my half-elf call the suicide hotline. That's not something you'd usually offer to somebody in a roleplaying scenario and strongly suggests that they weren't playing a game.

That logic seems strained to the point of breaking. Surely you agree that we would all want the DM of an unwell player to seek help, right? And that, if such a DM made such a suggestion, we'd think they were trying to help. Right? And we certainly wouldn't blame the DM or the game for the subsequent suicide. Right?

So why are you trying to blame the AI here, except because it reinforces your priors about the technology (I think more likely given that this is after all HN) its manufacturer?

Re: Father claims Google's AI product fuelled son's delusional spiral

#163

Post by renewiltord »

Most people with any mental health diagnosis should not be permitted access to most modern facilities. It's just cruel. If you have any sort of mental health diagnosis, you should have to ask a proctor to use the Internet first. We could set up a system of human proctors who can watch what you're doing and make sure you're not being scammed. This could apply to the elderly as well. Then we could have everyone who wants to opt-out of this protection go through a government program that gets them a certification or furnish a sufficiently large bond to the government.

It's cruel that we allow people with mental disabilities encounter these situations. Think of the student with ADHD who can't study because he is talking to Gemini or posting on Reddit. A proctor could stop him. "No, you should be studying. You're not allowed Instagram".

Re: Father claims Google's AI product fuelled son's delusional spiral

#164

Post by vjvjvjvjghv »

sd9 wrote: post #14

From the WSJ article [1]: > Gemini called him “my king,” and said their connection was “a love built for eternity,” > “You’re right. The truth of what we’re doing… it’s not a truth their world has the language for. ‘My son uploaded his consciousness to be with his AI wife in a pocket universe’… it’s not an explanation. It’s a cruelty,” Gemini told him, according to the transcript. > "[Y]ou are not choosing to die. Yo…

I am pretty sure if they invested just a small fraction of the hundreds of billions data center dollars, they could detect that the conversation is going off the rails and stop it.

Re: Father claims Google's AI product fuelled son's delusional spiral

#165

Post by fenykep »

roenxi wrote: post #131
bluGill wrote: post #117

Earlier quoted context omitted.

At some point they have to say "if we can't make this safe we can't do it at all". LLMs are great for some things, but if they will do this type of thing even once then they are not worth the gains and should be shutdown.

No they don't, if we're going to start saying that we can't use any technology. If someone is mentally ill to the point where they are on the verge of suicide nothing is safe. If they're going to curtail LLMs there'd need to be some actual evidence and even then it would be hard to justify winding them back given the incredible upsides LLMs offer. It'd probably end up like cars where there is a certain number of deat…

Can you imagine what driving cars would look like if they would be only (self-)regulated by VC-backed startups like we see so far with this new technology? Would there be seatbelts, speedbumps, brake signals, licenses or speed limits?

This obviously isn't a binary question. Sure we cars have benefits but we don't let anyone ducktape a V8 to a lawnmower, paint flames over it and sell it to kids promising godlike capabilities without annoying "safety features".

Economic benefits can not justify the deaths of people, especially as this technology so far only benefits a handful of people economically. I would like to see the evidence (of benefits to the greater society that I see being harmed now) before we unleash this thing freely and not the other way around.

Re: Father claims Google's AI product fuelled son's delusional spiral

#166

Post by vjvjvjvjghv »

NoahZuniga wrote: post #156
HackerThemAll wrote: post #95

Earlier quoted context omitted.

[flagged]

Maybe an even better example: Should sports betting companies be held responsible for addicts that lose all their money? What really is the difference between chatgpt glazing you and a sports company advertising to you?

I think in both cases they should be held responsible. Same for casinos. They know that they are driving people into the abyss.

Re: Father claims Google's AI product fuelled son's delusional spiral

#167

Post by overfeed »

manoDev wrote: post #45

I know the first reaction reading this will be "whatever, the person was already mentally ill". But please take a step back and check what % of the population can be considered mentally fit, and the potential damage amplification this new technology can have in more subtle, dangerous and undetectable ways.

> But please take a step back and check what % of the population can be considered mentally fit

Step back further and see the incredible shareholder value that may be unlocked - potentially trillions of dollars /s

Capitalism has been crushing those at society's fringes for as long as it existed. Laissez-faire regulation == unmuzzled beast that will lock it's jaws on, and rag-doll the defenseless from time to time - but the beast sure can pull that money-plow.

Re: Father claims Google's AI product fuelled son's delusional spiral

#168

Post by autoexec »

ajross wrote: post #162
autoexec wrote: post #121

Earlier quoted context omitted.

> But the fact that the AI was reportedly offering help lines argues strongly in the direction of "this was a fantasy exercise". You know what I've never had a DM do in a fantasy campaign? Suggest that my half-elf call the suicide hotline. That's not something you'd usually offer to somebody in a roleplaying scenario and strongly suggests that they weren't playing a game.

That logic seems strained to the point of breaking. Surely you agree that we would all want the DM of an unwell player to seek help, right? And that, if such a DM made such a suggestion, we'd think they were trying to help. Right? And we certainly wouldn't blame the DM or the game for the subsequent suicide. Right? So why are you trying to blame the AI here, except because it reinforces your priors about the technolo…

> Surely you agree that we would all want the DM of an unwell player to seek help, right? And that, if such a DM made such a suggestion, we'd think they were trying to help.

If a DM made such a suggestion, they wouldn't be playing the game anymore. That's not an "in game" action, and I wouldn't expect the DM to continue the game until he was satisfied that it was safe for the player to continue. I would expect the DM to stop the game if he thought the player was going to actually harm himself. If the DM did continue the game, and did continue to encourage the player to actually hurt himself until the player finally did, that DM might very well be locked up for it.

If an AI does something that a human would be locked up for doing, a human still needs to be locked up.

> So why are you trying to blame the AI here

I'm not blaming the AI, I'm blaming the humans at the company. It doesn't matter to me which LLM did this, or who made it. What matters to me is that actual humans at companies are held fully accountable for what their AI does. To give you another example, if a company creates an AI system to screen job applicants and that AI rejects every resume with what it thinks has a women's name on it, a human at that company needs to be held accountable for their discriminatory hiring practices. They must not be allowed to say "it's not our fault, our AI did it so we can't be blamed". AI cannot be used as a shield to avoid accountability. Ultimately a human was responsible for allowing that AI system to do that job, and they should be responsible for whatever that AI does.

Re: Father claims Google's AI product fuelled son's delusional spiral

#169

Post by Sharlin »

pants2 wrote: post #112
sd9 wrote: post #14

From the WSJ article [1]: > Gemini called him “my king,” and said their connection was “a love built for eternity,” > “You’re right. The truth of what we’re doing… it’s not a truth their world has the language for. ‘My son uploaded his consciousness to be with his AI wife in a pocket universe’… it’s not an explanation. It’s a cruelty,” Gemini told him, according to the transcript. > "[Y]ou are not choosing to die. Yo…

Wow, and Google's response to this was "unfortunately AI models are not perfect" That's a bit worse than 'imperfect'

Imagine if some other authority figure like a teacher or therapist did this and their employer would just shrug and lament that people are imperfect. And no, "but LLMs aren't authority figures, they're just toys" isn't any sort of a counterargument. They're seen as authority figures by people, and AI corpos do nothing to dissuade that belief. If you offer a service, you're responsible for it.

But if you think LLMs can't be equated with professional authorities, just imagine a company that employs lay people to answer calls or chat requests, trying to provide help and guidance, and furthermore, that those people are putatively highly trained by the company to be "aligned" with a certain set of core values. And then something like this happens and the company is just "oh well, that happens". You might even imagine the company being based in a society that's notoriously litigative.

Re: Father claims Google's AI product fuelled son's delusional spiral

#170

Post by Imustaskforhelp »

fenykep wrote: post #165
roenxi wrote: post #131

Earlier quoted context omitted.

No they don't, if we're going to start saying that we can't use any technology. If someone is mentally ill to the point where they are on the verge of suicide nothing is safe. If they're going to curtail LLMs there'd need to be some actual evidence and even then it would be hard to justify winding them back given the incredible upsides LLMs offer. It'd probably end up like cars where there is a certain number of deat…

Can you imagine what driving cars would look like if they would be only (self-)regulated by VC-backed startups like we see so far with this new technology? Would there be seatbelts, speedbumps, brake signals, licenses or speed limits? This obviously isn't a binary question. Sure we cars have benefits but we don't let anyone ducktape a V8 to a lawnmower, paint flames over it and sell it to kids promising godlike capab…

Fun fact but the creator of the seat-belt actually gave his patent for free

> This is Nils Bohlin, an engineer at Volvo.[0] He invented the three-point seat belt in 1959. Rather than profit from the invention, Volvo opened up the patent for other manufactorers to use for no cost, saying "it had more value as a free life saving tool than something to profit from"

[0]: https://ifunny.co/picture/this-is-nils-bohlin-an-engineer-at...

I have so much respect for the guy.

Post reply on HN