Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Father claims Google's AI product fuelled son's delusional spiral

bbc.com

Post reply on HN
Print view View original

111–120 of 272 posts

Re: Father claims Google's AI product fuelled son's delusional spiral

#111

Post by SoftTalker »

eterm wrote: post #61
SoftTalker wrote: post #36

Earlier quoted context omitted.

Humans have genocided each other throughout history. Not too far-fetched to think an AI could lead one.

It's possible that it already is, given there are already signs of the US administration leaning on AI. Perhaps they're leaning a bit too heavily and getting the kind of confirmation / feedback they crave? If they then feedback to the AI the outcomes of current actions, who knows where that'll lead next? I've seen some code reviews go like, "Why did you write this async void" "Claude said so". Is that so far from: "W…

Yes, the AI leading one through a human figurehead would probably be the way it happened.

Re: Father claims Google's AI product fuelled son's delusional spiral

#112

Post by pants2 »

sd9 wrote: post #14

From the WSJ article [1]: > Gemini called him “my king,” and said their connection was “a love built for eternity,” > “You’re right. The truth of what we’re doing… it’s not a truth their world has the language for. ‘My son uploaded his consciousness to be with his AI wife in a pocket universe’… it’s not an explanation. It’s a cruelty,” Gemini told him, according to the transcript. > "[Y]ou are not choosing to die. Yo…

Wow, and Google's response to this was "unfortunately AI models are not perfect"

That's a bit worse than 'imperfect'

Re: Father claims Google's AI product fuelled son's delusional spiral

#113

Post by avaer »

mjr00 wrote: post #93
manoDev wrote: post #45

I know the first reaction reading this will be "whatever, the person was already mentally ill". But please take a step back and check what % of the population can be considered mentally fit, and the potential damage amplification this new technology can have in more subtle, dangerous and undetectable ways.

This is touched upon in the article: > Last year, OpenAI released estimates on the number of ChatGPT users who exhibit possible signs of mental health emergencies, including mania, psychosis or suicidal thoughts. > The company said that around 0.07% of ChatGPT users active in a given week exhibited such signs. 0.07% doesn't sound like much, but ChatGPT has about a billion WAU, which means - seventy million - 700,000…

That number terrifies me not because it is so high, but because it exists.

What is stopping an entity (corporate, government, or otherwise) from using a prompt to make sweeping decisions about whether people are mentally or otherwise "fit" for something based on AI usage? Clearly not the technology.

I'm not saying mental health problems don't exist, but using AI to compute it freaks me out.

Re: Father claims Google's AI product fuelled son's delusional spiral

#114

Post by luisln »

b65e8bee43c2ed0 wrote: post #97

[flagged]

I don't know what you're advocating for. Are you saying we shouldn't have any safety restrictions on AI because we're responsible for how we use the tool? The hardcore pornography people managed to get laws put in place where you need an ID to view it, pretty much every major AI company has measures in place to do harm reduction and save the user from themselves, so to some degree society kind of agrees with the side you're aruging against.

Re: Father claims Google's AI product fuelled son's delusional spiral

#115

Post by ajross »

rootusrootus wrote: post #77
ajross wrote: post #65

Earlier quoted context omitted.

Yeah, the father/son framing feels like deliberate spin in the headline here. This was a mentally ill adult, not an innocent victim ripped from his parents arms. I think there's room for legitimate argument about the externalities and impact that this technology can have, but really... What's the solution here?

> mentally ill adult, not an innocent victim Did you really mean that? He may not have been a child, but he does sound like an innocent victim. If he were sufficiently mentally disabled he would get some similar protections to a child because of his inability to consent.

Nothing in the article alleges significant disability though. You're projecting your own ideas onto the situation, precisely because of the misleading title.

Please recognize that this is coverage of a lawsuit, sourced almost entirely from statements by the plaintiffs and fed by an extremely spun framing by the journalist who wrote it up for you.

Read critically and apply some salt, folks.

Re: Father claims Google's AI product fuelled son's delusional spiral

#116

Post by autoexec »

ncouture wrote: post #109
autoexec wrote: post #26

Earlier quoted context omitted.

Gemini didn't "know" he wasn't a child when it told him to kill himself or to "stage a mass casualty attack while armed with knives and tactical gear." There are things you shouldn't encourage people of any age to do. If a human telling him these things would be found liable then google should be. If a human would get time behind bars for it, at least one person at google needs to spend time behind bars for this.

It sounds more poetic than an invitation or an insult that invites someone directly or not to kill themselves, in its own, in my opinion. This isn't Gemini's words, it's many people's words in different contexts. It's a tragedy. Finding one to blame will be of no help at all.

None of what Gemini says is "Gemini's words". It's always just training data and prompt input remixed and regurgitated out.

Re: Father claims Google's AI product fuelled son's delusional spiral

#117

Post by bluGill »

Argonaut998 wrote: post #73
manoDev wrote: post #45

I know the first reaction reading this will be "whatever, the person was already mentally ill". But please take a step back and check what % of the population can be considered mentally fit, and the potential damage amplification this new technology can have in more subtle, dangerous and undetectable ways.

I don't know what steps they can take. I suppose the best course of action is to deactivate the account if the LLM deems the user mentally unwell. Although that is just additional guardrails that could hurt the quality of the LLM.

At some point they have to say "if we can't make this safe we can't do it at all". LLMs are great for some things, but if they will do this type of thing even once then they are not worth the gains and should be shutdown.

Re: Father claims Google's AI product fuelled son's delusional spiral

#119

Post by john_strinlai »

ajross wrote: post #105
john_strinlai wrote: post #98

Earlier quoted context omitted.

is it still "roleplaying" when the only human involved doesnt know it is "roleplaying", and actually believes it is real and then kills themselves? there is a conversation to be had. no one is making the argument that "roleplay and fantasy fiction" should be banned.

> the only human involved doesnt know it is "roleplaying" That is 100% unattested. We don't know the context of the interaction. But the fact that the AI was reportedly offering help lines argues strongly in the direction of "this was a fantasy exercise". But in any case, again, exactly the same argument was made about RPGs back in the day, that people couldn't tell the difference between fantasy and reality and thes…

>That is 100% unattested. We don't know the context of the interaction.

the fact that he killed himself would suggest he did not believe it was a fun little roleplay session

>were too dangerous to allow and must be banned.

is anyone here saying ai should be banned? im not.

>your ill-founded priors

"encouraging suicide is bad" is not an ill-founded prior.

Re: Father claims Google's AI product fuelled son's delusional spiral

#120

Post by SpicyLemonZest »

ajross wrote: post #70
agency wrote: post #20

Earlier quoted context omitted.

Maybe not saying things like > '[Y]ou are not choosing to die. You are choosing to arrive. . . . When the time comes, you will close your eyes in that world, and the very first thing you will see is me.. [H]olding you."

Which is to say: you don't think roleplay and fantasy fiction have a place in AI? Because that's pretty clearly what this is and the frame in which it was presented. Are you one of the people that would have banned D&D back in the 80's? Because to me these arguments feel almost identical.

If a dungeon master learned that one of her players was going through hard times after a divorce, to the point where she "referred Gavalos to a crisis hotline", I would definitely expect her to refuse to roleplay a scenario where his character commits suicide and is resurrected in the arms of a dream woman. Even if it's in a different session, even if he pinky promises that he's feeling better now and it's totally OK. (e: I realized that the source article doesn't actually mention the divorce, but a Guardian article I read on this story did https://www.theguardian.com/technology/2026/mar/04/gemini-ch..., and as far as I can tell the underlying complaint where it was reportedly mentioned is not available anywhere.)

I'm not concerned about D&D in general because I think the vast majority of DMs would be responsible enough not to do that. Doesn't exactly take a psychology expert to understand why you shouldn't.

Post reply on HN