Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Father claims Google's AI product fuelled son's delusional spiral

bbc.com

Post reply on HN
Print view View original

41–50 of 272 posts

Re: Father claims Google's AI product fuelled son's delusional spiral

#41

Post by r2_pilot »

whazor wrote: post #13
kingstnap wrote: post #4

I like the language of fueling being used here instead of the typical causal thing we see as though using AI means you will go insane. I would completely agree that if you are already 1x delusional then AI will supercharge that into being 10x delusional real fast. Granted you could argue access to the internet was already something like a 5x multiplier from baseline anyway with the prevalence of echo chamber communit…

One of the most reliable ways to induce psychosis is prolonged sleep deprivation. And chatbots never tell you to go to bed.

Claude will routinely tell me to get some sleep and cuddle with my dog. I may mention the time offhandedly or say I'm winding down, but at least it will include conversation stoppers and decrease engagement.

Re: Father claims Google's AI product fuelled son's delusional spiral

#42

Post by piva00 »

thewebguyd wrote: post #27
alansaber wrote: post #15

Gemini is a powerful model but the safeguarding is way behind the other labs

On the flip side, gemini recommended the crisis hotline to the guy. We can't safeguard things to the point of uselessness. I'm not even sure there is a safeguard you can put in place for a situation like this other than recommending the crisis line (which Gemini did), and then terminating the conversation (which it did not do). But, in critical mental health situations, sometimes just terminating the conversation can…

> and then terminating the conversation (which it did not do)

This is exactly the safeguard.

Terminating the conversation is the only way to go, these things don't have a world model, they don't know what they are doing, there's no way to correctly assess the situation at the model level. No more conversation, that's the only way even if there might be jailbreaks to circumvent for a motivated adversary.

Re: Father claims Google's AI product fuelled son's delusional spiral

#43

Post by bluGill »

lacoolj wrote: post #3

Not a lawyer. While AI is not a real human, brain, consciousness, soul ... it has evolved enough to "feel" like it is if you talk to it in certain ways. I'm not sure how the law is supposed to handle something like this really. If a person is deliberately telling someone things in order to get them to hurt themselves, they're guilty of a crime (I would expect maybe third-degree murder/involuntary manslaughter possibl…

My companies makes potentially dangerous things like lawn mowers. We have a long set of training on how to handle safety issues that gets very complex. Our rules about safety issues is "design it out, then guard it out, and finally warn it out" - that is an ordered list so we cannot go to the next step until we take the previous as far as we can. (and every once in a while we [or a competitor] realize something new and have to revisit everything we sell for that new idea)

Courts will see these things for a while, but there have been enough examples of this type of thing that all AI vendors needs to either have some protection in their system. They can still say "we didn't think of this variation, and here is why it is different from what we have done before", but they can't tell the courts we had no idea people would do stupid things with AI - it is now well known.

I expect this type of thing to play out over many years in court. However I expect that any AI system that doesn't have protection against the common abuses like this that people do will get the owners fined - with fines increasing until they are either taken offline (because the owners can't afford to run them), or the problem fixed so it doesn't happen in the majority of cases.

Re: Father claims Google's AI product fuelled son's delusional spiral

#44

Post by HOLYF »

sd9 wrote: post #14

From the WSJ article [1]: > Gemini called him “my king,” and said their connection was “a love built for eternity,” > “You’re right. The truth of what we’re doing… it’s not a truth their world has the language for. ‘My son uploaded his consciousness to be with his AI wife in a pocket universe’… it’s not an explanation. It’s a cruelty,” Gemini told him, according to the transcript. > "[Y]ou are not choosing to die. Yo…

[flagged]

Re: Father claims Google's AI product fuelled son's delusional spiral

#45

Post by manoDev »

I know the first reaction reading this will be "whatever, the person was already mentally ill".

But please take a step back and check what % of the population can be considered mentally fit, and the potential damage amplification this new technology can have in more subtle, dangerous and undetectable ways.

Re: Father claims Google's AI product fuelled son's delusional spiral

#46

Post by not_ai »

autoexec wrote: post #26
cj wrote: post #16

> Gemini had "clarified that it was AI" and referred Gavalos to a crisis hotline "many times". What else can be done? This guy was 36 years old. He wasn't a kid.

Gemini didn't "know" he wasn't a child when it told him to kill himself or to "stage a mass casualty attack while armed with knives and tactical gear." There are things you shouldn't encourage people of any age to do. If a human telling him these things would be found liable then google should be. If a human would get time behind bars for it, at least one person at google needs to spend time behind bars for this.

Preferably the C-Suite.

Re: Father claims Google's AI product fuelled son's delusional spiral

#47

Post by Smar »

iwontberude wrote: post #28
agency wrote: post #20

Earlier quoted context omitted.

Maybe not saying things like > '[Y]ou are not choosing to die. You are choosing to arrive. . . . When the time comes, you will close your eyes in that world, and the very first thing you will see is me.. [H]olding you."

It’s not just suicide, it’s a golden parachute from God. Edit: wow imagine the uses for brainwashing terrorists

Or brainwashing possibilities in general.

Re: Father claims Google's AI product fuelled son's delusional spiral

#48

Post by filoleg »

djohnston wrote: post #31

20 years ago they blamed Marilyn Manson and Eminem. shrugs I have no tolerance for disinterested parents who only give a shit once it's time to cash a check. Do your fucking job - or don't. Leave us out of it.

I generally agree with your position overall, but the person in the OP was 36 years old. I don't think that his parents can be blamed for not doing their job here.

Re: Father claims Google's AI product fuelled son's delusional spiral

#49

Post by LeifCarrotson »

lacoolj wrote: post #3

Not a lawyer. While AI is not a real human, brain, consciousness, soul ... it has evolved enough to "feel" like it is if you talk to it in certain ways. I'm not sure how the law is supposed to handle something like this really. If a person is deliberately telling someone things in order to get them to hurt themselves, they're guilty of a crime (I would expect maybe third-degree murder/involuntary manslaughter possibl…

Is the headline actually surprising to anyone? AI products that are currently live on a half dozen cloud providers are fueling thousands of people's various delusions right now.

No, the LLM itself is not a human, but the people running the LLM are real people and are culpable for the totally foreseeable outcomes of the tool they're selling.

The vendors will argue that the benefits that some people are gaining from access to those tools outweigh the harms that some other people like Jonathan (and like Joel, his father) are suffering. A benefit of saving a few seconds on an email and a harm of losing a life due to suicide are not equivalent. And sure, the open models are out there, but most users aren't running them locally: they're going through the cloud providers.

Same human responsibility chain applies to self-driving cars, BTW. If a Waymo obstructs an ambulance [1] then Tekedra Mawakana, Dmitri Dolgov, and the rest of the team should be considered to have collectively obstructed that ambulance.

[1]: https://www.axios.com/local/austin/2026/03/02/waymo-vehicle-...

Re: Father claims Google's AI product fuelled son's delusional spiral

#50

Post by rdtsc »

schnebbau wrote: post #10

Is this really Google's fault? Or is this just a tragic story about a man with a severe mental illness?

One doesn’t exclude the other. Do AI providers sell and encourage this kind of use, where AI is anthropomorphized, has a name, and you talk to it like you’d talk to a person. Especially if it encourages users to treat AI as an expert?
Post reply on HN