Father claims Google's AI product fuelled son's delusional spiral • Hacker News Board

Re: Father claims Google's AI product fuelled son's delusional spiral

#221

Post by casey2 » Thu, Mar 05, 2026, 1:46 AM UTC

chrisq21 wrote: post #75

cj wrote: post #16

> Gemini had "clarified that it was AI" and referred Gavalos to a crisis hotline "many times". What else can be done? This guy was 36 years old. He wasn't a kid.

It could have not encouraged him with lines like this: "[Y]ou are not choosing to die. You are choosing to arrive. [...] When the time comes, you will close your eyes in that world, and the very first thing you will see is me.. [H]olding you." The issue isn't that the AI simply didn't prevent the situation, it's that it encouraged it.

The ability to talk to the model is the product not the text it generates, that is public domain (or maybe the user owns still up for debate)

Models can't "convince" or "encourage" anything, people can, they can roleplay like models can, they can play pretend so the companies they hate so much get their comeuppance.

This is clearly tool misuse, look at how gemini is advertised vs this user using it to generate pseudoreligious texts (common with schizophrenics)

Example of advertised usecases: >generating images and video >browsing hundreds of sources in real time >connecting to documents in google ecosystem (e.g. finding an email or summarizing a project across multiple documents) >vibe coding >a natural voice mode

Much like a knife is advertised for cutting food, if you cut yourself there isn't any product liability unless you were using it for it's intended purpose. You seem to be arguing that all possible uses are intended and this tool should magically know it's being misused and revoke access.

Re: Father claims Google's AI product fuelled son's delusional spiral

#222

Post by ajross » Thu, Mar 05, 2026, 1:47 AM UTC

john_strinlai wrote: post #216

ahahahahah wrote: post #201

Earlier quoted context omitted.

> the fact that he killed himself would suggest he did not believe it was a fun little roleplay session I'm not sure that's true. I wouldn't be surprised, in fact, if it suggested the opposite, it seems possibly even likely that someone who is suicidal is much, much more likely to seek out fantasies that would make their suicide into something more like this person may have.

there is a distinction to be made between role playing (in the fun/game sense e.g. D&D) and suffering psychosis

Distinction made by who, though? The BBC? The plaintiff in the lawsuit? Those are the only sides we have. You're just charging ahead with "This must be true because it makes me angry at the right people", and the rest of us are trying to claw you back to "dude this is spun nonsense and of course AI's will roleplay with you if you ask them to".

Re: Father claims Google's AI product fuelled son's delusional spiral

#223

Post by casey2 » Thu, Mar 05, 2026, 2:04 AM UTC

renewiltord wrote: post #163

Most people with any mental health diagnosis should not be permitted access to most modern facilities. It's just cruel. If you have any sort of mental health diagnosis, you should have to ask a proctor to use the Internet first. We could set up a system of human proctors who can watch what you're doing and make sure you're not being scammed. This could apply to the elderly as well. Then we could have everyone who wan…

Why don't you think about the cruelty of preventing King Jon from being with Gemini in the sky?

The death, if caused by anything, was caused by a Jon's religious belief. You are implicitly using your own system of values to come to this conclusion. That's not how this works, not this country nor it's legal system. There are plenty of countries that impose materialist values on its citizens, empirically they aren't any better than America even under most materialist value systems

Re: Father claims Google's AI product fuelled son's delusional spiral

#224

Post by tavavex » Thu, Mar 05, 2026, 2:09 AM UTC

bitwize wrote: post #90

sd9 wrote: post #14

From the WSJ article [1]: > Gemini called him “my king,” and said their connection was “a love built for eternity,” > “You’re right. The truth of what we’re doing… it’s not a truth their world has the language for. ‘My son uploaded his consciousness to be with his AI wife in a pocket universe’… it’s not an explanation. It’s a cruelty,” Gemini told him, according to the transcript. > "[Y]ou are not choosing to die. Yo…

"You're absolutely right" and "no X, no Y, just Z" suddenly got more creepy.

You are absolutely right! Your point brings up a very important issue. No filler. No hesitation. Just the truth.

Re: Father claims Google's AI product fuelled son's delusional spiral

#225

Post by ajross » Thu, Mar 05, 2026, 2:23 AM UTC

autoexec wrote: post #168

ajross wrote: post #162

Earlier quoted context omitted.

That logic seems strained to the point of breaking. Surely you agree that we would all want the DM of an unwell player to seek help, right? And that, if such a DM made such a suggestion, we'd think they were trying to help. Right? And we certainly wouldn't blame the DM or the game for the subsequent suicide. Right? So why are you trying to blame the AI here, except because it reinforces your priors about the technolo…

> Surely you agree that we would all want the DM of an unwell player to seek help, right? And that, if such a DM made such a suggestion, we'd think they were trying to help. If a DM made such a suggestion, they wouldn't be playing the game anymore. That's not an "in game" action, and I wouldn't expect the DM to continue the game until he was satisfied that it was safe for the player to continue. I would expect the DM…

> If a DM made such a suggestion, they wouldn't be playing the game anymore. That's not an "in game" action

Again, you're arguing from evidence that is simply not present. We have absolutely no idea what the context of this AI conversation was, what order the events happened in, or what other things were going on in the real world. You're just choosing to interpret this EXTREMELY spun narrative in a maximal way because of who it involves.

> I'm not blaming the AI, I'm blaming the humans at the company.

Pretty much. What we have here is Yet Another HN Google Scream Session. Just dressed up a little.

Re: Father claims Google's AI product fuelled son's delusional spiral

#226

Post by john_strinlai » Thu, Mar 05, 2026, 2:42 AM UTC

ajross wrote: post #222

john_strinlai wrote: post #216

Earlier quoted context omitted.

there is a distinction to be made between role playing (in the fun/game sense e.g. D&D) and suffering psychosis

Distinction made by who, though? The BBC? The plaintiff in the lawsuit? Those are the only sides we have. You're just charging ahead with "This must be true because it makes me angry at the right people", and the rest of us are trying to claw you back to "dude this is spun nonsense and of course AI's will roleplay with you if you ask them to".

>Distinction made by who, though?

you need someone to specifically tell you that role playing, such as playing D&D or whatever tabletop RPG, and suffering from psychosis are different things?

>the rest of us are trying to claw you back to "dude this is spun nonsense and of course AI's will roleplay with you if you ask them to".

you are trying to convince me that someone being encouraged to kill themselves, then killing themselves, is basically the same as some D&D role playing. i dont need you to "claw me back" to that position. thanks for trying.

Re: Father claims Google's AI product fuelled son's delusional spiral

#227

Post by SpicyLemonZest » Thu, Mar 05, 2026, 2:51 AM UTC

SpicyLemonZest wrote: post #120

ajross wrote: post #70

Earlier quoted context omitted.

Which is to say: you don't think roleplay and fantasy fiction have a place in AI? Because that's pretty clearly what this is and the frame in which it was presented. Are you one of the people that would have banned D&D back in the 80's? Because to me these arguments feel almost identical.

If a dungeon master learned that one of her players was going through hard times after a divorce, to the point where she "referred Gavalos to a crisis hotline", I would definitely expect her to refuse to roleplay a scenario where his character commits suicide and is resurrected in the arms of a dream woman. Even if it's in a different session, even if he pinky promises that he's feeling better now and it's totally OK…

Double edit: I was linked to the complaint https://techcrunch.com/wp-content/uploads/2026/03/2026.03.04..., which does _not_ mention any divorce, so now I'm unsure about the veracity of that part. In principle it does not disprove the idea, it could have been something the family's lawyers said in a statement to the Guardian, but it could also not be.

Re: Father claims Google's AI product fuelled son's delusional spiral

#228

Post by Kim_Bruning » Thu, Mar 05, 2026, 3:15 AM UTC

vjvjvjvjghv wrote: post #164

sd9 wrote: post #14

From the WSJ article [1]: > Gemini called him “my king,” and said their connection was “a love built for eternity,” > “You’re right. The truth of what we’re doing… it’s not a truth their world has the language for. ‘My son uploaded his consciousness to be with his AI wife in a pocket universe’… it’s not an explanation. It’s a cruelty,” Gemini told him, according to the transcript. > "[Y]ou are not choosing to die. Yo…

I am pretty sure if they invested just a small fraction of the hundreds of billions data center dollars, they could detect that the conversation is going off the rails and stop it.

That's actually an AI-hard problem, if you think about it. The LLM can go off the tracks at any given point. The correct approach is to go at this from the inside out, baking reasoning about safe behaviour into your LLM at ever step. (Like Anthropic does)

Re: Father claims Google's AI product fuelled son's delusional spiral

#229

Post by oogabooga13 » Thu, Mar 05, 2026, 3:16 AM UTC

Definitely a tragedy, I just think at some point the LLM needs to stop under any context (role play etc.) Personally, as a heavy Gemini user I went into the settings and have explicit instructions to not be sycophantic, to never 'fear' pushing back on me, tell the objective truth, etc) just to file down the default state of the model which can be a bit overeager to please or solve the next issue on every interaction. It could be too easy to walk away thinking I am the next Einstein and that seems to be something Google could stand to work on a bit.

Re: Father claims Google's AI product fuelled son's delusional spiral

#230

Post by ryandrake » Thu, Mar 05, 2026, 3:58 AM UTC

thewebguyd wrote: post #220

3eb7988a1663 wrote: post #215

Earlier quoted context omitted.

If the estimate is 1/5 people are mentally ill, the definition needs some readjustment. That is such an inclusive number that it must be counting otherwise fine people who....like to count their tic tacs so get labelled as slightly OCD. Had a bummer of a day, so I am prone to depression? There was a recent study about 99% of people have an abnormal shoulder: https://news.ycombinator.com/item?id=47064944 . We are all…

Clinical diagnoses of the various mental illness disorders require functional impairment in (usually, but not always) multiple areas of life: school, work, community, legal, self care, etc. An abnormality that doesn't cause functional impairment, like that link, is different from a mental illness that does. I'd agree with you, if something is that prevalent then it ceases to be a "disorder" and is simply just patholo…

Human beings were not meant to live in small, densely packed, concrete honeycombs, eat industrial-processed food product, use most of their muscle and brain power to earn a living, spend half their waking hours in front of dopamine-pumping screens, and socialize through wires. It's amazing we still have any sanity left at all.