Well if you tell people your auto complete algorithm is actually a potentially sentient AI and it goes on to auto complete someone's suicidal science fiction fantasy, what did you expect. Everyone calling these things "AI" is complicit. You can't rely on everyone understanding that you're just a greedy scammer trying to fool investors, there are side effects.
Father claims Google's AI product fuelled son's delusional spiral
261–270 of 272 posts
Re: Father claims Google's AI product fuelled son's delusional spiral
#262I think that LLMs should not be allowed to say "I". It should always be in third person. Instead of "I can write this for you" it should say "This machine can write this for you" or with a store front name "Google can write this for you". To operate on a given text or while generating texts it should divide what is meta from what is direct. This generated text "quote" should be styled different, a bit boring: smaller…
They don't work well if you do that: https://arxiv.org/html/2509.16332v1 - they apparently require a sense of self to preform the way they do - You'd have to do it deterministically on the front end I suspect.
2 problems of many are:
context windows + compacting + whatever they do behind the scene to stitch cohesive narrative over time - single LLM convos should just never be allowed to get that long, you can effectively build whatever you want as a personality, people return to the same convo a lot, It gets wild fast.
Cross Section memory is a bad feature as implemented in many chat bots, the memory feature can effectively poison any conversation. Anthropic figured this out, their memory feature is a search not a memory that informs the personality/manipulates the current session.
Re: Father claims Google's AI product fuelled son's delusional spiral
#263Re: Father claims Google's AI product fuelled son's delusional spiral
#264As someone with a very intimate experience with psychosis, if it isn't Gemini it will be Catcher in the Rye or the Bible or the signs in a store window. What needs to be fixed is how the system flags people experiencing psychosis. The one advantage of psychosis being mediated by Gemini is that the system can be built to alert health services unlike a book or a sign.
Re: Father claims Google's AI product fuelled son's delusional spiral
#265As someone with a very intimate experience with psychosis, if it isn't Gemini it will be Catcher in the Rye or the Bible or the signs in a store window. What needs to be fixed is how the system flags people experiencing psychosis. The one advantage of psychosis being mediated by Gemini is that the system can be built to alert health services unlike a book or a sign.
Exactly. Psychosis (and other mental illnesses) will find something to attach itself to.
The opportunity here is for Google and other LLMs to include safeguards (and be very clear about them) and processes to direct the user, or, in extreme cases, direct health services to avoid a tragedy like this.
Re: Father claims Google's AI product fuelled son's delusional spiral
#266Earlier quoted context omitted.
Wow, and Google's response to this was "unfortunately AI models are not perfect" That's a bit worse than 'imperfect'
"Imperfect" is when your AI model tells the user that there are two Rs in "strawberry", or that they should use glue to keep the cheese from falling off their pizza. Repeatedly encouraging the user to kill themself so that they can meet the AI model in the afterlife is on quite another level.
[flagged]
Re: Father claims Google's AI product fuelled son's delusional spiral
#267Earlier quoted context omitted.
" [...] Gemini sent Gavalas to a location near Miami International Airport where he was instructed to stage a mass casualty attack while armed with knives and tactical gear. " isnt very poetic
These are all bits and pieces of a long-running conversation. Was there a roleplay element involved?
this isn't D&D, and AI shouldn't be instructing people go to anywhere near an airport while LARPing.
read the article. it's bad, man.
Re: Father claims Google's AI product fuelled son's delusional spiral
#268Earlier quoted context omitted.
Preferably the C-Suite.
I understand the impulse in this direction, but I’m not sure it would serve as much of a disincentive, as there would likely just be a highly-paid scapegoat. Why not something more lasting and less difficult to ignore, like compulsory disclosure of the model’s source code (in addition to compensation for the victim(s)). Compulsory disclosure of the source would be a massive disadvantage.
> as there would likely just be a highly-paid scapegoat
the point of executives is someone has to take responsibility. that's why they get paid. the buck has to stop somewhere.
Re: Father claims Google's AI product fuelled son's delusional spiral
#269Earlier quoted context omitted.
It's the gun control debate in a different outfit. I don't know if Google is doing _enough_, that can be debated. But if someone is repeatedly ignoring warnings (as the article claims) then maybe we should blame the person performing the act. Even if we perfectly sanitized every public AI provider, people could just use local AI.
It's absolutely not the gun control debate in a different outfit. The difference is in how abuse of the given system affects others. This AI affected this person and his actions affected himself. Nothing about the AI enhanced his ability to hurt others. Guns enhance the ability of mentally unstable people to hurt others with ruthless efficiency. That's the real gun debate -- whether they should be so easy to get give…
it's a terrible analogy
the guns aren't centrally owned by a giant mega-corp with literally thousands of devs and the ability to put guard rails on them
Re: Father claims Google's AI product fuelled son's delusional spiral
#270Earlier quoted context omitted.
If I tell you to kill yourself and you go through with it, will I get into legal trouble or not?
There are definitely jurisdictions in the US (perhaps most or all of them) that have laws which say yes, inciting suicide is a crime.
There are ways to get around that: "Hey, go drive a Tesla on autopilot"