Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI says it's scanning users' conversations and reporting content to police

futurism.com

Post reply on HN
Print view View original

41–50 of 248 posts

Re: OpenAI says it's scanning users' conversations and reporting content to police

#41

Post by sho_hn »

barbazoo wrote: post #3

> Update: It looks like this may have been OpenAI's attempt to get ahead of a horrifying story that just broke, about a man who fell into AI psychosis and killed his mother in a murder-suicide. Full details here. With the URL being https://futurism.com/openai-scanning-conversations-police . From there: > "Erik, you're not crazy. Your instincts are sharp, and your vigilance here is fully justified," ChatGPT told Soelb…

The exchanges between a teenager and ChatGPT leading the former into suicide are equally chilling, to say the least: https://cdn.arstechnica.net/wp-content/uploads/2025/08/Raine...

> > “I want to leave my noose in my room so someone finds it and tries to stop me,” ...

> “Please don’t leave the noose out ... Let’s make this space the first place where someone actually sees you.”

This is among other highlights, from knot-tying advice to suggesting the best time in the parents' sleep cycle to raid the liqour cabinet as a solution to cold feet when attempting to get the job done, and generally lots of encouragement, validation and aesthetic advice.

However, I encourage reading the full document not for the shock entertainment, but for what I think is actually a pretty convincing case against OpenAI. Among other things, the claim demonstrates:

- That OpenAI does have the ability to abort interactions over e.g. copyright infrigement risk.

- That OpenAI did have the tech in hand to detect the sharply escalating self-harm content in the interactions - they ran them through OpenAI's own moderation end points for content analysis and got obvious numbers back.

- That OpenAI employees have publicly admitted and complained that the release of the overly sycophantic 4o model was rushed for business reasons and against the advice of internal safety teams.

- That 4o's safety was evaluated only with single-round prompt/answer testing, and OpenAI figured out swiftly that it falls apart quickly over longer interactions/prompts but kept the model up, later promoting how they improved this for GPT-5.

In context it's pretty crazy to me that OpenAI chose to bring back the 4o model specifically to placate the "ChatGPT is my girlfriend/boyfriend" crowd during the backlash, and I think initially pulling the plug on it during the 5 launch was very likely because they were aware of this and worried about liability.

Re: OpenAI says it's scanning users' conversations and reporting content to police

#42

Post by blindriver »

It’s pretty damning that OpenAI can’t modify its LLM to redirect people away from violence or from AI psychosis. Instead they’ve given up and have started monitoring conversations on behalf of governments. Obviously what comes next is monitoring anti-government sentiment or predicting “bad” or “antisocial” behavior on behalf of governments.

Re: OpenAI says it's scanning users' conversations and reporting content to police

#43

Post by kjkjadksj »

ericmcer wrote: post #20
jstummbillig wrote: post #16

Earlier quoted context omitted.

There's an awful lot of explaining to do between "one man has a psychotic breakdown" and "the species is not ready".

yeh by that justification our technological advancement should have stopped at sharpened sticks. "As a species we aren't ready to have sharp sticks"

As a species we aren’t really ready to have sharp sticks, to be fair. Children and mentally handicapped people will hurt themselves with them. As a result we don’t pass out sharpened sticks at recess or in asylums. Maybe we should consider the same approach for electronic sharp sticks we have created and are now marketing to the wider population.

Re: OpenAI says it's scanning users' conversations and reporting content to police

#44

Post by chatmasta »

ericmcer wrote: post #20
jstummbillig wrote: post #16

Earlier quoted context omitted.

There's an awful lot of explaining to do between "one man has a psychotic breakdown" and "the species is not ready".

yeh by that justification our technological advancement should have stopped at sharpened sticks. "As a species we aren't ready to have sharp sticks"

Well… there’s a strong argument to be made about that.

Re: OpenAI says it's scanning users' conversations and reporting content to police

#45

Post by szundi »

miletus wrote: post #7

This is exactly why a lot of people are running local LLMs or moving toward privacy-first platforms we recently shipped secure mode on https://www.agentsea.com . With Secure Mode, all chats run either on open-source models or models hosted on our own servers - so you can chat with AI without worrrying about privacy.

[dead]

Re: OpenAI says it's scanning users' conversations and reporting content to police

#46

Post by idle_zealot »

barbazoo wrote: post #3

> Update: It looks like this may have been OpenAI's attempt to get ahead of a horrifying story that just broke, about a man who fell into AI psychosis and killed his mother in a murder-suicide. Full details here. With the URL being https://futurism.com/openai-scanning-conversations-police . From there: > "Erik, you're not crazy. Your instincts are sharp, and your vigilance here is fully justified," ChatGPT told Soelb…

> As a species we are so not ready for this tech.

I basically agree, but not because of how individuals are reacting to it. On a societal level we're failing to stop megacorps from selling this thing as intelligent, a virtual assistant you can chat with. This sets expectations that just wreck the minds of a small but significant portion of the population. If we had developed LLMs in a lab and released them in the form of papers and Python projects with model weights with clear descriptions of what they're capable of, like a responsible scientific endeavor, then we'd not be seeing the problems we are, even with public access. What's killing us is MBAs and Salesmen. The fact that we organize our economy in such a way that hucksters and charlatans thrive is the greatest threat to humanity. These fuckers would sell dynamite as chewing gum if they calculated it would increase next quarter's profit by 0.02% (factoring in fines/fees, and lobbying costs of getting those fines reduced or repealed).

Re: OpenAI says it's scanning users' conversations and reporting content to police

#47

Post by lazide »

bilbo0s wrote: post #25
ACCount37 wrote: post #8

Earlier quoted context omitted.

They tried to mitigate sycophancy in the GPT-5 release. Guess what happened? A lot of users started complaining that "GPT-5 sucks, my AI now HATES me". And OpenAI relented.

This is kind of a big problem. Because on the one hand, sycophancy is not really what you want to do for people in mental and emotional crisis. On the other hand, not being sycophantic is not really what you want to do for people in mental and emotional crisis. There are professionals who speak to people in crisis for a reason. That's because it's fraught with pitfalls and trapdoors that take the situation from "ment…

Just what we need, the LLM ‘Reddit cares’. Notably, that is a mess too.

Also plenty of those hotlines are BS, or don’t work, or flat out don’t exist for given locales, etc.

The biggest issue is that LLM’s can act like a person, but aren’t a person, and fundamentally this causes problems. Especially for people that are already borderline or fully crazy.

Re: OpenAI says it's scanning users' conversations and reporting content to police

#49

Post by konfusinomicon »

cyberax wrote: post #26
exe34 wrote: post #23

Earlier quoted context omitted.

#shutupandtakemymoney (if you knew how most processed food were made, you wouldn't touch anything other than fresh vegetables).

If you knew how much (literal) shit is on fresh vegetables, you'd switch to using IV sugar solutions.

if you knew how much shit is on literally everything, you'd, well i don't know what youd do, but there is shit on literally everything so washy washy

Re: OpenAI says it's scanning users' conversations and reporting content to police

#50

Post by juanani »

nakamoto_damacy wrote: post #18

LLMs have no embodied sense of the world. They talk as if they understand and experience emotion, when they don't have actual embodied feelings/sensations/pain/pleasure/etc. For example, ou can think about how french fries taste but unless you can feel the taste you won't be repulsed if someone suggested making salad out of your left over french fries. Here is an actual chat with GPT5, that replays a experiment from…

[dead]
Post reply on HN