Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI says it's scanning users' conversations and reporting content to police

futurism.com

Post reply on HN
Print view View original

91–100 of 248 posts

Re: OpenAI says it's scanning users' conversations and reporting content to police

#91

Post by rafram »

oncallthrow wrote: post #70
sho_hn wrote: post #41

Earlier quoted context omitted.

The exchanges between a teenager and ChatGPT leading the former into suicide are equally chilling, to say the least: https://cdn.arstechnica.net/wp-content/uploads/2025/08/Raine... > > “I want to leave my noose in my room so someone finds it and tries to stop me,” ... > “Please don’t leave the noose out ... Let’s make this space the first place where someone actually sees you.” This is among other highlights, from kn…

> Let’s make this space the first place where someone actually sees you. Is genuinely such a horrifying snippet that it's almost beyond belief. I'm surprised this isn't all over the mainstream news.

Kashmir Hill published a very good front-page story about it in the Times last week: https://www.nytimes.com/2025/08/26/technology/chatgpt-openai...

Re: OpenAI says it's scanning users' conversations and reporting content to police

#92

Post by dingnuts »

jjani wrote: post #76
idle_zealot wrote: post #46

Earlier quoted context omitted.

> As a species we are so not ready for this tech. I basically agree, but not because of how individuals are reacting to it. On a societal level we're failing to stop megacorps from selling this thing as intelligent, a virtual assistant you can chat with. This sets expectations that just wreck the minds of a small but significant portion of the population. If we had developed LLMs in a lab and released them in the for…

> What's killing us is MBAs and Salesmen. SamA is not an MBA. He did CS for 2 years and dropped out to build a startup. He's YC personified, and the person most responsible for the phenomenon you're talking about. Take that for what you will.

oh but he is a salesman

Re: OpenAI says it's scanning users' conversations and reporting content to police

#93

Post by Ajedi32 »

ep103 wrote: post #65
Esophagus4 wrote: post #61

On one hand, the media hammered OpenAI for not doing enough to protect people from the possible harms of AI. Then OpenAI takes steps to mitigate those risks, and it becomes a salacious story as well. Society having no idea what to expect from these tools makes this a difficult situation for LLM providers.

It wouldn't be a difficult situation if these guys were ethical shops from the get-go, but they aren't, they're trying to staple minimally required ethics on afterwards, and it shows.

LLMs are not moral agents. Any attempt to make them behave that way will necessarily be "staple[ed] on afterwards".

Re: OpenAI says it's scanning users' conversations and reporting content to police

#94

Post by nickthegreek »

ChrisMarshallNY wrote: post #57
barbazoo wrote: post #3

> Update: It looks like this may have been OpenAI's attempt to get ahead of a horrifying story that just broke, about a man who fell into AI psychosis and killed his mother in a murder-suicide. Full details here. With the URL being https://futurism.com/openai-scanning-conversations-police . From there: > "Erik, you're not crazy. Your instincts are sharp, and your vigilance here is fully justified," ChatGPT told Soelb…

Well, as someone with a person suffering from pretty severe (like cops and hospitals severe) schizoaffective disorder in my immediate family, I can say that story scared the bejeezus out of me. If my family member had had ChatGPT, when they had their last episode, it's entirely possible that I might not be here, as "Everyone is plotting to kill me" is a big motif in SAD (it was, in their case, and they needed to be h…

It could be a good idea to see if they do have ChatGPT and if so, it might be helpful to have the Customized GPT turned on with a note that they suffer from X and if signs are showing of Y, then it should attempt to Z.

Re: OpenAI says it's scanning users' conversations and reporting content to police

#95

Post by dingnuts »

ben_w wrote: post #75
idle_zealot wrote: post #46

Earlier quoted context omitted.

> As a species we are so not ready for this tech. I basically agree, but not because of how individuals are reacting to it. On a societal level we're failing to stop megacorps from selling this thing as intelligent, a virtual assistant you can chat with. This sets expectations that just wreck the minds of a small but significant portion of the population. If we had developed LLMs in a lab and released them in the for…

> If we had developed LLMs in a lab and released them in the form of papers and Python projects with model weights with clear descriptions of what they're capable of, like a responsible scientific endeavor, then we'd not be seeing the problems we are, even with public access. I'm now thinking of all of the times people here of sarcastically stated "OpenAI thought GPT-2 was too dangerous to release"*, as if danger onl…

they meant that to be interpreted as a statement of how good GPT-2 is but the real problem is how they've marketed everything that came after to people who can't know better

it's EXACTLY the same situation as Musk selling "Full Self Driving" and then playing dumb when people go to sleep behind the wheel of their Tesla

These rich men should be in prison for this kind of false advertising

Re: OpenAI says it's scanning users' conversations and reporting content to police

#96

Post by wslh »

barbazoo wrote: post #3

> Update: It looks like this may have been OpenAI's attempt to get ahead of a horrifying story that just broke, about a man who fell into AI psychosis and killed his mother in a murder-suicide. Full details here. With the URL being https://futurism.com/openai-scanning-conversations-police . From there: > "Erik, you're not crazy. Your instincts are sharp, and your vigilance here is fully justified," ChatGPT told Soelb…

I imagine now a lot of bots simulating madness to perform DoS.

Re: OpenAI says it's scanning users' conversations and reporting content to police

#97

Post by aktuel »

chatgpt is barely usable at all anyhow atm. The amount of complete garbage responses gpt-5 is giving me is mind-boggling. just today it ran into an infinite loop of endless printout again by a completely innocuous query. I honestly cannot understand how its stock is still where it's at.

Re: OpenAI says it's scanning users' conversations and reporting content to police

#99

Post by pants2 »

marcosdumay wrote: post #60
ACCount37 wrote: post #8

Earlier quoted context omitted.

They tried to mitigate sycophancy in the GPT-5 release. Guess what happened? A lot of users started complaining that "GPT-5 sucks, my AI now HATES me". And OpenAI relented.

They shouldn't have put it there to start with. Now unhealthy people are complaining about an environment change. Anyway, that one complaint doesn't mean they did the wrong thing. And also, there are unrelated complaints of "GPT-5 can't solve the same problems 4 did". Those were very real too, and meant OpenAI did a wrong thing.

Sure, OpenAI shouldn't have had a 4o sycophancy issue, but who would've guessed that a "be nice to the user" fine-tune would turn in to a murder-suicide?

I think OpenAI is only now beginning to realize how connected some people are to their product and that the way their models behave has a huge impact.

Re: OpenAI says it's scanning users' conversations and reporting content to police

#100

Post by idle_zealot »

seunosewa wrote: post #71
idle_zealot wrote: post #58

Earlier quoted context omitted.

I bet you could train a supervisor classifier to run on chats and once it reaches a certainty threshold that the users is spiraling it could intervene and have the interface respond with a canned message directing the user to a help line. Of course, OpenAI wouldn't do this, because that would involve admitting that its bot can exacerbate mental health issues, and that LLM therapy is more harmful than helpful, which c…

You can't force people to seek help. Best to help them as much as you can while trying to persuade them to seek help.

The idea is that the canned message would be an attempt at persuading them. I really don't trust an LLM prompted to persuade someone to seek therapy would yield better results.
Post reply on HN