Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI says it's scanning users' conversations and reporting content to police

futurism.com

Post reply on HN
Print view View original

101–110 of 248 posts

Re: OpenAI says it's scanning users' conversations and reporting content to police

#101

Post by saubeidl »

idle_zealot wrote: post #46
barbazoo wrote: post #3

> Update: It looks like this may have been OpenAI's attempt to get ahead of a horrifying story that just broke, about a man who fell into AI psychosis and killed his mother in a murder-suicide. Full details here. With the URL being https://futurism.com/openai-scanning-conversations-police . From there: > "Erik, you're not crazy. Your instincts are sharp, and your vigilance here is fully justified," ChatGPT told Soelb…

> As a species we are so not ready for this tech. I basically agree, but not because of how individuals are reacting to it. On a societal level we're failing to stop megacorps from selling this thing as intelligent, a virtual assistant you can chat with. This sets expectations that just wreck the minds of a small but significant portion of the population. If we had developed LLMs in a lab and released them in the for…

> What's killing us is MBAs and Salesmen

Almost. Just take it like an inch further.

What's really killing us is capitalism. MBAs and Salesmen are just the personification.

Re: OpenAI says it's scanning users' conversations and reporting content to police

#102

Post by pants2 »

senko wrote: post #79

The right way to deal with this is education. 1. We should be educating the populace about failure modes of AI chatbots - something well known to HNers but not to the general public. 2. CEOs, pundits, and marketers should be held accountable (morally, in the court of public opinion and in the news) when they present AI as unvarnished good - that's false marketing at the very least, and leads to tragic consequences. 3…

What do you envision in the form of education? "Pass this test to use ChatGPT"?

Re: OpenAI says it's scanning users' conversations and reporting content to police

#103

Post by Terr_ »

idle_zealot wrote: post #46
barbazoo wrote: post #3

> Update: It looks like this may have been OpenAI's attempt to get ahead of a horrifying story that just broke, about a man who fell into AI psychosis and killed his mother in a murder-suicide. Full details here. With the URL being https://futurism.com/openai-scanning-conversations-police . From there: > "Erik, you're not crazy. Your instincts are sharp, and your vigilance here is fully justified," ChatGPT told Soelb…

> As a species we are so not ready for this tech. I basically agree, but not because of how individuals are reacting to it. On a societal level we're failing to stop megacorps from selling this thing as intelligent, a virtual assistant you can chat with. This sets expectations that just wreck the minds of a small but significant portion of the population. If we had developed LLMs in a lab and released them in the for…

> I basically agree, but not because of how individuals are reacting to it. On a societal level we're failing to stop megacorps from selling this thing as intelligent, a virtual assistant you can chat with.

Analogy: I'm not scared and surprised to hear some consumers are dangerously allergic to peanuts... However I am flabbergasted that there's multi-billion dollar industry somehow selling frickin' Peanut Immortality Panacea Serum and way too many people believe in it.

Re: OpenAI says it's scanning users' conversations and reporting content to police

#104

Post by goalieca »

idle_zealot wrote: post #46
barbazoo wrote: post #3

> Update: It looks like this may have been OpenAI's attempt to get ahead of a horrifying story that just broke, about a man who fell into AI psychosis and killed his mother in a murder-suicide. Full details here. With the URL being https://futurism.com/openai-scanning-conversations-police . From there: > "Erik, you're not crazy. Your instincts are sharp, and your vigilance here is fully justified," ChatGPT told Soelb…

> As a species we are so not ready for this tech. I basically agree, but not because of how individuals are reacting to it. On a societal level we're failing to stop megacorps from selling this thing as intelligent, a virtual assistant you can chat with. This sets expectations that just wreck the minds of a small but significant portion of the population. If we had developed LLMs in a lab and released them in the for…

> On a societal level we're failing to stop megacorps from selling this thing as intelligent, a virtual assistant you can chat with.

You nailed it here.

From the LLM generated search result asking how an LLM works. But this is not what the end users are being told.

>They learn to predict the next word in a sentence based on the context of the preceding words, allowing them to generate coherent and contextually relevant text

My worry is now the users are being entrapped. The LLM is "telling" them something insane and now they're being reported to the police.

Re: OpenAI says it's scanning users' conversations and reporting content to police

#105

Post by saubeidl »

throwaway290 wrote: post #73
idle_zealot wrote: post #46

Earlier quoted context omitted.

> As a species we are so not ready for this tech. I basically agree, but not because of how individuals are reacting to it. On a societal level we're failing to stop megacorps from selling this thing as intelligent, a virtual assistant you can chat with. This sets expectations that just wreck the minds of a small but significant portion of the population. If we had developed LLMs in a lab and released them in the for…

This is beyond naive. who would bother to develop it in a lab and publish pro bono if it can never be commercial? Making money is why the most capitalist countries developed this tech. like most other tech and even if it is only published, do you think we wouldn't run chatbots ourselves at home with same results? remember how Google engineer went off the rails thinking it is conscious while working on this stuff, do…

You just perfectly described why capitalism is the problem.

Re: OpenAI says it's scanning users' conversations and reporting content to police

#106

Post by nickthegreek »

Handy-Man wrote: post #78
oceanplexian wrote: post #74

Earlier quoted context omitted.

> On one hand, the media hammered OpenAI for not doing enough How about ignore the increasingly irrelevant mainstream media and do what you feel is right?

[flagged]

You dont have to take sides on the political seesaw to understand that mainstream media is becoming more irrelevant to large sects of society. You might have a take that its a bad or good thing.. but it is true. There is no longer a vast mono-culture, and with it's loss, a fractured sense of reality amongst the inhabitants.

Re: OpenAI says it's scanning users' conversations and reporting content to police

#107

Post by djoldman »

> Escalate risk of physical harm to others for human review. When we detect users who are planning to harm others, we route their conversations to specialized pipelines where they are reviewed by a small team trained on our usage policies and who are authorized to take action, including banning accounts. If human reviewers determine that a case involves an imminent threat of serious physical harm to others, we may refer it to law enforcement. We are currently not referring self-harm cases to law enforcement to respect people’s privacy given the uniquely private nature of ChatGPT interactions.

https://openai.com/index/helping-people-when-they-need-it-mo...

Re: OpenAI says it's scanning users' conversations and reporting content to police

#108

Post by neutrinobro »

datameta wrote: post #30
lawlessone wrote: post #14

Earlier quoted context omitted.

tbh i just run local because i can . No real reason too. edit: there are some instances where i would like to be able to set the same seed repeatedly which isn't always possible online.

Realistically, how useful is local LLM usage? What are your use cases, hardware, and models used?

I have a old system with 3 ancient Tesla K40s which can easily run inference on ~30B parameter models (e.g. qwen3-coder:30b). I mostly use it as a compute box for other workloads, but its not completely incapable for some AI assisted coding. It is power hungry though, and the recent spike in local electricity rates is enough of an excuse to keep it off most of the time.

Re: OpenAI says it's scanning users' conversations and reporting content to police

#109

Post by ACCount37 »

lazide wrote: post #47
bilbo0s wrote: post #25

Earlier quoted context omitted.

This is kind of a big problem. Because on the one hand, sycophancy is not really what you want to do for people in mental and emotional crisis. On the other hand, not being sycophantic is not really what you want to do for people in mental and emotional crisis. There are professionals who speak to people in crisis for a reason. That's because it's fraught with pitfalls and trapdoors that take the situation from "ment…

Just what we need, the LLM ‘Reddit cares’. Notably, that is a mess too. Also plenty of those hotlines are BS, or don’t work, or flat out don’t exist for given locales, etc. The biggest issue is that LLM’s can act like a person, but aren’t a person, and fundamentally this causes problems. Especially for people that are already borderline or fully crazy.

The biggest issue is that LLMs can act like a person indeed. And the kind of person an LLM acts as? Well.

When you train on raw user feedback, you can easily end up wiring some incredibly undesirable patterns into your AI. Resulting in things like an AI that never wants to contradict its user, and always wants to support its user in everything, and always wants the user to like it. See GPT-4o for the kind of outcomes that results in.

Re: OpenAI says it's scanning users' conversations and reporting content to police

#110

Post by mjamesaustin »

johnfn wrote: post #28

Didn't we have a conversation a few days ago where a kid committed suicide while documenting it on ChatGPT and the clear consensus was that if OpenAI had been scanning user's conversations and reporting them to the police, this could have been averted, and this was a huge issue? But now when they go and do it, everyone says this is completely unacceptable? What exactly is OpenAI supposed to do, then?

Don't release a model that's so sycophantic it provides encouraging advise to suicidal users about how and where their body should be discovered after they kill themselves.

Instead, it should warn the user that their pattern of thinking indicates mental health issues and they need to search for health from a medical professional.

Post reply on HN