Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI says it's scanning users' conversations and reporting content to police

futurism.com

Post reply on HN
Print view View original

121–130 of 248 posts

Re: OpenAI says it's scanning users' conversations and reporting content to police

#121

Post by wrycoder »

bilbo0s wrote: post #25
ACCount37 wrote: post #8

Earlier quoted context omitted.

They tried to mitigate sycophancy in the GPT-5 release. Guess what happened? A lot of users started complaining that "GPT-5 sucks, my AI now HATES me". And OpenAI relented.

This is kind of a big problem. Because on the one hand, sycophancy is not really what you want to do for people in mental and emotional crisis. On the other hand, not being sycophantic is not really what you want to do for people in mental and emotional crisis. There are professionals who speak to people in crisis for a reason. That's because it's fraught with pitfalls and trapdoors that take the situation from "ment…

Perhaps flag the account and switch out the AI instance for Boring-BobGPT and further monitoring?

Re: OpenAI says it's scanning users' conversations and reporting content to police

#122

Post by lurking_swe »

sleazebreeze wrote: post #27
jonathanlb wrote: post #9

Earlier quoted context omitted.

Given that this is in response to a ChatGPT user who killed his mother and then himself, I'm not sure that positioning your product as being more secure than ChatGPT is wise, because your marketing here suggests either: 1. Profound tone-deafness about appropriate contexts for privacy messaging 2. Intentional targeting of users who want to avoid safety interventions 3. A fundamental misunderstanding of your ethical ob…

Or maybe I just want to be able to talk to an LLM without worrying about if its going to report me to the authorities.

that’s a good point, privacy is important.

To play devils advocate for a second, what if someone that’s mentally ill uses a local LLM for therapy and doesn’t get the help they need? Even if it’s against their will? And they commit suicide or kill someone because the LLM said it’s the right thing to do…

Is being dead better, or is having complete privacy better? Or does it depend?

I use local LLMs too, but it’s disingenuous to act like they solve the _real_ problem here. Mentally ill people trying to use an LLM for therapy. It can end catastrophically.

Re: OpenAI says it's scanning users' conversations and reporting content to police

#123

Post by ChrisMarshallNY »

nickthegreek wrote: post #94
ChrisMarshallNY wrote: post #57

Earlier quoted context omitted.

Well, as someone with a person suffering from pretty severe (like cops and hospitals severe) schizoaffective disorder in my immediate family, I can say that story scared the bejeezus out of me. If my family member had had ChatGPT, when they had their last episode, it's entirely possible that I might not be here, as "Everyone is plotting to kill me" is a big motif in SAD (it was, in their case, and they needed to be h…

It could be a good idea to see if they do have ChatGPT and if so, it might be helpful to have the Customized GPT turned on with a note that they suffer from X and if signs are showing of Y, then it should attempt to Z.

That's actually not a bad idea, but there's likely to be ramifications that would kill it.

One issue, is that these types of tools can also be abused. If you remember, the Soviet Union was notorious for getting dissidents committed, and the NSA|CIA also likes to brand leakers as "mentally unstable." It's a common foil. It would probably be fairly easy to "poison the context" like this, for any one of us. Could be a great way to go after folks we don't like.

Mental health treatment is a very complex (and fairly primitive) field.

Who's more mentally unfit? The person who's afraid that everyone is out to get them, or the person who thinks others should be locked permanently into a hellhole, on the off chance they could inconvenience them?

I'll lay odds that a bunch of folks on this very forum, are ones that would, at one time, have spent their lives in said hellhole. I'm one.

Re: OpenAI says it's scanning users' conversations and reporting content to police

#124

Post by cherioo »

johnfn wrote: post #28

Didn't we have a conversation a few days ago where a kid committed suicide while documenting it on ChatGPT and the clear consensus was that if OpenAI had been scanning user's conversations and reporting them to the police, this could have been averted, and this was a huge issue? But now when they go and do it, everyone says this is completely unacceptable? What exactly is OpenAI supposed to do, then?

Last week the think of the children groups were out in force, and this week it’s the privacy and liberty groups. Internet opinion is not a monoculture.

What does bug me though is that it was politically incorrect for the privacy folks to come out last week and for the two groups to fight. I have no solution.

Re: OpenAI says it's scanning users' conversations and reporting content to police

#125

Post by sho_hn »

thewebguyd wrote: post #117
bilbo0s wrote: post #25

Earlier quoted context omitted.

This is kind of a big problem. Because on the one hand, sycophancy is not really what you want to do for people in mental and emotional crisis. On the other hand, not being sycophantic is not really what you want to do for people in mental and emotional crisis. There are professionals who speak to people in crisis for a reason. That's because it's fraught with pitfalls and trapdoors that take the situation from "ment…

> I know that no one wants to hear this, but ChatGPT should probably be listening for people in crisis and, well, maybe not calling the cops, but maybe if there is a crisis line in their jurisdiction? A suicide hotline or something? Doesn't necessarily even need to call (particular in case of false positives) but there absolutely should be detection and a cutoff switch, where the chatbots just refuse to continue the…

I think a good first step would be if a ChatGPT user account could easily enter emergency contact information the system would notify in scenarios like this, making the escalation opt-in.

The moderation tech is already there, and if there's even a small amount of mentally ill who would fill this in on a good day and be saved by it on a bad day / during an episode, it'd be worth it.

Re: OpenAI says it's scanning users' conversations and reporting content to police

#126

Post by LamerBeeI »

aktuel wrote: post #97

chatgpt is barely usable at all anyhow atm. The amount of complete garbage responses gpt-5 is giving me is mind-boggling. just today it ran into an infinite loop of endless printout again by a completely innocuous query. I honestly cannot understand how its stock is still where it's at.

what stock?

Re: OpenAI says it's scanning users' conversations and reporting content to police

#127

Post by senko »

pants2 wrote: post #102
senko wrote: post #79

The right way to deal with this is education. 1. We should be educating the populace about failure modes of AI chatbots - something well known to HNers but not to the general public. 2. CEOs, pundits, and marketers should be held accountable (morally, in the court of public opinion and in the news) when they present AI as unvarnished good - that's false marketing at the very least, and leads to tragic consequences. 3…

What do you envision in the form of education? "Pass this test to use ChatGPT"?

As in informing the public, not adding to the school curriculum.

A sustained collaborative effort by the AI makers, (local) governments and journalists could work, IMHO.

Instead AI makers hype it up, governments have no clue wtf is going on and media just chases clicks.

Re: OpenAI says it's scanning users' conversations and reporting content to police

#128

Post by LamerBeeI »

sleazebreeze wrote: post #27
jonathanlb wrote: post #9

Earlier quoted context omitted.

Given that this is in response to a ChatGPT user who killed his mother and then himself, I'm not sure that positioning your product as being more secure than ChatGPT is wise, because your marketing here suggests either: 1. Profound tone-deafness about appropriate contexts for privacy messaging 2. Intentional targeting of users who want to avoid safety interventions 3. A fundamental misunderstanding of your ethical ob…

Or maybe I just want to be able to talk to an LLM without worrying about if its going to report me to the authorities.

I too think there should be no rules or attempts to derisk any situation, just let us die

Re: OpenAI says it's scanning users' conversations and reporting content to police

#129

Post by echelon_musk »

ericmcer wrote: post #20
jstummbillig wrote: post #16

Earlier quoted context omitted.

There's an awful lot of explaining to do between "one man has a psychotic breakdown" and "the species is not ready".

yeh by that justification our technological advancement should have stopped at sharpened sticks. "As a species we aren't ready to have sharp sticks"

The lyrics to 'Tool - Right In Two' come to mind.

Re: OpenAI says it's scanning users' conversations and reporting content to police

#130

Post by saubeidl »

idle_zealot wrote: post #120
saubeidl wrote: post #101

Earlier quoted context omitted.

> What's killing us is MBAs and Salesmen Almost. Just take it like an inch further. What's really killing us is capitalism . MBAs and Salesmen are just the personification.

I tend to shy away from that sort of rhetoric on HN. Saying that the problem is capitalism and we need socialism or whatever gets you downvoted to hell. Making the underlying arguments that lead to that conclusion gets you upvotes, exposing more people to those ideas. People, even smart people, especially in the US, hear socialism or communism and immediately think of the USSR or Maoist China. The terms are very unpo…

I find that part of being intellectually curious and a good engineer is questioning systems.

If we had a systems architecture in an engineering project that kept leading to bad results, we would consider rearchitecting how said system is run.

I believe it is important to make it clear to people that capitalism, too, is just a (imo deeply flawed) systems design. It is not a natural constant. If we find it leads to negative outcomes (which I strongly believe), we can and should talk about rearchitecting.

Post reply on HN