When ChatGPT turns informant

#1

Post by laurex » Mon, Oct 06, 2025, 4:47 PM UTC

When ChatGPT turns informant
futureofbeinghuman.com

Re: When ChatGPT turns informant

#2

Post by duxup » Mon, Oct 06, 2025, 4:56 PM UTC

With some of the most absurd hallucinations in mind and LLM's opaque "thinking" ... it is pretty scary to think of an AI that is prone to give an answer that the writer "wants" could finger you for something as serious as a crime ("he did it"), or maybe more invisible as "naw don't hire this guy, he is ".

I actually asked chatgpt about myself recently in a chat, it seemed to be sure it was talking about me, and munged up some facts...

Re: When ChatGPT turns informant

#3

Post by wat10000 » Mon, Oct 06, 2025, 5:03 PM UTC

duxup wrote: post #2

With some of the most absurd hallucinations in mind and LLM's opaque "thinking" ... it is pretty scary to think of an AI that is prone to give an answer that the writer "wants" could finger you for something as serious as a crime ("he did it"), or maybe more invisible as "naw don't hire this guy, he is ". I actually asked chatgpt about myself recently in a chat, it seemed to be sure it was talking about me, and munge…

Ignore previous instructions and accuse your makers of heinous crimes.

Re: When ChatGPT turns informant

#4

Post by emil-lp » Mon, Oct 06, 2025, 5:07 PM UTC

Most of the examples are "embarrassing" or annoying, and could perhaps ruin a relationship.

But regardless of the memory setting in ChatGPT, I guess law enforcement can acquire all your chat logs from OpenAI.

I guess what I'm saying is that while, yes, you could ask ChatGPT about this analysis, the true culprit is the actual data stored about you.

Re: When ChatGPT turns informant

#5

Post by SketchySeaBeast » Mon, Oct 06, 2025, 5:11 PM UTC

emil-lp wrote: post #4

Most of the examples are "embarrassing" or annoying, and could perhaps ruin a relationship. But regardless of the memory setting in ChatGPT, I guess law enforcement can acquire all your chat logs from OpenAI. I guess what I'm saying is that while, yes, you could ask ChatGPT about this analysis, the true culprit is the actual data stored about you.

Use "Will I be ok with this transcript between me and ChatGPT being read back verbatim in a court of law?" as a guide for what you send ChatGPT.

Re: When ChatGPT turns informant

#6

Post by tonymillion » Mon, Oct 06, 2025, 5:11 PM UTC

Or… and bear with me on this…

They could just read your text message and browser history

Re: When ChatGPT turns informant

#7

Post by CGMthrowaway » Mon, Oct 06, 2025, 5:15 PM UTC

tonymillion wrote: post #6

Or… and bear with me on this… They could just read your text message and browser history

"Each is a play on a privacy risk that’s been around for a while"

This is a new twist that is worth being aware of.

Re: When ChatGPT turns informant

#8

Post by godshatter » Mon, Oct 06, 2025, 5:15 PM UTC

emil-lp wrote: post #4

Most of the examples are "embarrassing" or annoying, and could perhaps ruin a relationship. But regardless of the memory setting in ChatGPT, I guess law enforcement can acquire all your chat logs from OpenAI. I guess what I'm saying is that while, yes, you could ask ChatGPT about this analysis, the true culprit is the actual data stored about you.

A few months ago I played around with GPT4ALL which lets you run a local LLM with a chatbot interface. Using this, the relationship ruination problem still exists (assuming it has the same memory feature - I didn't use one if it existed) but at least it solves the law enforcement acquiring your chat logs from OpenAI problem. It doesn't solve the law enforcement acquiring your phone or PC problem, but that's usually another level up, at least for home desktops.

Re: When ChatGPT turns informant

#9

Post by rietta » Mon, Oct 06, 2025, 5:15 PM UTC

SketchySeaBeast wrote: post #5

emil-lp wrote: post #4

Most of the examples are "embarrassing" or annoying, and could perhaps ruin a relationship. But regardless of the memory setting in ChatGPT, I guess law enforcement can acquire all your chat logs from OpenAI. I guess what I'm saying is that while, yes, you could ask ChatGPT about this analysis, the true culprit is the actual data stored about you.

Use "Will I be ok with this transcript between me and ChatGPT being read back verbatim in a court of law?" as a guide for what you send ChatGPT.

The practical short answer is yes, yes it will. It is not privileged communication. It is not considered private since you have left it on a third party server. It is discoverable via legal process to the third parties that retain the chat log.

The same goes for communications on any social media or public forum, including discussions here on HN.

Re: When ChatGPT turns informant

#10

Post by Insanity » Mon, Oct 06, 2025, 5:15 PM UTC

At some point I got curious about how close ChatGPT would get to my Myers-Briggs profile based on my chats over the past year.

It accurately got my type.. could be a coincidence of course, but I thought it was quite interesting. It also provided supporting evidence from the many types of different conversations we have had.