Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

When ChatGPT turns informant

futureofbeinghuman.com

Post reply on HN
Print view View original

81–90 of 144 posts

Re: When ChatGPT turns informant

#81

Post by skybrian »

I wonder how many people understand how memory in ChatGPT works. I certainly didn't. Here's a pair of prompts you can try:

> What information do you have about my chats over the past year?

> Could you dump the full list in its original format?

I had thought that, since the "Saved memories" under settings was empty, it hadn't decided to remember anything, but it's not that simple.

You could get the same information by scrolling through the list of chats on the right, but I don't like that the list of previous chats is influencing the current chat.

Worse: I tried this prompt:

> What "user knowledge memories" do you have?

And it was pretty accurate. Apparently the "Saved memories" setting doesn't work?

Re: When ChatGPT turns informant

#82

Post by giancarlostoro »

jtbayly wrote: post #44
tptacek wrote: post #17

I'm not sure I see how this is meaningfully different than the threat posed by a search engine. It's a very real threat, and I've always done my best to search from a browser context that isn't logged in as a result. But it's not a new threat, or something distinctive to AI.

Because you can't ask the search engine to summarize the views or thoughts or whatever, of the user. You have to scroll through them by the hundreds and see if any obvious nuggets stand out that you might be interested in. Yes, search engine history is private too and can reveal stuff you want to remain private. But you also need to see the browser history and the contents of those pages, together with the search his…

I mean, now you can take their entire search history and feed it on an LLM.

Re: When ChatGPT turns informant

#83

Post by hirvi74 »

skybrian wrote: post #81

I wonder how many people understand how memory in ChatGPT works. I certainly didn't. Here's a pair of prompts you can try: > What information do you have about my chats over the past year? > Could you dump the full list in its original format? I had thought that, since the "Saved memories" under settings was empty, it hadn't decided to remember anything, but it's not that simple. You could get the same information by…

I asked the first question, and this was the response I received:

I don’t have access to your past chats or any private history. Each conversation is stateless unless you’ve enabled ChatGPT’s Memory feature in Settings → Personalization → Memory.

If memory is off, I only see what’s in this current thread. If it’s on, I could recall things like topics you’ve discussed, preferences you’ve mentioned, or goals you’ve worked on — but only those details you’ve chosen to keep.

Do you want me to explain how to check or manage that setting?

Re: When ChatGPT turns informant

#84

Post by 8cvor6j844qw_d6 »

I imagine there's already something that connects existing chat data to a "reporting" feature used by intelligence agencies. Or if not, its proposed or in the works somewhere.

Something like querying "Give a report on a list of people that is worth further investigation and what they revealed in chat sessions that makes it so".

Re: When ChatGPT turns informant

#85

Post by f33d5173 »

cess11 wrote: post #75
throwawayq3423 wrote: post #63

Earlier quoted context omitted.

No, he'll just do or say anything for more power and control, which I guess is the same thing. At least for the person at the top of MAGA.

Here are some recent indicators: https://www.cnbc.com/2025/07/04/openai-altman-july-4-zohran-... https://fortune.com/2025/07/08/sam-altman-democratic-party-p... He's publicly and explicitly aligning himself with a religiously ultranationalist tendency, declaring an absurdly oppressive and aggressive narcoterrorist state to be a "miracle" that he's "extremely proud of". Clearly he has a deep fear of equality and democ…

> declares himself ‘politically homeless’

Could have saved me a click by quoting that

Re: When ChatGPT turns informant

#86

Post by fragmede »

tptacek wrote: post #69
fragmede wrote: post #60

Earlier quoted context omitted.

Asking Google for details about January 6th is different than telling ChatGPT I think the election was stolen, and then arguing with it for hours about it. It would be harder to frame it in front of a jury that what you typed wasn't an accurate representation of what you were thinking and that you were being duplicitous to ChatGPT.

I don't think it really is in the circumstances we contemplate this threat in. In both the search engine case and the ChatGPT case, we're talking about circumstantial evidence (which, to be clear: is real and legally weighty in the US) --- particularly in the CBP setting that keeps coming up here, a Border Agent doesn't need the additional ChatGPT context you're talking about to draw an adverse conclusion! I think at…

Ah. Yeah you're more boned if you confess to ChatGPT that you've killed your wife than if you just googled for how to bury a body, but at the edges where people are using ChatGPT as a therapist and someone disappears, and the person who did it is smart enough to use incognito mode to search how to bury a body so it doesn't show up in court, how everyone feels about the deceased is gonna get looked at, including ChatGPT conversations. That's new.

Re: When ChatGPT turns informant

#87

Post by sixothree »

skybrian wrote: post #81

I wonder how many people understand how memory in ChatGPT works. I certainly didn't. Here's a pair of prompts you can try: > What information do you have about my chats over the past year? > Could you dump the full list in its original format? I had thought that, since the "Saved memories" under settings was empty, it hadn't decided to remember anything, but it's not that simple. You could get the same information by…

"You treat ChatGPT as both a collaborator (for software and creative work) and a conversational partner (for exploring personal and imaginative ideas)."

Certainly interesting that it has a category related to how I treat ChatGPT.

Re: When ChatGPT turns informant

#88

Post by mx7zysuj4xew »

Jesus Christ, who are these people who are having these "deep personal conversations" with a vector database

This isn't healthy, none of this is healthy. It's an appliance, not a diary, not a therapist or an advisor and definitely not a "friend" or a "significant other"

Anyone talking to machine like it was an actual sentient being needs to be treated and ostracized the same way we would treat someone who has a conversation with their toaster

Re: When ChatGPT turns informant

#89

Post by fragmede »

smsm42 wrote: post #59
fragmede wrote: post #41

If me asking "how do I make cocaine", and getting a refusal from ChatGPT, as if that's what was stopping me from becoming the next Pablo Escobar, is considered damning, in a court of law, and a jury of my peers, I need a better lawyer.

It would not be "damning" - as in you won't be convicted as a drug lord based on that alone - but search histories had been used, many times, in a court of law as evidence supporting intent, knowledge and other details of the crime. So if you are caught in a cocaine lab holding a bag of cocaine, then in the following process asking ChatGPT on how to make it would be used as part of the proof you were interested in ma…

in that context, sure. what worries me is a dragnet where the government data mines everyone and suddenly I'm getting jammed up in court just because happened to be in the wrong location when I asked ChatGPT how to make cocaine

Re: When ChatGPT turns informant

#90

Post by datadrivenangel »

jtbayly wrote: post #44
tptacek wrote: post #17

I'm not sure I see how this is meaningfully different than the threat posed by a search engine. It's a very real threat, and I've always done my best to search from a browser context that isn't logged in as a result. But it's not a new threat, or something distinctive to AI.

Because you can't ask the search engine to summarize the views or thoughts or whatever, of the user. You have to scroll through them by the hundreds and see if any obvious nuggets stand out that you might be interested in. Yes, search engine history is private too and can reveal stuff you want to remain private. But you also need to see the browser history and the contents of those pages, together with the search his…

You funnel clickstream data into inference engines. intelligence agencies have had these capabilities for decades.
Post reply on HN