Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

When ChatGPT turns informant

futureofbeinghuman.com

Post reply on HN
Print view View original

111–120 of 144 posts

Re: When ChatGPT turns informant

#111

Post by krick »

> ChatGPT offers a “memory” personalization setting ... OpenAI are at pains to point out that this function can be turned off at any time, and that individual memories can be deleted

Uh… Isn't it just irrelevant (to the point of such remarks being actually misleading) anymore? AFAIK, it's been a couple of months already since OpenAI began storing all your conversations (because of that court order), whether you "delete" them or not, so while you can technically disable "memory" setting, it only means it won't be able to use your past responses to help you. But it surely would help anybody with, let's say, elevated access. Granted, the threat model in the post assumes that the author is only worried about what the user of the account can learn about other users of the account, and that he trusts OpenAI itself. But why would OpenAI be "at pains to point out that this function can be turned off" then?

Re: When ChatGPT turns informant

#112

Post by gitpusher »

The author makes it seem like we have two choices:

1) enable memory, and use ChatGPT like a confessional booth. Flood it with all of your deepest, darkest humiliations going all the way back to childhood ...

2) disable memory

Perhaps my age is showing. But memory or no memory, I would never tell ChatGPT anything compromising about myself. Nor would I tweet such things, write them in an email, or put them into a Slack message. This is just basic digital hygiene.

I've noticed a lot of people treat ChatGPT like a close confidant, which I find pretty interesting. Particularly the younger folks. I understand the allure – LLMs are the "friend" that never gets bored of listening, never judges you, and always says the right thing. Because of this people end up sharing even MORE than they would to their closest human friends.

Re: When ChatGPT turns informant

#113

Post by twobitshifter »

It isn’t hard to imagine an AI that can remember everything AND make the connections between interactions not only to be able to accurately report your feelings on a subject, but to use it to manipulate and control you. The belief that AI will always tell you the truth is foolish and we now know that all the leading AIs will resort to blackmail for self-preservation. https://www.bbc.com/news/articles/cpqeng9d20go

Now imagine an AI that has unlimited blackmail material on each and every citizen and either a permission or a survival instinct driving it to use it to manipulate the population. After all OpenAI, doesn’t only have access to one person’s interactions they have that for all users.

Re: When ChatGPT turns informant

#114

Post by cess11 »

f33d5173 wrote: post #85
cess11 wrote: post #75

Earlier quoted context omitted.

Here are some recent indicators: https://www.cnbc.com/2025/07/04/openai-altman-july-4-zohran-... https://fortune.com/2025/07/08/sam-altman-democratic-party-p... He's publicly and explicitly aligning himself with a religiously ultranationalist tendency, declaring an absurdly oppressive and aggressive narcoterrorist state to be a "miracle" that he's "extremely proud of". Clearly he has a deep fear of equality and democ…

> declares himself ‘politically homeless’ Could have saved me a click by quoting that

What does that tell you?

Re: When ChatGPT turns informant

#115

Post by JumpCrisscross »

throwawayq3423 wrote: post #61
JumpCrisscross wrote: post #32

Earlier quoted context omitted.

> Myers-Briggs has no scientific validity Is this an accuracy versus precision problem? In my anecdotal experience, most people will consistently test near one’s Myers-Briggs type on different tests. It’s just that this test doesn’t translate well to the real-world situations it was pitched for.

> test near one’s Myers-Briggs type He's saying that Myers-Briggs has no scientific basis. So if there's another test that does, you wouldn't be able to compare those results anyway.

> He's saying that Myers-Briggs has no scientific basis

Favourite-colour based personality theories have no scientific basis. It would still be interesting if an LLM could guess my favourite colour.

Re: When ChatGPT turns informant

#116

Post by krick »

edm0nd wrote: post #109
skybrian wrote: post #81

I wonder how many people understand how memory in ChatGPT works. I certainly didn't. Here's a pair of prompts you can try: > What information do you have about my chats over the past year? > Could you dump the full list in its original format? I had thought that, since the "Saved memories" under settings was empty, it hadn't decided to remember anything, but it's not that simple. You could get the same information by…

Claude reply: I don't have any information about your chats over the past year. Each conversation with me starts fresh - I don't have access to: Your previous conversations with Claude Chat history from other sessions Any personal data about you unless you share it in our current conversation Information about how you've used Claude in the past Every time you start a new conversation, it's like meeting me for the fir…

FWIW, it says the same when I asked deepseek that question. And while I cannot prove otherwise (I didn't try to specifically do that), I am under very strong impression that past chats influence the future ones. This could be some kind of cognitive bias, but there were some very suspicious coincidences.

I still somehow haven't tried Claude Chat, and while I wouldn't assume it lies about if it remembers anything, I wouldn't just trust whatever these things say about themselves either.

Re: When ChatGPT turns informant

#117

Post by janalsncm »

cortesoft wrote: post #104
janalsncm wrote: post #102

Earlier quoted context omitted.

In theory you could accomplish this by combing through search history. In practice, the scenario in OP is unlikely to be practical with search history alone. It’s much less convenient for CBP to ask someone to pull up their Google search history. And even if they did, it doesn’t work as well. Officers don’t have infinite time to assess every person. So I would call it a new threat.

They could also take your traditional search and chat history, feed it into an LLM, and ask it the same questions. Once you start doing that for one person... you could just feed everyone's chat and search history into an LLM, and ask it "who is the most dangerous" or whatever you want to ask. Its just another version of the classic computing problem "computers might not make a new thing possible, but it makes it pos…

I must not have understood the article correctly because I took ChatGPT to be a stand in for LLM technology in general. But I think I am wrong.

Re: When ChatGPT turns informant

#118

Post by hluska »

I read some of the supporting documents, namely the entire prompt that created the chat log and many pages of the chat transcript and with respect to the researcher, this isn’t a very good test.

John only uses the generative AI for pseudo-therapy. The conversations are repetitive and John has one hell of an awful life. Between the hardcore suicidal ideation, the drinking problem, the pathological lies, abuse and fraud, old John here has a lot of problems and only talks about them. I’m not a medical professional, but if I was I’d lobby to get “completely fucked” added to the next edition of the DSM. John is in such a crisis that if we were close friends and he told me this stuff, I would be in a hell of a state because he’s clearly crying for help, clearly needs professional help but he’s beyond the level of the typical police wellness check. I’d genuinely be concerned that he would get charged with something really bad, or do something drastic and get killed by a responding police officer.

I wonder if we need to test this on someone who is in such a severe crisis. I also wonder at what point privacy is less important than safety - I wouldn’t characterize John as “safe”; he’s an abuse victim who is obsessed with his abuser’s approval, a pathological liar who destroys careers over getting caught in lies and the kind of person who puts the ideation on suicidal ideation.

Fact is, if you told a therapist half of what John told ChatGPT they would have some ethical requirements around safety. A generative AI doesn’t, and with people like John maybe that’s a safety issue even more than a privacy issue.

Re: When ChatGPT turns informant

#120

Post by caminanteblanco »

gitpusher wrote: post #112

The author makes it seem like we have two choices: 1) enable memory, and use ChatGPT like a confessional booth. Flood it with all of your deepest, darkest humiliations going all the way back to childhood ... 2) disable memory Perhaps my age is showing. But memory or no memory, I would never tell ChatGPT anything compromising about myself. Nor would I tweet such things, write them in an email, or put them into a Slack…

My thoughts exactly. And even if there is some need to use the LLM for something sensitive, most platforms have some sort of incognito mode. Not that it's going to stop the government, etc. from accessing the chat, but useful for the same things you would use a browser's incognito mode for: something you don't want accessible with a basic history search
Post reply on HN