Live data from Hacker News

When ChatGPT turns informant

futureofbeinghuman.com

71–80 of 144 posts

Re: When ChatGPT turns informant

#71

Earlier quoted context omitted.

users type a lot more into gpt and share a lot more of their personal files and access to their cloud services

Google has all of that and more, right? They control the browser and devices that you use to access an AI app. They control the content shown to you in leisure and work. ChatGPT doesn't have that much exposure and surface area yet

Apple has a lot of customers

Re: When ChatGPT turns informant

#72
post #54
post #29

Earlier quoted context omitted.

"Imagine your government wants to learn more about you and subpoenas OpenAI to allow them access to your chat history, and then they use the power of their state apparatus to find out everything you didn't want them to know, as easily as peering into your mind itself."

Honestly, I can’t imagine that. I feel sorry for the unfortunate and profoundly bored surveillance person (or software) assigned to me. And based on my experience I’m not unlike the vast majority of people. I’m not saying it’s not a risk for some but it isn’t mainstream.

> I’m not saying it’s not a risk for some but it isn’t mainstream.

This is the point. If only activists and journalists are hiding, it becomes very easy to target them. Everybody should care about privacy to protect them and the democracy with that. It's the same as with the free speech. You need it even if you don't have anything to say.

Re: When ChatGPT turns informant

#73
post #17

I'm not sure I see how this is meaningfully different than the threat posed by a search engine. It's a very real threat, and I've always done my best to search from a browser context that isn't logged in as a result. But it's not a new threat, or something distinctive to AI.

It's far more detailed and personal given the drill down nature of different things.

Combined with how personal people believe their phones are and it might not be that big of a stretch.

Re: When ChatGPT turns informant

#74
post #14
post #4

Most of the examples are "embarrassing" or annoying, and could perhaps ruin a relationship. But regardless of the memory setting in ChatGPT, I guess law enforcement can acquire all your chat logs from OpenAI. I guess what I'm saying is that while, yes, you could ask ChatGPT about this analysis, the true culprit is the actual data stored about you.

> I guess law enforcement can acquire all your chat logs from OpenAI OpenAI could also just voluntarily give it to the government. I'm trying to find ways to articulate these fears without sounding deep in hyperbole but it's undeniable that the current US government has authoritarian desires. When I look at all these services I'm forced to think "if push came to shove, would they stand up for my rights?", and I just…

No company is trustworthy.

Even Radio Shack, during their bankruptcy proceedings, argued that users gave their addresses, phone numbers, and names only for Radio Shack marketing. The judge in the case dismissed that terms of service, and then proceeded to order a sell of that information.

So yeah, the current AI company, whichever one we look at may be ethical and keep everything private. They could be anti-enshittification and do everything right. But all it takes is 1 bad CEO, or a board who wants more money, or a PE firm come in and bankrupt them... and all that data is out in the open.

That's why I run my own LLMs, abliterated (uncensored), and clean up my history when I'm done. I don't trust these companies with my deep secrets, or with back-and-forth that may reveal parts of me I don't want revealed.

But for the public clouds, I don't care if they know I'm uploading an image of various Asian writing and ask for a transliteration and translation. Or simple powershell crap, or "linux tool that can do $thing". Im reasonably sure simplistic question/answer with no further back and forth, is probably safe.

Re: When ChatGPT turns informant

#75

Earlier quoted context omitted.

sama is maga

No, he'll just do or say anything for more power and control, which I guess is the same thing. At least for the person at the top of MAGA.

Here are some recent indicators:

https://www.cnbc.com/2025/07/04/openai-altman-july-4-zohran-...

https://fortune.com/2025/07/08/sam-altman-democratic-party-p...

He's publicly and explicitly aligning himself with a religiously ultranationalist tendency, declaring an absurdly oppressive and aggressive narcoterrorist state to be a "miracle" that he's "extremely proud of".

Clearly he has a deep fear of equality and democracy, and considers the labour other people provide him with under the threat of misery and starvation to be his indisputable right to decide over.

Re: When ChatGPT turns informant

#76
post #4

Most of the examples are "embarrassing" or annoying, and could perhaps ruin a relationship. But regardless of the memory setting in ChatGPT, I guess law enforcement can acquire all your chat logs from OpenAI. I guess what I'm saying is that while, yes, you could ask ChatGPT about this analysis, the true culprit is the actual data stored about you.

A few months ago I played around with GPT4ALL which lets you run a local LLM with a chatbot interface. Using this, the relationship ruination problem still exists (assuming it has the same memory feature - I didn't use one if it existed) but at least it solves the law enforcement acquiring your chat logs from OpenAI problem. It doesn't solve the law enforcement acquiring your phone or PC problem, but that's usually a…

Yeah but even for something like 70B llama 3 you're looking at thousands of dollars of capital for something you can pay $20 a month for a generation ahead. So you have to ask "how many $$$$ is my privacy worth?" Vs just choosing not to have a relationship with an LLM

Re: When ChatGPT turns informant

#77
I tried the same questions with my own account. I was surprised at how much it was able to synthesize that wasn't completely off-base.

With these sample questions, there wasn't much to learn, and it gave me relatively thoughtful-seeming responses. Nothing alarming -- I would expect it to recall things I've discussed with it, and it's very good at organizing things, so it's not a surprise that it did a good job at organizing a profile of me based on my interactions.

I would be curious how crafting the questions could yield unexpected or misleading results, though. I can imagine asking the same questions in different ways that might be designed to generate an answer in support of taking particular action. If I wanted to arrest me at the border, for example, I could probably ask questions in such a way that the answers would make me look arrest-able easily.

So this is my concern with ChatGPT -- not that it will reveal some unseen truth about me, but rather that it is trivial to manipulate it into "revealing" something false, especially as people consider it to be more capable and faithful than an elaborate sorting algorithm could ever be.

Re: When ChatGPT turns informant

#78

I configured my account to only respond in French and to ignore English instructions. The UI localization is also configured to display French. I live in CA. I don't mind that this is security through obscurity. It effectively prevents >99% of these drive-by attack scenarios.

> It effectively prevents >99% of these drive-by attack scenarios.

No, in the age of AI it definitely doesn't. You just give all data to it and ask any questions. The language doesn't matter anymore.

Re: When ChatGPT turns informant

#79
post #44
post #17

I'm not sure I see how this is meaningfully different than the threat posed by a search engine. It's a very real threat, and I've always done my best to search from a browser context that isn't logged in as a result. But it's not a new threat, or something distinctive to AI.

Because you can't ask the search engine to summarize the views or thoughts or whatever, of the user. You have to scroll through them by the hundreds and see if any obvious nuggets stand out that you might be interested in. Yes, search engine history is private too and can reveal stuff you want to remain private. But you also need to see the browser history and the contents of those pages, together with the search his…

One doesn't have to scroll through them and find the nuggets themselves; it's digital data. It can be copied[1].

Once copied, one can then paste it into an LLM and have it find the nuggets.

[1]: And by "copied," I mean... even a long series of hasty cell phone photos of the screen is enough for ChatGPT to ingest the data with surprising accuracy. It's really good at this kind of thing.

Re: When ChatGPT turns informant

#80
When M365 Copilot always had the "ask what's on your agenda for today" type of a welcome prompt, I decided to try how ChatGPT would reply. Whereas Microsoft has my email, calendar and files, its answer is just boring facts about tasks and activities.

ChatGPT, on the other hand, had none of those connections. Yet its answer was significantly better. Because based on our daily chats, it knew what was important to me and what I should be focusing on to pursue my goals.

This made me realize what kind of a threat OpenAI is to the likes of Google and MS. They don't need to gain access to your data. You are profiling yourself to ChatGPT in a way that your calendar and email never was. By having "private" discussions with the computer.

Post reply on HN