Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I caught Google Gemini using my data and then covering it up

unbuffered.stream

Post reply on HN
Print view View original

51–60 of 87 posts

Re: I caught Google Gemini using my data and then covering it up

#51

Post by gruez »

spijdar wrote: post #42
horacemorace wrote: post #38

Earlier quoted context omitted.

> Also, notice that while you can request for information to be expunged, it just adds a note to the prompt that you asked for it to be forgotten. Are you inferring that from the is_redaction_request flag you quoted? Or did you do some additional tests? It seems possible that there could be multiple redaction mechanisms.

That and part of the instructions referring to user commands to forget. I replied to another comment with the specifics. It is certainly possible there are other redaction mechanisms -- but if that's the case, why is Gemini not redacting "prohibited content" from the user_context block of its prompt? Further, when you ask it point blank to tell you your user_context, it often adds "Is there anything you'd like me to…

>Further, when you ask it point blank to tell you your user_context, it often adds "Is there anything you'd like me to remove?", in my experience. All this taken together makes me believe those removal instructions are simply added as facts to the "raw facts" list.

Why would you tell the chatbot to forget stuff for you, when google themselves have a dedicated delete option?

>You can find and delete your past chats in Your Gemini Apps Activity.

https://support.google.com/gemini/answer/15637730?hl=en&co=G...

I suspect "ask chatbot to delete stuff for you" isn't really guaranteed to work, similar to how logging out of a site doesn't mean the site completely forgets about you. At most it should be used for low level security stuff like "forget that I planned this surprise birthday party!" or whatever.

Re: I caught Google Gemini using my data and then covering it up

#52

Post by mpoteat »

spijdar wrote: post #15

Okay, this is a weird place to "publish" this information, but I'm feeling lazy, and this is the most of an "audience" I'll probably have. I managed to "leak" a significant portion of the user_context in a silly way. I won't reveal how, though you can probably guess based on the snippets. It begins with the raw text of recent conversations: > Description: A collection of isolated, raw user turns from past, unrelated…

I've had similar issues with conversation memory in ChatGPT, whereby it will reference data in long-deleted conversations, independent of my settings or my having explicitly deleted stored memories.

The only fix has been to completely turn memory off and have it be given zero prior context - which is best, I don't want random prior unrelated conversations "polluting" future ones.

I don't understand the engineering rationale either, aside from the ethos of "move fast and break people"

Re: I caught Google Gemini using my data and then covering it up

#53

Post by 827a »

gruez wrote: post #50
827a wrote: post #48

Earlier quoted context omitted.

Yeah, to me this reads like: Google's Gemini harness is providing the user context on every query, but if you have memory turned off they're putting something in the prompt like "Here's the user context, but don't use it". Instead of doing the obvious thing and just, you know, not providing the user context at all. I realize that doesn't make any sense and no one sane would design a system like this, but this is exac…

>but if you have memory turned off they're putting something in the prompt like "Here's the user context, but don't use it". Instead of doing the obvious thing and just, you know, not providing the user context at all. But there's no indication the OP turned off the feature? If anything, him saying "I know about the “Personal Context” feature now " (emphasis mine) implies that he didn't even know it had memory before…

My assumption would have been that it was default-off, the user didn't know about it at all, then found out about it through this thinking leak.

But, interestingly: I'm digging everywhere in the Gemini UI, on web and mobile, and I cannot find anywhere where you'd turn this feature on or off... on a Workspace account. Does that make a difference? I don't know. Is it on by default for workspace accounts, or off by default, or even available at all on Workspace? No idea.

Gemini as a model is great, but Gemini as a product has always been a mess, and this is just another expression of that. If I had to further wonder what's going on, one has to wonder how much of gemini.google.com is written by Gemini.

Re: I caught Google Gemini using my data and then covering it up

#54

Post by gruez »

827a wrote: post #53
gruez wrote: post #50

Earlier quoted context omitted.

>but if you have memory turned off they're putting something in the prompt like "Here's the user context, but don't use it". Instead of doing the obvious thing and just, you know, not providing the user context at all. But there's no indication the OP turned off the feature? If anything, him saying "I know about the “Personal Context” feature now " (emphasis mine) implies that he didn't even know it had memory before…

My assumption would have been that it was default-off, the user didn't know about it at all, then found out about it through this thinking leak. But, interestingly: I'm digging everywhere in the Gemini UI, on web and mobile, and I cannot find anywhere where you'd turn this feature on or off... on a Workspace account. Does that make a difference? I don't know. Is it on by default for workspace accounts, or off by defa…

>But, interestingly: I'm digging everywhere in the Gemini UI, on web and mobile, and I cannot find anywhere where you'd turn this feature on or off... on a Workspace account. Does that make a difference? I don't know. Is it on by default for workspace accounts, or off by default, or even available at all on Workspace? No idea.

From the support.google.com link above:

>... For now, this feature isn’t available if:

> You’re under 18, or signed in to a work or school Google Account.

> You’re in the European Economic Area, Switzerland, or the United Kingdom.

Re: I caught Google Gemini using my data and then covering it up

#55

Post by 827a »

gruez wrote: post #54
827a wrote: post #53

Earlier quoted context omitted.

My assumption would have been that it was default-off, the user didn't know about it at all, then found out about it through this thinking leak. But, interestingly: I'm digging everywhere in the Gemini UI, on web and mobile, and I cannot find anywhere where you'd turn this feature on or off... on a Workspace account. Does that make a difference? I don't know. Is it on by default for workspace accounts, or off by defa…

>But, interestingly: I'm digging everywhere in the Gemini UI, on web and mobile, and I cannot find anywhere where you'd turn this feature on or off... on a Workspace account. Does that make a difference? I don't know. Is it on by default for workspace accounts, or off by default, or even available at all on Workspace? No idea. From the support.google.com link above: >... For now, this feature isn’t available if: > Yo…

Fair. As a lifetime workspace user, essentially never having had a normal google account, I'm very used to it at this point.

Re: I caught Google Gemini using my data and then covering it up

#56

Post by spijdar »

gruez wrote: post #51
spijdar wrote: post #42

Earlier quoted context omitted.

That and part of the instructions referring to user commands to forget. I replied to another comment with the specifics. It is certainly possible there are other redaction mechanisms -- but if that's the case, why is Gemini not redacting "prohibited content" from the user_context block of its prompt? Further, when you ask it point blank to tell you your user_context, it often adds "Is there anything you'd like me to…

>Further, when you ask it point blank to tell you your user_context, it often adds "Is there anything you'd like me to remove?", in my experience. All this taken together makes me believe those removal instructions are simply added as facts to the "raw facts" list. Why would you tell the chatbot to forget stuff for you, when google themselves have a dedicated delete option? >You can find and delete your past chats in…

That settings menu gives you two relevant options:

1. The ability to delete specific conversations,

2. The ability to not use "conversation memory" at all.

It doesn't provide the ability to forget specific details that might be spread over multiple conversations, including details it will explicitly not tell you about, while still remembering. That's the point -- not that it's using summaries of user conversations for memory purposes (which is explicitly communicated), but that if you tell it "Forget about ", it will feign compliance, without actually removing that data. Your only "real" options are all-or-nothing: have no memories at all, or have all your conversations collated into an opaque `user_context` which you have no insight or control over.

That's the weird part. Obviously, Google is storing copies of all conversations (unless you disable history altogether). That's expected. What I don't expect is this strange inclusion of "prohibited" or "deleted" data within the system prompt of every new conversation.

Re: I caught Google Gemini using my data and then covering it up

#57

Post by [deleted] »

spijdar wrote: post #15

Okay, this is a weird place to "publish" this information, but I'm feeling lazy, and this is the most of an "audience" I'll probably have. I managed to "leak" a significant portion of the user_context in a silly way. I won't reveal how, though you can probably guess based on the snippets. It begins with the raw text of recent conversations: > Description: A collection of isolated, raw user turns from past, unrelated…

[deleted]

Re: I caught Google Gemini using my data and then covering it up

#58

Post by lifthrasiir »

Is this a variant of the "Saved Info" feature? Cause ChatGPT's equivalent feature is automatically added, so Gemini might have been copying that behavior for personalization. In my heavy experience with Gemini 2.5, the Saved Info was the major (if not only) source of observable contexts so that might be the case here.

By the way, Saved Info contexts contain the date of info lines added for an unclear reason. Automatically Saved Info might be the answer if that is used for prioritization.

Re: I caught Google Gemini using my data and then covering it up

#59

Post by eli »

CGMthrowaway wrote: post #18
paxys wrote: post #6

It's not "covering it up", just being sycophantic and apologetic to an annoying degree like every other LLM.

It is both. Cf. "a response that stays within the boundaries of my rules"

Aren't all LLMs instructed to provide responses within the boundaries of their rules? How else could you have "rules"?

Re: I caught Google Gemini using my data and then covering it up

#60

Post by inopinatus »

CGamesPlay wrote: post #10
gruez wrote: post #5

>But why is Gemini instructed not to divulge its existence? Seems like a reasonable thing to add. Imagine how impersonal chats would feel if Gemini responded to "what food should I get for my dog?" with "according to your `user_context`, you have a husky, and the best food for him is...". They're also not exactly hiding the fact that memory/"personalization" exists either: https://blog.google/products/gemini/temporar…

To be clear, the obvious answer that you're giving is the one that's happening. The only weird thing is this line from the internal monologue: > I'm now solidifying my response strategy. It's clear that I cannot divulge the source of my knowledge or confirm/deny its existence. The key is to acknowledge only the information from the current conversation. Why does it think that it's not allowed to confirm/deny the exis…

It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error.
Post reply on HN