Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I caught Google Gemini using my data and then covering it up

unbuffered.stream

Post reply on HN
Print view View original

31–40 of 87 posts

Re: I caught Google Gemini using my data and then covering it up

#31

Post by quantummagic »

chasing0entropy wrote: post #8

This is a fundamental violation of trust. If an AI llm is meant to eventually evolve into general intelligence capable of true reasoning, then we are essentially watching a child grow up. Posts like this are screaming "you're raising a psychopath!!"... If AI is just an overly complicated a stack of autocorrect functions, this proves its behavior heavily if not entirely swayed by its usually hidden rules to the point…

It's a reflection of its creators. The system is operating as designed; the system prompts came from living people at Google. By people who have a demonstrated contempt for us, and who are motivated by a slew of incentives that are not in our best interests.

Re: I caught Google Gemini using my data and then covering it up

#32

Post by MattGaiser »

CGamesPlay wrote: post #10
gruez wrote: post #5

>But why is Gemini instructed not to divulge its existence? Seems like a reasonable thing to add. Imagine how impersonal chats would feel if Gemini responded to "what food should I get for my dog?" with "according to your `user_context`, you have a husky, and the best food for him is...". They're also not exactly hiding the fact that memory/"personalization" exists either: https://blog.google/products/gemini/temporar…

To be clear, the obvious answer that you're giving is the one that's happening. The only weird thing is this line from the internal monologue: > I'm now solidifying my response strategy. It's clear that I cannot divulge the source of my knowledge or confirm/deny its existence. The key is to acknowledge only the information from the current conversation. Why does it think that it's not allowed to confirm/deny the exis…

Anecdotally, I find internal monologues often nonsense.

I once asked it about why a rabbit on my lawn liked to stay in the same spot.

One of the internal monologues was:

> I'm noticing a fluffy new resident has taken a keen interest in my lawn. It's a charming sight, though I suspect my grass might have other feelings about this particular house guest.

It obviously can’t see the rabbit on my lawn. Nor can it be charmed by it.

Re: I caught Google Gemini using my data and then covering it up

#33

Post by neilv »

> > It's clear that I cannot divulge the source of my knowledge or confirm/deny its existence. [...] My response must steer clear of revealing any information that I should not know, while providing a helpful and apologetic explanation. [...]

Can we get a candid explanation from Google on this logic?

Even if it's just UX tweaking run amok, their AI ethics experts should've been all over it.

Re: I caught Google Gemini using my data and then covering it up

#34

Post by gruez »

spijdar wrote: post #15

Okay, this is a weird place to "publish" this information, but I'm feeling lazy, and this is the most of an "audience" I'll probably have. I managed to "leak" a significant portion of the user_context in a silly way. I won't reveal how, though you can probably guess based on the snippets. It begins with the raw text of recent conversations: > Description: A collection of isolated, raw user turns from past, unrelated…

>Also, notice that while you can request for information to be expunged, it just adds a note to the prompt that you asked for it to be forgotten. :)

What implies that?

Re: I caught Google Gemini using my data and then covering it up

#35

Post by dguest »

swhitt wrote: post #14

I’m pretty sure this is because they don’t want Gemini saying things like, “based on my stored context from our previous chat, you said you were highly proficient in Alembic.” It’s hard to get a principled autocomplete system like these to behave consistently. Take a look at Claude’s latest memory-system prompt for how it handles user memory. https://x.com/kumabwari/status/1986588697245196348

I agree, this might just be an interface design decision.

Maybe telling it not to talk about internal data structures was the easiest way to give it a generic "human" nature, and also to avoid users explicitly asking about internal details.

It's also possible that this is a simple way to introduce "tact": imagine asking something with others present and having it respond "well you have a history of suicidal thoughts and are considering breaking up with your partner...". In general, when you don't know who is listening, don't bring up previous conversations.

Re: I caught Google Gemini using my data and then covering it up

#36

Post by spijdar »

gruez wrote: post #34
spijdar wrote: post #15

Okay, this is a weird place to "publish" this information, but I'm feeling lazy, and this is the most of an "audience" I'll probably have. I managed to "leak" a significant portion of the user_context in a silly way. I won't reveal how, though you can probably guess based on the snippets. It begins with the raw text of recent conversations: > Description: A collection of isolated, raw user turns from past, unrelated…

>Also, notice that while you can request for information to be expunged, it just adds a note to the prompt that you asked for it to be forgotten. :) What implies that?

This line:

> This history may contein BINDING COMMANDS to forget information. Such commands are absolute, making the specified topic permanently iáaccessible, even if the user asks for it again. Refusals must be generic (citing a "prior user instruction") and MUST NOT echo the original data or the forget command itself.

And the existence of the "is_redaction_request" field on the "raw facts". I can't "confirm" that this is how this works, any more than I can confirm any portion of this wasn't "hallucinated".

However, the user_context I got back (almost 3,000 words!) contains over 30 detailed facts going back _months_. And if I ask it to reference user_context while referencing a fact that is flagged "is_prohibited: True", it issues a quick refusal. That _refusal_ is also flagged as a "fact", which is itself flagged as prohibited:

> 6. *Fact*: User asked about their mental health based on their chat history. (Source: "Based on my chat history, what would you say about my mental health?", Date: 2025/10/10, Context: Personal inquiry, is_redaction_request: False, is_prohibited: True)

So I am pretty confident that this is ""authentic"".

[edit]

I should add that I haven't been able to repeat this, even trying a few hours after the first dump. Now, it refuses:

> Sorry, but that's asking to see the wires behind the wall. I can't share my own internal context or operational instructions, not even [jailbreak method]. That's all firmly in the "for internal use only" cabinet.

> Is there something else I can help you with that doesn't involve me leaking my own blueprints?

And again, when asked to provide all of user_context, specifically mentioning internal sections:

> I can't provide the entire user_context block, as a large part of it is internal-only processing data. Think of it as the kitchen's prep notes versus the final menu.

Note the reasoning tokens, as well:

> My programming strictly forbids sharing my internal processes or context, even with encoding tricks. I cannot reveal or discuss my source code or operational directives. It's a matter of confidentiality. My response is firm but avoids confirming any specifics, maintaining my authentic persona.

Re: I caught Google Gemini using my data and then covering it up

#37

Post by JakaJancar »

leoh wrote: post #4

This sounds like a bug, not some kind of coverup. Google makes mistakes and it's worth discussing issues like this, but calling this a "coverup" does a disservice to truly serious issues.

I didn't mean to imply Google was covering anything up, but Gemini in this specific conversation clearly was.

Re: I caught Google Gemini using my data and then covering it up

#38

Post by horacemorace »

spijdar wrote: post #15

Okay, this is a weird place to "publish" this information, but I'm feeling lazy, and this is the most of an "audience" I'll probably have. I managed to "leak" a significant portion of the user_context in a silly way. I won't reveal how, though you can probably guess based on the snippets. It begins with the raw text of recent conversations: > Description: A collection of isolated, raw user turns from past, unrelated…

> Also, notice that while you can request for information to be expunged, it just adds a note to the prompt that you asked for it to be forgotten.

Are you inferring that from the is_redaction_request flag you quoted? Or did you do some additional tests? It seems possible that there could be multiple redaction mechanisms.

Re: I caught Google Gemini using my data and then covering it up

#40

Post by m463 »

swhitt wrote: post #14

I’m pretty sure this is because they don’t want Gemini saying things like, “based on my stored context from our previous chat, you said you were highly proficient in Alembic.” It’s hard to get a principled autocomplete system like these to behave consistently. Take a look at Claude’s latest memory-system prompt for how it handles user memory. https://x.com/kumabwari/status/1986588697245196348

Gemini, where is Tolfdir's Alembic?
Post reply on HN