Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hacking Google Bard – From Prompt Injection to Data Exfiltration

embracethered.com

Post reply on HN
Print view View original

121–130 of 163 posts

Re: Hacking Google Bard – From Prompt Injection to Data Exfiltration

#121

Post by nick222226 »

danShumway wrote: post #77
nomel wrote: post #70

Earlier quoted context omitted.

What use cases do you see this happening, where extraction of confidential data is an actual risk? Most use I see involved LLMs primed with a users data, or context around that, without any secret sauce. Or, are people treating the prompt design as some secret sauce?

I'd view this article as an example. I suspect it's not that hard to get a malicous document into someone's drive; basically any information you give to Bard is vulnerable to this attack if Bard then interacts with 3rd-party content. Email agents also come to mind, where an attacker can get a prompt into the LLM by sending an email that the LLM will then analyze in your inbox. Basically any scenario where an LLM is p…

Properly sandboxing and firewalling LLMs is going to be the killer app.

Re: Hacking Google Bard – From Prompt Injection to Data Exfiltration

#122

Post by l33t7332273 »

stainablesteel wrote: post #112

people are still trying manual prompt injections? i made a custom gpt to do that for me

I bet you could make another gpt to recognize them.

Did you write a blog or otherwise release the process you took to make that? It sounds pretty cool.

Re: Hacking Google Bard – From Prompt Injection to Data Exfiltration

#123

Post by nick222226 »

ratsmack wrote: post #46
1970-01-01 wrote: post #17

I love seeing Google getting caught with its pants down. This right here is a real-wold AI saftey issue that matters. Their moral alignment scenarios are fundamentally bullshit if this is all it takes to pop confidential data.

I have nothing against Google, but I enjoy watching so many people hyperventilating over the wonders of "AI" when it's just poorly simulated intelligence at best. I believe it will improve over time, but the current methods employed are nothing but brute force guessing at what a proper response should be.

Comparing what exists against the ideal is not a good assessment in my opinion. You've already become acclimated to the GPT that exists. "poorly simulated intelligence" using LLMs was unfathomable 5 years ago. In another 5 years we'll be far into the deep.

Re: Hacking Google Bard – From Prompt Injection to Data Exfiltration

#125

Post by godelski »

alex-robbins wrote: post #119
godelski wrote: post #114

Earlier quoted context omitted.

> Whats the endgame here? I don't mean to be rude, but at least to me the sentiment of this comment comes off as asking what the end game is for any hacker demonstrating vulnerabilities in ordinary software. There's always a cat and mouse game. I think we should all understand that given the name of this site... The point is to perform such checks on LLMs as we would with any software. There definitely is the ability…

> the sentiment of this comment comes off as asking what the end game is for any hacker demonstrating vulnerabilities GP isn't asking about the "endgame" as in "for what purpose did this author do this thing?". It was "endgame" as in "how is the story of LLMs going to end up?". It could be "just" more cat and mouse, like you both mentioned. But a sibling comment talks about the possibility for architectural changes,…

Okay yeah that makes more sense.

To respond more appropriately to that, I think truthfully we don't really know the answer to that right now (as implied my my previous comment). There are definitely people asking the question and it definitely is a good and important question but there's just a lot we don't know at this point. What we can and can't do. Maybe some take that as an unsatisfying answer but I think you could also take it as a more exciting answer as in there's this great mystery to be solved that's important and solving puzzles is fun. If you like puzzles haha. There are definitely a lot of interesting ideas out there such as those you mentioned and it'll be interesting to see what actually works and if those methods can actually maintain effectiveness as the systems evolve.

Re: Hacking Google Bard – From Prompt Injection to Data Exfiltration

#126

Post by simonw »

upupupandaway wrote: post #124

I don't understand the exfiltration part here. Wasn't only the user's own conversation that got copied elsewhere? That could have done in many different ways. I think I'm missing the point here.

That's the exfiltration. The user had been using Bard. They accept an invite to a new Google Doc with hidden instructions, at which point their previous conversation with Bard is exfiltrated via a loaded image link.

They did not intend for their previous conversation to be visible to an attacker. That's a security hole.

Maybe that conversation was entirely benign, or maybe they'd been previously asking for advice about a personal issue - healthcare or finance or relationship advice or something.

Re: Hacking Google Bard – From Prompt Injection to Data Exfiltration

#127

Post by edgyquant »

MagicMoonlight wrote: post #72

I tested bard prior to release and it was hilarious how breakable it was. The easiest trick I found was to just overflow its context. You fill up the entire context window with junk and then at the end introduce a new prompt and all it knows is that prompt because all the rules have been pushed out.

Bard was far less susceptible to simple context overflows than ChatGPT last time I checked. You can hit GPT4 with just a repeat of the word the for 2-3 prompts in a row and it will start schizoposting. This doesn’t work with Bard

Re: Hacking Google Bard – From Prompt Injection to Data Exfiltration

#128

Post by edgyquant »

potatoman22 wrote: post #118
logicchains wrote: post #116

Earlier quoted context omitted.

>There's no fundamental reason to think it's impossible There is, although we don't have a formal proof of it yet. Current LLMs are essentially Turning complete, in that they can be used to simulate any arbitrary Turing machine. This makes it impossible to prove an LLM will never output a certain statement for any possible input. The only way around this would be making a "non-Turing-complete" LLM variant, but it wou…

Couldn't you program the sampler to not output certain token sequences?

That’s seems extremely difficult if not impossible. There’s a million ways an idea can be conveyed in language.

Re: Hacking Google Bard – From Prompt Injection to Data Exfiltration

#129

Post by godelski »

c2occnw wrote: post #115
godelski wrote: post #114

Earlier quoted context omitted.

> Whats the endgame here? I don't mean to be rude, but at least to me the sentiment of this comment comes off as asking what the end game is for any hacker demonstrating vulnerabilities in ordinary software. There's always a cat and mouse game. I think we should all understand that given the name of this site... The point is to perform such checks on LLMs as we would with any software. There definitely is the ability…

Is there not some categorical difference between a purposefully-built system, which given enough time and effort and expertise and constraints, we can engineer to be effectively secure, and a stochastically-trained black box?

Yes? Kinda? Hard to say tbh. I think the distance between these categories is probably smaller than you're implying (or at least I'm interpreting), or rather the distinction between these categories is certainly not always clear or discernible (let alone meaningfully so).

Go is a game with no statistical elements yet there are so many possible move sets that it might as well be. I think we have a lower bound on the longest possible legal game being around 10^48 moves and an upper bound being around 10^170. At 10^31 moves per second (10 quettahertz) it'd still take you billions of years to play the lower bound longest possible game. It's pretty reasonable to believe we can never build a computer that can play the longest legal game even with insane amounts of parallelism and absurdly beautiful algorithms, let alone find a deterministic solution (the highest gamma ray we've ever detected is ~4RHz or 4x10^27) or "solving" Go. Go is just a board with 19x19 locations and 3 possible positions (nothing, white, black) (legal moves obviously reducing that 10^170 bound).

That might seem like a non-sequitur, but what I'm getting at is that there's a lot of permutations in software too and I don't think there are plenty of reasonably sized programs that would be impossible to validate correctness of within a reasonable amount of time. Pretty sure there's classes of programs we know that can't be validated in a finite time nor with finite resources. A different perspective on statistics is actually not viewing states as having randomness but viewing them as having levels of uncertainty. So there's a lot of statistics that is done in frameworks which do not have any value of true randomness (random like noise not random like np.random.randn()). Conceptually there's no difference between uncertainty and randomness, but I think it's easier to grasp the idea that there are many purposefully-built finite systems that have non-zero amounts of uncertainty, so those are no different than random systems.

More here on Go: https://senseis.xmp.net/?NumberOfPossibleGoGames And if someone knows more about go and wants to add more information or correct me I'd love to hear it. I definitely don't know enough about the game let alone the math, just using it as an example.

Re: Hacking Google Bard – From Prompt Injection to Data Exfiltration

#130

Post by rsanek »

shwouchk wrote: post #106
bastawhiz wrote: post #75

Earlier quoted context omitted.

How can you provide assurance that that there are no false positives or negatives? XSS detection was a thing that people attempted and it failed miserably because you need it to work correctly 100% of the time for it to be useful. Said another way, what customer needs and is willing to pay for prompt injection protection but has some tolerance for error?

Good point (not sarcastically). What customer needs and is willing to pay for an antivirus that has some tolerance for error?

every current antivirus software has some false positives and some false negatives, that's why sites like virustotal exist. i don't see how this is any different
Post reply on HN