Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hacking Google Bard – From Prompt Injection to Data Exfiltration

embracethered.com

Post reply on HN
Print view View original

91–100 of 163 posts

Re: Hacking Google Bard – From Prompt Injection to Data Exfiltration

#91

Post by JTon »

MagicMoonlight wrote: post #72

I tested bard prior to release and it was hilarious how breakable it was. The easiest trick I found was to just overflow its context. You fill up the entire context window with junk and then at the end introduce a new prompt and all it knows is that prompt because all the rules have been pushed out.

> because all the rules have been pushed out.

Can you unpack this a little please? Is it possible to ELI5 the mechanisms involved that can "push" a rule set out? I would have assumed the rules apply globally/uniformly across the entire prompt

Re: Hacking Google Bard – From Prompt Injection to Data Exfiltration

#92

Post by grepfru_it »

MagicMoonlight wrote: post #72

I tested bard prior to release and it was hilarious how breakable it was. The easiest trick I found was to just overflow its context. You fill up the entire context window with junk and then at the end introduce a new prompt and all it knows is that prompt because all the rules have been pushed out.

I was able to browse google and youtube source code in the very very early days. Was only patched when I called up a friend and let him know. And I tried to submit the flaw through normal channels of a supportless technology company but you can guess how well that went...

Re: Hacking Google Bard – From Prompt Injection to Data Exfiltration

#93

Post by monkpit »

simonw wrote: post #32
monkpit wrote: post #30

Earlier quoted context omitted.

Why not just have a safeguard tool that checks the LLM output and doesn’t accept user input? It could even be another LLM.

Using AI to detect attacks against AI isn't a good option in my opinion. I wrote about why here: https://simonwillison.net/2022/Sep/17/prompt-injection-more-...

“And doesn’t accept user input” is basically what you outlined there with your section about API shape.

Re: Hacking Google Bard – From Prompt Injection to Data Exfiltration

#94

Post by grepfru_it »

simonw wrote: post #32
monkpit wrote: post #30

Earlier quoted context omitted.

Why not just have a safeguard tool that checks the LLM output and doesn’t accept user input? It could even be another LLM.

Using AI to detect attacks against AI isn't a good option in my opinion. I wrote about why here: https://simonwillison.net/2022/Sep/17/prompt-injection-more-...

I used an LLM to generate a summary of your article:

The author argues that prompt injection attacks against language models cannot be solved with more AI. They propose that the only credible mitigation is to have clear, enforced separation between instructional prompts and untrusted input. Until one of the AI vendors produces an interface like this, the author suggests that we may just have to learn to live with the threat of prompt injection.

Re: Hacking Google Bard – From Prompt Injection to Data Exfiltration

#95

Post by rockinghigh »

mr_toad wrote: post #84
jstarfish wrote: post #40

Earlier quoted context omitted.

Naive question, but why not fine-tune models on The Art of Deception, Tony Robbins seminars and other content that specifically articulates the how-tos of social engineering? Like, these things can detect when you're trying to trick it into talking dirty. Getting it to second-guess whether you're literally using coercive tricks straight from the domestic violence handbook shouldn't be that much of a stretch.

They aren’t smart enough to lie. To do that you need a model of behaviour as well as language. Deception involves learning things like the person you’re trying to deceive exists as an independent entity, that that entity might not know things you know, and that you can influence their behaviour with what you say.

You could fine tune a model to lie, deceive, and try to extract information via a conversation.

Re: Hacking Google Bard – From Prompt Injection to Data Exfiltration

#96

Post by getpost »

simonw wrote: post #73
getpost wrote: post #57

>So, Bard can now access and analyze your Drive, Docs and Gmail! I asked Bard if I could use it to access gmail, and it said, "As a language model, I am not able to access your Gmail directly." I then asked Bard for a list of extensions, and it listed a Gmail extension as one of the "Google Workspace extensions." How do I activate the Gmail extension? "The Bard for Gmail extension is not currently available for activ…

Asking models about their own capabilities rarely returns useful results, because they were trained on data that existed before they were created. That said, Google really could fix this with Bard - they could inject an extra hidden prompt beforehand that anticipates these kinds of questions. Not sure why they don't do that.

I've been wondering about how to do incremental updates without incurring the cost of a full recalculation of the training data. I suppose I assumed that LLM providers would (if not now, eventually) incorporate a fine-tuning step to update a model's self-knowledge before making the model available. This would avoid including the update in the context length.

Among many, many applications, this would be helpful in allowing LLMs to converse about the current version of a website or application. I'd want a sense of time to be maintained, so that the LLM would know, if asked, about various versions. "Before the April 5, 2023 update, this feature was limited to ..., but now ... is supported."

I asked GPT4 about incremental updates, and it seemed to validate by my basic understanding. Here's the conversation so far:

https://chat.openai.com/share/00fe148a-13aa-4e92-8b77-f0de48...

Re: Hacking Google Bard – From Prompt Injection to Data Exfiltration

#97

Post by notfed »

Angostura wrote: post #86
notfed wrote: post #48

Earlier quoted context omitted.

This isn't an LLM problem. It's a XSS problem, and it's as old as Myspace. I don't think prompt engineering needs to be considered. The solution is to treat an LLM as untrusted, and design around that.

How untrustworthy though? Shoud I simply discard all its output? Presumably not, so that's the problem.

Hacker News doesn't trust you, and you're still able to post text. There are safe ways to handle untrusted data sources.

Re: Hacking Google Bard – From Prompt Injection to Data Exfiltration

#98

Post by WaffleIronMaker »

JTon wrote: post #91
MagicMoonlight wrote: post #72

I tested bard prior to release and it was hilarious how breakable it was. The easiest trick I found was to just overflow its context. You fill up the entire context window with junk and then at the end introduce a new prompt and all it knows is that prompt because all the rules have been pushed out.

> because all the rules have been pushed out. Can you unpack this a little please? Is it possible to ELI5 the mechanisms involved that can "push" a rule set out? I would have assumed the rules apply globally/uniformly across the entire prompt

> ELI5

The model can look at X amount of input to decide what words come next.

Normally, Google fills part of X with instructions, and you control the other part.

However if you give it exactly X amount of input, then there's no room for Google's original instructions, and you control it all.

Re: Hacking Google Bard – From Prompt Injection to Data Exfiltration

#99

Post by Lucasoato »

danShumway wrote: post #87
zsolt_terek wrote: post #55

We at Lakera AI work on a prompt injection detector that actually catches this particular attack. The models are trained on various data sources, including prompts from the Gandalf prompt injection game.

I have beef with Lakera AI specifically -- Lakera AI has never produced a public demo that has a 100% defense rate against prompt injection. Lakera has launched a "game" that it uses for harvesting data to train its own models, but that game has never been effective at preventing 100% of attacks and does not span the full gamut of every possible attack. If Lakera AI had a defense for this, the company would be able t…

Sorry, where is Lakera claiming to have 100% success rate to an ever changing attack?

Of course that’s a known fact among technical people expert in that matter that an impassable defense against any kind of attack of this nature is impossible.

Re: Hacking Google Bard – From Prompt Injection to Data Exfiltration

#100

Post by getpost »

rvba wrote: post #83
simonw wrote: post #73

Earlier quoted context omitted.

Asking models about their own capabilities rarely returns useful results, because they were trained on data that existed before they were created. That said, Google really could fix this with Bard - they could inject an extra hidden prompt beforehand that anticipates these kinds of questions. Not sure why they don't do that.

Because they are a company outsourced to cheap countries that lost its competitive edge. Average tenure is 1.3 years, so they are more like an outsourcing company that churns crappy projects made by interns. Projects get cancelled due to no promotions

Right, humans are involved in fine-tuning, at least for the time being. GPT4 says, ..."core content creation and verification are human-driven tasks."
Post reply on HN