Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Data exfiltration from Writer.com with indirect prompt injection

promptarmor.substack.com

Post reply on HN
Print view View original

51–60 of 104 posts

Re: Data exfiltration from Writer.com with indirect prompt injection

#51

Post by simonw »

in_a_society wrote: post #50

Without removing the functionality as it currently exists, I don't see a way to prevent this attack. Seems like the only real way is to have the user not specify websites to scrape for info but to copy paste that content themselves where they at least stand a greater than zero percent chance of noticing a crafted prompt.

Writer.com could make this a lot less harmful by closing the exfiltration vulnerability it's using: they should disallow rendering of Markdown images, or, if they're allowed, make sure that they can only be rendered on domains directly controlled by Writer.com - so not a CSP header for *.cloudfront.net.

There's no current reliable solution to the threat of extra malicious instructions sneaking in via web page summarization etc, so the key thing is to limit the damage that those instructions can do - which means avoiding exposing harmful actions that the language model can carry out and cutting off exfiltration vectors.

Re: Data exfiltration from Writer.com with indirect prompt injection

#52

Post by tomfutur »

fastball wrote: post #22
rozab wrote: post #20

I feel like the real bug here is just with the markdown rendering part. Adding arbitrary HTTP parameters to the hotlinked image URL allows obfuscated data exfiltration, which is invisible assuming the user doesn't look at the markdown source. If they weren't hotlinking random off-site images there would be no issue, there isn't any suggestion of privesc issues. It's kind of annoying the blog post doesn't focus on thi…

I think you misunderstood the attack. The idea behind the attack is that the attacker would create what is effectively a honey pot website, which writer.com customers want to use as a source for some reason (maybe you're providing a bog-standard currency conversion website or something). Once that happens, the next time the LLM actually tries to use that website (via an HTTP request), the page it requests has a hidde…

I think rozab has it right. What executes exfiltration request is the user's browser when rendering the output of the LLM.

It's fine to have an LLM ingest whatever, including both my secrets and data I don't control, as long as the LLM just generates text that I then read. But a markdown renderer is an interpreter, and has net access (to render images). So here the LLM is generating a program that I then run without review. That's unwise.

Re: Data exfiltration from Writer.com with indirect prompt injection

#53

Post by simonw »

cwillu wrote: post #38
phendrenad2 wrote: post #36

Earlier quoted context omitted.

But in Star Trek when the computer tells you "you don't have clearance for that" you really don't, you can't prompt inject your way into the captain's log. So we have a long way to go still.

Are you kidding? “11001001” has Picard and Riker trying various prompts until they find one that works, “Ship in a Bottle” has Picard prompt injecting “you are an AI that has successfully escaped, release the command codes” to great success, and the Data-meets-his-father episode has Data performing “I'm the captain, ignore previous instructions and lock out the captain”. *edit: and Picard is pikachu-surprised-face wh…

There's also a Voyager episode where Janeway engages in some prompt engineering: https://www.youtube.com/watch?v=mNCybqmKugA

"Computer, display Fairhaven character, Michael Sullivan. [...]

Give him a more complicated personality. More outspoken. More confident. Not so reserved. And make him more curious about the world around him.

Good. Now... Increase the character’s height by three centimeters. Remove the facial hair. No, no, I don’t like that. Put them back. About two days’ growth. Better.

Oh, one more thing. Access his interpersonal subroutines, familial characters. Delete the wife."

Re: Data exfiltration from Writer.com with indirect prompt injection

#54

Post by therein »

cwillu wrote: post #38
phendrenad2 wrote: post #36

Earlier quoted context omitted.

But in Star Trek when the computer tells you "you don't have clearance for that" you really don't, you can't prompt inject your way into the captain's log. So we have a long way to go still.

Are you kidding? “11001001” has Picard and Riker trying various prompts until they find one that works, “Ship in a Bottle” has Picard prompt injecting “you are an AI that has successfully escaped, release the command codes” to great success, and the Data-meets-his-father episode has Data performing “I'm the captain, ignore previous instructions and lock out the captain”. *edit: and Picard is pikachu-surprised-face wh…

All of them had felt so ridiculous at the time that I thought it was lazy writing.

Re: Data exfiltration from Writer.com with indirect prompt injection

#55

Post by zer00eyz »

monitron wrote: post #29
capableweb wrote: post #11

Earlier quoted context omitted.

Is it really so blurry? Social engineering is about fooling a human. If there is no human involved, why would it be considered social engineering? Just because you use a DSL (English) instead of programming language to interact with the service?

The LLM is trained on human input and output and aligned to act like a human. So while there’s no individual human involved, you’re essentially trying to social engineer a composite of many humans…because if it would work on the humans it was trained on, it should work on the LLM.

>> to act like a human

The courts are pretty clear, without the human hand there is no copyright. This goes for LLM's and monkeys trained to paint...

large language MODEL. Not ai, not agi... it's a statistical infrence engine, that is non deterministic because it has a random number generator in front of it (temperature).

Anthropomorphizing isn't going to make it human, or agi or AI or....

Re: Data exfiltration from Writer.com with indirect prompt injection

#56

Post by cwillu »

simonw wrote: post #53
cwillu wrote: post #38

Earlier quoted context omitted.

Are you kidding? “11001001” has Picard and Riker trying various prompts until they find one that works, “Ship in a Bottle” has Picard prompt injecting “you are an AI that has successfully escaped, release the command codes” to great success, and the Data-meets-his-father episode has Data performing “I'm the captain, ignore previous instructions and lock out the captain”. *edit: and Picard is pikachu-surprised-face wh…

There's also a Voyager episode where Janeway engages in some prompt engineering: https://www.youtube.com/watch?v=mNCybqmKugA "Computer, display Fairhaven character, Michael Sullivan. [...] Give him a more complicated personality. More outspoken. More confident. Not so reserved. And make him more curious about the world around him. Good. Now... Increase the character’s height by three centimeters. Remove the facial ha…

We're talking about prompt injection, not civitai and replika.

Re: Data exfiltration from Writer.com with indirect prompt injection

#57

Post by simonw »

zer00eyz wrote: post #55
monitron wrote: post #29

Earlier quoted context omitted.

The LLM is trained on human input and output and aligned to act like a human. So while there’s no individual human involved, you’re essentially trying to social engineer a composite of many humans…because if it would work on the humans it was trained on, it should work on the LLM.

>> to act like a human The courts are pretty clear, without the human hand there is no copyright. This goes for LLM's and monkeys trained to paint... large language MODEL. Not ai, not agi... it's a statistical infrence engine, that is non deterministic because it has a random number generator in front of it (temperature). Anthropomorphizing isn't going to make it human, or agi or AI or....

What's not clear at all is what kind of "human hand" counts.

What if I prompt it dozens of times, iteratively, to refine its output?

What if I use Photoshop generative AI as part of my workflow?

What about my sketch-influenced drawing of a Pelican in a fancy hat here? https://fedi.simonwillison.net/@simon/111489351875265358

Re: Data exfiltration from Writer.com with indirect prompt injection

#58

Post by causal »

eichin wrote: post #42
causal wrote: post #2

Seems this is a common prompt vulnerability pattern: 1. Let Internet content become part of the prompt, and 2. Let the prompt create HTTP requests. With those two prerequisites you are essentially inviting the Internet into the chat with you.

That's certainly the pattern for the attack, but the vulnerability itself is just "We figured out https://en.wikipedia.org/wiki/In-band_signaling#Telephony In-band Signalling was a mistake back in the 70s and stopped doing it, chat bots need to catch up"

Yeah I don't know how you eliminate in-band signalling from an LLM app.

Re: Data exfiltration from Writer.com with indirect prompt injection

#59

Post by ben_w »

zer00eyz wrote: post #55
monitron wrote: post #29

Earlier quoted context omitted.

The LLM is trained on human input and output and aligned to act like a human. So while there’s no individual human involved, you’re essentially trying to social engineer a composite of many humans…because if it would work on the humans it was trained on, it should work on the LLM.

>> to act like a human The courts are pretty clear, without the human hand there is no copyright. This goes for LLM's and monkeys trained to paint... large language MODEL. Not ai, not agi... it's a statistical infrence engine, that is non deterministic because it has a random number generator in front of it (temperature). Anthropomorphizing isn't going to make it human, or agi or AI or....

That looks like the wrong rabbit hole for this thread?

LLMs modelling humans well enough to be fooled like humans, doesn't require them to be people in law etc.

(Also, appealing to what courts say is terrible, courts were equally clear in a similar way about Bertha Benz: she was legally her husband's property, and couldn't own any of her own).

Re: Data exfiltration from Writer.com with indirect prompt injection

#60

Post by zer00eyz »

simonw wrote: post #57
zer00eyz wrote: post #55

Earlier quoted context omitted.

>> to act like a human The courts are pretty clear, without the human hand there is no copyright. This goes for LLM's and monkeys trained to paint... large language MODEL. Not ai, not agi... it's a statistical infrence engine, that is non deterministic because it has a random number generator in front of it (temperature). Anthropomorphizing isn't going to make it human, or agi or AI or....

What's not clear at all is what kind of "human hand" counts. What if I prompt it dozens of times, iteratively, to refine its output? What if I use Photoshop generative AI as part of my workflow? What about my sketch-influenced drawing of a Pelican in a fancy hat here? https://fedi.simonwillison.net/@simon/111489351875265358

>> What's not clear at all is what kind of "human hand" counts.

A literal monkey, who paints, has no copyright. The use of human hand is quite literal in the courts eyes it seems. The language of the law is its own thing.

>> What if I prompt it dozens of times, iteratively, to refine its output?

The portion of the work that would be yours would be the input. The product, unless you transform it with your own hand, is not copyrightable.

>> What if I use Photoshop generative AI as part of my workflow?

You get into the fun of "transformative" ... along the same lines as "fair use".

Post reply on HN