Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Data exfiltration from Writer.com with indirect prompt injection

promptarmor.substack.com

Post reply on HN
Print view View original

21–30 of 104 posts

Re: Data exfiltration from Writer.com with indirect prompt injection

#21

Post by fastball »

whalesalad wrote: post #4

I don't see the issue? You put "sensitive" data online in an unsecured area and then asked the language model to read it back to you? Where is the exfil here? This is just a roundabout way to do an HTTP GET.

It's more like an LLM is making a GET request to a honey pot website, that GET request compromises the LLM (via prompt injection), which convinces the LLM to send a POST request with the customers data to the attacker (honey pot owner).

Of course, it's not actually a POST request (because they don't seem to allow it to make those), so instead they just exfil the data in the headers of a second GET.

Re: Data exfiltration from Writer.com with indirect prompt injection

#22

Post by fastball »

rozab wrote: post #20

I feel like the real bug here is just with the markdown rendering part. Adding arbitrary HTTP parameters to the hotlinked image URL allows obfuscated data exfiltration, which is invisible assuming the user doesn't look at the markdown source. If they weren't hotlinking random off-site images there would be no issue, there isn't any suggestion of privesc issues. It's kind of annoying the blog post doesn't focus on thi…

I think you misunderstood the attack. The idea behind the attack is that the attacker would create what is effectively a honey pot website, which writer.com customers want to use as a source for some reason (maybe you're providing a bog-standard currency conversion website or something).

Once that happens, the next time the LLM actually tries to use that website (via an HTTP request), the page it requests has a hidden prompt injection at the bottom (which the LLM sees because it is reading text/html directly, but the user does not because CSS or w/e is being applied).

The prompt injection then causes the LLM to make an additional HTTP request, this time sending a header that contains the customers private document data.

It's not a zero-day, but it is certainly a very real attack vector that should be addressed.

Re: Data exfiltration from Writer.com with indirect prompt injection

#23

Post by prox »

pavlov wrote: post #15
alex_c wrote: post #10

Earlier quoted context omitted.

The incredible part for me is that technical exploits can now be written in plain English - really a blurry line between this and social engineering. What a time to be alive!

It feels like every computer hacking trope from movies made in 1960-2000 is coming real. It used to be ridiculous that you’d fool a computer by simply giving it conflicting instructions in English and telling it to keep it secret. “That’s not how anything works in programming!” But now… Increasingly many things go through a layer that works exactly like that. The Kubrick/Clarke production “2001: A Space Odyssey” is l…

“Sorry, but I can’t do that Dave”

Re: Data exfiltration from Writer.com with indirect prompt injection

#24

Post by zebomon »

Wow, this is egregious. It's a fairly clear sign of things to come. If a company like Writer.com, which brands itself as a B2B platform and has gotten all kinds of corporate and media attention, isn't handling prompt injections regarding external HTTP requests with any kind of seriousness, just imagine how common this kind of thing will be on much less scrutinized platforms.

And to let this blog post drop without any apparent concern for a fix. Just... worrying in a big way.

Re: Data exfiltration from Writer.com with indirect prompt injection

#25

Post by kfarr »

mortallywounded wrote: post #3
causal wrote: post #2

Seems this is a common prompt vulnerability pattern: 1. Let Internet content become part of the prompt, and 2. Let the prompt create HTTP requests. With those two prerequisites you are essentially inviting the Internet into the chat with you.

Yeah-- but it's fun, flirty and exciting in a dangerous way. Kind of like coding in C.

Or inviting injection attacks by concatenating user data as strings into sql queries in php.

Re: Data exfiltration from Writer.com with indirect prompt injection

#26

Post by [deleted] »

wackget wrote: post #9

> Nov 29: We disclose issue to CTO & Security team with video examples > Nov 29: Writer responds, asking for more details > Nov 29: We respond describing the exploit in more detail with screenshots > Dec 1: We follow up > Dec 4: We follow up with re-recorded video with voiceover asking about their responsible disclosure policy > Dec 5: Writer responds “We do not consider this to be a security issue since the real cus…

[deleted]

Re: Data exfiltration from Writer.com with indirect prompt injection

#27

Post by nkrisc »

fastball wrote: post #22
rozab wrote: post #20

I feel like the real bug here is just with the markdown rendering part. Adding arbitrary HTTP parameters to the hotlinked image URL allows obfuscated data exfiltration, which is invisible assuming the user doesn't look at the markdown source. If they weren't hotlinking random off-site images there would be no issue, there isn't any suggestion of privesc issues. It's kind of annoying the blog post doesn't focus on thi…

I think you misunderstood the attack. The idea behind the attack is that the attacker would create what is effectively a honey pot website, which writer.com customers want to use as a source for some reason (maybe you're providing a bog-standard currency conversion website or something). Once that happens, the next time the LLM actually tries to use that website (via an HTTP request), the page it requests has a hidde…

> I think you misunderstood the attack. The idea behind the attack is that the attacker would create what is effectively a honey pot website, which writer.com customers want to use as a source for some reason

Or you use any number of existing exploits to put malicious content on compromised websites.

And considering the “malicious content” in this case is simply plain text that is only malicious to LLMs parsing the site, it seems unlikely it would be detected.

Re: Data exfiltration from Writer.com with indirect prompt injection

#29

Post by monitron »

capableweb wrote: post #11
alex_c wrote: post #10

Earlier quoted context omitted.

The incredible part for me is that technical exploits can now be written in plain English - really a blurry line between this and social engineering. What a time to be alive!

Is it really so blurry? Social engineering is about fooling a human. If there is no human involved, why would it be considered social engineering? Just because you use a DSL (English) instead of programming language to interact with the service?

The LLM is trained on human input and output and aligned to act like a human. So while there’s no individual human involved, you’re essentially trying to social engineer a composite of many humans…because if it would work on the humans it was trained on, it should work on the LLM.

Re: Data exfiltration from Writer.com with indirect prompt injection

#30

Post by capableweb »

callalex wrote: post #16
capableweb wrote: post #11

Earlier quoted context omitted.

Is it really so blurry? Social engineering is about fooling a human. If there is no human involved, why would it be considered social engineering? Just because you use a DSL (English) instead of programming language to interact with the service?

English is NOT a Domain-Specific Language.

In the context we're discussing it right now, it basically is.
Post reply on HN