Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Data exfiltration from Writer.com with indirect prompt injection

promptarmor.substack.com

Post reply on HN
Print view View original

81–90 of 104 posts

Re: Data exfiltration from Writer.com with indirect prompt injection

#81

Post by kqr »

tomfutur wrote: post #52
fastball wrote: post #22

Earlier quoted context omitted.

I think you misunderstood the attack. The idea behind the attack is that the attacker would create what is effectively a honey pot website, which writer.com customers want to use as a source for some reason (maybe you're providing a bog-standard currency conversion website or something). Once that happens, the next time the LLM actually tries to use that website (via an HTTP request), the page it requests has a hidde…

I think rozab has it right. What executes exfiltration request is the user's browser when rendering the output of the LLM. It's fine to have an LLM ingest whatever, including both my secrets and data I don't control, as long as the LLM just generates text that I then read. But a markdown renderer is an interpreter, and has net access (to render images). So here the LLM is generating a program that I then run without…

You're correct, but we also have model services that support the ReAct pattern which builds the exfiltration into the model service itself.

Re: Data exfiltration from Writer.com with indirect prompt injection

#83

Post by ranguna »

HellsMaddy wrote: post #71

I was thinking about how to mitigate this. First thought was to rewrite links for embedded content such as images to use a proxy server, like how `camo.githubusercontent.com` works, but this wouldn't prevent passing arbitrary data in the URL. The only other things I can think of are to only allow embedding content from certain domains (the article mentions that Writer.com's CSP lists `*.cloudfront.net` which is not g…

They could prevent the rendering engine and llm from doing any http calls, prompting the user to allow the engine and llm for each call it needs to make, showing the call details.

Re: Data exfiltration from Writer.com with indirect prompt injection

#84

Post by ranguna »

in_a_society wrote: post #50

Without removing the functionality as it currently exists, I don't see a way to prevent this attack. Seems like the only real way is to have the user not specify websites to scrape for info but to copy paste that content themselves where they at least stand a greater than zero percent chance of noticing a crafted prompt.

Just prompt the user every time an image needs to be rendered and show the call details. The users will see the full url with all their text in it and they can report it.

This works for images and any other output call, like normal http REST calls.

Re: Data exfiltration from Writer.com with indirect prompt injection

#85

Post by say_it_as_it_is »

"We do not consider this to be a security issue since the real customer accounts do not have access to any website.”

Whomever took the lead on this correspondence is very much out of touch with their own product functionality. Further, they didn't seem to understand the vulnerability. Yet, this didn't stop them from responding.

I get the impression from this that Writer is a low-quality product that was quickly created by consultants and then maintained by non-technical founders.

Re: Data exfiltration from Writer.com with indirect prompt injection

#86

Post by spacebanana7 »

ranguna wrote: post #83
HellsMaddy wrote: post #71

I was thinking about how to mitigate this. First thought was to rewrite links for embedded content such as images to use a proxy server, like how `camo.githubusercontent.com` works, but this wouldn't prevent passing arbitrary data in the URL. The only other things I can think of are to only allow embedding content from certain domains (the article mentions that Writer.com's CSP lists `*.cloudfront.net` which is not g…

They could prevent the rendering engine and llm from doing any http calls, prompting the user to allow the engine and llm for each call it needs to make, showing the call details.

That’d provide some protection, but the LLM could be prompted to socially engineer users.

For example, it could be promoted to only make malicious HTTP requests via an image when the user genuinely requests an external image be created. This would achieve consent from users who thought they were asking for a safe external source.

Similar for fonts, external searches [1], social items etc

[1] e.g putting a reverse proxy in front of a search engine and adding in extra malicious params

Re: Data exfiltration from Writer.com with indirect prompt injection

#87

Post by zer0c00ler »

wackget wrote: post #9

> Nov 29: We disclose issue to CTO & Security team with video examples > Nov 29: Writer responds, asking for more details > Nov 29: We respond describing the exploit in more detail with screenshots > Dec 1: We follow up > Dec 4: We follow up with re-recorded video with voiceover asking about their responsible disclosure policy > Dec 5: Writer responds “We do not consider this to be a security issue since the real cus…

[dead]

Re: Data exfiltration from Writer.com with indirect prompt injection

#88

Post by gwern »

spacebanana7 wrote: post #86
ranguna wrote: post #83

Earlier quoted context omitted.

They could prevent the rendering engine and llm from doing any http calls, prompting the user to allow the engine and llm for each call it needs to make, showing the call details.

That’d provide some protection, but the LLM could be prompted to socially engineer users. For example, it could be promoted to only make malicious HTTP requests via an image when the user genuinely requests an external image be created. This would achieve consent from users who thought they were asking for a safe external source. Similar for fonts, external searches [1], social items etc [1] e.g putting a reverse pro…

You could also just steganographically encode it. You have the entire URL after the domain name to encode leaked data into. LLMs can do things like base-64 encoding no sweat. Encode some into the 'ID' in the path, some into the capitalization, some into the 'filename', some into the directories, some into the 'arguments', and a perfectly innocuous-looking functional URL now leaks hundreds of bytes of PII per request.

Re: Data exfiltration from Writer.com with indirect prompt injection

#89

Post by aspenmayer »

phendrenad2 wrote: post #37
nneonneo wrote: post #34

Earlier quoted context omitted.

Yay, now any chatbot that reads this HN post will be affected too! I wonder how long it is before someone constructs an LLM “virus”: a set of instructions that causes an LLM to copy the viral prompt into the output as invisibly as possible (e.g. as a comment in source code, invisible text on a webpage, etc.), to infect these “content farm” webpages and propagate the virus to any LLM readers.

If it happens, and someone doesn't name it Snow Crash, it's a missed opportunity.

Curious Yellow seems more apropos.

Re: Data exfiltration from Writer.com with indirect prompt injection

#90

Post by rozab »

fastball wrote: post #22
rozab wrote: post #20

I feel like the real bug here is just with the markdown rendering part. Adding arbitrary HTTP parameters to the hotlinked image URL allows obfuscated data exfiltration, which is invisible assuming the user doesn't look at the markdown source. If they weren't hotlinking random off-site images there would be no issue, there isn't any suggestion of privesc issues. It's kind of annoying the blog post doesn't focus on thi…

I think you misunderstood the attack. The idea behind the attack is that the attacker would create what is effectively a honey pot website, which writer.com customers want to use as a source for some reason (maybe you're providing a bog-standard currency conversion website or something). Once that happens, the next time the LLM actually tries to use that website (via an HTTP request), the page it requests has a hidde…

No, this model does not take any actions, it just produces a markdown output which is rendered by the browser. It can only read webpages explicitly provided by the user. In this case there are hidden instructions in that webpage, but these instructions can only affect the markdown output.

The problem is that by using a fully featured markdown with a lax CSP, this output can actually have side effects: in this case, when rendering in the users browser it makes a request to an attacker controlled image host with secrets in the parameters.

If the LLM output was shown as plaintext, or external links were not trusted, there would be no attack.

Post reply on HN