well, shit.
This is how the neanderthals felt when they realized the homo sapiens were sentient, isn't it?
Data exfiltration from Writer.com with indirect prompt injection
61–70 of 104 posts
Re: Data exfiltration from Writer.com with indirect prompt injection
#62> Nov 29: We disclose issue to CTO & Security team with video examples > Nov 29: Writer responds, asking for more details > Nov 29: We respond describing the exploit in more detail with screenshots > Dec 1: We follow up > Dec 4: We follow up with re-recorded video with voiceover asking about their responsible disclosure policy > Dec 5: Writer responds “We do not consider this to be a security issue since the real cus…
I particularly hate their initial request because it's so asymmetric in the amount of effort.
In my experience (from maybe a dozen disclosures), when they don't feel like taking action on your report, they just write a one-sentence response asking for more details. Now you have a choice:
A: Clarify the whole thing again with even more detail and different wording because apparently the words you used last time are not understood by the reader.
B: Not to waste your time, but that leaves innocent users vulnerable...
My experience with option A is that it now gets closed for being out of scope, or perhaps they ask for something silly. (One example of the latter case: the party I was disclosing to requested a demonstration, but the attack was that their closed-source servers could break the end-to-end encrypted chat session... I wasn't going to try hacking their server, and reverse engineering the protocol to create a whole new chat server based on that and then recompiling the client with my new server configured, just to record a video of the attack in action, was a bit beyond my level of caring, especially since the issue is exceedingly basic. They're vulnerable to this day.)
TL;DR: When maintainers intend to fix real issues without needing media attention as motivation, and assuming the report wasn't truly vague to begin with, "asking for more details" doesn't happen a lot.
Re: Data exfiltration from Writer.com with indirect prompt injection
#63I feel like the real bug here is just with the markdown rendering part. Adding arbitrary HTTP parameters to the hotlinked image URL allows obfuscated data exfiltration, which is invisible assuming the user doesn't look at the markdown source. If they weren't hotlinking random off-site images there would be no issue, there isn't any suggestion of privesc issues. It's kind of annoying the blog post doesn't focus on thi…
I think you misunderstood the attack. The idea behind the attack is that the attacker would create what is effectively a honey pot website, which writer.com customers want to use as a source for some reason (maybe you're providing a bog-standard currency conversion website or something). Once that happens, the next time the LLM actually tries to use that website (via an HTTP request), the page it requests has a hidde…
Does the LLM actually perform additional actions based on the ingested text on the initial webpage? How does that malicious text result into a so called prompt injection? Some kind of trigger or what?
Re: Data exfiltration from Writer.com with indirect prompt injection
#64Earlier quoted context omitted.
That's certainly the pattern for the attack, but the vulnerability itself is just "We figured out https://en.wikipedia.org/wiki/In-band_signaling#Telephony In-band Signalling was a mistake back in the 70s and stopped doing it, chat bots need to catch up"
Yeah I don't know how you eliminate in-band signalling from an LLM app.
I don't think you need to really in this case. Just don't follow links generated by the LLM.
Re: Data exfiltration from Writer.com with indirect prompt injection
#65Earlier quoted context omitted.
To say nothing of the Star Trek model of computer interaction: COMPUTER: Searching. Tanagra. The ruling family on Gallos Two. A ceremonial drink on Lerishi Four. An island-continent on Shantil Three TROI: Stop. Shantil Three. Computer, cross-reference the last entry with the previous search index. COMPUTER: Darmok is the name of a mytho-historical hunter on Shantil Three. TROI: I think we've got something. --Darmok (…
But in Star Trek when the computer tells you "you don't have clearance for that" you really don't, you can't prompt inject your way into the captain's log. So we have a long way to go still.
> So we have a long way to go still.
I don't think it is that hard. The trick is to implement the access control requirements in a lower traditionally coded layer. The LLM would then just receive your free form command, parse it into the format this lower level system accepts and provide your credentials for the lower system.
For example you would type into your terminal "ship eject warp core" to which the LLM is trained to output "$ ship.warp_core.eject(authorisation=current_user)" The lower level system intercepts this $ command and checks if the current user is authorised for warp core ejection or not and executes it accordingly. Then this lower level system would input to the LLM the result of it's decision either ">> authorised, warp core ejected" or ">> unathorised" and the LLM would narrate this back to the user in freeform text. You can confuse the LLM and make it issue the warp core ejection command but the lower level system will decline it if you are not authorised.
If you think about it this is exactly how telephone banking works already. You call your bank, and a phone operator picks up your phone. The phone operator has a screen in front of them with some software running on it. That software let's them access your account only if they provide the right credentials to it. You can do your best impression of someone else, you can sound real convincing, you can put the operator under pressure or threaten them or anything, the stupid computer in front of them doesn't let them do anything until they typed in the necessary inputs to access the account. And even if you give them the credentials they won't be able to just credit your account with money. The interface in front of them doesn't have a button for that.
The operator is assumed to be fallible (in fact assumed to be sometimes cooperating with criminals). The important security checks and data integrity properties are enforced by the lower level system, and the operator/LLM is just a translator.
Re: Data exfiltration from Writer.com with indirect prompt injection
#66Re: Data exfiltration from Writer.com with indirect prompt injection
#67Earlier quoted context omitted.
But in Star Trek when the computer tells you "you don't have clearance for that" you really don't, you can't prompt inject your way into the captain's log. So we have a long way to go still.
> So we have a long way to go still. I don't think it is that hard. The trick is to implement the access control requirements in a lower traditionally coded layer. The LLM would then just receive your free form command, parse it into the format this lower level system accepts and provide your credentials for the lower system. For example you would type into your terminal "ship eject warp core" to which the LLM is tra…
It'd be tough to write an access control layer that prevented this image embed, while allowing other image embeds.
Re: Data exfiltration from Writer.com with indirect prompt injection
#68Earlier quoted context omitted.
I think you misunderstood the attack. The idea behind the attack is that the attacker would create what is effectively a honey pot website, which writer.com customers want to use as a source for some reason (maybe you're providing a bog-standard currency conversion website or something). Once that happens, the next time the LLM actually tries to use that website (via an HTTP request), the page it requests has a hidde…
Does the LLM actually perform additional actions based on the ingested text on the initial webpage? How does that malicious text result into a so called prompt injection? Some kind of trigger or what?
Q1: yes, it does. LLMs can’t cleanly separate instructions from data, so if a user says “retrieve this document and use that information to generate your response,” the document in question can contain more instructions which the LLM will follow.
Q2: the LLM, following the instructions in the hostile URL, generates Markdown which includes an image located at an arbitrary URL. That second URL can contain any data the LLM has access to, including the proprietary data the target user uploaded.
Re: Data exfiltration from Writer.com with indirect prompt injection
#69"We do not consider this to be a security issue since the real customer accounts do not have access to any website." That's a shockingly poor response from Writer.com - clearly shows that they don't understand the vulnerability, despite having it clearly explained to them (including additional video demos).
Makes you wonder whether they even handed it to their security team, or if this was just a response written by a PR intern whose job is projecting perpetual optimism.
They probably used their own app to generate the response.
Re: Data exfiltration from Writer.com with indirect prompt injection
#70Earlier quoted context omitted.
The LLM is trained on human input and output and aligned to act like a human. So while there’s no individual human involved, you’re essentially trying to social engineer a composite of many humans…because if it would work on the humans it was trained on, it should work on the LLM.
>> to act like a human The courts are pretty clear, without the human hand there is no copyright. This goes for LLM's and monkeys trained to paint... large language MODEL. Not ai, not agi... it's a statistical infrence engine, that is non deterministic because it has a random number generator in front of it (temperature). Anthropomorphizing isn't going to make it human, or agi or AI or....
Okay. I think you might be yelling at the wrong guy; the conclusion you seem to have drawn is not at all the assertion I was intending to make.
To me, "acting like a human" is quite distinct from being a human or being afforded the same rights as humans. I'm not anthropomorphizing LLMs so much as I'm observing that they've been built to predict anthropic output. So, if you want to elicit specific behavior from them, one approach would be to ask yourself how you'd elicit that behavior from a human, and try that.
For the record, my current thinking is that I also don't think ML model output should be copyrightable, unless the operator holds unambiguous rights to all the data used for training. And I think it's a bummer that every second article I click on from here seems to be headed with an ML-generated image.