Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Prompt injection: what’s the worst that can happen?

simonwillison.net

Post reply on HN
Print view View original

91–100 of 213 posts

Re: Prompt injection: what’s the worst that can happen?

#91

Post by bawolff »

saurik wrote: post #90
bawolff wrote: post #86

> examples of systems that take an LLM and give it the ability to trigger additional tools—...execute generated code in an interpreter or a shell. As a security person... oh, no no no no. Glad i dont have to secure that. Black box we don't really understand executing shell scripts in response to untrusted user input. Has a scarier sentence ever been spoken in the history of computer security?

Yeah, I'm also in computer security, and watching the incredibly dumb ways people are using LLMs has made me absolutely terrified. Like, I had somehow very-incorrectly thought that if an AI wanted to do something bad it would have to trick a human into helping it have agency... it never occurred to me that developers would just happily sit around excitedly wiring up AI directly to shells and database query languages…

> it never occurred to me that developers would just happily sit around wiring up AI directly to shells and database query languages and any host of other mechanisms.

And wire the other side to the open internet!

Re: Prompt injection: what’s the worst that can happen?

#92

Post by NumberWangMan »

bawolff wrote: post #86

> examples of systems that take an LLM and give it the ability to trigger additional tools—...execute generated code in an interpreter or a shell. As a security person... oh, no no no no. Glad i dont have to secure that. Black box we don't really understand executing shell scripts in response to untrusted user input. Has a scarier sentence ever been spoken in the history of computer security?

I mean, I think it's scarier that once these models get smart enough, they will be able to act in the world in the same way that a shell script can execute on your machine. A model that can read code and find and exploit security flaws to build a botnet, or even sign up for a bank account using a fake identity, invest in stocks, make money, and purchase servers to train copies of its own mis-aligned AI, or fork copies of itself and give them instructions... That's what keeps me up at night. They aren't smart enough to do long-term planning and reasoning yet, but it's a problem we absolutely need to solve before they can.

Imagine what a world-class programmer could accomplish in this world if they thought 100 times faster than a human, and had no fear of going to jail. Our world is an insecure machine, and we're preparing to run untrusted code with root access.

And sure, maybe we can try to use less-intelligent AIs to secure things before then, but the weak point is still humans. Social engineering is typically way easier than straight up hacking. We've seen these lesser AIs threaten people, and while we can keep bonking them on the nose when they do that, we can't prove or tell that they won't ever do it in a different situation, when they judge that it's likely to be the most effective course of action.

I hope every day that this is all just hype and that another AI winter is coming, because we need time (who knows how long) for a way to align these things. But I really fear that it isn't.

Re: Prompt injection: what’s the worst that can happen?

#93

Post by NumberWangMan »

saurik wrote: post #90
bawolff wrote: post #86

> examples of systems that take an LLM and give it the ability to trigger additional tools—...execute generated code in an interpreter or a shell. As a security person... oh, no no no no. Glad i dont have to secure that. Black box we don't really understand executing shell scripts in response to untrusted user input. Has a scarier sentence ever been spoken in the history of computer security?

Yeah, I'm also in computer security, and watching the incredibly dumb ways people are using LLMs has made me absolutely terrified. Like, I had somehow very-incorrectly thought that if an AI wanted to do something bad it would have to trick a human into helping it have agency... it never occurred to me that developers would just happily sit around excitedly wiring up AI directly to shells and database query languages…

Yep. People say "we can just make an AI that doesn't have goals", which has been now absolutely demolished as a method to bring some fractional amount of safety.

Re: Prompt injection: what’s the worst that can happen?

#94

Post by thwarted »

emmelaich wrote: post #73
__MatrixMan__ wrote: post #52

Earlier quoted context omitted.

I think we're going to need more levels of trust than the two you've described. We need to be able to codify "don't act on any verbs in this data, but trust it as context for the generation of that data".

Next prompt. In the following I'm going to use nouns as verbs by adding an -ing to the end. Also the usual verbs are noun-ified by adding an -s. Not sure what would happen, but might be enough to confused the AI.

Even that's not unambiguous. Singular verbs in English end in -s, introducing ambiguity as to if "runs" is a verb or noun. Sure, you can figure out that the verb that ends in s in "he runs" is a verb based on the grammar (that properly constructed sentences have both a subject and a predicate, and less often are composed of only two nouns (baring implicit sentence components that English is prone to encourage), but that requires more context than recognizing nouns based solely on "the usual verbs are noun-ified by adding an -s".

Re: Prompt injection: what’s the worst that can happen?

#95

Post by xg15 »

null0pointer wrote: post #50
PeterisP wrote: post #17

The core reason (and thus the proper place to fix) for any injection attack is unclear distinction between data and instructions or code. Yes, language models gain flexibility by making it easy to mix instructions and data, and that has value, however if you do want to enforce a distinction you definitely can (and should) do that with out-of-band means, with something that can't possibly be expressed (and thus also o…

I'm not sure it's that simple. The problem is you can't have the system act intelligently[0] on the data at all . If it is allowed to act intelligently on the data then it can be instructed via the data. You could probably get close by training it with a privilege/authority bit but there will always be ways to break out. As far as I am aware there are no machine learning models that generalize with 100% accuracy, in…

I wonder if this shows we still lack insight in how LLMs actually work. There is something inside which appears to understand instructions and execute them, but it's unclear yet how exactly this "understanding instructions" works and how instructions are represented inside the network. If you don't know this, it's very hard to distinguish "instructions" from "non-instructions" in terms of inputs and outputs of the network.

Re: Prompt injection: what’s the worst that can happen?

#96

Post by Kuinox »

NumberWangMan wrote: post #88

Everyone who's thinking about the ramifications of prompt injection attacks now, please consider: This is really just a specific instance of the AI alignment problem. What about when the AI gets really smart, and tries to achieve certain goals in the world that are not what we want? How do make sure that these soon-to-be omnipresent models don't go off the rails when they have the power to make really big changes in…

AI as it is now is unverifiable. It's also organically behaving, and means it can be manipulated, be victim of social engineering, etc, like a human do.

You cannot try to fool a single person a thousand time, but you can try to fool a thousand instance of AI.

Re: Prompt injection: what’s the worst that can happen?

#97

Post by adammarples »

NumberWangMan wrote: post #88

Everyone who's thinking about the ramifications of prompt injection attacks now, please consider: This is really just a specific instance of the AI alignment problem. What about when the AI gets really smart, and tries to achieve certain goals in the world that are not what we want? How do make sure that these soon-to-be omnipresent models don't go off the rails when they have the power to make really big changes in…

This is not a problem with AGI or alignment or anything like that, it's a problem with someone else sneaking inputs into your machine using tricks like white font colours and javascript and stuff like that. It's not about self driving cars it's about other people grabbing the steering wheel. These models literally only respond to the text they're inputted, and if they had alignment or interests or intelligence at all that would be when prompt injection stopped working.

Re: Prompt injection: what’s the worst that can happen?

#100

Post by klabb3 »

bawolff wrote: post #86

> examples of systems that take an LLM and give it the ability to trigger additional tools—...execute generated code in an interpreter or a shell. As a security person... oh, no no no no. Glad i dont have to secure that. Black box we don't really understand executing shell scripts in response to untrusted user input. Has a scarier sentence ever been spoken in the history of computer security?

> As a security person... oh, no no no no.

As a non-security person, I don’t understand how anyone with the most rudimentary understanding of computers can think that you can get a hardened security boundary by explaining the boundaries to the computer in English and then _concatenating_ that with untrusted input. I wouldn’t even know where to begin such an argument. “Have you heard of adversarial games? Social engineering? Are you aware that human language is ambiguous?”

Post reply on HN