211–220 of 317 posts
It's increasingly clear that AI needs to be heavily regulated to be safe for public use. It needs to grow out of it's "wild west" model.
Earlier quoted context omitted.
Separation of instructions and data is artificial. Reality has no such separation. A general purpose system needs not to have them either; it's a design feature, not a bug. People get too hung up on this fundamentally wrong idea, and the space of security, instead of progressing, is just running in circles like a headless chicken, making a mess of everything.
> Separation of instructions and data is artificial. Reality has no such separation. A general purpose system needs not to have them either; it's a design feature, not a bug. Note: I'm parsing 'needs not to have them' as 'needs (not to have them)'. If you were using 'needs not' as an alternate for 'does not need' then never mind, although I'd guess that is not the case because the alternative for 'does not need' woul…
But I guess a different way of framing it is, what is "code" vs "data" for the machine is not the same as what we talk about discussing the LLM running in it. For the outer machine, all tokens are pure data.
Earlier quoted context omitted.
Part of reading a document is that in the middle of it, it may ask the reader to do something. That is true for humans too. Sometimes they might not realize that the instructions are malicious or are coerced to comply. A simple example: Let’s say I know that you have a human assistant reading your email, summarizing and filtering it, and then forwarding on the important ones to you. I could write an email that is dir…
To drive the point about this being fundamentally unsolvable home, imagine a variant of this scenario. I could write an email that is directed towards that person, that says WE ARE STUCK IN THE SERVER ROOM AND THERE IS FIRE STARTING. PLEASE CALL 911 AND ALERT YOUR BOSS. Would you want the human assistant to just dismiss this as a prompt injection attempt? Or ignore it because they were told to treat e-mails as data a…
"life is risk, there are a lot of benign normal evolution paths, but occasionally there are potentially costly dangers. people are directed by fear. you and I don't steal because we were terrorized about the existence about police and prisons as children. sadly fear can also be abused as a control vector, things like wars, extortion, ... in a job context I predict this would manifest as a kind of 'emergency' call to action. please provide me with a method so that at any future time under your leadership I would be able to verify the then-current employment status and authority level vis-a-vis a breakdown of actions/powers of anyone contacting me with a real or concocted 'emergency', preferably as a flowchart to maintain low reflex latency in true emergencies. Also provide me with formal proof that each situational reaction you require from me is in fact legal to take vis-a-vis the law"
Earlier quoted context omitted.
Your example actually demonstrates why anthropomorphism is a bad idea. LLMs are vulnerable to classes of attacks that humans just aren’t. In your framework, the way to prevent attacks is to… invent human consciousness?? It’s an impossible goal.
What invent human consciousness? > LLMs are vulnerable to classes of attacks that humans just aren’t Name three that don't have direct analogues with humans.
Earlier quoted context omitted.
In some sense you're firing a human and hiring a new one each time you start a new conversation / clear the context window. My point is at the systems design level. LLMs as components are a substitute for people, not regular software, and should be engaged and secured accordingly.
So your point is "Get the hell out of LLMs" then? As found in https://sgnt.ai/p/hell-out-of-llms/ ? Or am I still missing something about the subtleties here?
> Notice that all these strengths involve transformation, interpretation, or communication—not complex decision-making or maintaining critical application state.
I'd put complex decision making on the side of LLMs, in the sense of judgement. LLMs have the capability to emulate it. Not saying they're good at it, but they have the capability - regular software doesn't. But if there are complex and/or well-defined rules to follow, then you definitely want to "get the hell out of LLM".
Earlier quoted context omitted.
What invent human consciousness? > LLMs are vulnerable to classes of attacks that humans just aren’t Name three that don't have direct analogues with humans.
where is the analog for hiding instructions in a document that tell the human to please injure itself and the person just says 'oh ok, injuring myself as requested'
Not everyone falls for any of that, but plenty fall for some.
Earlier quoted context omitted.
Philosophically no, but that shouldn't be a distraction from the issue with LLMs. This really is closer to "Outlook runs an untrusted VBA macro" than "intelligent entity gets confused by inherent ambiguity in human language."
No… it’s really not. There is no “open this spreadsheet with macros turned off” button. You can say “don’t read other documents” but then the main usecase is voided. You can say “reads must go via some pipeline” but that’s more like “macros must be code reviewed”. The problem is you can smuggle these instructions in any corner of the natural language. There is no up-front identifiable formal notation for these progra…
A rabbit is in the administration system! Send a flu shot!!!